The Digital Deluge Lecture 1 Learning in Retirement David Coll Professor Emeritus Department of Systems and Computer Engineering Winter 2009 Welcome to “The Digital Deluge” • My name is David Coll and • In the next few weeks I’m going to try and give you a feel for what’s going on in a world awash in digital information. • We’ll look at – – – – – – the jargon, the sources, the volumes, the technology, the societal impact, and how some are coping. • This is a lecture series about the flood of information created by new technologies. • The problem in offering a course about this subject is that there is way too much information to handle. • But – we have ways of coping • For example • Clear and simple instructions … First - My Co-ordinates • Professor Emeritus in the Department of Systems and Computer Engineering, Carleton University • [email protected] • http://www.sce.carleton.ca/faculty/coll.htm • 613-225-4229 Information • The world is awash in INFORMATION. • Our senses: sight, hearing, smell, touch, and taste are bombarded with inputs from all sides • Information is essential for our survival. • But … • Let’s narrow it down somewhat • We might limited ourselves to information that comes to us through a medium: mediated communications – Print • newspapers, magazines, books, flyers … – Posters – Radio – Television – Recordings – music, images, videos – The Telephone • As well as through • the Internet and • the World Wide Web • We will exclude information communicated to us through spectral channels by mediums wearing star-studded gowns and too many bracelets … Introduction • The rapid growth and deployment of digital information processing and communications technology is changing our world, the way we do business, and the way we live. • Information be it text, data, images, music, speech, or television is acquired, stored, processed, communicated, received, and presented in digital form, i.e., as numbers. • The resulting torrent of information is so vast that some have dubbed it the “The Digital Deluge”. • del·uge –1 • a: an overflowing of the land by water • b: a drenching rain – 2: an overwhelming amount or number, as in:received a deluge of offers The Deluge from Various Perspectives The extent, rapidity of acceptance, and pervasiveness of digital technology is almost (?) beyond comprehension. The Digital Deluge • “Data is omnipresent. Everywhere we go, we encounter it, in our business transactions, subscription lists, email, etc.” • “To put the growth of data in perspective, every 18 months the processing capacity of the world doubles, but at the same time data has been doubling every nine months.” • http://itmanagement.earthweb.com/datbus/article.php/1495861 The Digital Deluge • “One of the scariest statistics in recent memory is IDC’s prediction that the amount of digital data between now and 2010, reaching a whopping 988 billion gigabytes.” • “(That’s a figure so huge, it would be helpful to have one of those goofy yet helpful references to put it into context. How many football fields it would fill, for example.)” • http://www.itbusinessedge.com/blogs/tve/?p=98 ScienceDaily (Nov. 8, 2007) • “Most people have a few gigabytes of files on their PC. • In the next decade, astronomers expect to be processing 10 million gigabytes of data every hour from the Square Kilometre Array telescope”. • http://www/sciencedaily.com/releases/2007/11/071107091524.htm • “When the Large Hadron Collider [at CERN] starts running in 2008, it will be the biggest scientific instrument in the world. • Thousands of scientists across the planet will be clamouring for access to the streams of data that will come out of the instrument. • The LHC will produce about 15 Petabytes of data every year, which is more than 1000 [times] the amount of information in book form printed every year around the world”. • If written to CDs, the stack of CDs would be about 20 km in height! • http://gridcafe.web.cern.ch/gridcafe/GridatCERN/gridatcern.html From IDC Report • “With a compound annual growth rate of almost 60%, the digital universe is growing faster and is projected to be nearly 1.8 zettabytes (1,800 exabytes) in 2011, a 10-fold increase over the next five years • “At 281 billion gigabytes (281 exabytes), the digital universe in 2007 was 10% bigger than originally estimated • IDC’s new research shows the digital universe is growing more rapidly than original estimates as a result of accelerated growth in worldwide shipments of – digital cameras, – digital surveillance cameras, and – digital televisions • as well as – a better understanding of information replication trends. • They didn’t mention traffic to and from cell phones , iPhones, Blackberries and the ilk.. • The digital universe in 2007 was equal to almost 45 gigabytes (GB) of digital information for every person on earth – or the equivalent of over 17 billion 8 GB iPhones. • Other fast-growing corners of the digital universe include those related to – – – – Internet access in emerging countries sensor-based applications data centers supporting “cloud computing” social networks comprised of digital content created by many millions of online users. The Digital Deluge • What’s it all about and what can we do about it? • Maybe we can not only survive the Digital Deluge, but use it to our advantage. • Hey! Didn’t Noah’s kids learn to water ski behind the Arc? Let’s Start with Five Questions • • • • • Where does information come from? How does it get to be “digital”? How does it get to us? Why is it a problem? What is being done to cope with it? Where Does the Information Come From? Information Generators • • • • • • • • • Arts Literature History Language Education Religion Science and Technology Government/Bureaucracy Commerce • • • • • • • • Administration Medicine and Pharmacology Geography and Cartography Entertainment News … … … Traditional Sources • Media – Print: Books, Newspapers – Film – Radio and TV Broadcast • Accumulated Knowledge – Libraries – Repositories – Folk Lore • Publications – Books, Newspapers, Magazines, Professional Journals • Radio and TV • Personal Communications and Correspondence • Sensors and Instrumentation • . • Libraries and Repositories – Publications, Reference Material, Dictionaries, Encyclopedia, Catalogs • Personal Creation – Records • Bills, Accounts, Investments, Wills, Warranties, Contracts, … – Libraries • Books, Magazines, Clippings, … – Diaries and Writings – Recordings • Audio, Photo, Film, Video • Computer Files How Does It Get To Us? • Purchase and Own • Delivery Services – Mail, Newspapers, Magazines • Broadcast Services – Radio – Television • Message Services – Telegraph – Teletype • Data Communications – File Transfers, Data Collection – Transactions Where did this “Deluge” come from? • The major reason the world is awash in information in digital form is the COMPUTER • Actually the ability to represent and process information on digital form • But, that is only half the story. • An isolated computer is just a fancy adding machine or typewriter with memory • So, the deluge results from the fact that all the world’s computers are connected! – Well, most all. • The ability to communicate information is an essential factor behind the deluge, and • Communications is done “digitally”.. What’s the Problem? • There is so much information/data*. – Entertainment, medical, scientific research, educational, literature, news, personal … • We are not always sure about just what “information” there is in the data *“information” often implies “knowledge” whereas we are talking about the volumes of data that are generated – whether they make sense or not. The Culprit? • A mixture of – The Internet • networked computers, ubiquitous communications, – Applications • the WWW, information processing – and Smart Machines • Enabled by the evolution of common digital technologies throughout. So What’s New? • A common technological basis for the – acquisition, – distribution and – handling • of information in digital formats. • A basis that has very recently reached levels of maturity that have unleashed information in a flood of incomprehensible immensity. • That’s the Punch Line • But Why The Deluge Now? • Because we can! Enabling Technologies ELECTRONICS COMMUNICATIONS DIGITAL TECHNOLOGY HARDWARE SOFTWARE • But, first • Let us pause for a short break …. What will we do? • We’ll look at sources of the information explosion in areas that affect us: – – – – – – – – – Mobile telephones Digital cameras Music: iPods, CDs, DVDs HDTV The Internet and Web Applications Libraries and other repositories Smart terminals Home automation Scientific Research, and so on. • We will look at what “digital” means – How the adoption of digital technology is used to create information, store and process it, communicate and display it. • Why we are deluged with digital information, i.e., – What’s behind the sudden onslaught of information, where this surge is coming from, how it’s to be handled, and what the future holds. • Then, hopefully we can discuss some ways of coping, and even enjoying the deluge – • Let’s start with some definitions Terminology • Misuse of terminology can lead to lead to misunderstanding. • Being precise in one's terminology can be important. • It is also important to understand the historical origin of terms, to better understand their meaning, and to realize how different points of view can modify meaning. Scale and Scope of the Deluge • How deep is the flood? • If we are in water that is 6 feet deep we have a feel for what sort of situation we are facing • But, what’s an exafoot? • What are these funny names for data rates and volumes? • The basic digital (binary) unit is the bit, a single binary digit with values 0 or 1. • In Digital Communications, one usually refers to data rates in terms of bits per second, or bps. • Data rates, often mistakenly termed bandwidth, are expressed as the number of bits transmitted per second – as in 56 kbits per second (kbps) or 1.544 megabits per sec (mbps) Common Data Rates • Data Rate speeds derive from teletype and telephone services • Originally, everything was a multiple of the telegraph speed of 75 bits per second – 300, 1200, 2400, 4800, 9600 bps • Then multiples of the basic telephone sampling speed of 64,000 kbps took over, giving services at the T1 TDM carrier speed of 1.544 Mbps, and multiples of it. Common Optical FibreTransmission Rates – in Megabytes per second. Optical OC-1 OC-3 OC-12 OC-48 OC-192 OC-768 Line Rates 51.84 155.52 622.08 2,488.32 9,953.28 39,813.12 Optical Fibre Transmission Rates - in Megabytes per second- MBps • The OC192 rate is 9,953.28 megabits per second, or almost 10,000 Mbps = 10 gigabits per second. • This number is 192 times 51.84 mbps • 51.84 = 810 x 8,000 x 8, or 810 x 64,000 • The number of bits per second in a basic SONET frame used for the transmission of 810 voice channels. Prefix Symbol Base 10 Base 2 yotta Y +24 +80 zetta Z +21 +70 exa E +18 +60 peta P +15 +50 tera T +12 +40 giga G +09 +30 mega M +06 +20 kilo k +03 +10 hecto h +02 - deka da +01 - deci d -01 - centi c -02 - milli m -03 - micro m -06 - nano n -09 - pico p -12 - femto f -15 - -18 - atto zepto z -21 - yocto y -24 - In communications, electronics, and physics, multipliers are defined in powers of 10 from 10-24 to 1024, proceeding in increments of three orders of magnitude (103 or 1,000). In IT and data storage, multipliers are defined in powers of 2 from 210 to 280, proceeding in increments of ten orders of magnitude (210 or 1,024). But, how many is that in base 10?* • • • • • • • Kilo Mega Giga Tera Exa Peta Zetta = 1 thousand = 1 million = 1 billion = 1 trillion = 1 quadrillion = 1 quintillion = 1 sextillion = 103 = 106 = 109 = 1012 = 1015 = 1018 = 1021 * By the “US short scale”: check http://en.wikipedia.org/wiki/Names_of_large_numbers Computer Memory and Processing • In digital computers, the basic element of information storage is the byte, an 8-bit binary number. • The basic storage element is a word, and is usually 8, 16, 32 or 64 bits long, or 1, 2, 4, or 8 bytes. • It is common to measure the size of a computer memory in terms of powers of 2, • e.g. • 1 Kilobyte = 210 = 1024 bytes. Counting Bytes • Kilobyte: 1 KB = 210 = 1024 bits ~ 1 x 103 • Megabyte: 1MB = 220 bits, or 1,048,576 bits ~ 1 x 106 • Gigabyte: 1GB = 230 bits = 1,073,741,824 bits • Terabyte: 1TB = 240 bits = 1,099,511,627,776 bits • Exabyte: 1 EB =250 bits = 1,125,899,906,842,624 bits ~ 1 x 1015 • Petabyte 1 PB = 260 bits = 1,152,921,504,606,846,976 bits ~ 1 x 1018 Bits and Bytes • Transmission speeds invariably refer to bits per second • e.g. 1.544 Mbps • File sizes refer to bytes • e.g. 1 Megabyte of RAM = 1024 Kilobytes or 1,048,576 bits; but communications rarely, if ever, refers to bits at all. The “Digital” Thing: What’s the Big Idea, Anyway? • “Digital” is a word (mis)used to describe information (often called data) in discrete form, and the subsequent processing of that information, as opposed to information that occurs in a continuous, or “analog” form.