Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
What is “Computation”? Computation is about “processing information” - Comp411 – Spring 2007 Transforming information from one form to another Deriving new information from old Finding information associated with a given input “Computation” describes the motion of information through time “Communication” describes the motion of information through space 1/16/2007 L02 - Information 1 What is “Information”? “ 10 Problem sets, 2 quizzes, and a final!” information, n. Knowledge communicated or received concerning a particular fact or circumstance. Carolina won again. Tell me something new… A Computer Scientist’s Definition: Information resolves uncertainty. Information is simply that which cannot be predicted. The less predictable a message is, the more information it conveys! Comp411 – Spring 2007 1/16/2007 L02 - Information 2 Real-World Information Why do unexpected messages get allocated the biggest headlines? … because they carry the most information. Comp411 – Spring 2007 1/16/2007 L02 - Information 3 What Does A Computer Process? • A Toaster processes bread and bagels • A Blender processes smoothies and margaritas • What does a computer process? • 2 allowable answers: – Information – Bits • What is the mapping from information to bits? Comp411 – Spring 2007 1/16/2007 L02 - Information 4 Quantifying Information (Claude Shannon, 1948) Suppose you’re faced with N equally probable choices, and I give you a fact that narrows it down to M choices. Then I’ve given you log2(N/M) bits of information Information is measured in bits (binary digits) = number of 0/1’s required to encode choice(s) Examples: w information in one coin flip: log2(2/1) = 1 bit w roll of a single die: log2(6/1) = ~2.6 bits w outcome of a Football game: 1 bit (well, actually, “they won” may convey more information than “they lost”…) Comp411 – Spring 2007 1/16/2007 L02 - Information 5 Example: Sum of 2 dice i2 = log2(36/1) = 5.170 bits i3 = log2(36/2) = 4.170 bits i4 = log2(36/3) = 3.585 bits i5 = log2(36/4) = 3.170 bits i6 = log2(36/5) = 2.848 bits i7 = log2(36/6) = 2.585 bits i8 = log2(36/5) = 2.848 bits i9 = log2(36/4) = 3.170 bits i10 = log2(36/3) = 3.585 bits i11 = log2(36/2) = 4.170 bits i12 = log2(36/1) = 5.170 bits 2 3 4 5 6 7 8 9 10 11 12 The average information provided by the sum of 2 dice: iave Comp411 – Spring 2007 12 i 2 Mi N Entropy N) log2 ( M pi log2 ( pi ) 3.274 bits i i 1/16/2007 L02 - Information 6 Show Me the Bits! Can the sum of two dice REALLY be represented using 3.274 bits? If so, how? The fact is, the average information content is a strict *lower-bound* on how small of a representation that we can achieve. In practice, it is difficult to reach this bound. But, we can come very close. Comp411 – Spring 2007 1/16/2007 L02 - Information 7 Variable-Length Encoding • Of course we can use differing numbers of “bits” to represent each item of data • This is particularly useful if all items are *not* equally likely • Equally likely items lead to fixed length encodings: – Ex: Encode a particular roll of 5? – {(1,4), (2,3), (3,2), (4,1)} which are equally likely if we use fair dice 4 4 – Entropy = p(roll i roll 5) log 2 ( p(roll i roll 5)) 14 log 2 ( 14 ) 2 i 1 i 1 bits – 00 – (1,4), 01 – (2,3), 10 – (3,2), 11 – (4,1) • Back to the original problem. Let’s use this encoding: 2 - 10011 6 - 111 10 - 1000 Comp411 – Spring 2007 3 - 0101 7 - 101 11 - 0100 1/16/2007 4 - 011 8 - 110 12 - 10010 5 - 001 9 - 000 L02 - Information 8 Variable-Length Encoding • Taking a closer look 2 - 10011 3 - 0101 4 - 011 5 - 001 6 - 111 7 - 101 8 - 110 9 - 000 10 - 1000 11 - 0100 12 - 10010 Unlikely rolls are encoded using more bits More likely rolls use fewer bits • Decoding 2 5 3 6 5 8 3 Example Stream: 1001100101011110011100101 Comp411 – Spring 2007 1/16/2007 L02 - Information 9 Huffman Coding • A simple *greedy* algorithm for approximating an entropy efficient encoding 1. 2. 3. 4. Find the 2 items with the smallest probabilities Join them into a new *meta* item whose probability is the sum Remove the two items and insert the new meta item Repeat from step 1 until there is only one item Huffman decoding tree 36/36 15/36 21/36 7/36 8/36 9 4/36 5 11/36 4 4/36 11 2/36 3 2/36 6/36 10 8 5/36 6 5/36 2/36 3/36 12 Comp411 – Spring 2007 7 5/36 3/36 4/36 10/36 1/16/2007 1/36 2 1/36 L02 - Information 10 Converting Tree to Encoding Once the *tree* is constructed, label its edges consistently and follow the paths from the largest *meta* item to each of the real item to find the encoding. 2 - 10011 6 - 111 10 - 1000 3 - 0101 7 - 101 11 - 0100 0 0 0 9 4/36 8/36 15/36 4 - 011 8 - 110 12 - 10010 36/36 5 4/36 0 11 2/36 1 3 2/36 4 3/36 0 1 5/36 0 10 3/36 1 0 12 Comp411 – Spring 2007 1 11/36 1 0 4/36 21/36 0 7/36 1 Huffman decoding tree 1 1 5 - 001 9 - 000 1/16/2007 1/36 2/36 7 6/36 0 8 5/36 10/36 1 6 5/36 1 2 1/36 L02 - Information 11 Encoding Efficiency How does this encoding strategy compare to the information content of the roll? 1 5 2 4 3 3 4 3 5 3 6 3 bave 36 36 36 36 36 36 5 3 4 3 3 4 2 4 1 5 36 36 36 36 36 bave 3.306 Pretty close. Recall that the lower bound was 3.274 bits. However, an efficient encoding (as defined by having an average code size close to the information content) is not always what we want! Comp411 – Spring 2007 1/16/2007 L02 - Information 12 Encoding Considerations - Encoding schemes that attempt to match the information content of a data stream remove redundancy. They are data compression techniques. - However, sometimes our goal in encoding information is increase redundancy, rather than remove it. Why? Make the information easier to manipulate (fixed-sized encodings) Make the data stream resilient to noise (error detecting and correcting codes) -Data compression allows us to store our entire music collection in a pocketable device Comp411 – Spring 2007 1/16/2007 -Data redundancy enables us to store that *same* information *reliably* on a hard drive L02 - Information 13 Error detection using parity Sometimes we add extra redundancy so that we can detect errors. For instance, this encoding detects any single-bit error: ex: 2-1111000 3-1111101 4-0011 5-0101 6-0110 7-0000 8-1001 9-1010 10-1100 11-1111110 12-1111011 Comp411 – Spring 2007 1: 2: 3: 4: 4 001111100000101 4 6* 7 4 9* 7 4 10* 7 001111100000101 001111100000101 001111100000101 4 2* There’s something peculiar about those codings 5 001111100000101 Same bitstream – w/4 possible interpretations if we allow for only one error 1/16/2007 L02 - Information 14 Property 1: Parity The sum of the bits in each symbol is even. (this is how errors are detected) 2-1111000 = 1 + 1 + 1+ 1 + 0 + 0 + 0 = 4 3-1111101 = 1 + 1 + 1 + 1 + 1 + 0 + 1 = 6 4-0011 = 0 + 0 + 1 + 1 = 2 5-0101 = 0 + 1 + 0 + 1 = 2 6-0110 = 0 + 1 + 1 + 0 = 2 7-0000 = 0 + 0 + 0 + 0 = 0 8-1001 = 1 + 0 + 0 + 1 = 2 9-1010 = 1 + 0 + 1 + 0 = 2 10-1100 = 1 + 1 + 0 + 0 = 2 11-1111110 = 1 + 1 + 1 + 1 + 1 + 1 + 0 = 6 12-1111011 = 1 + 1 + 1 + 1 + 0 + 1 + 1 = 6 Comp411 – Spring 2007 1/16/2007 How much information is in the last bit? L02 - Information 15 Property 2: Separation Each encoding differs from all others by at least two bits in their overlapping parts 2 1111000 3 4 5 6 7 8 9 10 11 1111101 0011 0101 0110 0000 1001 1010 1100 1111110 3 1111101 4 0011 5 0101 6 0110 7 0000 8 1001 9 1010 10 1100 11 1111110 12 1111011 1111x0x xx11 x1x1 x11x xxxx 1xx1 1x1x 11xx 1111xx0 11110xx xx11 x1x1 0xx1 x11x 0x1x 01xx xxxx 00xx 0x0x 0xx0 1xx1 x0x1 xx01 xxxx x00x 1x1x x01x xxxx xx10 x0x0 10xx 11xx xxxx x10x x1x0 xx00 1x0x 1xx0 11111xx xx11 x1x1 x11x xxxx 1xx1 1x1x 11xx 1111xx1 xx11 x1x1 x11x xxxx 1xx1 1x1x 11xx 1111x1x This difference is called the “Hamming distance” “A Hamming distance of 1 is needed to uniquely identify an encoding” Comp411 – Spring 2007 1/16/2007 L02 - Information 16 A Short Detour It is illuminating to consider Hamming distances geometrically. 10 00 11 01 Given 2 bits the largest Hamming distance that we can achieve between 2 encodings is 2. This allows us to detect 1-bit errors if we encode 1 bit of information using 2 bits. With 3 bits we can find 4 encodings with a Hamming distance of 2, allowing the detection of 1-bit errors when 3 bits are used to encode 2 bits of information. 110 010 111 011 100 000 101 001 We can also identify 2 encodings with Hamming distance 3. This extra distance allows us to detect 2-bit errors. However, we could use this extra separation differently. Encodings separated by a Hamming distance of 2. Comp411 – Spring 2007 110 010 111 011 100 000 101 001 Encodings separated by a Hamming distance of 3. 1/16/2007 L02 - Information 17 Error correcting codes We can actually correct 1-bit errors in encodings separated by a Hamming distance of 2. This is possible because the sets of bit patterns located a Hamming distance of 1 from our encodings are distinct. 110 However, attempting error correction with such a small separation is dangerous. Suppose, we have a 2-bit error. Our error correction scheme will then misinterpret the encoding. Misinterpretations also occurred when we had 2-bit errors in our 1-bit-error-detection (parity) schemes. A safe 1-bit error correction scheme would correct all 1-bit errors and detect all 2-bit errors. What Hamming distance is needed between encodings to accomplish this? Comp411 – Spring 2007 1/16/2007 010 111 011 100 000 101 001 This is just a voting scheme L02 - Information 18 Information Summary Information resolves uncertainty • Choices equally probable: • N choices down to M log2(N/M) bits of information •Choices not equally probable: • choicei with probability pi log2(1/pi) bits of information • average number of bits = pilog2(1/pi) • use variable-length encodings Comp411 – Spring 2007 1/16/2007 L02 - Information 19 Computer Abstractions and Technology 1. Layer Cakes 2. Computers are translators 3. Switches and Wires (Read Chapter 1) Comp411 – Spring 2007 1/16/2007 L02 - Information 20 Computers Everywhere The computers we are used to Desktops Laptops Embedded processors Cars Mobile phones Toasters, irons, wristwatches, happy-meal toys Comp411 – Spring 2007 1/16/2007 L02 - Information 21 A Computer System What is a computer system? Where does it start? Where does it end? for (i = 0; i < 3; i++) m += i*i; Assembler and Linker addi $8, $6, $6 sll $8, $8, 4 CPU Compiler A Module B ALU A Cells CO B FACI S Gates Transistors Comp411 – Spring 2007 1/16/2007 L02 - Information 22 Computer Layer Cake Applications Systems software Shared libraries Operating System Hardware – the bare metal Computers are digital Chameleons Apps Systems S/W Libraries Operating System Hardware Comp411 – Spring 2007 1/16/2007 L02 - Information 23 Computers are Translators User-Interface (visual programming) High-Level Languages Compilers Interpreters Assembly Language Machine Language int x, y; y = (x-3)*(y+123456) Comp411 – Spring 2007 x: y: c: .word 0 .word 0 .word 123456 ... lw addi lw lw add mul sw 1/16/2007 $t0, $t0, $t1, $t2, $t1, $t0, $t0, x $t0, -3 y c $t1, $t2 $t0, $t1 y L02 - Information 24 Computers are Translators User-Interface (visual programming) High-Level Languages Compilers Interpreters Assembly Language Machine Language x: y: c: .word 0 .word 0 .word 123456 ... lw addi lw lw add mul sw Comp411 – Spring 2007 $t0, $t0, $t1, $t2, $t1, $t0, $t0, 0x04030201 0x08070605 0x00000001 0x00000002 0x00000003 0x00000004 0x706d6f43 x $t0, -3 y c $t1, $t2 $t0, $t1 y 1/16/2007 L02 - Information 25 Why So Many Languages? Application Specific Historically: COBOL vs. Fortran Today: C# vs. Java Visual Basic vs. Matlab Static vs. Dynamic Code Maintainability High-level specifications are easier to understand and modify Code Reuse Code Portability Virtual Machines Comp411 – Spring 2007 1/16/2007 L02 - Information 26 Under the Covers Input Output Storage Processing Datapath Control Comp411 – Spring 2007 1/16/2007 L02 - Information 27 Under the Covers Input Output Storage Processing Datapath Control Comp411 – Spring 2007 1/16/2007 L02 - Information 28 Under the Covers Cathode Ray Tube (CRT) Input Output Storage Processing The “last vacuum tube” Now nearing extinction Datapath Control Comp411 – Spring 2007 1/16/2007 L02 - Information 29 Under the Covers Input Output Storage Processing Liquid Crystal Displays (LCDs) Datapath Control Comp411 – Spring 2007 1/16/2007 L02 - Information 30 Under the Covers Input Output Storage Processing Datapath Control Comp411 – Spring 2007 1/16/2007 L02 - Information 31 Under the Covers Input Output Storage Processing Datapath Control Comp411 – Spring 2007 1/16/2007 L02 - Information 32 Under the Covers Input Output Storage Processing Intel Pentium III Xeon Datapath Control Comp411 – Spring 2007 1/16/2007 L02 - Information 33 Implementation Technology Relays Vacuum Tubes Transistors Integrated Circuits Gate-level integration Medium Scale Integration (PALs) Large Scale Integration (Processing unit on a chip) Today (Multiple CPUs on a chip) Nanotubes?? Quantum-Effect Devices?? Comp411 – Spring 2007 1/16/2007 L02 - Information 34 Implementation Technology Common Links? A controllable switch Computers are wires and switches open open closed control Comp411 – Spring 2007 1/16/2007 L02 - Information 35 Chips Silicon Wafers Chip manufactures build many copies of the same circuit onto a single wafer. Only a certain percentage of the chips will work; those that work will run at different speeds. The yield decreases as the size of the chips increases and the feature size decreases. Wafers are processed by automated fabrication lines. To minimize the chance of contaminants ruining a process step, great care is taken to maintain a meticulously clean environment. Comp411 – Spring 2007 1/16/2007 L02 - Information 36 Field Effect Transistors (FETs) Modern silicon fabrication technology is optimized to build a particular type of transistor. The flow of electrons from the VDS source to the drain is controlled by a gate voltage. Source Gate Drain IDS = 0 n+ n+ p Bulk Comp411 – Spring 2007 1/16/2007 L02 - Information 37 Chips Silicon Wafers IBM photomicrograph (Si has been removed!) Metal 2 M1/M2 via Metal 1 Polysilicon Diffusion Mosfet (under polysilicon gate) Comp411 – Spring 2007 1/16/2007 L02 - Information 38 How Hardware WAS Designed 20 years ago I/O Specification Truth tables State diagrams Logic design Circuit design Circuit Layout Comp411 – Spring 2007 1/16/2007 L02 - Information 39 How Hardware IS Designed Today (with software) High-level hardware specification languages Verilog VHDL Comp411 – Spring 2007 1/16/2007 L02 - Information 40 Reconfigurable Chips Programmable Array Logic (PALs) Fixed logic / programmable wires Field Programmable Gate Arrays (FPGAs) Repeated reconfigurable logic cells Comp411 – Spring 2007 1/16/2007 L02 - Information 41 Next Time Computer Representations How is X represented in computers? X = text X = numbers X = anything else Encoding Information Comp411 – Spring 2007 1/16/2007 L02 - Information 42