Download here

What is “Computation”?
Computation is about “processing information”
-
Comp411 – Spring 2007
Transforming information from one form
to another
Deriving new information from old
Finding information associated with a
given input
“Computation” describes the motion of
information through time
“Communication” describes the motion of
information through space
1/16/2007
L02 - Information 1
What is “Information”?
“ 10 Problem sets, 2
quizzes, and a final!”
information, n. Knowledge
communicated or received
concerning a particular
fact or circumstance.
Carolina won again.
Tell me something
new…
A Computer Scientist’s Definition:
Information resolves uncertainty.
Information is simply that which
cannot be predicted.
The less predictable a message is, the
more information it conveys!
Comp411 – Spring 2007
1/16/2007
L02 - Information 2
Real-World Information
Why do unexpected messages
get allocated the biggest
headlines?
… because they carry the
most information.
Comp411 – Spring 2007
1/16/2007
L02 - Information 3
What Does A Computer Process?
• A Toaster processes bread and bagels
• A Blender processes smoothies
and margaritas
• What does a
computer process?
• 2 allowable answers:
– Information
– Bits
• What is the mapping from
information to bits?
Comp411 – Spring 2007
1/16/2007
L02 - Information 4
Quantifying Information
(Claude Shannon, 1948)
Suppose you’re faced with N equally probable choices, and
I give you a fact that narrows it down to M choices. Then
I’ve given you
log2(N/M) bits of information
Information is measured
in bits (binary digits) =
number of 0/1’s required
to encode choice(s)
Examples:
w information in one coin flip: log2(2/1) = 1 bit
w roll of a single die: log2(6/1) = ~2.6 bits
w outcome of a Football game: 1 bit
(well, actually, “they won” may convey more
information than “they lost”…)
Comp411 – Spring 2007
1/16/2007
L02 - Information 5
Example: Sum of 2 dice
i2 = log2(36/1) = 5.170 bits
i3 = log2(36/2) = 4.170 bits
i4 = log2(36/3) = 3.585 bits
i5 = log2(36/4) = 3.170 bits
i6 = log2(36/5) = 2.848 bits
i7 = log2(36/6) = 2.585 bits
i8 = log2(36/5) = 2.848 bits
i9 = log2(36/4) = 3.170 bits
i10 = log2(36/3) = 3.585 bits
i11 = log2(36/2) = 4.170 bits
i12 = log2(36/1) = 5.170 bits
2
3
4
5
6
7
8
9
10
11
12
The average information provided by the sum of 2 dice:
iave 
Comp411 – Spring 2007
12

i 2
Mi
N
Entropy
N)
log2 ( M
 pi log2 ( pi )  3.274 bits
i
i
1/16/2007
L02 - Information 6
Show Me the Bits!
Can the sum of two dice REALLY be represented
using 3.274 bits? If so, how?
The fact is, the average
information content is a
strict *lower-bound* on how
small of a representation
that we can achieve.
In practice, it is difficult
to reach this bound. But,
we can come very close.
Comp411 – Spring 2007
1/16/2007
L02 - Information 7
Variable-Length Encoding
• Of course we can use differing numbers of “bits” to
represent each item of data
• This is particularly useful if all items are *not* equally
likely
• Equally likely items lead to fixed length encodings:
– Ex: Encode a particular roll of 5?
– {(1,4), (2,3), (3,2), (4,1)} which are equally likely if we use fair dice
4
4
– Entropy =   p(roll i roll  5) log 2 ( p(roll i roll  5))   14 log 2 ( 14 )  2
i 1
i 1
bits
– 00 – (1,4), 01 – (2,3), 10 – (3,2), 11 – (4,1)
• Back to the original problem. Let’s use this encoding:
2 - 10011
6 - 111
10 - 1000
Comp411 – Spring 2007
3 - 0101
7 - 101
11 - 0100
1/16/2007
4 - 011
8 - 110
12 - 10010
5 - 001
9 - 000
L02 - Information 8
Variable-Length Encoding
• Taking a closer look
2 - 10011 3 - 0101
4 - 011
5 - 001
6 - 111
7 - 101
8 - 110
9 - 000
10 - 1000 11 - 0100 12 - 10010
Unlikely rolls are encoded using more bits
More likely rolls use fewer bits
• Decoding
2
5
3
6 5 8
3
Example Stream: 1001100101011110011100101
Comp411 – Spring 2007
1/16/2007
L02 - Information 9
Huffman Coding
•
A simple *greedy* algorithm for approximating an
entropy efficient encoding
1.
2.
3.
4.
Find the 2 items with the smallest probabilities
Join them into a new *meta* item whose probability is the sum
Remove the two items and insert the new meta item
Repeat from step 1 until there is only one item
Huffman decoding
tree
36/36
15/36
21/36
7/36
8/36
9
4/36
5
11/36
4
4/36
11
2/36
3
2/36
6/36
10
8
5/36
6
5/36
2/36
3/36
12
Comp411 – Spring 2007
7
5/36
3/36
4/36
10/36
1/16/2007
1/36
2
1/36
L02 - Information 10
Converting Tree to Encoding
Once the *tree* is constructed, label its edges consistently
and follow the paths from the largest *meta* item to
each of the real item to find the encoding.
2 - 10011
6 - 111
10 - 1000
3 - 0101
7 - 101
11 - 0100
0
0
0
9
4/36
8/36
15/36
4 - 011
8 - 110
12 - 10010
36/36
5
4/36
0
11
2/36
1
3
2/36
4
3/36
0
1
5/36
0
10
3/36
1
0
12
Comp411 – Spring 2007
1
11/36
1
0
4/36
21/36
0
7/36
1
Huffman decoding
tree
1
1
5 - 001
9 - 000
1/16/2007
1/36
2/36
7
6/36
0
8
5/36
10/36
1
6
5/36
1
2
1/36
L02 - Information 11
Encoding Efficiency
How does this encoding strategy compare to the
information content of the roll?
1 5  2 4  3 3  4 3  5 3  6 3
bave  36
36
36
36
36
36
5 3  4 3  3 4  2 4  1 5
 36
36
36
36
36
bave  3.306
Pretty close. Recall that the lower bound was 3.274 bits.
However, an efficient encoding (as defined by having an
average code size close to the information content) is
not always what we want!
Comp411 – Spring 2007
1/16/2007
L02 - Information 12
Encoding Considerations
- Encoding schemes that attempt to match the
information content of a data stream remove
redundancy. They are data compression techniques.
- However, sometimes our goal in encoding information
is increase redundancy, rather than remove it. Why?
Make the information easier to manipulate
(fixed-sized encodings)
Make the data stream resilient to noise
(error detecting and correcting codes)
-Data compression allows us to
store our entire music collection
in a pocketable device
Comp411 – Spring 2007
1/16/2007
-Data redundancy
enables us to store
that *same*
information
*reliably* on
a hard drive
L02 - Information 13
Error detection using parity
Sometimes we add extra redundancy so that we
can detect errors. For instance, this encoding
detects any single-bit error:
ex:
2-1111000
3-1111101
4-0011
5-0101
6-0110
7-0000
8-1001
9-1010
10-1100
11-1111110
12-1111011
Comp411 – Spring 2007
1:
2:
3:
4:
4
001111100000101
4
6*
7
4
9*
7
4
10*
7
001111100000101
001111100000101
001111100000101
4
2*
There’s something
peculiar about
those codings
5
001111100000101
Same bitstream –
w/4 possible interpretations
if we allow for only one error
1/16/2007
L02 - Information 14
Property 1: Parity
The sum of the bits in each symbol is even.
(this is how errors are detected)
2-1111000 = 1 + 1 + 1+ 1 + 0 + 0 + 0 = 4
3-1111101 = 1 + 1 + 1 + 1 + 1 + 0 + 1 = 6
4-0011 = 0 + 0 + 1 + 1 = 2
5-0101 = 0 + 1 + 0 + 1 = 2
6-0110 = 0 + 1 + 1 + 0 = 2
7-0000 = 0 + 0 + 0 + 0 = 0
8-1001 = 1 + 0 + 0 + 1 = 2
9-1010 = 1 + 0 + 1 + 0 = 2
10-1100 = 1 + 1 + 0 + 0 = 2
11-1111110 = 1 + 1 + 1 + 1 + 1 + 1 + 0 = 6
12-1111011 = 1 + 1 + 1 + 1 + 0 + 1 + 1 = 6
Comp411 – Spring 2007
1/16/2007
How much
information
is in the
last bit?
L02 - Information 15
Property 2: Separation
Each encoding differs from all others by at least
two bits in their overlapping parts
2
1111000
3
4
5
6
7
8
9
10
11
1111101
0011
0101
0110
0000
1001
1010
1100
1111110
3
1111101
4
0011
5
0101
6
0110
7
0000
8
1001
9
1010
10
1100
11
1111110
12
1111011
1111x0x
xx11
x1x1
x11x
xxxx
1xx1
1x1x
11xx
1111xx0
11110xx
xx11
x1x1
0xx1
x11x
0x1x
01xx
xxxx
00xx
0x0x
0xx0
1xx1
x0x1
xx01
xxxx
x00x
1x1x
x01x
xxxx
xx10
x0x0
10xx
11xx
xxxx
x10x
x1x0
xx00
1x0x
1xx0
11111xx
xx11
x1x1
x11x
xxxx
1xx1
1x1x
11xx
1111xx1
xx11
x1x1
x11x
xxxx
1xx1
1x1x
11xx
1111x1x
This difference is called
the “Hamming distance”
“A Hamming distance of 1
is needed to uniquely
identify an encoding”
Comp411 – Spring 2007
1/16/2007
L02 - Information 16
A Short Detour
It is illuminating to consider Hamming distances geometrically.
10
00
11
01
Given 2 bits the largest Hamming distance
that we can achieve between 2 encodings is 2.
This allows us to detect 1-bit errors if we
encode 1 bit of information using 2 bits.
With 3 bits we can find 4 encodings with a Hamming distance of 2,
allowing the detection of 1-bit errors when
3 bits are used to encode 2 bits of information.
110
010
111
011
100
000
101
001
We can also identify 2 encodings
with Hamming distance 3. This
extra distance allows us to detect
2-bit errors. However, we could
use this extra separation
differently.
Encodings separated by
a Hamming distance of 2.
Comp411 – Spring 2007
110
010
111
011
100
000
101
001
Encodings separated by
a Hamming distance of 3.
1/16/2007
L02 - Information 17
Error correcting codes
We can actually correct 1-bit errors in encodings separated by a
Hamming distance of 2. This is possible because the sets of bit
patterns located a Hamming distance of 1 from our encodings are
distinct.
110
However, attempting error correction
with such a small separation is dangerous.
Suppose, we have a 2-bit error. Our error
correction scheme will then misinterpret
the encoding. Misinterpretations also
occurred when we had 2-bit errors in our
1-bit-error-detection (parity) schemes.
A safe 1-bit error correction scheme
would correct all 1-bit errors and detect
all 2-bit errors. What Hamming distance
is needed between encodings to
accomplish this?
Comp411 – Spring 2007
1/16/2007
010
111
011
100
000
101
001
This is just
a voting
scheme
L02 - Information 18
Information Summary
Information resolves uncertainty
• Choices equally probable:
• N choices down to M 
log2(N/M) bits of information
•Choices not equally probable:
• choicei with probability pi 
log2(1/pi) bits of information
• average number of bits = pilog2(1/pi)
• use variable-length encodings
Comp411 – Spring 2007
1/16/2007
L02 - Information 19
Computer Abstractions
and Technology
1. Layer Cakes
2. Computers are translators
3. Switches and Wires
(Read Chapter 1)
Comp411 – Spring 2007
1/16/2007
L02 - Information 20
Computers Everywhere
The computers we are used to
Desktops
Laptops
Embedded processors
Cars
Mobile phones
Toasters, irons, wristwatches, happy-meal toys
Comp411 – Spring 2007
1/16/2007
L02 - Information 21
A Computer System
What is a computer system?
Where does it start?
Where does it end?
for (i = 0; i < 3; i++)
m += i*i;
Assembler and Linker addi $8, $6, $6
sll $8, $8, 4
CPU
Compiler
A
Module
B
ALU
A
Cells
CO
B
FACI
S
Gates
Transistors
Comp411 – Spring 2007
1/16/2007
L02 - Information 22
Computer Layer Cake
Applications
Systems software
Shared libraries
Operating System
Hardware – the bare metal
Computers are
digital Chameleons
Apps
Systems S/W
Libraries
Operating System
Hardware
Comp411 – Spring 2007
1/16/2007
L02 - Information 23
Computers are Translators
User-Interface (visual programming)
High-Level Languages
Compilers
Interpreters
Assembly Language
Machine Language
int x, y;
y = (x-3)*(y+123456)
Comp411 – Spring 2007
x:
y:
c:
.word 0
.word 0
.word 123456
...
lw
addi
lw
lw
add
mul
sw
1/16/2007
$t0,
$t0,
$t1,
$t2,
$t1,
$t0,
$t0,
x
$t0, -3
y
c
$t1, $t2
$t0, $t1
y
L02 - Information 24
Computers are Translators
User-Interface (visual programming)
High-Level Languages
Compilers
Interpreters
Assembly Language
Machine Language
x:
y:
c:
.word 0
.word 0
.word 123456
...
lw
addi
lw
lw
add
mul
sw
Comp411 – Spring 2007
$t0,
$t0,
$t1,
$t2,
$t1,
$t0,
$t0,
0x04030201
0x08070605
0x00000001
0x00000002
0x00000003
0x00000004
0x706d6f43
x
$t0, -3
y
c
$t1, $t2
$t0, $t1
y
1/16/2007
L02 - Information 25
Why So Many Languages?
Application Specific
Historically: COBOL vs. Fortran
Today: C# vs. Java
Visual Basic vs. Matlab
Static vs. Dynamic
Code Maintainability
High-level specifications are
easier to understand and modify
Code Reuse
Code Portability
Virtual Machines
Comp411 – Spring 2007
1/16/2007
L02 - Information 26
Under the Covers
Input
Output
Storage
Processing
Datapath
Control
Comp411 – Spring 2007
1/16/2007
L02 - Information 27
Under the Covers
Input
Output
Storage
Processing
Datapath
Control
Comp411 – Spring 2007
1/16/2007
L02 - Information 28
Under the Covers
Cathode Ray Tube (CRT)
Input
Output
Storage
Processing
The “last vacuum tube”
Now nearing extinction
Datapath
Control
Comp411 – Spring 2007
1/16/2007
L02 - Information 29
Under the Covers
Input
Output
Storage
Processing
Liquid Crystal Displays (LCDs)
Datapath
Control
Comp411 – Spring 2007
1/16/2007
L02 - Information 30
Under the Covers
Input
Output
Storage
Processing
Datapath
Control
Comp411 – Spring 2007
1/16/2007
L02 - Information 31
Under the Covers
Input
Output
Storage
Processing
Datapath
Control
Comp411 – Spring 2007
1/16/2007
L02 - Information 32
Under the Covers
Input
Output
Storage
Processing
Intel Pentium III Xeon
Datapath
Control
Comp411 – Spring 2007
1/16/2007
L02 - Information 33
Implementation Technology
Relays
Vacuum Tubes
Transistors
Integrated Circuits
Gate-level integration
Medium Scale Integration (PALs)
Large Scale Integration (Processing unit on a chip)
Today (Multiple CPUs on a chip)
Nanotubes??
Quantum-Effect Devices??
Comp411 – Spring 2007
1/16/2007
L02 - Information 34
Implementation Technology
Common Links?
A controllable switch
Computers are wires and switches
open
open
closed
control
Comp411 – Spring 2007
1/16/2007
L02 - Information 35
Chips
Silicon Wafers
Chip manufactures build many copies of
the same circuit onto a single wafer.
Only a certain percentage of the chips
will work; those that work will run at
different speeds. The yield decreases
as the size of the chips increases and
the feature size decreases.
Wafers are processed by automated
fabrication lines. To minimize the
chance of contaminants ruining a
process step, great care is taken to
maintain a meticulously clean
environment.
Comp411 – Spring 2007
1/16/2007
L02 - Information 36
Field Effect Transistors (FETs)
Modern silicon fabrication technology is optimized to build a
particular type of transistor. The flow of electrons from
the
VDS
source to the drain
is
controlled
by
a
gate
voltage.
Source
Gate
Drain
IDS = 0
n+
n+
p
Bulk
Comp411 – Spring 2007
1/16/2007
L02 - Information 37
Chips
Silicon Wafers
IBM photomicrograph (Si has been removed!)
Metal 2
M1/M2 via
Metal 1
Polysilicon
Diffusion
Mosfet (under polysilicon gate)
Comp411 – Spring 2007
1/16/2007
L02 - Information 38
How Hardware WAS Designed
20 years ago
I/O Specification
Truth tables
State diagrams
Logic design
Circuit design
Circuit Layout
Comp411 – Spring 2007
1/16/2007
L02 - Information 39
How Hardware IS Designed
Today (with software)
High-level hardware specification languages
Verilog
VHDL
Comp411 – Spring 2007
1/16/2007
L02 - Information 40
Reconfigurable Chips
Programmable Array Logic (PALs)
Fixed logic / programmable wires
Field Programmable Gate Arrays (FPGAs)
Repeated reconfigurable logic cells
Comp411 – Spring 2007
1/16/2007
L02 - Information 41
Next Time
Computer Representations
How is X represented in computers?
X = text
X = numbers
X = anything else
Encoding Information
Comp411 – Spring 2007
1/16/2007
L02 - Information 42