Download Part A

CS 208: Computing Theory
Assoc. Prof. Dr. Brahim Hnich
Faculty of Computer Sciences
Izmir University of Economics
Contact Info



Room 415
Email: [email protected]
Office hours

Tuesdays: 14—16
Textbooks

The course lecture are based on the following
books

Harry R. Lewis and C.H. Papadimitriou. "Elements of
the Theory of Computation", ISBN 0-13-272741-2,
Prentice Hall. (Available in library)

Michael Sipser. "Introduction to the Theory of
Computation", ISBN 053494728X, PWS Publishing
Company.
Textbooks

The course lecture are based on the following
books

Michael R. Garey and David S. Johnson. “Computer
and Intractability", ISBN 0-7167-1045-5, W.H.
Freeman and Company.

C.H. Papadimitriou. “Computational Complexity",
ISBN 0-201-53082-1, Addison-Wesley Publishing
Company, Inc.
Grading and exams



6 Homeworks: 30%
Midterm Exam 30%
Final Exam : 40%
Programme

Part A: Preliminaries

Introduction

Mathematical Notations for Computations
Programme

Part B: Automata Theory

Finite State Automata

Context Free Grammars
Programme

Part C: Computability Theory

Turing Machines

Decidability and Undecidability
Programme

Part D: Complexity Theory

Time Complexity

Intractability
Programme

Part E: Summary and Concluding
Remarks
Good Luck!
Outline

Introduction to the
course




Goal
Organization
Outline
Mathematical
Notations for
Computations
What is computation?

Definition:
Processing of
information based on
a finite set of
operations or rules
What is computation?

Paper and pencil Arithmetic






99 - 8 = 91
Calculators
Bar code scanner
Digital camera
Computer programs in C and Java
…
What do we want in a “theory”

Generality



Technology-independent
Abstraction: ignores inessential details
Precision


Mathematical, formal
Can prove theorems about computation


Positive: what can be computed
Negative: what cannot be computed
So you want to be a good computer
scientist/ software engineer?

Learn the rules


Learn basic principles



Algorithms, data structures, …
Data abstraction, …
Understand which problems you
can solve with computers and
which you simply cannot
… and why!
So you want to be a good computer
scientist/ software engineer?

Learn the rules


Learn basic principles


Algorithms, data structures, …
Data abstraction, …
Understand which problems you
can solve efficiently with
computers and which you
cannot!
Course Overall Objective


Understand the
fundamental capabilities
and limitations of
computers
Make a theory out of the
idea of computation
Why study theory?

Pragmatic Reasons

Apply efficient algorithms
to tractable problems

Avoid intractable or
impossible problems

Learn to tell the
difference
Why study theory?

Mathematical education

Automata Theory

Computability Theory

Complexity Theory
Why study theory?

Philosophy

What is computation?

What is provable?
Course of two parts
The Good News

You only have to take this
course once


Except those of you who
will not pay attention to
the lectures
At the end, you can say
that you understand what
computers are really
about

Good for postgraduate
and professional life
2nd half of the course
The Bad News

Theory course




Formal
Requires mathematics
Understanding Proofs
Analytical skills
The Bad News

Theory course




Formal
Requires mathematics
Understanding Proofs
Analytical skills
But, we will try to make it fun because it is really interesting!
End of Introduction
Next: Topics to be covered
Three major topics

What is a computer?


What can (cannot) be
computed?


Automata Theory
Computability Theory
What can (cannot) be
computed efficiently?

Complexity Theory
Three major topics

What is a computer?


What can (cannot) be
computed?


Automata Theory
Computability Theory
What can (cannot) be
computed efficiently?

Complexity Theory
Complexity Theory
Complexity Theory

Key notion: tractable vs. intractable
problems
Complexity Theory

Key notion: tractable vs. intractable
problems
EASY
HARD
Complexity Theory

A problem is a general question:



Description of parameters
Description of solution
E.g.


Input: Given a list of integers
Can you sort the list in ascending order?
Complexity Theory

An algorithm is a step-by-step procedure:



A recipe
A computer program
A mathematical object
Complexity Theory

We want the most efficient algorithm as a
function of problem size:

Fastest (mostly)

Most economical with memory (sometimes)
Example: Traveling Salesman
Problem

Given



A set of m cities
A set of inter-city
distances
Find a permutation of
cities which makes a tour
of minimum length
Example: TSP instance
a
10
5
6
b
9
c
3
9
(not drawn to scale)
d
Example: TSP instance
a
10
5
6
b
9
c
9
A solution:
The tour a,b,d,c,a has length 27
3
d
Problem Size

What is an appropriate measure of
problem size?


m the number of nodes?
m(m+1)/2 distances?
Problem Encoding

Use an encoding of the problem


Alphabet of symbols
Strings: abcd//10/5/9//6/9//3
Measures


Problem size: length of the encoding
Time complexity: how long an algorithm
takes, as a function of problem size
Time Complexity

What is tractable?

A function f(n) is O(g(n)) whenever
f(n) ≤ c. g(n)
for n > some constant
Time Complexity
A polynomial-time algorithm is one whose
time complexity is O(p(n)) for some
polynomial p(n)
e.g. O(n2)
Time Complexity
An exponential-time algorithm is one whose
time complexity cannot be bounded by a
polynomial
e.g. O(nlog n)
Tractability

Basic distinction


Polynomial time means tractable
Exponential time means intractable
Tractability
Execution time
10
30
60
n
.00001 sec
.00003 sec
.00006 sec
n2
.00001 sec
.00009 sec
.00036 sec
n3
.00001 sec
.027 sec
.216 sec
2n
.001 sec
17.9 min
3n
0.59 sec
6.5 years
336
centuries
1.3 1013
centuries
Effect of Speed-Ups
Let us wait for faster hardware!
Present
100 times
problem size faster
1000 times
faster
n
N1
100N1
1000N1
n2
N2
10N2
31.6N2
n3
N3
4.64N3
10N3
2n
N4
N4+6.64
N4+9.97
3n
N5
N5+4.19
N5+6.29
Back to TSP

Your boss says:


“Get me an efficient traveling-salesman
algorithm, or else!”
What are you going to do?
Response

“Yes Ma’am, expect it this afternoon!”
Response


“Yes Ma’am, expect it this afternoon!”
Problem is



All known algorithms (essentially) check all
possible paths
Exhaustive checking is exponential!
… so best of luck!
Response

“hah!, I will prove that no polynomial
algorithm is possible”
Response


“hah!, I will prove that no polynomial
algorithm is possible”
Problem is


Proving intractability is very hard
Many important problems have


No known tractable algorithms
No known proof for intractability
Response

“I can’t find an efficient algorithm, I guess
I’m just a pathetic loser!”
Response


“I can’t find an efficient algorithm, I guess
I’m just a pathetic loser!”
Bad for job security
Perfect Response


“The problem is NP-Complete. I cant find
an efficient algorithm, but neither can any
of these famous people …!”
Advantage is


The problem is “just as hard” as other
problems smart people can’t solve efficiently.
So it would no good to fire you and fire
someone else to do the job
Perfect Response

“Would you settle for a pretty good
algorithm, but that does not give you the
optimal solution?”
Perfect Response


“Would you settle for a pretty good
algorithm, but that does not give you the
optimal solution?”
Intractability isn’t the end of the story



Find an approximate solution
Use randomization
Parallelism can help
Computability Theory
Computability Theory

Complexity theory divides problems into
two classes



Tractable
Intractable
Computability theory divides problems
into two classes


Decidable
Un-decidable
Computability Theory:
important discovery

In the first half of the 20th century,
mathematicians like Kurt Goedel, Alan
Turing, and Alonzo Church discovered
that some problems do not have a
computable solution!
Example of undecidable
problem

Is a mathematical statement true or
false?


N.B. what could be more amenable to
automation?
But, no computer algorithm can perform this
task!
Example of undecidable
problem

Is a mathematical statement true or
false?


N.B. what could be more amenable to
automation?
But, no computer algorithm can perform this
task!
These results require theoretical models
for computers Automata Theory  construction of real
computers
Another example

Hilbert’s tenth problem (1900): Find a
finite algorithm that decides whether a
polynomial has an integer root.
Another example




A polynomial is a sum of terms, each term is a
product of variables and constants (coefficients)
A root of a polynomial is an assignment of
values to the variables such the value of the
polynomial is 0
x=2 and y=2 is a root for 5x +15y=25
There is no such algorithm (1970)!
More example of undecidable
problems





Does a program run forever?
Is a program correct?
Are two programs equivalent?
Is a program optimal?
…
Automata Theory
Automata Theory


Deals with the definitions and properties
of mathematical models of computation
Several models


Finite automata: is used in text processing,
compilers, and hardware design
Context-free grammar: is used in
programming language and artificial
intelligence (language processing)
Automata Theory

Complexity and Computability theory
require a precise definition of a computer

Thus Automata theory is the best starting
point to begin the study of the theory of
computation!
Mathematical Notations and Terminology
Outline






Sets
Functions and relations
Graphs
Strings and languages
Boolean logic
Proofs
Sets

Sets are defined by its members


E.g. {1,2,4} or {blue, green, red}
Order is not important
Set Cardinality

Sets can be




finite such as {1, 2}
infinite such as {x| x is even} i.e. the set of
even naturals
If A is finite then its cardinality (or size)
denoted by |A| is the number of its
elements
The empty Ø set has cardinality 0
Set operations

Membership


Subset


{a,b} U {c,d} = {a,b,c,d}
Intersection


A=B means that for x, x Є A iff x Є B
Union


A ⊆ B iff for every x, x Є A implies x Є A
Equality


x Є A means that x is a member of the set A
{a,b} ∩ {a,c} = {a}
Difference

{a,b} - {b,c} = {a}
Set Operations


A and B are disjoint iff A ∩ B = Ø
The power set of A, P(A) = {B | B ⊆ A}


E.g., A= {1, 2} and P(A)={Ø, {1}, {2}, {1,2}}
Cartesian Product


A X B ={{a,b} | a Є A and b Є B}
E.g. {1,2} x {3,4} = {(1,3), (1,4), (2,3), (2,4)}
Relations and functions

A k-ary relation R on A1,…,Ak is a subset
of A1 x A2 X … X Ak

A binary relation on A is a subset of A x A
Example of a relation
a
b
c
d
R= {(a, b), (a, c), (b, d), (d, b), (d, d)}
Example of a relation
a
a
b
b
d
d
c
c
R= {(a, b), (a, c), (b, d), (d, b), (d, d)}
Properties of binary relations

Reflexive

(a,a) Є R for each a Є A
a
Properties of binary relations

Symmetric

If
a
If (a,b) Є R then (b,a) Є R
b
then
b
a
Properties of binary relations

Transitive

If (a,b) Є R and (b,c) Є R then (a,c) Є R
a
If
b
Then
And
b
c
a
c
Equivalence relation

An equivalence relation is a relation that
is



Reflexive
Symmetric
Transitive
Examples
Transitive
Not symmetric
Not reflexive
Examples
Not Transitive
Symmetric
Not reflexive
Examples
Not Transitive
Symmetric
Reflexive
Examples
Transitive
Symmetric
Reflexive
Partial Functions

A partial function f: S  T satisfies



f⊆SXT
For all s Є S there is at most one t Є T with
(s, t) Є f
Note that (s, t) Є f we write f(s)=t
Partial function example
a
a
f:{a, b, c, d}  {a, b, c, d}
b
b
f= {(a, b), (b, d), (d, d)}
d
d
c
c
Total Functions

A partial function f: S  T satisfies





f⊆SXT
For all s Є S there is exactly one t Є T with
(s, t) Є f
Note that (s, t) Є f we write f(s)=t and we say
that t is the image of s under f
S is called the domain of f
T is called the range of f
Total function example
a
a
f:{a, b, c, d}  {a, b, c, d}
b
b
f= {(a, b), (b, d), (c,d), (d, d)}
d
d
c
c
Injections

A function f:ST is injective if distinct
elements in its domain S have distinct
images

For all a,b Є S. a ≠ b  f(a) ≠ f(b)
Injections

A function f:ST is injective if distinct
elements in its domain S have distinct
images

1
2
3
For all a,b Є S. a ≠ b  f(a) ≠ f(b)
1
2
3
1
2
3
1
2
3
Surjections

A function f:ST is surjective if every b
in T is the image of some a in S

For all b Є T. there exists a Є S: f(a) =b
Surjections

A function f:ST is surjective if every b
in T is the image of some a in S

1
2
3
For all b Є T. there exists a Є S: f(a) =b
1
2
3
1
2
3
1
2
3
Bijections

A function f:ST is bijective iff it is
injective and surjective. Also called a 1-1
correspondence
Bijections

1
2
A function f:ST is bijective iff it is
injective and surjective. Also called a 1-1
correspondence
1
2
3
1
2
3
1
2
3
Relations
Relations
Partial functions
Relations
Partial functions
Total functions
Relations
Partial functions
Injections
Total functions
Relations
Partial functions
Injections
Surjections
Total functions
Relations
Partial functions
Injections
Bijections
Surjections
Total functions
Graphs

Graph G=<V,E>, where



V is the set of nodes or vertices
E is the set of edges
Degree of a vertex is the number of
edges including that vertex
Graphs

Graph G=<V,E>, where


1
2
V is the set of nodes or vertices
E is the set of edges
b
a
5
4
3
d
c
Graph
Subgraph
Path
Graph
path
A path in a graph is a sequence of nodes connected by edge
A simple path is a path that doesn’t repeat any node
A graph is connected if every two nodes have a path between them
Fully connected graph
1
2
Not fully connected graph
b
a
5
4
3
d
c
Cycle
cycle
A cycle is a path that starts and ends at the same node
A simple cycle is a cycle that doesn’t repeat any node except the
first and the last
root
tree
leaves
A graph is a tree if it is connected and has no simple cycles
The nodes of degree 1 are called leaves
There is a specially designated node called a root
Directed Graphs

Graph G=<V,E> is directed if it has arrows
instead of edges


V is the set of nodes or vertices
E is the set of directed edges
b
a
d
c
Directed Graphs


The number of arrows pointing from a particular
node is the outdegree
The number of arrows pointing to a particular
node is the indegree
b
a
d
c
Strings and languages




An alphabet is a finite set of symbols
A string over an alphabet is a finite
sequence of symbols from the alphabet
The length of a string is the number of
symbols in the string
The empty string is denoted by ε and has
length 0
Strings and languages





Reverse: abcd vs. dcba
Substring: bc is a substring of abcd
Concatenation: of ab and cd is abcd
xk is x...x, k times
A language is a set of strings
Boolean logic

Boolean logic is a mathematical system
built around the two values TRUE (1) and
FLASE (0)


Foundation of digital electronics and
computer design
TRUE and FALSE are called the Boolean
values
Boolean operations


Negation: not(1)=0 and NOT(0)=1
Conjunction:




1 AND 0 = 0
1 AND 1 = 1
0 AND 0 = 0
0 AND 1 = 0
Boolean expressions

Use Boolean operations for combining simple
statements





Let P stand for the truth value of “The sun is shining”
Let Q stand for the truth value of “Today is Tuesday”
“P AND Q” stands for “The sun is shining and Today is
Tuesday”
“P OR Q” stands for “The sun is shining or Today is
Tuesday”
P and Q are called operands of the operation
More logic operations


Equality: P ↔ Q is 1 iff both of its
operands have the same truth value
Implication: P  Q is equivalent NOT(P)
OR Q
Distributive law
P AND (Q OR R) equals
(P AND Q) OR (P AND R)
P OR (Q AND R) equals
(P OR Q) AND (P OR R)
Definition

Definitions describe the objects and the
notions that we use.



Simple or complex
Precise and concise
When defining some object we must make
clear what constitutes that object and what
does not!
Proofs

Proofs determine the truth or falsity of a
mathematical statement



Finding proofs isn’t always easy
There is no recipe
… but there are some guidelines

Actually the more proofs you do the better you get
at it!
Proofs: Guidelines






Use examples and pictures
Be patient
Come back to it
Be neat
Be concise
Use a strategy



I’ll attempt a proof by induction strategy
It seems that I can find a counter-example easily
etc
Proof types





Proof by construction
Proof by contradiction
Proof by induction
…
Or a combination
Proof by construction


Many theorems state that a particular
type of objects exist.
Proof by construction demonstrates how
to build such an object

To show that there exists an O(nlog n) sorting
algorithm, we can construct the quick-sort
algorithm
Proof by contradiction

Proof by contradiction: we assume that
the theorem is false and then show that
this assumption leads to a contradiction
Proof by contradiction

Prove by contradiction that the square
root of 2 is an irrational number


N.B. A number is rational if it is a fraction m/n
where m and n are integers.
A number is irrational iff it is not rational
Proof by contradiction
Let us assume that sqrt(2) is rational
This means there exists relatively prime
integers m and n such that
sqrt(2)= m/n
Note that m and n cannot both be even
Now rewrite sqrt(2)= m/n as
n sqrt(2) = m
Proof by contradiction
Let us take the square
(n sqrt(2))2 = m2
n 2 2 = m2
Thus m2 is even and so is m (= 2 k)
So we get n2 2 = (2k)2 =4 k2
Proof by contradiction
So we get n2 2 = (2k)2 =4 k2
Which simplifies to n2=2k2
Thus n2 is even and so is n
This is a contradiction! QED
Proof by induction

To prove P(n)

Base case: prove P(0)

Induction hypothesis: assume P(k) holds for
some k≤n

Induction step: Given the induction
hypothesis show that P(n+1) holds
Proof by induction

Prove by induction


1+2+…..+n = n(n+1)/2
Any volunteers?
Conclusions


Introduction
Course overview




Complexity Theory
Computability Theory
Automata Theory
Formal background