Download Triangulation and Embedding using Small Sets of Beacons

Instabilities and Oscillations in
Networks of Queues
Matthew Andrews
Bell Labs
Joint work with Alex Slivkins (Cornell)
September 27, 2005
1
Network of routers and sessions
S1


S2
S3
router A
router B
t2
t3
t1
session: sender  fixed flow path  receiver
router: intersection inside the network
2
Queueing Model
S1
S2
S3




t1
t2
Model router as simple queue
Each queue has a capacity =
service rate
if total arrival rate  capacity
then all traffic is routed
without congestion or queuing
else, congestion occurs:
queue builds up,
traffic is delayed
t3
3
Network of Queues
S1
S2
S3
router A
router B
t2
S1
t1
S2
modeled as
S3
t3
t2
t3
t1
4
FIFO Queue
S1
S2
S3



t1
t2
FIFO queue serves packets in
order received
We consider idealized model in
which packet size  0
Study rate at which data
arrives at/departs from queue
t3
5
Fluid-based FIFO queue


session = continuous flow, not a discrete stream of packets
what is a fluid-based FIFO queue?
[Bramson]
S1
S2
S3


h

C
t1
t2

t3
h(t) = height of the queue at time t
= [total amount of fluid] / [capacity C]
given arrival rates in1(t), in2(t), in3(t)
what is h(t) and what are exit rates outi(t)?
if h(t) = 0 and j inj(t)  C then
all flow just comes through: outi(t) = ini(t)
else, capacity is divided proportionally,
and the excessive fluid is queued up
 outi(t+h(t)) = ini(t)  C / j inj(t)
6
Talk Outline


Part 1: Traffic generated by external arrival process
 Study “queueing theoretic stability”
 Can we keep the queue sizes bounded?
 Result: FIFO can be unstable at arbitrarily small network loads
Part 2: Traffic controlled by end-to-end congestion control (e.g. TCP)
 Study “control theoretic stability”
 Does the system oscillate?
 Result: end-to-end congestion control can cause oscillations
7

Network of FIFO queues, external traffic
arrivals
Definitions
 Stable: queuesizes remain bounded

Unstable: queuesizes are unbounded
Stable
Time
Total
queuesize

Total
queuesize
Part 1: FIFO stability
Unstable
Time

We would like queues to be stable as long
as no queue is overloaded
??
8
Permanent sessions: Constant arrivals
0.45
S1
S2
0.45
r = 0.9
S3 0.45
Thm: (Bramson ‘96)
FIFO stable for all r  1
t2
t3





t1
Each session i has rate ri
Injection rate into session i equals ri at all times
Sum of session rates passing through capacity C queue  rC
r = network load parameter
Interesting case is r  1
9
Permanent sessions: Bounded arrivals
0.45
S1
S2
0.45
r = 0.9
S3 0.45
t2
t3




t1
Each session i has rate ri
Injection rate into session i at most ri at all times
Sum of session rates passing through capacity C queue  rC
Standard model in networking literature, e.g.
 Fair Queueing literature (e.g. Parekh-Gallager),
 Network calculus (e.g. Cruz, Le Boudec-Thiran book)
10
Permanent sessions: Bounded arrivals
0.2
S1
S2
0.45
r = 0.9
S3
0.1
t3
t1
t2




Each session i has rate ri
Injection rate into session i at most ri at all times
Sum of session rates passing through capacity C queue  rC
Standard model in networking literature, e.g.
 Fair Queueing literature (e.g. Parekh-Gallager),
 Network calculus (e.g. Cruz, Le Boudec-Thiran book)
11
Permanent sessions: Bounded arrivals
0.45
S1
S2
0.3
r = 0.9
S3 0.45
Thm: (‘00)
FIFO can be unstable
for r > 0.999999997
t2
t3




t1
Each session i has rate ri
Injection rate into session i at most ri at all times
Sum of session rates passing through capacity C queue  rC
Standard model in networking literature, e.g.
 Fair Queueing literature (e.g. Parekh-Gallager),
 Network calculus (e.g. Cruz, Le Boudec-Thiran book)
12
Main Result



What happens at small network loads?
Is there some network load below which FIFO is stable?
 No!!!!
Thm: For all e > 0, there exists network and injection pattern s.t.
 Network load is r < e
 The network is unstable
13
Transfer Process

How can we move fluid from one FIFO queue to another?
Injection rate e
Existing fluid
of height 1
14
Transfer Process

How can we move fluid from one FIFO queue to another?
Arrival rate
Injection rate e
1/1+e
Total arrival
rate 1 + e
Departure rate 1
Departure rate
1/1+e
Departure rate
e/1+e
15
Transfer Process

How can we move fluid from one FIFO queue to another?
Arrival rate
Injection rate e
1/1+e
Total arrival
rate 1 + e
Departure rate 1
Departure rate
1/1+e
Departure rate
e/1+e
16
Transfer Process

How can we move fluid from one FIFO queue to another?
Arrival rate
Injection rate e
1/1+e
Total arrival
rate 1 + e
Departure rate 1
Departure rate
1/1+e
Departure rate
e/1+e
17
Transfer Process

How can we move fluid from one FIFO queue to another?
Injection rate e
18
Transfer Process

How can we move fluid from one FIFO queue to another?
Injection rate 0
19
Transfer Process

How can we move fluid from one FIFO queue to another?
Injection rate 0
20
Transfer Process

How can we move fluid from one FIFO queue to another?
Injection rate 0
21
Repeated Transfer Process

Can repeat transfer process indefinitely
22
Repeated Transfer Process

Can repeat transfer process indefinitely
23
Repeated Transfer Process

Can repeat transfer process indefinitely
24
Repeated Transfer Process


Can repeat transfer process indefinitely
Two problems:
 Want amount of fluid to grow!
 Sessions are of finite length: some fluid eventually departs
25
Transfer Process Revisited

Replace departing fluid
Injection rate e
Injection rate r
Departure rate < r
26
Transfer Process Revisited

Replace departing fluid
Injection rate e
Injection rate r
Departure rate < r
27
Transfer Process Revisited

Replace departing fluid
Injection rate e
Injection rate r
Departure rate < r
28
Transfer Process Revisited

Replace departing fluid
Works but network load r >> 1
Injection rate e
Injection rate r
Departure rate < r
29
Parallel Transfers
Arrival rate ~ r / 1 - e
Injection rate r
1
2
3
4
Departure rate ~ r / 1 - e




Transfer process between queues 1 & 3
Transfer process between queues 2 & 4
Newly injected traffic passes through queues 1&2
Queues 1&2 are decreasing: hence rate amplification!!!
30
Parallel Transfers
Arrival rate ~ r / 1 - e
Injection rate r
1
2
3
4
Departure rate ~ r / 1 - e

For a sufficiently large number of parallel transfers
 Can make arriving traffic have higher rate than departing traffic
 Instability!!!
31
Part 2: Stability of congestion control
S


router A
router B
t
Part 1 of talk:
 External arrival process
 Injection patterns could be independent of queueing behavior
Part 2 of talk:
 Injection patterns governed by congestion control (e.g. TCP)
 Does TCP converge? Can it oscillate?
32
End-to-end congestion control
rate
?
rate
S
router A
router B
t
?



sender does not know in advance how much bandwidth
will be available at a given time, needs to adjust dynamically
 goal: use available bandwidth, but avoid congestion
if all routers on session path have some bandwidth available,
source should increase the sending rate
if one of the routers on session path is congested,
source should decrease the sending rate
33
End-to-end congestion control
rate
?
rate
S
router A
router B
t
?



design principle: routers do not give feedback to senders
 sender does not know what goes on inside the Internet
however, sender can get limited feedback from the receiver
 can determine whether congestion occurred on the path
TCP protocol: additive increase, multiplicative decrease
 no congestion  increase sending rate by 1
 congestion
 decrease sending rate by half
34
Network as dynamical system
S1


S2
S3
router
router
t2
t3
t1
system = {network of routers and fixed set of sessions}
 state = {sending rates of sessions, composition of queues}
 dynamics: TCP-like congestion control, FIFO queues
choose initial state and let the system run
 we’d like the system to converge, preferably to some nice state
35
Convergence vs oscillations
S1
S2
S3
router
router
t2


t3

t1
Prior work: for a version of TCP,
system converges to a social optimum
starting from any initial state

[Kelly, Maulloo, Tan ’98],[Low et al.]
and a large body of follow-up work
However, this work uses an approximation of FIFO dynamics:
congestion of a queue Q = F(sending rates of sessions that use Q)
 in reality it is a function of arrival rates, which may be different
we model congestion as a function of arrival rates
36
Example prior work

Kelly, Maulloo, Tan

xi (t )   wi - xi (t ) Q:iS (Q )  Q (t )
 Q (t )  pQ

iS ( Q )
xi (t )


pQ ( y )  ( y - 1  e )  / e 2


xi (t) = injection rate into flow i at time t
pQ = “shadow price” for congestion at router Q
37
Convergence vs oscillations
S1
S2
S3
router
router
t2

t3

t1
Prior work: for a version of TCP,
system converges to a social optimum
starting from any initial state

[Kelly, Maulloo, Tan ’98] + follow-ups

approximation of FIFO dynamics
Our contribution: for any TCP-like congestion control mechanism,
there exists a network of routers and a set of sessions such that
starting from a certain initial state, system oscillates
at all times, for each router, the total sending rate
of all sessions that use this router is less than its capacity
38
Fluid-based FIFO queue

session = continuous flow, not a discrete stream of packets
S1
S2
S3


FIFO
queue
h

C
t1
t2

h(t) = height of the queue at time t
= [total amount of fluid] / [capacity C]
given arrival rates in1(t), in2(t), in3(t)
what is h(t) and what are exit rates?
if h(t) = 0 and j inj(t)  C then
all flow just comes through
else, capacity is divided proportionally,
and the excessive fluid is queued up
t3
39
Sending rates vs arrival rates
S1
S2
S3
ini(t)


FIFO
queue
h

due to queues that are upstream,
in general arrival rates in1(t), in2(t), in3(t)
are different from sending rates
all convergence results from prior work
use sending rates instead of arrival rates
we use the true arrival rates
C
t1
t2
t3
40
Parameterized congestion control

??
sender detects congestion using feedback from receiver
 yes  decrease sending rate; no  increase sending rate

congestion: at least one queue is non-empty
S

t
no congestion: all queues are empty
S
t
41
Parameterized congestion control




sender detects congestion using feedback from receiver
 yes  decrease sending rate; no  increase sending rate
discrete-time TCP: additive increase, multiplicative decrease
 yes  decrease rate by half; no  increase rate by 1
we use a continuous-time approximation
-t
 yes congestion  sending rate f(t) = f(0)  e
 no congestion  sending rate g(t) = g(0) + t
  is the damping parameter that controls the rate of change
generalization: (almost) any flow increase function g(t),
any f(t) that satisfies some axioms, e.g. f(t) = f(0) - t
42
Main result
Theorem For any given congestion control {f(t), g(t)} there exists:

damping parameter  (same  for all sessions)

a network of routers, a set of sessions, and an initial state
such that system oscillates and for each router, the total sending
rate of all sessions that use this router is always less than its capacity
Note routers are almost FIFO:

each session is in one of the two priority classes

FIFO scheduling for sessions within the same class
we are extending our construction to the all-FIFO case
43
The basic gadget: high-level idea
S
capacity
Q1
1
Q2
<1
Q3
<1
Q4
t
1
start
S
Q1
1
Q2
Q3
Q4
t
finish
S
Q1
Q2
Q3
Q4
1
t
44
The basic gadget: animation
init rate e
S’
init rate 0
S
capacity
Q1
1
1
Q2
<1
Q3
Q4
<1
t
1
t’
45
The basic gadget: animation
init rate e
S’
rate = 0
S
capacity
Q1
1
Q2
<1
Q3
Q4
<1
t
1
t’
46
The basic gadget: animation
rate
S’
rate = 0
S
capacity
Q1
1
Q2
<1
Q3
Q4
<1
t
1
t’
47
The basic gadget: animation
rate = 0
S’
rate = 0
S
capacity
Q1
1
Q2
<1
Q3
Q4
1
<1
t
1
t’
48
The basic gadget: animation
rate
S’
rate = 0
S
capacity
Q1
1
Q2
<1
Q3
Q4
<1
t
1
t’
49
Connect two gadgets
identify Q4 from one gadget with Q1 from another


before
after
1
1
2
2
3
3
4
1
2
3
4
1
2
3
4
50
Row of gadgets
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
fluid eventually reaches its destination, and the process stops
not too exciting!
51
Row of gadgets
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
a better process
52
Row of gadgets
refil
l
S1
1
2
and now a refill happens
3
1
2
3
1
2
3
1
2
3
t1
a better process
53
Row of gadgets
refil
l
S1
1
2
... and so on
3
1
2
3
1
2
3
1
2
3
t1
a better process
54
Rows of gadgets
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
S2
1
2
3
1
2
3
1
2
3
1
2
3
t2
S3
1
2
3
1
2
3
1
2
3
1
2
3
t3
each row is running the same process, shifted in time
55
Rows of gadgets
refil
l
S1
now a refill happens
1
2
3
1
2
3
1
2
3
1
2
3
t1
1
2
3
1
2
3
1
2
3
1
2
3
t2
1
2
3
1
2
3
1
2
3
1
2
3
t3
refill
S2
refill
S3
each row is running the same process, shifted in time
56
Rows of gadgets
refil
l
... and so on
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
S2
1
2
3
1
2
3
1
2
3
1
2
3
t2
S3
1
2
3
1
2
3
1
2
3
1
2
3
t3
Wait a sec... how do the refills happen ???
57
Refills and vertical sessions
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
S2
1
2
3
1
2
3
1
2
3
1
2
3
t2
S3
1
2
3
1
2
3
1
2
3
1
2
3
t3
1
2
3
1
2
3
1
2
3
1
2
3
t4
rate >0
S4
58
Refills and vertical sessions
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
S2
1
2
3
1
2
3
1
2
3
1
2
3
t2
S3
1
2
3
1
2
3
1
2
3
1
2
3
t3
1
2
3
1
2
3
1
2
3
1
2
3
t4
rate >0
S4
59
Refills and vertical sessions
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
S2
1
2
3
1
2
3
1
2
3
1
2
3
t2
S3
1
2
3
1
2
3
1
2
3
1
2
3
t3
1
2
3
1
2
3
1
2
3
1
2
3
t4
rate >0
S4
60
Refills and vertical sessions
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
S2
1
2
3
1
2
3
1
2
3
1
2
3
t2
S3
1
2
3
1
2
3
1
2
3
1
2
3
t3
S4
1
2
3
1
2
3
1
2
3
1
2
3
t4
rate
61
Refills and vertical sessions
refilled !!!
S1
1
2
3
1
2
3
1
2
3
1
2
3
t1
S2
1
2
3
1
2
3
1
2
3
1
2
3
t2
S3
1
2
3
1
2
3
1
2
3
1
2
3
t3
1
2
3
1
2
3
1
2
3
1
2
3
t4
rate = 0
S4
62
Conclusions and open questions
End-to-end congestion control can cause oscillations

prior work proved convergence but queueing dynamics not fully
captured

with true queueing dynamics convergence is desirable but
impossible
Our example is paranoid, but we need to replicate initial state exactly

our result suggests that in practice convergence can be hard, too

possible solution: integrate congestion control into routers,
e.g. Greedy Primal-Dual algorithm of Sasha Stolyar
Open questions:

extension to the all-FIFO case

non-FIFO schedulers, e.g. Generalized Processor Sharing
63