Download Chapter 6: Multimedia Networking

4/14/16
Ch. 5: Multimedia Networking
Multimedia Systems
Prof. Ben Lee
School of Electrical Engineering and Computer Science
Oregon State University
Outline
—
—
Introduction
Wireless LANs
◦ What is the Internet?
◦ 802.11
– Physical layer
– Medium Access Control (MAC) protocol
– Mobility
—
Dealing with packet jitter and loss
◦ Issues
◦ Reducing Delay Jitter
– Fixed vs. Adaptive Playout
◦ Recovering from Packet Loss
– FEC
– Interleaving
—
Case Study: Loss Resilience Features in H.264
Chapter 6: Multimedia Networking
2
1
4/14/16
Introduction
—
—
Explosive growth of network applications that
transmit audio and video over the Internet.
A new class continuous media applications:
◦ Entertainment video, IP telephony, Internet radio,
Multimedia WWW sites, teleconferencing, interactive
games, virtual worlds, distance learning ...
—
Service requirements are significantly different
from those of traditional data oriented
applications.
◦ A need for a network architecture (multimedia
networking) that can support these applications.
Chapter 6: Multimedia Networking
3
Multimedia Over Internet
—
Multimedia applications are highly sensitive to
end-to-end delay and delay variations (jitter) but
typically loss tolerant.
◦ In contrast to applications, such as Web text/image, email, FTP, and Telnet, where long delays are annoying
but not harmful.
—
IP protocol only provides a best-effort service:
◦ No guarantees of end-to-end delay.
◦ No guarantees of variations of packet delay within a
packet stream.
— Will concentrate
on WLANs!
Chapter 6: Multimedia Networking
4
2
4/14/16
Multimedia Networking Issues
—
Application classes:
◦ Streaming stored audio/video
◦ Multicast streaming of real-time audio/video
◦ Real-time interactive audio/video
—
Application issues:
◦ Packet jitter
◦ Packet loss/recovery
—
Internet protocols for multimedia
◦ RTSP, RTP/RTCP
—
Network QoS and Resource Management
◦ Intserv/RSVP, Diffserv
—
Congestion avoidance
◦ Avoiding packet loss
Chapter 6: Multimedia Networking
5
Multimedia Networking
—
Need to understand how E-to-E communication is
achieved:
◦ Internet - TCP/UDP, IP, Link Layer
◦ Various technologies that comprise the Internet
◦ …and how they affect the end-to-end delay, jitters, packet losses
of media data being transmitted.
—
Need something beyond best effort:
◦ Making the best of best effort - buffering
◦ Recovering from packet loss - redundancy
◦ Protocols for Real-time Interactive applications
– RTP, RTCP
◦ Congestion Avoidance
◦ QoS - end-to-end guarantees
Chapter 6: Multimedia Networking
6
3
4/14/16
Outline
—
—
Introduction
Wireless LANs
◦ What is the Internet?
◦ 802.11
– Physical layer
– Medium Access Control (MAC) protocol
– Mobility
—
Dealing with packet jitters and loss
◦ Issues
◦ Reducing Delay Jitters
– Fixed vs. Adaptive Playout
◦ Recovering from Packet Loss
– FEC
– Interleaving
—
Case Study: Loss Resilience Features in H.264
Chapter 6: Multimedia Networking
7
What is the Internet?
Core Network
Edge Network
Chapter 6: Multimedia Networking
8
4
4/14/16
Internet Protocol Architecture
Host A
Application Layer
TCP identify applications
Using 16-bit port numbers
App X
App Y
Transport Layer
Logical TCP connection
TCP/UDP
Network Layer
Link Layer
Host B
Global network address
IP
Software
Network Access
Type A
Physical Layer
Physical
Type A
Hardware
Router
App X
App Y
Application Layer
TCP/UDP
Transport Layer
IP
Network Layer
Network Access
Type B
Physical
Type B
Link Layer
Physical Layer
IP
Network
Type A
NA
NA
Type A Type B
Physical Physical
Type A Type B
Network
Type B
Chapter 6: Multimedia Networking
9
Protocol Data Units
User Data
Application
TCP
Header
TCP Segment
IP
Header
Preamble
Frame
Header
IP Datagram
CRC
Frame Packet
Frame format depends on the specific type of network.
Chapter 6: Multimedia Networking
10
5
4/14/16
Wireless LANs
—
—
Wireless LANs: untethered (often mobile)
networking.
IEEE 802.11 standard:
◦ MAC protocol
◦ Unlicensed frequency spectrum: 900 MHz, 2.4 GHz, 5
GHz
—
Not meant to replace wired networks:
◦ A way to cover locations that are difficult to cover.
– Manufacturing plants
– Historical buildings, etc.
—
Provides mobility, relocation,Ad Hoc networking:
◦ Home networking
◦ PANs
◦ Sensor networks
Chapter 6: Multimedia Networking
11
WLAN Architecture
—
Basic Service Set (BSS)
contains:
◦ Mobile Stations (MSs)
◦ Access point (AP): base
station.
– Defined by Service Set
Identifier (SSID)
—
Extended Service Set
(ESS)
◦ A set of infrastructured
BSSs.
Typical Range: 30~300 ft
Chapter 6: Multimedia Networking
12
6
4/14/16
Ad Hoc Networks
—
—
Ad hoc network: IEEE 802.11 stations can
dynamically form network without AP.
Applications:
◦
◦
◦
◦
—
“laptop” meeting in conference room, car.
Interconnection of “personal” devices.
Battlefield
Sensor networks
IETF MANET (Mobile Ad hoc Networks)
working group
Chapter 6: Multimedia Networking
13
What is IEEE 802.11?
—
—
—
Family of standards set forth by IEEE to define the
specifications for WLANs (adopted in 1997).
1 and 2 Mbps at distances of 150-2000 ft.
IEEE 802.11 standard:
◦ Medium Access Control (MAC) protocol
◦ Physical Layer (PHY) specifications
–
–
–
–
–
802.11b
802.11a
802.11g
802.11n
802.11ac
◦ Power Saving Mechanism
Chapter 6: Multimedia Networking
14
7
4/14/16
802.11x Comparison
802.11b
802.11a
802.11g
802.11n
Standard Ratified
Sept. 1999
Sept. 1999
June 2003
Sept. 2009
Frequency
2.4 GHz
5 GHz
2.4 GHz
2.4 or 5 GHz
Range (typical)
100-150 ft
100-150 ft
~165 ft
Data Rates
1, 2, 5.5, 11
6, 9, 12, 18, 24,
36, 48, 54
6, 9, 12, 18, 24,
36, 48, 54
540 (max.)
Realistic
Throughput
~4-6 Mbps
~27 Mbps
~20 Mbps
~200 Mbps
Modulation
CCK
OFDM
OFDM
OFDM
# of channels
Non Overlapping
11/3
12/8
11/3
11/3
# of users
32
64
64
96
25-35 ft
Chapter 6: Multimedia Networking
15
802.11ac
—
Gigabit Wi-Fi
—
5GHz band only
—
Wider BW
◦ 6.93 Gbps max. data rate
◦ Less interference and more non-overlapping channels.
◦ 80 MHz (433 Mbps) and 160 MHz (867 Mbps) channels
– Compared to 20 MHz and 40 MHz of 802.11n
—
—
—
—
Higher order modulation
◦ OFDM 256 QAM, 8 bits per symbol
Higher order MIMO
◦ Up to 8 spatial streams, max. 4 streams per STA.
Multi-User MIMO (MU-MIMO)
◦ Allows AP to simultaneously service multiple STAs.
Beamforming
◦ Improves data rate for a given range.
Chapter 6: Multimedia Networking
16
8
4/14/16
IEEE 802.11 MAC Layer
—
Same MAC layer for all variants (802.11b, a, g, n,
ac)
◦ DCF (Distributed Coordination Function)
– CSMA/CA (Carrier Sense Multiple Access with collision
avoidance)
– All stations may not hear each other:
– Hidden node problem
– Exposed node problem
◦ PCF (Point Coordination Function)
– Polling (optional)
Chapter 6: Multimedia Networking
17
Joining the Network
—
Scanning
◦ Need to find an AP to join.
—
Authentication request/response
◦ Need to authenticate users that are authorized to
use the network.
—
(Re-)Assocation request/response
◦ A MS needs to register, or associate, with an AP. The
server can then use the registration information to
determine which AP to use to communicate with MS.
Chapter 6: Multimedia Networking
18
9
4/14/16
Passive Scanning
MS waits for a beacon every 102.4 ms.
— Must check all 11 channels.
— Allows MS to power down if needed
—
Beacon Frame
2
2
Frame
Duration
Control
6
6
6
DA
SA
BSSID
2
Seq.
Control
0 - 23 12
4
payload
CRC
Ti mes ta mp ,
Beac on int er val,
Cap abili ty i nfo , e tc.
Na me assi gn ed to AP
Sou rce (AP ) MAC ad dr ess
Desti na tio n MAC ad dr ess (Br oa dc ast )
NAV (N etw or k Allo cati on Vec to r)
Fr a me Ty pe :
Ma na ge m ent , c ont rol ,
or da ta
Chapter 6: Multimedia Networking
19
Active Scanning (1)
MS probes for APs.
— Must check all 11 channels! (i.e., perform a full scan)
—
Probe request/response Frame
2
2
Frame
Duration
Control
Ch. 1
6
6
6
DA
SA
BSSID
2
Seq.
Control
0 - 23 12
4
payload
CRC
SSID ( ne two rk na m e),
Sup po rte d rat es
Probe request
Probe response
Name assigned to AP
Sou rce ( MS) MA C a dd re ss
Probe response
Desti na tio n MAC ad dr ess (Br oa dc ast )
Probe request
NAV (N etw or k Allo cati on Vec to r)
Ch. 6
Fr a me Ty pe :
Ma na ge m ent , c ont rol ,
or da ta
Chapter 6: Multimedia Networking
20
10
4/14/16
Active Scanning (2)
Ava ila bl e
Ch an n els
Ho st
Ini ti at e
Sca n ni ng
Ch. Switch Time
—
MS has to wait if there is no
responses from APs
Channel # 1
No APs in Ch. #1
MinChann elTime
Ch. Switch Time
Channel # 2
MaxChann elTime
…
Multiple APs in
Ch. #2
…
MinChannelTime - Time
MaxChannelTime -Time
MS has to wait if a
response is heard
— Selects AP with best
signal strength.
—
Ch. Switch Time
Channel #N
MinChann elTime
No AP in Ch. #N
Act iv e s ca n
com pl et e
Chapter 6: Multimedia Networking
21
Authentication
—
Open-system:
◦ Does not verify the identity of MS.
—
—
Username and passwd
Address filtering:
◦ Authorized MAC address list
◦ Kept in a centralized server (OSU uses “Maintain”)
—
Shared-key
◦ WEP
Chapter 6: Multimedia Networking
22
11
4/14/16
(Re-)Association
—
—
Allows the new AP to keep track of the location
of MS.
The Switch becomes aware of location MS, i.e.,
needs to forward frames to new AP.
◦ More on this later…
—
MS can now communicate with the world!
Chapter 6: Multimedia Networking
23
Comm. with Outside World
Switch Table
MAC ad dr ess : xx :xx: xx:x x:xx :xx
Router
Internet
Not aw ar e o f APs!
1
ARP Table
IP Address
MAC Address
128.193.143. 14 8
00:19:E3:0E:65: 18
…
…
…
…
4
Interface
3
xx:xx:xx:xx:xx:xx
4
Server
Switch
2
MAC Address
00:19:E3:0E:65: 18
3
0 - 23 12
4
Payload
CRC
MAC ad dr ess of
IP Datagram
ro ute r i nte rf ace to
whic h AP is
att ach ed
Only us ed in
MAC ad dr ess
ad ho c mo de
of MS or AP
tra ns mitti ng thi s f ra me
Atrium
APy
APz
APx
Data Frame
2
2
6
6
6
2
6
Frame
Address Address Address Seq. Address
Duration
Control
1
2
3
Control
4
MAC ad dr ess
of MS or AP to
rec eiv e t his f ra m e
MAC ad dr ess : 0 0: 19:E 3: 0E:6 5: 18
IP a dd res s: 12 8.1 93 .1 43. 14 8
Classrooms
Ethernet (802.3) frame Û 8 02.11 frame
Chapter 6: Multimedia Networking
24
12
4/14/16
Roaming
Example Roaming Path of 1 st floor of KEC
Switch has to
forward to APz
Switch Table
MAC Address
Interface
00:19:E3:0E:65: 18
2®3
4
1
Server
Switch
2
3
3. Update Switch Table
Atrium
2. Broadcast( 00:19:E 3:0E :6 5:1 8)
APy
APz
MS
1. Scan, authentication, reassociation
MAC ad dr ess : 0 0: 19:E 3: 0E:6 5: 18
IP a dd res s: 12 8.1 93 .1 43. 14 8
APx
Classrooms
SSID=“OSU_PUB”
Chapter 6: Multimedia Networking
25
Handoff Delay
Ava ila bl e
Ch an n els
Ho st
Ini ti at e
Ha nd o ff
—
Ch. Switch Time
Time MS has to wait
if there is no
responses from APs
Channel # 1
No APs in Ch. #1
MinChann elTime
Ch. Switch Time
Channel # 2
MaxChann elTime
Time MS has to wait if a
response is heard
—
…
◦ Channel Switch Time = 5~19 ms
◦ MinChannelTime = 1~7 ms
◦ MaxChannelTime = ~11 ms
Multiple APs in
Ch. #2
…
—
Resulting in scanning delay of
350~500 ms!
—
Represents > 90% of overall
handoff delay.
Ch. Switch Time
Channel #N
MinChann elTime
Act iv e s ca n
com pl et e
◦ Will loose packets!
No AP in Ch. #N
Channel # 2
No set standard for these
parameters.
A recent study of 802.11b based
on Intersil Prism2 and Atheros
5212 NICs:
◦ Authentication & Association each
take < 10 ms.
◦ Wired network update takes < 20
ms
Ha nd o ff c om p le te
Chapter 6: Multimedia Networking
26
13
4/14/16
Medium Access Control
—
How do you control the
access to wireless medium?
Signals attenuate as function
of distance:
◦ Carrier Sense detects signals
at the transmitter.
◦ But, collision occurs at the
receiver.
—
Cannot listen while you send:
◦ Transceiver is half duplex.
◦ Rx cannot tune to Tx
frequency.
—
All you hear is yourself!
◦ Your own signal strength at
your antenna is much stronger
than anybody else’s signal.
Chapter 6: Multimedia Networking
27
IEEE 802.11 Carrier Sensing
—
802.11 uses both physical (at PHY) and virtual
(at MAC) carrier sensing.
◦ Physical carrier sensing
– Medium consider busy when transmitting
◦ Virtual carrier sensing
– Maintained by each station using Network Allocation Vector
(NAV), i.e., 16-bit counter in µs increments.
– Set based on Duration field in received frames (e.g., RTS,
CTS).
– When nonzero, station thinks medium is busy.
—
Medium considered busy whenever either
indicates carrier.
Chapter 6: Multimedia Networking
28
14
4/14/16
IEEE 802.11 MAC Protocol:
CSMA/CA
Physical layer monitors the
802.11 CSMA: sender
If sense channel idle for DISF (Distributed
Inter Frame Space)
Then transmit entire frame (no collision
detection)
If sense channel busy
then binary backoff
If ACK not received
then retransmit
energy level on the radio
frequency
802.11 CSMA receiver:
If received OK
return ACK after SIFS (Short Inter
Frame Space)
Link-layer ACK
Chapter 6: Multimedia Networking
29
Inter Frame Spacing
—
Priorities are defined through different inter frame spacing (IFS)
◦ SIFS (Short Inter Frame Spacing)
– Highest priority, for ACK, CTS, and polling response
◦ PIFS (PCF IFS)
– Medium priority, for time-bounded service using PCF
◦ DIFS (DCF IFS)
– Lowest priority, for best-effort data
◦ EIFS (Extended IFS)
– Lowest priority interval used to report bad or unknown frame.
Chapter 6: Multimedia Networking
30
15
4/14/16
IEEE 802.11 MAC Protocol
DIFS
contention window
(randomized back-off
mechanism)
DIFS
medium busy
next frame
t
direct acc ess if
medium is free ³DIF S
slot time
Station ready to send starts sensing the medium.
If the medium is free for the duration of DIFS the station can
start sending.
— If the medium is busy, the station has to wait for a DIFS, then
the station must additionally wait a random back-off time
(collision avoidance, multiple of slot-time).
— If another station occupies the medium during the back-off
time of the station, the back-off timer stops (fairness).
—
—
Chapter 6: Multimedia Networking
31
Link Layer ACK
SIFS = 10 µs
short slot_time = 9 µs (802.11g only)
long sl ot_time = 20 µs (802.11b/g)
DIFS = 2 x sl ot_time + SIFS
DIFS
Sender
data
SIFS
Receiver
ACK
slot_time
DIFS
Other
stations
NAV
…
data
t
defer access
Back off delay
Chapter 6: Multimedia Networking
32
16
4/14/16
Random Backoff
—
Backoff delay uniformly chosen in the range [0,
CW-1], where CW is called Contention Window.
◦ Backoff timer is decreased as long as medium is sensed to
be idle for DIFS.
◦ Backoff timer is frozen medium is busy.
•
•
•
•
CW reaches 0, station transmits.
Initially CW =1. After each unsuccessful
transmission (no CTS or ACK) CW is doubled (up
to a max. value).
Once CW reach CW MAX, it stays at that value until
reset.
CW is reset after every successful transmission of a
data or RTS packet.
Chapter 6: Multimedia Networking
33
Hidden Terminal Effect
—
Hidden terminals:A, C cannot hear each other
◦ Obstacles, signal attenuation
◦ Collisions at B
—
—
Goal: avoid collisions at B
CSMA/CA: CSMA with Collision Avoidance
Chapter 6: Multimedia Networking
34
17
4/14/16
Hidden & Exposed Terminal
Problem
Hidden Terminal
Currently transmitting
Currently transmitting
A
C
Exposed Terminal
B
B cannot hear A
A
B
C
D
C hears B’s
Transmission:
Cannot transmit to D!
Chapter 6: Multimedia Networking
35
Collision Avoidance (1)
—
CSMA/CA - Explicit
channel reservation:
◦ Sender: send short (fixed
size) RTS: request to
send.
◦ Receiver: reply with short
(fixed size) CTS: clear to
send.
—
—
CTS reserves channel for
sender, notifying (possibly
hidden) stations
Avoid hidden station
collisions
Chapter 6: Multimedia Networking
36
18
4/14/16
Collision Avoidance (2)
NAV set
RTS
CTS
NAV set
NAV set
Data
ACK
NAV set
Chapter 6: Multimedia Networking
37
RTS-CTS Exchange
DIFS
sender
RTS
data
SIFS
receiver
other
stations
SIFS
SIFS
CTS
ACK
DIFS
NAV (RTS)
data
NAV (CTS)
t
defer access
contention
Chapter 6: Multimedia Networking
38
19
4/14/16
802.11e
—
QoS support in WLANs:
◦ Also referred to as WiFi MultiMedia (WMM).
—
—
Backward compatible with DCF and PCF.
Enhanced Distributed Channel Access (EDCA):
◦ Evolution of DCF
◦ Introduces traffic priority
◦ Favors VoIP applications
Chapter 6: Multimedia Networking
39
EDCA
—
Traffic differentiation:
—
—
New Inter-frame Spec for each AC:
—
—
4 Access Categories (ACs) with 8 Traffic Classes
(TCs).
Arbitration Inter Frame Space (AIFS)
Transmission Opportunity (TXOP)
Chapter 6: Multimedia Networking
40
20
4/14/16
Access Categories (ACs)
—
—
Media access is based on the AC of frames.
4 AC’s are defined:
◦ AC_BK (AC0) – Background
◦ AC_BE (AC1) – Best-effort
◦ AC_VI (AC2) – Video
◦ AC_VO (AC3) – Voice
—
The size of Contention Window (CW) and
Inter-frame space (IFS) is dependent on AC.
Chapter 6: Multimedia Networking
41
Priority
ACs map directly from Ethernet-level Class of
Service (CoS) priority levels
Priority
AC
Designation
1
AC_BK (0)
Background
2
AC_BK (0)
Background
0
AC_BE (1)
Best Effort
3
AC_BE (1)
Best Effort
4
AC_VI (2)
Video
5
AC_VI (2)
Video
6
AC_VO (3)
Voice
7
AC_VO (3)
Voice
Chapter 6: Multimedia Networking
42
21
4/14/16
Arbitration Interframe Space
(AIFS)
—
—
AIFS is used to defer the contention window or
transmission for each AC.
The value of AIFS differ corresponding to AC:
◦ AIFS[AC] = AIFSN[AC] x SlotTime + SIFS
– where AIFSN[AC] is an AIFS number
◦ AIFSN for each AC is broadcast via beacon frame
containing “EDCA Parameter Set” element.
—
DIFS = 2 x SlotTime + SIFS
Chapter 6: Multimedia Networking
43
AC-based Backoff
Chapter 6: Multimedia Networking
44
22
4/14/16
CWmin Differentiation
CW is a function of
CW min[AC] and
CW max[AC].
— Higher priority stations
have smaller CW min and
CW max.
— Smaller CW min and
CW max correspond to
on average fewer backoff slots being traversed
per transmission.
—
AC
CW min
CW max
AC_BK
CW min
CW max
AC_BE
CW min
CW max
AC_VI
(CW min+1)/2 -1
CW min
AC_VO
(CW min+1)/4 -1
(CW min+1)/2 -1
Chapter 6: Multimedia Networking
45
Chapter 6: Multimedia Networking
46
Priority
23
4/14/16
Transmission Opportunity
(TXOP)
The interval of time when a particular STA has the right
to initiate transmissions.
— Frames exchange sequences are separated by SIFS.
— When will a STA get a TXOP ?
—
◦ Win a contention in EDCA, or
◦ Receive a CF-poll (“polled TXOP”)
Chapter 6: Multimedia Networking
47
Default EDCA Parameters
AC
CWmin
CWmax
AIFSN
Max TXOP
Background (AC_BK)
31
1023
7
0
Best Effort (AC_BE)
31
1023
3
0
Video (AC_VI)
15
31
2
3.008 ms
Voice (AC_VO)
7
15
2
1.504 ms
Legacy DCF
15
1023
2
0
Chapter 6: Multimedia Networking
48
24
4/14/16
Other Features of 802.11 MAC
—
Power management
◦ Informs AP it wants to sleep: AP buffers frames.
◦ Periodically wakes up to receive beacon signals:
– Indicates whether frames have been buffered.
– Requests awaiting frames from AP.
Chapter 6: Multimedia Networking
49
Outline
—
—
Introduction
Wireless LANs
◦ What is the Internet?
◦ 802.11
– Physical layer
– Medium Access Control (MAC) protocol
– Mobility
—
Dealing with packet jitter and loss
◦ Issues
◦ Reducing Delay Jitter
– Fixed vs. Adaptive Playout
◦ Recovering from Packet Loss
– FEC
– Interleaving
—
Case Study: Loss Resilience Features in H.264
Chapter 6: Multimedia Networking
50
25
4/14/16
Issues
—
Real-time interactive multimedia applications, e.g.,
Internet phone and real-time video conferencing are
very sensitive to
◦
◦
◦
◦
Delay
Loss
Delay-Jitter
Out-of-Order Arrivals
End-to-End Delay
Jitter
< 150 ms not perceived by user.
150 ms - 400 ms acceptable but not ideal.
> 400 ms seriously hinders interactivity.
> Jitter max
Send
X
X
Receive
Perfect
Delivery
Packet
Loss
Delay-Jitter
Out-of-order
Arrivals
Chapter 6: Multimedia Networking
51
Sources of Packet Loss, Jitter,
and Out-of-Order Arrival
Core Network
Edge Network
Chapter 6: Multimedia Networking
52
26
4/14/16
Packet Delay and Loss in Core
Network
End-to-end delays:
DE_to_E = DTx_proc + DQueuing + DPropagation + DRx_proc
— Delay Jitter:
—
◦ Random queuing delays in the routers.
—
Packet Loss:
◦ Packet drop due to overflowed queues in the routers.
—
Out-of-Order Arrivals: No TCP
◦ Packets taking different paths.
◦ Random queue delays in the routers.
Chapter 6: Multimedia Networking
53
Packet Delay and Loss in Edge
Network
WLANs are pervasive.
— Carrier Sense interference
—
◦ Access serialization to
medium => delay
—
Hidden Node interference
◦ Packet collision = > loss
◦ MAC retry (up to Retry Limit
= 7 times)
—
Home Entertainment
Increase in peer-to-peer
HD Video Streaming over
WLANs:
◦ e.g., Apple AirPlay, Google
Chromecast, Intel WiDi, etc.
N-Screen
Chapter 6: Multimedia Networking
54
27
4/14/16
Current Solutions
—
To mitigate the impact of “best-effort”
protocols, we can:
◦ Use UDP to avoid TCP’s slow-start phase and
retransmission.
◦ Buffer content at client and control playback to
remedy jitter.
◦ Timestamp packets at the sender (so receiver knows
when the packets should be played back).
◦ Prefetch data during playback (when client has extra
storage and bandwidth).
◦ Adapt compression level to available bandwidth.
Chapter 6: Multimedia Networking
55
Internet Phone Example
Voice sampling:
8 kHz @ 8 bits
Talkspurt
Chunk = 160 bytes
t
20 ms
Silent Periods
> 20 ms
UDP segment
Transmit
Header
Will need
• Sequence number
• Timestamp
…but wait...UDP and IP do not have this...
Chapter 6: Multimedia Networking
56
28
4/14/16
Reducing Delay-Jitter (1)
p = packaging period of the media at the sender
p
2p
2p
3p
Send
a
b
c
d
e
f
g
h
Receive
Playout
Display
Queue
a
d
c
b
e
d
e
f
g
h
Playing the samples as soon as they arrive ensures minimal
end-to-end latency but at the cost of an increased gap-rate.
Chapter 6: Multimedia Networking
57
Reducing Delay-Jitter (2)
3p
Send
a
b
c
d
e
f
g
h
Receive
Playout
Display
Queue
a
b
a
c
b
a
d
c
b
e
d
c
e
d
f
e
g
f
g
h
—
Enqueuing the first sample for some duration better ensures
continuous playout.
—
Which initial playout strategy is better? It depends…
◦ But at the cost of higher playout latency throughout the conference.
Chapter 6: Multimedia Networking
58
29
4/14/16
Fixed Playout Delay
—
—
When should it playout? p - r
or p’ - r?
Packets
However, playout delay should
not exceed some threshold,
e.g.,
Packets
generated
Missed
playout
◦ 250 ms for audio
◦ 10-15 frames/s for video (66.6
ms - 100 ms)
—
—
Tradeoff between playout
delay vs. packet loss.
For interactive services, such
as internet phone, long delays
can be bothersome if not
intolerable.
Packets
received
Playout schedule
p-r
Playout schedule
pʼ - r
Time
s
r
pʼ
p
Chapter 6: Multimedia Networking
59
Adaptive Playout Delay (1)
—
Often end-to-end delay distribution of packets
within a talk spurt is not known.
◦ …and can change over time.
—
—
Estimate the network delay and its variance, and
adjust the playout delay accordingly.
Determine play out delay on a per-talkspurt
basis.
Chapter 6: Multimedia Networking
60
30
4/14/16
Adaptive Playout Delay (2)
qi
di
Send
Receive
Kvi
si
si+1
ri
Playout
r i+1
pi
pi+1
qi
si = timestamp of the i th packet generated by the sender
r i = time packet i is received by the receiver
pi= time packet i is played at the receiver
d i= average network delay
vi= average deviation of network delay
Chapter 6: Multimedia Networking
61
Adaptive Playout Delay (3)
—
E-to-E delay for the ith packet will be ri - si . The average
network delay of the ith packet d i is (similar to how TCP
RTT is estimated)
d i = (1 - u)d i-1 + u(ri - s i),
where u is fixed constant (e.g., u = 0.01)
—
The average delay deviation v i is
vi = (1 - u)vi-1 + u|ri - s i - d i|,
—
The receiver then uses the following
pi = s i + d i + Kvi,
where K is called a congestion estimator (e.g., 4).
Chapter 6: Multimedia Networking
62
31
4/14/16
Adaptive Playout Delay (4)
When should playout delay be changed?
— For voice, transmission occurs in alternating talk spurts
and silence periods.
— If no packet loss, receiver can determine that a packet is
the first packet of the talk spurt.
—
◦ e.g., s i - s i-i > 20 ms, then receiver knows ith packet start a new
talkspurt
—
What if there is a packet loss? Two successive packets
received may have timestamp that differ by more than
20 ms.
◦ Use sequence numbers.
Chapter 6: Multimedia Networking
63
Real-Time Protocol
—
—
Internet-standard protocol for transporting
real-time data, including audio and video.
A standardized packet structure that includes
fields for audio/video data, sequence number,
timestamp, an other useful fields.
◦ PCM or GSM for audio
◦ MPEG1 and MPEG2 for video
—
Runs on top of UDP.
Chapter 6: Multimedia Networking
64
32
4/14/16
RTP Packet Format
Version # (2 bits)
Padding Bit
Extension Bit
# of Contributing Sources
Marker bit
Payload type (7 bits)
V=2P X
3 words
CC
M
PT
Seque nce Num ber
Timestamp
Synchron izatio n Source (SSRC) Ide ntifier
0-15 words
Contrib uting So urce (CSRC) Ide ntifier
…
Extension H ead er
RTP Payload
Chapter 6: Multimedia Networking
65
Payload Type
—
Indicates the type of multimedia data carried by
the packet.
◦ Audio: PCM 𝜇-law, GSM, MPEG Audio, etc.
◦ Video: Motion JPEG, H.261, MPEG-1, MPEG-2, etc.
—
Defines sampling rate of timestamp.
◦ Audio
– 8 kHz sampling rate (every 125 𝜇s)
– 20 ms minimum inter-frame time.
– Marker (M) bit set on the first packet after a silence period.
Chapter 6: Multimedia Networking
66
33
4/14/16
Sequence No.
—
Allows the receiver to detect missing and outof-ordered packets.
◦ 16 bits long.
—
—
Sender increments the sequence number by
one for each transmitted packet.
RTP does not do anything when a packet is
detected to be lost.
◦ Instead, it is left to the application to decide what
should be done.
Chapter 6: Multimedia Networking
67
Timestamp
Enables the receiver to playback samples at appropriate
intervals and to enable different media streams to be
synchronized.
— 32 bits long.
— A counter of “ticks”, where time between ticks is
dependent on encoding in use.
— For example,
—
◦ Audio application that samples data every 125 𝜇s uses this as its
clock resolution.
◦ Multiple audio samples can be transmitted together.
◦ If packet is generated every 10 ms (each containing 10 ms/125 𝜇
s = 80 samples), timestamp between successive packets would
have timestamp differ by 80.
Chapter 6: Multimedia Networking
68
34
4/14/16
Timestamp vs. Sequence No.
—
Timestamp relates packet to real-time:
◦ Timestamp values sampled from a media specific
clock.
—
—
Sequence number relates packet to other
packets.
Allows many packets to have the same
timestamp but different sequence numbers.
Chapter 6: Multimedia Networking
69
MPEG Example
—
How does the timestamp/sequence-number
mechanism support MPEG?
◦ Out-of-order transmission.
– Sequence numbers increase monotonically.
– Timestamps reflect reference relationships.
◦ Large frames.
– One video frame likely to be split into parts and packed into
multiple RTP packets.
– Timestamps associate packets together as part of the same
frame, while seq. no distinguish packets from each other.
Chapter 6: Multimedia Networking
70
35
4/14/16
Audio Silence example
—
—
Nothing is sent during silent periods.
Why might this cause problems?
◦ Other side needs to distinguish between loss and
silence.
—
How does the timestamp/seq. no mechanism
help?
◦ After receiving no packets for a while, next packet
received will reflect a big jump in timestamp, but have
the correct next seq. no. Thus, receiver knows what
happened.
Chapter 6: Multimedia Networking
71
Recovering from Packet Loss
Packet is considered lost if it never arrives or arrives
after its scheduled playout time.
— Sending packets over TCP may not viable:
—
◦ Retransmission => unacceptable end-to-end delay.
◦ Congestion control => transmission rate is reduced
Packet loss rate of 1% - 20% is tolerable (depending
how voice is encoded and transmitted).
— Forward Error Correction and Interleaving can help conceal
loss packets.
— If packet loss exceeds 10 - 20%, due to congestion...
—
Chapter 6: Multimedia Networking
72
36
4/14/16
Forward Error Correction
Redundancy is
introduced into the
stream to enable the
receiver to recover from
errors due to loss.
— Forms of redundancy
—
RTP Packet
RTP Header
header extension
RTP payload
◦ Simple replication and retransmission of original
data.
◦ k-way XOR
◦ Replication, recoding, and
re-transmission of original
data.
FEC payload
Chapter 6: Multimedia Networking
73
FEC Basic Concept
f, e
Send
g, f
h, g
i, h
j, i
k, j
X
l, k
m, l
X
X
Receive
Display e
Queue d
c
—
e
d
g
f
e
g
f
i
h
g
j
i
h
j
i
j
m
l
Key issue: If a sample is lost, how do we ensure that the redundant
information necessary for the repair arrives?
◦ How much bandwidth should we dedicate to FEC?
◦ Where should we place the redundant information in the stream?
Chapter 6: Multimedia Networking
74
37
4/14/16
FEC Method 1: n-way XOR
Send a redundant encoded chunk after every n chunks.
Transmit the word-by-word XOR of groups of n chunks.
— If a single packet is lost, receiver can fully reconstruct the
lost packet.
— Example: 3-way XOR
—
—
10100011 ⊕ 10100011 ⊕ 00000011 = 00000011
Sample 1
Sample 2
Lost!
Sample 3
Redundant Code
10100011 ⊕ 00000011 ⊕ 00000011 = 10100011
Sample 1
—
Redundant Code
Sample 3
Reconstructed
Sample 2
n can’t be too large, higher prob. of packet loss and delay. n
can’t be too small, higher transmission rate.
Chapter 6: Multimedia Networking
75
FEC Method 2: Sending Lowbit-Rate Encoded Chunks
Can cope with non-consecutive packet loss.
Nominal stream
(e.g., PCM @ 64 kbps)
Low-bit-rate stream
(e.g., GSM @ 13 kbps)
Occasional low-bit-rate chunks in between high quality
give overall audio quality.
Chapter 6: Multimedia Networking
76
38
4/14/16
Interleaving
—
—
—
—
—
Divide 20 ms chunks into
four 5 ms units and
interleave.
Upon loss have a set of
partially filled chunks.
Multiple small gaps as
opposed to a single large gap!
Low overhead, but can
increase latency.
Maybe not great for Internet
phone, but streaming stored
audio or video.
Chapter 6: Multimedia Networking
77
Outline
—
—
Introduction
Wireless LANs
◦ What is the Internet?
◦ 802.11
– Physical layer
– Medium Access Control (MAC) protocol
– Mobility
—
Dealing with packet jitter and loss
◦ Issues
◦ Reducing Delay Jitter
– Fixed vs. Adaptive Playout
◦ Recovering from Packet Loss
– FEC
– Interleaving
—
Case Study: Loss Resilience Features in H.264
Chapter 6: Multimedia Networking
78
39
4/14/16
Introduction
—
Two layers of H.264:
◦ Video Coding Layer (VCL) – translates video data into
bitstream (discussed in Chapter 4).
◦ Network Abstraction Layer (NAL) – maps and
packetizes VCL bitstream into units for storage or
transmission.
—
NAL - “Network friendly” features:
◦ Error resiliency (encoder) – improves the decoder’s
capability to recover/conceal errors.
◦ Error concealment (decoder) – recovers from errors.
Chapter 6: Multimedia Networking
NAL Units
Sequence
Parameter
Set
Picture
Parameter
Set
79
Picture
Parameter
Set
Chapter 6: Multimedia Networking
80
40
4/14/16
Loss Resilience
—
H.264 adds Baseline and Extended Profiles to
provide error resiliency:
◦ Add redundancy to the bitstream.
◦ Facilitates a better error concealment (i.e., recover
lost parts of the picture).
—
Baseline Profile:
◦ Arbitrary Slice Ordering (ASO)
◦ Flexible Macroblock Ordering (FMO)
◦ Redundant Pictures or Slices
—
Extended Profile:
◦ Data Partitioning
Chapter 6: Multimedia Networking
81
Arbitrary Slice Order
—
—
Slices can be arranged in any order.
Aids in error concealment at the decoder.
No ASO
Received Order: 0, 1, 2, 3
Slice 0
ASO
Received Order: 0, 2, 1, 3
Slice 0
Predicted from Slice 0 & 2 Lost!
Slice 1
Slice 1
Slice 2
Lost!
Slice 2
Slice 3
Lost!
Slice 3
Predicted from Slice 2
Lost!
Errors tend to occur in spurts
Chapter 6: Multimedia Networking
82
41
4/14/16
Flexible Macro Ordering
—
A picture can be subdivided into regions, called
slice groups:
◦ Each slice group can contain one or more slices.
—
Each slice is transmitted independently in
separate packets:
◦ Within a slice, MBs are usually processed in raster
scan order
—
—
Using FMO, a frame can be split into many MB
scanning patterns that results in good error
robustness.
FMO together with advanced resilience tools
can maintain visual quality even with packet loss
rate of 10%!
Chapter 6: Multimedia Networking
83
7 FMO Types
Each color represents a slice group
Run-length
Grows\shrinks in
circular pattern
Regions of interests
Scattered
Row-wise
Column-wise
Type 6 is the most general and allows full flexibility to the user.
Chapter 6: Multimedia Networking
84
42
4/14/16
FMO Example
—
Type 1 (scattered or dispersed) is especially useful in
concealing errors in networks with high packet loss
rate.
Easier to conceal
Slice 1 lost!
Slice 0
Slice 1
Slice 2
No FMO (3 slices)
0
1
2
3
0
1
2
3
0
1
2
2
3
0
1
2
3
0
1
2
3
0
0
1
2
3
0
1
2
3
0
1
2
2
3
0
1
2
3
0
1
2
3
0
0
1
2
3
0
1
2
3
0
1
2
2
3
0
1
2
3
0
1
2
3
0
0
1
2
3
0
1
2
3
0
1
2
2
3
0
1
2
3
0
1
2
3
0
0
1
2
3
0
1
2
3
0
1
2
FMO Type 1 (4 slice groups)
Chapter 6: Multimedia Networking
85
Redundant Slices
—
Allows an encoder to send an extra
representation of a picture region (typically at
lower fidelity) that can be used if the primary
representation is corrupted or lost:
◦ Also referred to as Forward Error Correction (FEC).
—
May improve performance, but at a cost of
reduced compression.
Chapter 6: Multimedia Networking
86
43
4/14/16
Data Partitioning (DP)
Separate more important and less important syntax
elements.
— Each compressed slice is partitioned into:
—
◦ Partition A - contains the most important information, such as
slice header, MB types, quantization parameters, and MVs.
◦ Partition B (intra partition) - contains intra-coded block pattern
(CBP) and coefficients.
◦ Partition C (inter partition) - contains only inter CBPs and inter
coefficients.
—
Partitions can be treated differently by the delivery
layer:
◦ Unequal Error Protection in PHY and MAC
◦ Prioritized or Guaranteed Services in MAC (e.g., 802.11e)
Chapter 6: Multimedia Networking
87
DP Illustration
Partition A
If lost, slice cannot
be reconstructed!
If lost, error
concealment can
be used
Partition B
(I- & SI-slice)
Partition C
(P-, B-, & SP-slice)
Chapter 6: Multimedia Networking
88
44
4/14/16
Error Concealment
—
Error resilience features of H.264 allows for
error concealment:
◦ Reconstruct lost information (within manageable
complexity).
—
Types of error concealment:
◦
◦
◦
◦
—
Spatial – Weighted Pixel Averaging, MV recovery
Temporal – Copy-Paste, MV recovery
Hybrid
Others…
Can be applied to both lost MBs and frames.
Chapter 6: Multimedia Networking
89
Error Characteristics
—
Depends on the severity and type of loss, error
resilience features used, and characteristics of video:
◦ I-slice – Intra prediction mode and residual
◦ P-slice – Motion vector and residual
Slices lost due
to packet loss
Sent frame
Received frame
Figure 9: Slices 2 and 4 lost.
Error Concealment
Error propagation 5 frames later
Figure 11: Chapter
Error propagation
three frames
later due to poor EC.
6: Multimedia
Networking
used
to conceal
lostfor
slices
and 4. work [10].
dia Player.Figure
It was 10:
alsoFrame
the thecopy
media
player
of choice
our 2previous
FFmpeg is a collection of libraries including libavcodec, which contains an H.264
decoder. The H.264 decoder contains an EC module, where different MV EC
instance, consider a car moving across a screen. Putting that single frame in
techniques can be found. The FFmpeg H.264 decoder applies spatial mean MV,
the context of neighboring frames adds a time dimension thus revealing whether
spatial median MV, last MV, and zero MV for P-frames [11]. Spatial mean MV
the car is stationary or if it is indeed moving in a particular direction. Such
simply takes the average of surrounding MVs to recover a lost MV while spatial
conclusions can be drawn from correlations witnessed within a single frame or in
median MV involves using the median of surrounding MVs instead. Using the
the context of multiple frames. The underlying concept of these correlations is
last MV for MV recovery involves copying the MV from the same spatial locausually motion. By exploiting the spatial and temporal relationships that exist
tion of the previous inter-predicted reference frame (usually a P-frame) and then
among MVs, it is possible to perform EC via MV recovery.
pasting in place of the lost MV. Zero MV simply uses a zero MV to estimate a
lost MV, and is also referred to as frame copy.
2.3Corrupted
ErrorB-frames
Concealment
Techniques
are handled
differently. Since B-MBs are bi-directionally
predicted,
each lost
MB in the
a B-frame
two MV used
estimates.
The decoders.
two MVs
This subsection
discusses
defaultrequires
EC techniques
by H.264
are
temporally
weighting
the
MVs
of a future
P-frame
as shown
Ourgenerated
discussionby
is based
on FFmpeg,
which
is the
underlying
software
of VLC
Mein Fig. 12. The first set of estimated MVs refers to a past reference frame. This
is generated by multiplying the MVs from the next P-frame by tt12 . The second
set of estimated MVs, which points backwards, is generated by multiplying the
1 t2
same MVs from the next P-frame by t11
t2 .
90
45
4/14/16
DP – Perceptual Quality
Original
Only residual
(Partition C)
is lost!
I-frame
w/o residual
P-frame
w/o residual
Chapter 6: Multimedia Networking
91
Copy-Paste
—
—
—
—
—
Temporal technique
Replaces the missing image part with the
spatially corresponding part inside a previously
decoded frame.
Very effective when there is very little
movement.
Can be applied to lost MBs or frames (called
frame copy).
Can be applied to I- or P-frames.
Chapter 6: Multimedia Networking
92
46
4/14/16
Copy-Paste - Perceptual
Quality
Original
Error
Copy-Paste
Chapter 6: Multimedia Networking
93
Weighted Pixel Averaging
—
—
—
Based on assumption of
continuity of natural scene
content in space
Spatial concealment technique. p
L
Used for I-frames:
pT
dL
dT
dR
p̂
pR
◦ Can also be used for P-frames
—
—
dB
Use pixel values of
surrounding available MBs
Disadvantage – blurred
reconstruction
pB
p̂ =
pT d B + pB dT + pL d R + pR d L
d B + dT + d R + d L
Chapter 6: Multimedia Networking
94
47
4/14/16
WPA - Perceptual Quality
Original
Error
Weighted
Pixel
Averaging
Chapter 6: Multimedia Networking
95
MV Recovery
—
For P- and B-frames:
◦ MV can be lost
◦ Residual (error) can be lost
◦ Or both can be lost
—
FFmpeg – decoder for VLC Media
Player:
Spatial technique
◦ Spatial - Left MV, Right MV, Top
MV, Bottom MV, Mean MV, Median
MV
◦ Temporal - Last MV, Zero MV
(frame copy)
—
Each estimated MV generates an
estimated MB, which is compared
against surrounding MBs using
Boundary Matching Algorithm
(BMA).
◦ The best one is used as the final
MB!
Substitute
MB
Reference
Frame
Missing
MB
Current
Frame
Temporal technique
Chapter 6: Multimedia Networking
96
48
4/14/16
Boundary Matching Algorithm
Calculates the total
absolute pixel value
difference between the
edge pixels of the
estimated MB and the
adjacent pixels of
surrounding MBs.
— Choose the one with
minimum error.
MVs neighboring blocks
—
Median/Average MV
from neighboring blocks
Es timate
d
MB
Es timate
d
MB
Boundary Pixels
MV from previous frame
Es timate
d
MB
No MV
Es timate
d MB
Chapter 6: Multimedia Networking
97
MV Recovery - Perceptual
Quality
Original
Error
MV Recovery
Chapter 6: Multimedia Networking
98
49
4/14/16
Iterative Process of EC
—
Each MB in a lost region is concealed in the
order shown below.
Good slice
Lost slice
Good slice
Lost MB
Consider a simplified lost slice with good slices above and below it.
Chapter 6: Multimedia Networking
99
Chapter 6: Multimedia Networking
100
Questions?
50