Download ppt - CSE Labs User Home Pages

Introduction to Multimedia
Networking
•
•
•
•
Classify multimedia applications
Identify the network services the apps need
Making the best of best effort service
Streaming Stored Multimedia vs. Interactive
Applications
– Adaptive Playback and Smart Error Recovery Algorithms
• Some Common Protocols:
– RTSP, RTP/RTCP, SIP
Required Readings:
Relevant chapters/sections from your
csci5211/csci4221 Textbook
CSci5221:
Multimedia
1
Multimedia and Quality of Service
Multimedia applications:
network audio and video
(“continuous media”)
QoS
network provides
application with level of
performance needed for
application to function.
CSci5221:
Multimedia
2
Digital Audio
• Sampling the analog signal
– Sample at some fixed rate
– Each sample is an arbitrary real number
• Quantizing each sample
– Round each sample to one of a finite number of values
– Represent each sample in a fixed number of bits
4 bit representation
(values 0-15)
CSci5221:
Multimedia
3
Audio Examples
• Speech
– Sampling rate: 8000 samples/second
– Sample size: 8 bits per sample
– Rate: 64 kbps
• Compact Disc (CD)
– Sampling rate: 44,100 samples/second
– Sample size: 16 bits per sample
– Rate: 705.6 kbps for mono,
1.411 Mbps for stereo
CSci5221:
Multimedia
4
Why Audio Compression
• Audio data requires too much bandwidth
– Speech: 64 kbps is too high for a dial-up modem user
– Stereo music: 1.411 Mbps exceeds most access rates
• Compression to reduce the size
– Remove redundancy
– Remove details that human tend not to perceive
• Example audio formats
– Speech: GSM (13 kbps), G.729 (8 kbps), and G.723.3 (6.4
and 5.3 kbps)
– Stereo music: MPEG 1 layer 3 (MP3) at 96 kbps, 128
kbps, and 160 kbps
CSci5221:
Multimedia
5
A few words about audio compression
• Analog signal sampled at
constant rate
– telephone: 8,000
samples/sec
– CD music: 44,100
samples/sec
• Each sample quantized, i.e.,
rounded
– e.g., 28=256 possible
quantized values
• Each quantized value
represented by bits
– 8 bits for 256 values
CSci5221:
Multimedia
• Example: 8,000
samples/sec, 256 quantized
values --> 64,000 bps
• Receiver converts it back
to analog signal:
– some quality reduction
Example rates
• CD: 1.411 Mbps
• MP3: 96, 128, 160 kbps
• Internet telephony: 5.3 13 kbps
6
Digital Video
• Sampling the analog signal
– Sample at some fixed rate (e.g., 24 or 30 times per sec)
– Each sample is an image
• Quantizing each sample
– Representing an image as an array of picture elements
– Each pixel is a mixture of colors (red, green, and blue)
– E.g., 24 bits, with 8 bits per color
CSci5221:
Multimedia
7
The
2272 x 1704
hand
CSci5221:
Multimedia
The
320 x 240
hand
8
A few words about video compression
• Video is sequence of
images displayed at
constant rate
Examples:
• MPEG 1 (CD-ROM) 1.5
Mbps
– e.g. 24 images/sec
• MPEG2 (DVD) 3-6 Mbps
• Digital image is array of
• MPEG4 (often used in
pixels
Internet, < 1 Mbps)
• Each pixel represented
Research:
by bits
• Layered (scalable) video
• Redundancy
– adapt layers to available
bandwidth
– spatial
– temporal
CSci5221:
Multimedia
9
Video Compression: Within an Image
• Image compression
– Exploit spatial redundancy (e.g., regions of same color)
– Exploit aspects humans tend not to notice
• Common image compression formats
– Joint Pictures Expert Group (JPEG)
– Graphical Interchange Format (GIF)
Uncompressed: 167 KB
CSci5221:
Multimedia
Good quality: 46 KB
Poor quality: 9 KB
10
Video Compression: Across Images
• Compression across images
– Exploit temporal redundancy across images
• Common video compression formats
– MPEG 1: CD-ROM quality video (1.5 Mbps)
– MPEG 2: high-quality DVD video (3-6 Mbps)
– Proprietary protocols like QuickTime and RealNetworks
CSci5221:
Multimedia
11
MM Networking Applications
Classes of MM applications:
1) Streaming stored audio
and video
2) Streaming live audio and
video
3) Real-time interactive
audio and video
Jitter is the variability
of packet delays within
the same packet stream
CSci5221:
Multimedia
Fundamental
characteristics:
• Typically delay sensitive
– end-to-end delay
– delay jitter
• But loss tolerant:
infrequent losses cause
minor glitches
• Antithesis of data,
which are loss intolerant
but delay tolerant.
12
Application Classes
• Streaming
– Clients request audio/video files from servers and
pipeline reception over the network and display
– Interactive: user can control operation (similar to VCR:
pause, resume, fast forward, rewind, etc.)
– Delay: from client request until display start can be 1 to
10 seconds
CSci5221:
Multimedia
13
Application Classes (more)
• Unidirectional Real-Time:
– similar to existing TV and radio stations, but delivery on the
network
– Non-interactive, just listen/view
• Interactive Real-Time :
– Phone conversation or video conference
– More stringent delay requirement than Streaming and
Unidirectional because of real-time nature
– Video: < 150 msec acceptable
– Audio: < 150 msec good, <400 msec acceptable
CSci5221:
Multimedia
14
Streaming Stored Multimedia
Streaming:
• media stored at source
• transmitted to client
• streaming: client playout begins
before all data has arrived
• timing constraint for still-to-be
transmitted data: in time for playout
CSci5221:
Multimedia
15
Streaming Stored Multimedia:
What is it?
1. video
recorded
2. video
sent
network
delay
3. video received,
played out at client
time
streaming: at this time, client
playing out early part of video,
while server still sending later
part of video
CSci5221:
Multimedia
16
Streaming Stored Multimedia:
Interactivity
• VCR-like functionality: client can
pause, rewind, FF, push slider bar
– 10 sec initial delay OK
– 1-2 sec until command effect OK
– RTSP often used (more later)
• timing constraint for still-to-be transmitted data:
in time for playout
CSci5221:
Multimedia
17
Streaming Live Multimedia
Examples:
• Internet radio talk show
• Live sporting event
Streaming
• playback buffer
• playback can lag tens of seconds after
transmission
• still have timing constraint
Interactivity
• fast forward impossible
• rewind, pause possible!
CSci5221:
Multimedia
18
Interactive, Real-Time Multimedia
• applications: IP telephony,
video conference, distributed
interactive worlds
• end-end delay requirements:
– audio: < 150 msec good, < 400 msec OK
• includes application-level (packetization) and network
delays
• higher delays noticeable, impair interactivity
• session initialization
– how does callee advertise its IP address, port number,
encoding algorithms?
CSci5221:
Multimedia
19
Multimedia Over Today’s Internet
TCP/UDP/IP: “best-effort service”
• no guarantees on delay, loss
?
?
?
?
?
?
But you said multimedia apps requires ?
QoS and level of performance to be
?
? effective!
?
?
Today’s Internet multimedia applications
use application-level techniques to mitigate
(as best possible) effects of delay, loss
CSci5221:
Multimedia
20
Challenges
• TCP/UDP/IP suite provides best-effort, no
guarantees on expectation or variance of packet
delay
• Streaming applications delay of 5 to 10 seconds is
typical and has been acceptable, but performance
deteriorate if links are congested (transoceanic)
• Real-Time Interactive requirements on delay and
its jitter have been satisfied by over-provisioning
(providing plenty of bandwidth), what will happen
when the load increases?...
CSci5221:
Multimedia
21
Challenges (more)
• Most router implementations use only
First-Come-First-Serve (FCFS) packet
processing and transmission scheduling
• To mitigate impact of “best-effort”
protocols, we can:
– Use UDP to avoid TCP and its slow-start phase…
– Buffer content at client and control playback to remedy
jitter
– Adapt compression level to available bandwidth
CSci5221:
Multimedia
22
How should the Internet evolve to
better support multimedia?
Integrated services philosophy:
• Fundamental changes in
Internet so that apps can
reserve end-to-end
bandwidth
• Requires new, complex
software in hosts & routers
Laissez-faire
• no major changes
• more bandwidth when
needed
• content distribution,
application-layer multicast
– application layer
CSci5221:
Multimedia
Differentiated services
philosophy:
• Fewer changes to Internet
infrastructure, yet provide
1st and 2nd class service.
Will discuss QoS later!
23
Solution Approaches in IP
Networks
• Just add more bandwidth and enhance caching capabilities
(over-provisioning)!
• Need major change of the protocols :
– Incorporate resource reservation (bandwidth, processing, buffering),
and new scheduling policies
– Set up service level agreements with applications, monitor and enforce
the agreements, charge accordingly
• Need moderate changes (“Differentiated Services”):
– Use two traffic classes for all packets and differentiate service
accordingly
– Charge based on class of packets
– Network capacity is provided to ensure first class packets incur no
significant delay at routers
CSci5221:
Multimedia
24
Streaming Stored Multimedia
Application-level streaming
techniques for making the
best out of best effort
service:
– client side buffering
– use of UDP versus TCP
– multiple encodings of
multimedia
CSci5221:
Multimedia
Media Player
•
•
•
•
jitter removal
decompression
error concealment
graphical user interface
w/ controls for
interactivity
25
Internet Multimedia: Simplest Approach
• audio or video stored in file
• files transferred as HTTP object
– received in entirety at client
– then passed to player
audio, video not streamed:
• no, “pipelining,” long delays until playout!
CSci5221:
Multimedia
26
Internet multimedia: Streaming Approach
• browser GETs metafile
• browser launches player, passing metafile
• player contacts server
• server streams audio/video to player
CSci5221:
Multimedia
27
Streaming from a streaming server
• This architecture allows for non-HTTP protocol
between server and media player
• Can also use UDP instead of TCP.
CSci5221:
Multimedia
28
User Control of Streaming Media: RTSP
What it doesn’t do:
HTTP
• does not define how
• Does not target multimedia
audio/video is encapsulated
content
for streaming over network
• No commands for fast
• does not restrict how
forward, etc.
streamed media is
transported; it can be
RTSP: RFC 2326
transported over UDP or
• Client-server application
TCP
layer protocol.
• For user to control display: • does not specify how the
media player buffers
rewind, fast forward,
audio/video
pause, resume,
repositioning, etc…
CSci5221:
Multimedia
29
RTSP: out of band control
FTP uses an “out-ofband” control channel:
• A file is transferred over
one TCP connection.
• Control information
(directory changes, file
deletion, file renaming,
etc.) is sent over a
separate TCP connection.
• The “out-of-band” and “inband” channels use
different port numbers.
CSci5221:
Multimedia
RTSP messages are also
sent out-of-band:
• RTSP control messages
use different port numbers
than the media stream:
out-of-band.
– Port 554
• The media stream is
considered “in-band”.
30
RTSP Example
Scenario:
• metafile communicated to web browser
• browser launches player
• player sets up an RTSP control connection,
data connection to streaming server
CSci5221:
Multimedia
31
Metafile Example
<title>Twister</title>
<session>
<group language=en lipsync>
<switch>
<track type=audio
e="PCMU/8000/1"
src = "rtsp://audio.example.com/twister/audio.en/lofi">
<track type=audio
e="DVI4/16000/2" pt="90 DVI4/8000/1"
src="rtsp://audio.example.com/twister/audio.en/hifi">
</switch>
<track type="video/jpeg"
src="rtsp://video.example.com/twister/video">
</group>
</session>
CSci5221:
Multimedia
32
RTSP Operation
CSci5221:
Multimedia
33
RTSP Exchange Example
C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0
Transport: rtp/udp; compression; port=3056; mode=PLAY
S: RTSP/1.0 200 1 OK
Session 4231
C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
Range: npt=0C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
Range: npt=37
C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0
Session: 4231
S: 200 3 OK
CSci5221:
Multimedia
34
Streaming Multimedia: Client Buffering
variable
network
delay
client video
reception
constant bit
rate video
playout at client
buffered
video
constant bit
rate video
transmission
client playout
delay
time
• Client-side buffering, playout delay compensate
for network-added delay, delay jitter
CSci5221:
Multimedia
35
Streaming Multimedia: Client Buffering
“constant”
drain
rate,
d
variable
fill
rate, x(t)
buffered
video
• Client-side buffering, playout delay compensate
for network-added delay, delay jitter
CSci5221:
Multimedia
36
Streaming Multimedia: UDP or TCP?
UDP
• server sends at rate appropriate for client (oblivious to
network congestion !)
– often send rate = encoding rate = constant rate
– then, fill rate = constant rate - packet loss
• short playout delay (2-5 seconds) to compensate for network
delay jitter
• error recover: time permitting
TCP
•
•
•
•
send at maximum possible rate under TCP
fill rate fluctuates due to TCP congestion control
larger playout delay: smooth TCP delivery rate
HTTP/TCP passes more easily through firewalls
CSci5221:
Multimedia
37
Streaming Multimedia: client rate(s)
1.5 Mbps encoding
28.8 Kbps encoding
Q:
how to handle different client receive rate
capabilities?
– 28.8 Kbps dialup
– 100Mbps Ethernet
A:
server stores, transmits multiple copies of video,
encoded at different rates
CSci5221:
Multimedia
38
Real-time interactive
applications
• PC-2-PC phone
– instant messaging
services are providing
this, e.g. Yahoo
messenger, googletalk
– Skype
• PC-2-phone
Going to now look
at a PC-2-PC
Internet phone
example in detail
– Dialpad
– Net2phone
– Skype
• videoconference with
Webcams
CSci5221:
Multimedia
39
Voice Over IP (VoIP)
• Delivering phone calls over IP
– Computer to computer
– Analog phone to/from computer
– Analog phone to analog phone
• Motivations for VoIP
– Cost reduction
– Simplicity
– Advanced applications
•
•
•
•
CSci5221:
Web-enabled call centers
Collaborative white boarding
Do Not Disturb, Locate Me, etc.
Voicemail sent as e-mail
Multimedia
40
Traditional Telecom
Infrastructure
7040
External line
7041
Corporate/Campus
Private Branch
Exchange
7042
212-8538080
Telephone
switch
Another
switch
7043
Corporate/Campus LAN
CSci5221:
Multimedia
Internet
41
VoIP Gateways
Corporate/Campus
7040
Another campus
8151
External line
8152
7041
PBX
PBX
8153
7042
7043
VoIP Gateway
VoIP Gateway
Internet
LAN
8154
LAN
IP Phone Client
CSci5221:
Multimedia
42
VoIP With an Analog Phone
• Adapter
– Converts between analog and digital
– Sends and receives data packets
– Communicates with the phone in standard way
CSci5221:
Multimedia
43
Skype
• Niklas Zennström and
Janus Friis in 2003
• Developed by KaZaA
• Instant Messenger
(IM) with voice
support
• Based on peer-topeer (P2P) networking
technology
CSci5221:
Multimedia
44
Skype Network Architecture
• Login server is the only
central server (consisting of
multiple machines)
• Both ordinary host and super
nodes are Skype clients
• Any node with a public IP
address and having
sufficient resources can
become a super node
• Skype maintains their own
super nodes
CSci5221:
Multimedia
45
Challenges of Firewalls and NATs
• Firewalls
– Often block UDP traffic
– Usually allow hosts to initiate connections on port 80
(HTTP) and 443 (HTTPS)
• NAT
– Cannot easily initiate traffic to a host behind a NAT
– … since there is no unique address for the host
• Skype must deal with these problems
– Discovery: client exchanges messages with super node
– Traversal: sending data through an intermediate peer
CSci5221:
Multimedia
46
Interactive Multimedia: Internet Phone
Introduce Internet Phone by way of an example
• speaker’s audio: alternating talk spurts, silent
periods.
– 64 kbps during talk spurt
• pkts generated only during talk spurts
– 20 msec chunks at 8 Kbytes/sec: 160 bytes data
• application-layer header added to each chunk.
• Chunk+header encapsulated into UDP segment.
• application sends UDP segment into socket every
20 msec during talkspurt.
CSci5221:
Multimedia
47
Internet Phone: Packet Loss and Delay
• network loss: IP datagram lost due to
network congestion (router buffer
overflow)
• delay loss: IP datagram arrives too late for
playout at receiver
– delays: processing, queueing in network; end-system
(sender, receiver) delays
– typical maximum tolerable delay: 400 ms
• loss tolerance: depending on voice encoding,
losses concealed, packet loss rates
between 1% and 10% can be tolerated.
CSci5221:
Multimedia
48
Delay Jitter
variable
network
delay
(jitter)
client
reception
constant bit
rate playout
at client
buffered
data
constant bit
rate
transmission
client playout
delay
time
• Consider the end-to-end delays of two consecutive packets:
difference can be more or less than 20 msec
CSci5221:
Multimedia
49
Internet Phone: Fixed Playout Delay
• Receiver attempts to playout each chunk
exactly q msecs after chunk was
generated.
– chunk has time stamp t: play out chunk at t+q .
– chunk arrives after t+q: data arrives too late
for playout, data “lost”
• Tradeoff for q:
– large q: less packet loss
– small q: better interactive experience
CSci5221:
Multimedia
50
Fixed Playout Delay
• Sender generates packets every 20 msec during talk spurt.
• First packet received at time r
• First playout schedule: begins at p
• Second playout schedule: begins at p’
packets
loss
packets
generated
packets
received
playout schedule
p' - r
playout schedule
p-r
time
r
CSci5221:
Multimedia
p
p'
51
Adaptive Playout Delay, I
• Goal: minimize playout delay, keeping late loss rate low
• Approach: adaptive playout delay adjustment:
– Estimate network delay, adjust playout delay at beginning of
each talk spurt.
– Silent periods compressed and elongated.
– Chunks still played out every 20 msec during talk spurt.
t i  timestamp of the ith packet
ri  the time packet i is received by receiver
p i  the time packet i is played at receiver
ri  t i  network delay for ith packet
d i  estimate of average network delay after receiving ith packet
Dynamic estimate of average delay at receiver:
di  (1  u)di 1  u(ri  ti )
where u is a fixed constant (e.g., u = .01).
CSci5221:
Multimedia
52
Adaptive playout delay II
Also useful to estimate the average deviation of the delay, vi :
vi  (1  u)vi 1  u | ri  ti  di |
The estimates di and vi are calculated for every received packet,
although they are only used at the beginning of a talk spurt.
For first packet in talk spurt, playout time is:
pi  ti  di  Kvi
where K is a positive constant.
Remaining packets in talkspurt are played out periodically
CSci5221:
Multimedia
53
Adaptive Playout, III
Q: How does receiver determine whether
packet is first in a talkspurt?
• If no loss, receiver looks at successive
timestamps.
– difference of successive stamps > 20 msec -->talk spurt
begins.
• With loss possible, receiver must look at
both time stamps and sequence numbers.
– difference of successive stamps > 20 msec and sequence
numbers without gaps --> talk spurt begins.
CSci5221:
Multimedia
54
Recovery from packet loss (1)
forward error correction
(FEC): simple scheme
• for every group of n chunks
create a redundant chunk by
exclusive OR-ing the n original
chunks
• send out n+1 chunks, increasing
the bandwidth by factor 1/n.
• can reconstruct the original n
chunks if there is at most one
lost chunk from the n+1 chunks
CSci5221:
Multimedia
• Playout delay needs to
be fixed to the time to
receive all n+1 packets
• Tradeoff:
– increase n, less
bandwidth waste
– increase n, longer
playout delay
– increase n, higher
probability that 2 or
more chunks will be lost
55
Recovery from packet loss (2)
2nd FEC scheme
• “piggyback lower
quality stream”
• send lower resolution
audio stream as the
redundant information
• for example, nominal
stream PCM at 64 kbps
and redundant stream
GSM at 13 kbps.
• Whenever there is non-consecutive loss, the
receiver can conceal the loss.
• Can also append (n-1)st and (n-2)nd low-bit rate
chunk
CSci5221:
Multimedia
56
Recovery from packet loss (3)
Interleaving
•
•
•
chunks are broken
up into smaller units
for example, 4 5 msec units per
chunk
Packet contains small units from
different chunks
CSci5221:
Multimedia
•
•
•
if packet is lost, still have most
of every chunk
has no redundancy overhead
but adds to playout delay
57
Summary: Internet Multimedia: bag
of tricks
• use UDP to avoid TCP congestion control (delays)
for time-sensitive traffic
• client-side adaptive playout delay: to compensate
for delay
• server side matches stream bandwidth to available
client-to-server path bandwidth
– chose among pre-encoded stream rates
– dynamic server encoding rate
• error recovery (on top of UDP)
– FEC, interleaving
– retransmissions, time permitting
– conceal errors: repeat nearby data
CSci5221:
Multimedia
58
Real-Time Protocol (RTP)
• RTP specifies a packet
structure for packets
carrying audio and video
data
• RFC 1889.
• RTP packet provides
– payload type
identification
– packet sequence
numbering
– timestamping
CSci5221:
Multimedia
• RTP runs in the end
systems.
• RTP packets are
encapsulated in UDP
segments
• Interoperability: If two
Internet phone applications
run RTP, then they may be
able to work together
59
RTP runs on top of UDP
RTP libraries provide a transport-layer interface
that extend UDP:
• port numbers, IP addresses
• payload type identification
• packet sequence numbering
• time-stamping
CSci5221:
Multimedia
60
RTP Example
• Consider sending 64
kbps PCM-encoded
voice over RTP.
• Application collects
the encoded data in
chunks, e.g., every 20
msec = 160 bytes in a
chunk.
• The audio chunk along
with the RTP header
form the RTP packet,
which is encapsulated
into a UDP segment.
CSci5221:
Multimedia
• RTP header indicates
type of audio encoding
in each packet
– sender can change
encoding during a
conference.
• RTP header also
contains sequence
numbers and
timestamps.
61
RTP and QoS
• RTP does not provide any mechanism to
ensure timely delivery of data or provide
other quality of service guarantees.
• RTP encapsulation is only seen at the end
systems: it is not seen by intermediate
routers.
– Routers providing best-effort service do not make any
special effort to ensure that RTP packets arrive at the
destination in a timely matter.
CSci5221:
Multimedia
62
RTP Header
Payload Type (7 bits): Indicates type of encoding currently being
used. If sender changes encoding in middle of conference, sender
informs the receiver through this payload type field.
•Payload type 0: PCM mu-law, 64 kbps
•Payload type 3, GSM, 13 kbps
•Payload type 7, LPC, 2.4 kbps
•Payload type 26, Motion JPEG
•Payload type 31. H.261
•Payload type 33, MPEG2 video
Sequence Number (16 bits): Increments by one for each RTP packet
sent, and may be used to detect packet loss and to restore packet
sequence.
CSci5221:
Multimedia
63
RTP Header (2)
• Timestamp field (32 bytes long). Reflects the
sampling instant of the first byte in the RTP data
packet.
– For audio, timestamp clock typically increments by one
for each sampling period (for example, each 125 usecs
for a 8 KHz sampling clock)
– if application generates chunks of 160 encoded samples,
then timestamp increases by 160 for each RTP packet
when source is active. Timestamp clock continues to
increase at constant rate when source is inactive.
• SSRC field (32 bits long). Identifies the source
of the RTP stream. Each stream in a RTP session
should have a distinct SSRC.
CSci5221:
Multimedia
64
Real-Time Control Protocol (RTCP)
• Works in conjunction
• Statistics include
with RTP.
number of packets
• Each participant in
sent, number of
RTP session
packets lost,
periodically transmits
interarrival jitter, etc.
RTCP control packets
• Feedback can be used
to all other
to control
participants.
performance
• Each RTCP packet
– Sender may modify its
contains sender and/or
transmissions based on
feedback
receiver reports
– report statistics useful to
application
CSci5221:
Multimedia
65
RTCP - Continued
- For an RTP session there is typically a single multicast address; all RTP
and RTCP packets belonging to the session use the multicast address.
- RTP and RTCP packets are distinguished from each other through the use of
distinct port numbers.
- To limit traffic, each participant reduces his RTCP traffic as the number
of conference participants increases.
CSci5221:
Multimedia
66
RTCP Packets
Receiver report packets:
• fraction of packets
lost, last sequence
number, average
interarrival jitter.
Sender report packets:
• SSRC of the RTP
stream, the current
time, the number of
packets sent, and the
number of bytes sent.
CSci5221:
Multimedia
Source description
packets:
• e-mail address of
sender, sender's name,
SSRC of associated
RTP stream.
• Provide mapping
between the SSRC and
the user/host name.
67
Synchronization of Streams
• RTCP can synchronize
different media streams
within a RTP session.
• Consider videoconferencing
app for which each sender
generates one RTP stream
for video and one for audio.
• Timestamps in RTP packets
tied to the video and audio
sampling clocks
– not tied to the wallclock time
CSci5221:
Multimedia
• Each RTCP sender-report
packet contains (for the
most recently generated
packet in the associated
RTP stream):
– timestamp of the RTP
packet
– wall-clock time for when
packet was created.
• Receivers can use this
association to synchronize
the playout of audio and
video.
68
RTCP Bandwidth Scaling
• RTCP attempts to limit • The 75 kbps is equally shared
among receivers:
its traffic to 5% of
– With R receivers, each
the session bandwidth.
receiver gets to send RTCP
traffic at 75/R kbps.
Example
• Sender gets to send RTCP
• Suppose one sender,
traffic at 25 kbps.
sending video at a rate of 2
Mbps. Then RTCP attempts • Participant determines RTCP
to limit its traffic to 100
packet transmission period by
Kbps.
calculating avg RTCP packet
size (across the entire
• RTCP gives 75% of this
session) and dividing by
rate to the receivers;
allocated rate.
remaining 25% to the
sender
CSci5221:
Multimedia
69
SIP
• Session Initiation Protocol
• Comes from IETF
SIP long-term vision
• All telephone calls and video conference calls take
place over the Internet
• People are identified by names or e-mail
addresses, rather than by phone numbers.
• You can reach the callee, no matter where the
callee roams, no matter what IP device the callee
is currently using.
CSci5221:
Multimedia
70
SIP Services
• Setting up a call
– Provides mechanisms for
caller to let callee know
she wants to establish a
call
– Provides mechanisms so
that caller and callee can
agree on media type and
encoding.
– Provides mechanisms to
end call.
CSci5221:
Multimedia
• Determine current
IP address of
callee.
– Maps mnemonic
identifier to current IP
address
• Call management
– Add new media streams
during call
– Change encoding during
call
– Invite others
– Transfer and hold calls
71
Setting up a call to a known IP address
Bob
Alice
167.180.112.24
INVITE bob
@193.64.2
10.89
c=IN IP4 16
7.180.112.2
4
m=audio 38
060 RTP/A
VP 0
193.64.210.89
port 5060
port 5060
Bob's
terminal rings
200 OK
.210.89
c=IN IP4 193.64
RTP/AVP 3
3
75
m=audio 48
ACK
port 5060
m Law audio
port 38060
GSM
time
CSci5221:
Multimedia
port 48753
time
• Alice’s SIP invite message
indicates her port number
& IP address. Indicates
encoding that Alice prefers
to receive (PCM ulaw)
• Bob’s 200 OK message
indicates his port number,
IP address & preferred
encoding (GSM)
• SIP messages can be sent
over TCP or UDP; here sent
over RTP/UDP.
•Default SIP port number
is 5060.
72
Setting up a call (more)
• Codec negotiation:
– Suppose Bob doesn’t have
PCM ulaw encoder.
– Bob will instead reply with
606 Not Acceptable
Reply and list encoders he
can use.
– Alice can then send a new
INVITE message,
advertising an appropriate
encoder.
CSci5221:
Multimedia
• Rejecting the call
– Bob can reject with
replies “busy,” “gone,”
“payment required,”
“forbidden”.
• Media can be sent over
RTP or some other
protocol.
73
Example of SIP message
INVITE sip:[email protected] SIP/2.0
Via: SIP/2.0/UDP 167.180.112.24
From: sip:[email protected]
To: sip:[email protected]
Call-ID: [email protected]
Content-Type: application/sdp
Content-Length: 885
• Here we don’t know
Bob’s IP address.
Intermediate SIP
servers will be
necessary.
• Alice sends and
c=IN IP4 167.180.112.24
m=audio 38060 RTP/AVP 0
receives SIP messages
using the SIP default
port number 506.
Notes:
• HTTP message syntax
• sdp = session description protocol
• Call-ID is unique for every call.
• Alice specifies in Via:
header that SIP client
sends and receives
SIP messages over UDP
CSci5221:
Multimedia
74
Name translation and user locataion
• Caller wants to call
callee, but only has
callee’s name or e-mail
address.
• Need to get IP
address of callee’s
current host:
– user moves around
– DHCP protocol
– user has different IP
devices (PC, PDA, car
device)
CSci5221:
Multimedia
• Result can be based on:
– time of day (work, home)
– caller (don’t want boss to
call you at home)
– status of callee (calls sent
to voicemail when callee is
already talking to
someone)
Service provided by SIP
servers:
• SIP registrar server
• SIP proxy server
75
SIP Registrar
• When Bob starts SIP client, client sends SIP
REGISTER message to Bob’s registrar server
(similar function needed by Instant Messaging)
Register Message:
REGISTER sip:domain.com SIP/2.0
Via: SIP/2.0/UDP 193.64.210.89
From: sip:[email protected]
To: sip:[email protected]
Expires: 3600
CSci5221:
Multimedia
76
SIP Proxy
• Alice sends invite message to her proxy server
– contains address sip:[email protected]
• Proxy responsible for routing SIP messages to
callee
– possibly through multiple proxies.
• Callee sends response back through the same set
of proxies.
• Proxy returns SIP response message to Alice
– contains Bob’s IP address
• Note: proxy is analogous to local DNS server
CSci5221:
Multimedia
77
Example
SIP registrar
intel.com
.
Caller [email protected]
with places a
call to [email protected]
SIP registrar
google.com
2
SIP proxy
.
umn.edu
4
(1) Zhili sends INVITE
message to UMN SIP
1
5
7
proxy. (2) Proxy forwards
8
request to Intel
6
registrar server.
(3) Intel server returns
9
SIP client
redirect response,
197.87.54.21
SIP client
indicating that it should 217.123.56.89
try [email protected]
(4) UMN proxy sends INVITE to Google registrar. (5) Google registrar
forwards INVITE to 197.87.54.21, which is running Dan’s SIP client. (6-8)
SIP response sent back (9) media sent directly
between clients.
Note: also a SIP ack message, which is not shown.
CSci5221:
Multimedia
3
78
Comparison with H.323
• H.323 is another signaling
protocol for real-time,
interactive
• H.323 is a complete,
vertically integrated suite
of protocols for multimedia
conferencing: signaling,
registration, admission
control, transport and
codecs.
• SIP is a single component.
Works with RTP, but does
not mandate it. Can be
combined with other
protocols and services.
CSci5221:
Multimedia
• H.323 comes from the ITU
(telephony).
• SIP comes from IETF:
Borrows much of its
concepts from HTTP.
• SIP has a Web flavor,
whereas H.323 has a
telephony flavor.
• SIP uses the KISS
principle: Keep it simple
stupid.
79