Download Immersive Media Delivery and Access Grid

Video over IP
as a VoIP extension
JongWon Kim, Ph.D.
Advanced Network Conference (APAN Busan 2003)
Aug 27th, 2003
Networked Media Laboratory
Dept. of Information & Communications
Kwang-Ju Institute of Science & Technology (K-JIST)
[email protected]
http://netmedia.kjist.ac.kr
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Contents
 Media Delivery over IP
 Network-adaptive Media Delivery
o Networking Support
o Network Adaptation
 Video over IP as a VoIP extension
 Future of Video over IP - Toward Access Grid &
Immersive Media
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Multimedia Delivery
over IP
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Multimedia over IP Networks
Media Broadcasting
Hosted Streaming
Interactive Conferencing
Conferencing
Proxy
Streaming
Sender
RTN
Proxy
Internet
Manager
Receivers
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Multimedia Delivery - Technologies
 Efficient Media Representation Techniques:
 Scalable source representation to deal with dynamically varying
bandwidth
 Robust source representation to deal with high error/loss rate
 Dynamic and Reliable Media Delivery Techniques:
 Network adaptation to map scalable and robust source represent
ations onto the MM characteristic-aware transport protocols
 Adaptive resource provisioning/management to guarantee the de
sired QoS / multicast service
 Universal Media Access with Security Techniques:
 Unified integrated media support with a scalable information str
ucture and a choice of efficient file format for MM
 End-system and application customization with security concerns
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Multimedia Application’s QoS Requirements
Interactive
Responsive
Timely
Non-critical
Packet Loss
5%
Conversational
voice and video
0%
100 msec
Zero
loss
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Comman
d/ control
(eg Telnet,
Interactive
games)
Voice/video
messaging
1 sec
Transactions
(eg E-commerce,
Web-browsing, Email access)
Streaming
audio/video
10 sec
Paging,
Download
s
(eg FTP,
still image)
Delay
Fax
100 sec
Background
(eg Usenet)
Media Delivery – Application vs Network
Network Characteristics and Policies:
Rate (CBR, VBR, ABR, …), Error
(random/bursty in BER/PLR/CRL),
Delay/Jitter (tolerance)
Content-aware Transport
Application/Source
Application/Source Characteristic :
steady/burst, real-time?, lossresilient?
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Network
Network-adaptive Delivery
Network Adaptive Media
Delivery
- Networking Support Part -
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Media Delivery – Adaptation & Support
Congestion
(Rate, Flow)
Control
Media Representation
(Compression)
Media
Synchronizati
on
System
Support
(Transport,
OS,
Storage)
Dynamic
Network
Adaptation
Framework
Error Control
Network QoS
Support
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Interoperable
Protocol
Other
Middleware
Support
Network
Multicast
Support
Inter-operable Delivery with Protocol
Support
Media Contents
Shared Tools
RTP Payload
RSVP
Security
Protocols
RTCP
Conference Control
SDP
SAP
SIP
RTSP
HTTP
RTP
UDP
TCP
IPv4, IPv6
 Standard-based vs. Proprietary Protocols?
 Related issue: Standardized media content
representation (e.g. MPEG-X) vs. Others?
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
SMTP
Required Network Infrastructure:
Multicast, QoS, and others
Paid services with
Copyright management,
authentication,
billing, etc
Content
broadcast
network
End-to-end QoS guarantee with
QoS provisioning (admission control
and QoS controls) &
Network adaptation
Economical and reliable network with
The Internet
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Scalable multicasting & Optimized
server location and load sharing
including contents routing
Multicast Support for Multipoint
Media Distribution
Multicast streaming?
Model (ASM, SSM, ALM/Overlay/P2P)
Source filtering
capability
Internet
232/8
IGMPv3
Reliable Multicast Transport
Mobile/Wireless Multicast
Resilient Multicasting of continuous media
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
QoS-network Support for Quality
Media Delivery
QoS Model: IntServ, DiffServ, Traffic
Engineering (MPLS), …
Configuration
Reservation
request
QoS
monitor
TCP/IP
Console or
Client User
Resource
Manager-aware
Application
CAC
module
Execution module
Monitoring
module
SNMP module
Route
Re source
Mon.
KOREN21
K-JIST
Reservation and Allocation
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
SLA
module
Route
Info
module
DB Interface
JAVA Interface
Program
DB
SLA
TCP/IP
Browsing
QoS signaling
Auth.
module
BBTP
Resource
Manager
EF
MIB
BE
WFQ
DiffServ
Network
MIB
MIB
KISTI
Toward Consolidated Support via
Middleware?
 Issues to be covered +++
o
o
o
o
Identification
Directories, …
Security: Authentication, Authorization, …
…
 Middleware:
o Specialized networked services that are shared by
applications and users
o A set of core software components that permit scaling
of applications and networks
 GRID Middleware
 Home Networking Middleware
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Network Adaptive Media
Delivery
- Network Adaptation Part:
Bandwidth (rate), loss, and delay -
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Media Delivery – Adaptation & Support
Congestion
(Rate, Flow)
Control
Media Representation
(Compression)
Media
Synchronizati
on
System
Support
(Transport,
OS,
Storage)
Dynamic
Network
Adaptation
Framework
Error Control
Network QoS
Support
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Interoperable
Protocol
Other
Middleware
Support
Network
Multicast
Support
Required Network Adaptation
Functionalities
 In case of Reliable Unicast with feedback
o Network Congestion Control => Quality Adaptation: Source
Rate Control / Network Rate Adaptation
o Error control: Error Resiliency and Error Recovery
o Quality Recovery: Post-processing & Concealment
o Synchronization Control & Adaptive Playout
Quality Adaptation
Error Control
Quality Recovery
Adaptable
Source
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Media
Pump
Congestion
Control
(TCP-friendly)
Network
Client
Synchronization
Control
TCP-Friendly Congestion Control /
Rate Control
TCP-Friendly Congestion Control: equationbased …
Toward unified congestion control handling:
Congestion Manager (CM)
Congestion Control / Rate Control / Flow
Control
Multicast congestion control?
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
TCP-Friendly Congestion Control /
Rate Control (Cont.)
Rate Control by
o Source Rate Control (on-line encoding only)
o Source Rate Adaptation (or Shaping)
o Smoothing – via Buffering
o Packet Scheduling
o Pre-fetching
I
Transmission
Rate
B
B B P
(4)
time
New connection
request
I-picture
Starting point
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
I BB P BBP BB P BB I BB P
B
B
Time
Ravg
0
(3)
B
Transmission
Rate
period
(2)
B
Td 2Td 3Td 4Td 5Td 6Td 7Td 8Td 9Td
Superposed stream
(existing sources)
(1)
B
Non-Smoothed
VBR stream
P
P
BB P
B B
Smoothed
near-CBR
stream
I
Td 2Td 3Td 4Td 5Td 6Td 7Td 8Td 9Td
Time
Proactive and Reactive Error Controls
Network error controls
o Proactive (FEC), Reactive (ARQ), and Hybrid
o Delay is key controlling factor
Network (Channel) error control vs Source
error resiliency?
Quality Recovery: Error Concealment and Postprocessing
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Error Control and Quality Recovery
Source
Network
Adaptation
Encoder
Network
Adaptation Decoder
CRC
Sink
Error Detection
Sequence No.
Loss Detection
Retransmission
Error Correction
FEC
Data Structuring
(Synchronization)
Erasure Correction
Resynchronization
Quality Control
(Layered Coding)
Loss Concealment
Network Framing
Post Processing
Application Framing
Coding
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Courtesy of Dr. Noh NVW 2001
Synchronization Control and Adaptive
Playout
Synchronization
o Intra-media
o Inter-media
o Inter-client
o Tight vs Loosely-coupled
Adaptive playout
o Utilizing silence
o Utilizing time-scale
modification
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Media Delivery – Network Adaptation
MM Content
Source encoding
Feedback from Host/Application
Scalable stream
in rate/error
Source transcoding &
Framing/Multiplexing
Integrated &
Feedback from Network
Channel
stream
prioritized
stream
Trans-MUX &
Channel coding
with FEC/ARQ
Network
Transmission
< Source prioritization for content-aware network adaptation >
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Media Delivery – Network Adaptation
Layered Video Encoding
Network Adaptation &
Prioritized Packetization
R-D /
Corruption
Model
R-D
Analysis
Video Layered
Preenprocesscoding Corruping
tion
Analysis
Frame
Complexity
(Quality
Constraint)
Constant
Quality
Rate
Control
Target
Minimum
Bandwidth
(Network
Constraint)
Target
Minimum
Buffer Size
(Receiver
Constraint)
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Network
Adaptation
Layered
RPI
Wired/Wireless
Networks
Network
Adaptation
(Source Prioritized
Rate/Error PacketiResilience) zation
(Network
Rate/Loss/
(only if
Delay)
applicable)
Estimated
Available
Bandwidth,
Loss/Delay
Network
Adaptation
Network
(Network Adaptation
Filtering)
Network Feedback
Network
Monitoring
&
Feedback
Handling
Receiver/
User
(Receiver /
User
Adaptation)
Network Feedback (end-to-end)
Application Feedback
Video over IP as VoIP
extension (V2oIP)
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
VoIP Extension: Voice + Video + Data
 We want to achieve IP TELEPHONY (a.k.a.
voice/video/data convergence): Real time voice,
video and data communication over Internet
Protocol (IP) networks
 However, latest high-end VC system supports
o
o
o
o
o
o
o
o
H.263+ or H.263++ (w/ Annex U for background restoration)
384 kbit/s or 768 kbit/s
TDM/ISDN (H.320) or IP (H.323)
CIF (352x288) or CIF-Interlace 2x(352x288)
Capture and rendering typically interlace
25/30 fps, 50/60 fields per second
Round-trip delay around 400ms +
Round-trip delay for multipoint (w/ MCU) more than one second
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
V2oIP Building Blocks




Protocol stacks (H.323, MGCP, SIP)
Switching cores (e.g. Gatekeepers, Call Agents)
Media engines (Announcement, Mixing, Gateway)
Service creation environment
Call Agents
Proxy
SIP
Service
Service
Creation
Announcement
Conference Bridges Creation
Player
Announcement
Toolkit
Multi-point Control
MGCP
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
GateGatekeepers
keepers
Gatekeeper Toolkit
H.323
Gateways
Gateways
V2oIP Deployment
MCU
POTS
PSTN
Multicast Audio and Video
Decentralized
Unicast Audio and Video
Centralized
H.323
Entity
TC
Non-H.323
Entity
TP
xDSL
Non-H.323
System
Administrative
Domain A
Physical
Connection
GKA1
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
BECH
GKC/
BEC
GW
Signaling
H.323
Entity
TA
Clearing
House
GKC1
GKA/BEA
Cable
BEB/GKB
Administrative
Domain A
GKB1
Administrative
Domain B
Backend
Servers (services)
H.323
Entity
TB
Video Quality and Function Issues (1)
Improve Video Quality and Size
 Upcoming H.264 video encoding
o
o
o
Product design stage now, 640x480 at full frame rate on one high-end DSP
Bit rate around 384 kbit/s yields acceptable picture quality
Subjective quality significantly better than H.263++ w/ CIF at same bit rate
o
Linking with region-oriented (maybe combined with segmentation) encoding on top
of variable frame skipping
 Variable, content-dependent capture rate
 Wide-screen video support
o
Can support video resolution like 2560x480 with single CCD camera with custom,
wide-angle lens; Multiple DSPs for distributed codec is required; 1.5 – 3 Mbps
 Progressive scan hardware
o
No A/D and D/A losses; Better quality and good delay characteristics; 250ms
round-trip delay over IP demonstrated?
 Enhance error resilience with ERPS and so on
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Video Quality and Function Issues (2)
Provide more natural, cooperative
environment for multi-site conferencing
 Let’s avoid transcoding and enhance user experience
o Create all representations needed in the sending endpoint and simulcast
them (in line with layered multicast video)
o Ideally suited for IP multicast environment; But can be with overlay
(application-layer) multicast;
o May need some innovative changes in the control and mux protocols
o No transcoding artifacts whatsoever
o No additional end-to-end delay for multipoint when using IP multicast
 Let’s get rid of fixed environment that discourages
uncooperative use of the technology
o Toward flexible meeting environment without fixed camera position,
fixed background, studio lighting, seating
 Let’s add more functionalities while keeping user interface
simple and straightforward
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
VoIP Extension: Real-time Collaboration
 Bring Voice, Screen, Computation together to improve: Realtime Collaboration Vision from Microsoft
o
o
o
o
o
Rich presence and IM
PC and phone integration
Data/screen conferencing
Information agent
Enhanced Meetings
incoming
communi
cation
context
to web
phone
to task
list
Automatic
Broadcast and
Recording
Remote Participation
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Note taking
2003 Gurdeep Singh Pall from Microsoft Corporation
Future of Video over IP
- Access Grid & Immersive Media -
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
The Access Grid
Access Grid does for people what the
computational Grid does for machines
Much more than teleconferencing, if possible
at commodity prices
To enable groups of people to interact with
Grid resources and to use the Grid
technology to support group to group
collaboration at a distance
o
o
o
o
Distributed Lectures and seminars
Remote participation in panel discussions
Virtual site visits meetings
Complex distributed grid based demonstrations
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Usage Areas
 The Academic and Research, Government, Private Sectors
 Making Remote Collaborations Work across Boundaries
 Check “Multi-Sector Collaboration over the Access Grid” by J.
T. von Hoffman (Boston Univ)”
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Access Grid
Components of an AG Node
RGB Video
Digital Video
Display
Computer
Digital Video
NETWORK
Digital Audio
Video
Capture
Computer
NTSC Video
Audio
Capture
Computer
Analog Audio
Mixer
Control
Computer
Presenter
mic
Echo
Canceller
Stream
Type
Max.
latency
Max.
Jitter
Min. BW
Max.
Loss
Multi
cast
Text
100 ms
N/A
64 kbps
0%
No
Audio
400 ms
60 ms
64 kbps
5%
Yes
256 kbps x 4 = 1 Mbps
25
%
Video
400 ms
30 ms
Yes
Presenter
camera
Ambient mic
(tabletop)
Audience camera
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Requires
Multicast
AG extended toward ACE





Enable group-to-group interactions at a
distance
Improve the user experience
Enable complex multi-site visual and
collaborative experiences
Build on integrated grid services
architecture
Use quality but affordable digital IP based
audio/video
Immersive Media Supported
Advanced Collaboration Environment
Advanced Collaboration Environment
Access Grid 2.0
Access Grid
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Broadband Content Requires More…
Browsing
144 Kbps
½ Screen Video
300Kbps
Music CDs
160 Kbps
Full Screen
600Kbps – 1.5Mbps
TV Shows
Music
High Resolution
6-8 Mbps
Live Events
Pay Per View
 What kinds of broadband contents
o
o
o
o
High speed Internet access: WWW + Mail + Messaging
Video focused: Basic TV + Pay TV + Personalized TV
Networked games + Music downloads + Online gambling
…
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Immersive
Media
> 100 Mbps
Future Broadband Contents Service &
Immersive Media
 Interactive collaboration - Collaborative design and engineering; Remote
customer support; Distance learning
 Remote visualization & Large-scale, multi-site computation and data
mining
 Shared virtual reality - Military and industrial team training / simulation;
Multiplayer games; Virtual shopping malls (e-commerce); Mobile
entertainment; Online tradeshows and conferences; Virtual Heritage
 Any combination of the above
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Core Techniques for Immersive Media
Service Integration
3D Video Synthesis &
Reconstruction
Immersive Media Storage
& Query/Browsing
3D Geometry
Generation
& Modeling
Immersive Media
Content Adaptation
Interactive User
Interface
Immersive Media
Compression &
Integration
Immersive Media
Delivery
Contents Service
(Protocol & System)
3D Audio Generation
& Reconstruction
Multi-modal Immersive
Media Service Technology
Multi-D Haptic
Generation &
Reconstruction
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST
Digital Contents
Management &
Protection
Thank You!
Send Inquiries to
[email protected]
NETWORKed MEDIA LAB.
DEPT. OF INFO. & COMM., K-JIST