Download More Pixels and Samples: High Resolution Media Streaming

More Pixels and Samples:
High Resolution Media Streaming
Roger Zimmermann
Data Management Research Laboratory
University of Southern California
Los Angeles, CA 90089
http://dmrl.usc.edu
Outline
• Motivation
• Background
– Remote Media Immersion
– Distributed Immersive Performance
• High-performance Data Recording
Architecture
• Demonstration
• Conclusions
APAN, January 2004
Integrated Media Systems Center, USC
Motivation
• The charter of the Integrated Media
Systems Center (IMSC) is
“Immersipresence”
– Immerse real (e.g. people) and virtual
elements into a common space
• Becomes much more interesting in a
distributed environment
– Many sub-problems: tracking, gesture
recognition, data management, …
– Video and audio are an important
component
APAN, January 2004
Integrated Media Systems Center, USC
What is the problem?
• Live streaming is either
– Low to medium quality, or
– Very expensive, i.e., there are only a few
people to call …
• Other obstacles
– Complicated (not like the telephone)
– Often requires room engineering
– Network bandwidth is not available
• Some of the technical constraints can
and will be solved
APAN, January 2004
Integrated Media Systems Center, USC
Ex.: Network Infrastructure
• UTOPIA (Utah Telecommunications Open
Infrastructure Agency): public works
project to provide fiber to the home
(FTTH).
• SuperNet, Alberta, Canada. Public
project to provide a high speed Internet
infrastructure.
• NSF sponsored workshop, Oct. 23-24, 2003,
Chicago, Illinois. The importance of
“broaderband” networks is recognized.
APAN, January 2004
Integrated Media Systems Center, USC
Research Timeline
2002
Jun 2-3
Unveiling of RMI Demonstration
Oct 29
Internet2 Meeting: RMI Demonstration
Dec 28
DIP Experiment 1: Distributed Duet
2003
Jan 18
Recording from Stream
Jan 19
DIP Experiment 2: Remote Master Class
Jun 2-3
DIP Experiment 3: Duet with Audience
2004
Jan 29
APAN Meeting: HYDRA Experiment
APAN, January 2004
Integrated Media Systems Center, USC
What is the RMI?
“The goal of the Remote Media Immersion
system is to build a testbed for the creation of
immersive applications.”
Immersive application aspects:
1. Multi-model environment (aural, visual, haptic, …)
2. Shared space with virtual and real elements
3. High fidelity
4. Geographically distributed
5. Interactive
APAN, January 2004
Integrated Media Systems Center, USC
RMI Challenges

Immersive, high-quality video
acquisition and rendering
 High Definition video 1080i and
720p (40 Mb/s)

Immersive, high-quality audio
acquisition and rendering
 10.2 channels of uncompressed
audio (12 Mb/s)

Storage and transmission of media
streams across networks

Synchronization between streams
(A/V, A/A, V/V)!
APAN, January 2004
Integrated Media Systems Center, USC
RMI Architecture
APAN, January 2004
Integrated Media Systems Center, USC
RMI Experimental Setup
• Synchronized immersive audio and HDTV streamed playback
from Yima server over Internet2
– 16 channels of immersive audio, uncompressed at 16 Mb/s
– 1920x1080i HDTV content, MPEG-2 compressed at 40 Mb/s
• Control of end-to-end process: capturing, network interface,
transmission, rendering
ISI East
IMSC
APAN, January 2004
Integrated Media Systems Center, USC
Internet2 Fall ‘02
Member Meeting
Video: HDTV 1280x720p
Audio: 10.2 channel,
immersive sound
system
New World Symphony, Miami, FL
APAN, January 2004
Integrated Media Systems Center, USC
Distributed Immersive Performance
• Outgrowth of Remote Media Immersion (RMI)
– Create seamless immersive environment for
distributed musicians, conductor (active) and
audience (passive)
– Compelling relevance for any human interaction
scenario: education, journalism, communications
• Scenario:
– Orchestra not available in town
– Famous soloist cannot fit travel into schedule
– Multiple soloists in different places
APAN, January 2004
Integrated Media Systems Center, USC
APAN, January 2004
Integrated Media Systems Center, USC
APAN, January 2004
Integrated Media Systems Center, USC
APAN, January 2004
Integrated Media Systems Center, USC
APAN, January 2004
Integrated Media Systems Center, USC
60 ms
20 ms
30 ms
40 ms
10 ms
30 ms
Challenge: network latency
APAN, January 2004
Integrated Media Systems Center, USC
• Key observations:
– Network latency maps to audio delay on stage
– Video delay is zero
• Challenge:
– Synchronization
– Transmitting low latency video of conductor to players
and audience
– Maintaining constant delay between players
Player 1
15m: 45ms
15m: 45ms
Conductor
Player 2
APAN, January 2004
10m: 30ms
Integrated Media Systems Center, USC
Barriers and Requirements
1. Real-time continuous media (CM) stream
transmission (network protocol) with low latency
2. Precise timing: GPS clock, synchronization
3. Data loss management: error concealment, FEC,
retransmission, multi-path streaming
4. Many-to-many transmission capability
5. Low latency, high-quality real-time video and
audio acquisition and rendering
6. Real-time CM stream recording
7. User experiments, requirements, specifications,
performance evaluation
APAN, January 2004
Integrated Media Systems Center, USC
Distributed Immersive Performance
v.1.0-The Duet
• Experiments and Objectives
– Experimental testbed and demonstration system
– Demonstrate and document a distributed musical performance
with two musicians (a duet)
– Two-way interactive video and 10.2 channel immersive audio
capability
– Explore other applications involving passive and active participants,
such as two-site interactive meetings
– Evaluate technical barriers and psychophysical effects of latency
and fidelity on music and other forms of human interaction
between two interconnected sites
• Dennis Thurmond - USC Thornton School of Music
• Elaine Chew - USC Industrial and Systems Engineering
APAN, January 2004
Integrated Media Systems Center, USC
Distributed Immersive Performance
v.1.0-The Duet
Linux PC
Linux PC
DV FireWire
Camera
DV FireWire
Camera
100BaseT
campus net
100BaseT
IMSC net
350 meters
Ramo Hall of Music (RHM 106)
Powell Hall (PHE 106)
• Video: NTSC resolution, 31 Mb/s DV, software decode, one-way
latency: 110 ms due to DV camera compression + < 5 ms network
• Audio: uncompressed, 16 or more channels at 1 Mb/s each, one-way
latency: < 10 ms due to audio processing + < 5 ms network
APAN, January 2004
Integrated Media Systems Center, USC
Distributed Immersive
Performance v.1.0-The Duet
APAN, January 2004
Integrated Media Systems Center, USC
HYDRA Streaming Architecture
• Most previous work in streaming media has focused on the
retrieval and playback functionality.
• More and more devices directly output digital media
streams:
– E.g., camcorders (FireWire, USB, SDI),
microphones (Bluetooth), mobile handsets (3G)
• Need for a backend data stream recording /
playback system (“Super TiVo”)
 HYDRA (High-performance Data Recording Architecture)
[ICEIS 2003]
APAN, January 2004
Integrated Media Systems Center, USC
Challenges
• Variable bit rate media streams
• Multi-zoned disks
• Different read and write
transfer rates
APAN, January 2004
Integrated Media Systems Center, USC
Live Streaming
• Latency is a crucial limiting factor:
– Only ~ 20-40 ms is unnoticeable (for
universal interactive applications)
• Tradeoff: Latency versus bandwidth
– Compression reduces bandwidth
– But: high compression increases latency
(e.g., interframe MPEG compression)
• Approach:
– Perform experiments within this design space
e.g. DV: NTSC resolution, 31Mb/s, SW/HW codecs
e.g. uncompressed audio and video
APAN, January 2004
Integrated Media Systems Center, USC
Architecture
HYDRA HD Live Streaming
JVC HD10U
FireWire
MPEG TS
Extractor
HD-SDI
RTP/
UDP/IP
VGA Display
MPEG-2
Decoder
• Acquisition and rendering PC are both Linux
based (RH 9 includes kernel support for FireWire).
• MPEG transport stream extraction.
• Data transport via UDP packets with single
retransmissions
APAN, January 2004
Integrated Media Systems Center, USC
Rendering
• Solution 1: Software based rendering
• Use X11 hw acceleration: XvMC (libmpeg2)
– Motion compensation and iDCT with GPU
• Our hw: NVIDIA FX 5200 ($100)
• Performance: ~ 90 fps @ 1280x720 with 3 GHz P4
APAN, January 2004
Integrated Media Systems Center, USC
Rendering
• Issues with software rendering
– Precise timing: 29.97 fps
– Decoding time for I, P, and B frames varies
– Buffering of decoded frames necessary to
achieve precise timing
– Transport stream splitter and audio decoding
– Video card refresh rate (timing) is
independent of MPEG timing, but
• Non-standard display modes are possible:
720p on Linux (16x9)
– Decoding latency
APAN, January 2004
Integrated Media Systems Center, USC
Rendering
• Solution 2: Hardware based rendering
• E.g.: CineCast HD board from Vela Research
– Digital HD-SDI and analog RGB/YPrPb outputs
• Great and stable picture (but $$$)
• Genlock input for synchronization
APAN, January 2004
Integrated Media Systems Center, USC
Rendering
• Issues with hardware rendering
– Linux drivers hard to come by
– CineCast HD board uses SCSI interface
• Wrote our own SCSI extensions to the Linux
SCSI Generic driver (/dev/sg0)
– Decoding latency: requires 8 x 64 kB to start
decoding
– Consumer HD card:
Telemann HiPix ($400)
But: No Linux drivers
(no Windows filters?)
– New Vela card:
CineCast HD LE
APAN, January 2004
Integrated Media Systems Center, USC
Live HD Video Streaming
(1280x720p)
APAN, January 2004
Integrated Media Systems Center, USC
Distributed Immersive Performance
v.2.0-Extended Architecture
• Conflicting requirements: Low latency and low
bandwidth (i.e., use of compression)
• Solution - two-tier architecture:
• Between performers
– Low latency stereo audio streaming
– Low latency video streaming
• Between performers and audience
– High definition video streaming
– Multichannel audio streaming (10.2 channel)
• Recording of all streams sychronously for archival
purposes and later playback.
APAN, January 2004
Integrated Media Systems Center, USC
Multichannel audio
Stereo audio
Low latency, low resolution video
High latency, high resolution video
Performer 1
Performer 2
Playback and
Recording
Audience
APAN, January 2004
Integrated Media Systems Center, USC
Thank You! Questions?
• More info at:
– Data Management Research Lab
• http://dmrl.usc.edu
– Integrated Media Systems Center
• http://imsc.usc.edu
• Acknowledgments:
– Kun Fu, Beomjoo Seo, Shihua Liu, Dwipal A.
Desai, Didi Shu-Yuen Yao, Mehrdad Jahangiri,
Farnoush Banaei-Kashani, Rishi Sinha, Hong
Zhu, Nitin Nahata, Sahitya Gupta, Vasan N.
Sundar,
APAN, January 2004
Integrated Media Systems Center, USC