Download A Semantic-based Middleware for Multimedia Collaborative

A Semantic-based Middleware for
Multimedia Collaborative Applications
Agustín J. González
Advisor: Dr. Hussein Abdel-Wahab
Doctoral Dissertation Defense
Old Dominion University
February 2000
Outline
Introduction
Middleware
Objectives
Extension of operating systems network services
Stream synchronization
Floor control framework
Protocol for dynamic image transmission
Experimental results
Conclusions
Questions
2
Introduction
• Large-scale Multimedia Applications
more
* Desktop computer performance increase
* Internet growth in bandwidth and # of hosts
• A challenging class of applications
*
*
*
*
Processing power & bandwidth
Scalability more
Heterogeneity (Ethernet/modem, WinNT/Solaris, MPEG/H263)
Timely data delivery
• Traditional services
* Network layer: UDP & TCP (real time was not a concern)
* Operating systems: Abstractions are not adequate for multimedia.
» Example: Real time is not well supported.
• Gap between multimedia requirements and system
more
services
Outline
3
Multimedia Resource Requirements
Processing
Quality 
Q1
Q2
Q3
Bandwidth
CPU performance
4
Processor
Increas e
ProcessorPerformance
Performance Increase
CPU
µProc
60%/yr.
100
1
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
10
1980
1981
Performance
1000
Time
3
Source: Dr. David Patterson University of Virginia Distinguished Lecture Series,
May 19,1998. http://www.cs.berkeley.edu/~pattrsn/talks/Stanford.pdf
Effect on MM
5
Multimedia Resource Requirements
Processing
Multimedia
Requirements
Quality 
Q1
Q2
Q3
Bandwidth
Bandwidth
6
Multimedia Resource Requirements
Processing
Multimedia
Requirements
Quality 
Q1
Q2
Q3
Bandwidth
High processing + high bandwidth + Others
Introductio
n
7
Internet Growth
56 M
Introductio
n
8
Gap between system services and application
requirements
Developers need to fill this gap by implementing
common services for multimedia applications.
Real-time
Scalability
Heterogeneity
Multimedia applications
Middleware
Trad. Application
Traditional requirements
Trad. Services
outline
9
Objective
“Our main objective is to investigate and propose
heterogeneous, scalable, reliable, flexible, and
reusable solutions and enhancements to common
needs in developing multimedia collaborative
applications.”
Needs we addressed:
* Extension of network services
* Media synchronization
* Floor control
* Data sharing
Outline
10
Extension of Network Services
• New services
* Asynchronous data reception
* Quality of service monitoring
* Transmission traffic rate control
more
more
more
• New convenient facilities
more
* Unified Multicast/Unicast API
* Efficient buffer management for Application Data
more
Unit
Outline
11
Asynchronous data reception
Event-driven model
Network packet arrivals
packet
packet
“application function”
Thread watching
Ext. Net. Srvs
12
Quality of services monitoring
Packet Size
Traffic monitoring
ti-3
7
3,67 8 Bps
9
packet
packet
Window k=3
si
packet
packet
7
2,15 8 Bps
9
ti-2
ti-1
ti
t
Time
i
STTRk (t ) 
s
j  i  k 1
j
t  t i k
“application function”
“application function”
Ext. Net. Srvs
13
Transmission traffic rate control
si
Packet Size
Traffic Limit
2,200 Bps
ti-2
i-1
packet
i
packet
7
2,15 8 Bps
9
ti-1
t`i
ti
Send() call
Time
Actual Tx time
“application function”
Ext. Net. Srvs
14
Unified Multicast/Unicast API




Multicast address 

,
 Unicast address 
port




• Datagram transmission
* A send to a machine or a multicast group does not make a difference.
• Datagram reception
* if the given IP address is a multicast, join group.
* if address is not multicast, do not bind (I’m client).
Ext. Net. Srvs
15
Efficient buffer management for Application
Data Unit
* Goal: to prevent payload movements in memory
* Sender modules create an output buffer that can hold
following “headers” and “tails” .
* Receiver module needs to allocate worst case buffer size.
A
H
A
B
H
Tx
B
A
H
B
A
H
B
Rx
Ext. Net. Srvs
16
Stream Synchronization
• Problem: processing times and network delays are not
deterministic.
• The objective of synchronization is to faithfully
reconstruct the temporal relationship between events
(“pieces of data”).
• Main characteristics of our solution:
* It depends on one-way messages only
» No need of feedback
* It only requires sender’s and receivers’ clock rates to be constant.
» These clocks might be off.
» These clocks might even have different rates of change.
» No need of globally synchronized clocks
* It supports policies to handle late packets and delay adjustments.
Details
17
Stream Synchronization (details)
•
•
•
•
more
Time model
Intra-stream synchronization more
more
Inter-stream synchronization
Clock skew estimation and removal
more
Outline
18
Time Model
Virtual Observer
Networ
k
Sync
19
Intra-stream Synchronization (model)
ci
ci-1
Sender
Perception
ci
ci-1
“Seen” by virtual observer
Timestamping
ti-1
ti
ai-1
Arrival
ai
ai
Receiver
qi
Equalization
qi-1
pi
Playout
pi-1
Synchronization condition:
pi  ci  cte
pi-1
pi
“Seen” by receiver
cte
Virtual delay
Tradeoff:
% late packets
Adaptive compromise
Solution
Total delay
20
Intra-stream Synchronization (solution)
Adjust “virtual delay” to achieve a given % of late packets
Estimator for % of late packets: l   l  1   * 1 for late arrival 


i
i 1
0
otherwise 
NASA MBone 1% late packets
Slow start
1 min !
more
sync
21
Fast start refinement
Less than 5 s !
sync
22
Inter-stream Synchronization
Global synchronization model v/s Differentiated synchronization model
Actual network delay
Global Sync Model
Synchronizes streams
coming from anywhere
with worst case delay
Differentiated
Synchronizes streams
coming from one virtual
observer
Solution
23
Inter-stream Synchronization (solution)
Data
Sync
Virtual delay
Audio
Sync
Inter-stream
coordinator
Video
Sync
Max. virtual delay
sync
24
Clock skew estimation and removal
Goal: Remove differences in clock frequencies
Correction
Before
After
The algorithm adjusts a straight line as new packets arrive
sync
25
Lightweight Framework for Floor Control
•
•
Problem: How to manage exclusive resources in large-scale multimedia
applications?
We recognize two cases:
User
User
Resource
Resource
Communication Channel
1 
n :   : 1
n 
Everywhere Resource
Node (participant)
“Audio”
Localized Resource
Active Resource
Inactive resource
Solution
“Shared tool”
26
Floor Control (Solution)
* We propose two protocols for floor control, one per
architecture.
* Features: lightweight, scalable, robust
Delayed
preemptive
Preemptive
(1) Request
(2) Release
(1) Request
(1) Request
(2) Granted
Coordinator
*
*
(2) Taken
Floor holder
(3) Granted
(3) Taken
or
Released
Participant
(4) Granted
TCP connection
Heartbeat
The coordinator is stationary for localized resources.
The coordinator migrates with floor for everywhere resources
Localized res.
27
Architecture for localized resources
RequesterListener
ResourceUserListener
HolderRefresh
Taken
Granted
Release
Requester Control
Request
Released
Granted
Release
Requester
Request
Released
Withdrew
Policy
RequestNoti
WithdrawalNoti
HolderTimeout
SelectNextHolder
*
Monitor/LogListener
log
Granted
Taken
Release
Heartbeat
Heartbeat
Coordinator
log
NewHolder
*
NewHolderListener
*
getResourceInfo
ResourceInfo
*
Monitor/LogListener
*
Everywhere res.
x
*
Object implementing interface x
Object related with floor architecture
Main floor architecture objects
Optional Object
1-1 reliable remote invocation
1-N unreliable remote invocation
Local invocation
Outline
28
Architecture for everywhere resources
RequesterListener
Request
Released
Requester Control
*
Monitor/LogListener
log
HolderRefresh
Taken
Granted
Granted
Release
Release
ResourceListener
*
*
getResourceInfo
Granted
Taken
Release
Requester
Request
Withdrew
Monitor/LogListener
log
Coordinator
Activate
Request
Withdrew
Released
Heartbeat
Policy
RequestNoti
WithdrawalNoti
HolderTimeout
SelectNextHolder
After Granted
Granted
To all Requesters
Heartbeat
Coordinator
Before Granted
Other Objects
(Same Architecture as above)
1-1 Temporary connection for reliable remote invocation
1-N unreliable remote invocation
Local invocation
Outline
29
Protocol for Dynamic Image
Transmission
• Problem: In addition to audio and video, multimedia
sessions needs a component to convey the main idea of
discussion.
• Traditional solutions:
* Use video (size limitation & high bandwidth)
* Shared tools: XTV, co-browsers, VNC,.. (do not scale well)
• Our solution:
* Video-like protocol tuned to send dynamic images
Solution
30
Protocol for Dynamic Image Transmission
• Sender:
* Temporal redundancy removal
» Sample image at regular period
» Divide image in tiles
» Process only changed tiles
* Spatial redundancy removal
» compress and send changed tiles
• Receiver:
» Receive data unit
» Decompress tile
» Update tile in image
Losses?
31
Overcoming losses
• Each tile is retransmitted after a random time
• This also accommodates late comers
Performance Study
*
*
*
*
*
How to select a tile compression technique? (JPEG, GIF, PNG?)
Is there a “best” tile size? What does it depend on?
How often to sample the image?
How can two tiles be compared efficiently?
Maximum data transmission rate? What does it depend on?
Outline
32
Implementation and Experimental
Results
• Implementation:
* Network support implemented
* Synchronization: implemented and used with real RTP data in off-line
analysis
* Floor control: partially implemented for localized resources
* Image protocol implemented
• Putting everything together: Odust
* A prototypical sharing tool built on top of the middleware.
It uses:
* Network support, floor control, dynamic image protocol, other
application specific modules.
Odust
33
Odust Description
User: Rodrigo
OS: WinNT
User: Cecilia
OS: Solaris
Multicast
Network
User: Agustín
OS: Solaris
User: Eduardo
OS: WinNT
Architecture
34
Odust Description: Cecilia’s view
UNIX
User: Rodrigo
OS: WinNT
User: Cecilia
OS: Solaris
Multicast
Network
Architecture
User: Agustín
OS: Solaris
User: Eduardo
OS: WinNT
35
Odust Description: Rodrigo’s view
WinNT
User: Rodrigo
OS: WinNT
User: Cecilia
OS: Solaris
Multicast
Network
User: Agustín
OS: Solaris
Architecture
User: Eduardo
OS: WinNT
36
Odust Description: Eduardo’s view
User: Rodrigo
OS: WinNT
User: Cecilia
OS: Solaris
Multicast
Network
User: Agustín
OS: Solaris
User: Eduardo
OS: WinNT
WinNT
Architecture
37
Odust Description: Agustín’s view
User: Rodrigo
OS: WinNT
UNIX
User: Cecilia
OS: Solaris
Multicast
Network
User: Agustín
OS: Solaris
User: Eduardo
OS: WinNT
Architecture
38
Odust Architecture
Application
B’s View
Application
A’s View
Application A
JDesktop
WinNT
Java VM
Native
Library
d
Capture and
Dynamic Compound
Image Protocol
Sender
e
Mx
a
Token
Manager
j
b
n
Token
Client
m
l
Event
Injector
c
Application A Sender
Sharing Tool
i
Dx
h
g
k
Dynamic Compound
Image Protocol
Receiver and Display
f
Sender
Multicast
Event
Capture
Application A Receiver
Sharing Tool
Temporary TCP
Receiver
Outline
Method Invocation
39
Conclusion
• We observed the convenience of a middleware
Real-time
Scalability
Heterogeneity
Multimedia applications
Middleware
Trad. Application
Traditional requirements
Trad. Services
• It offers:
*
*
*
*
Multimedia network services
Synchronization
Floor control
Dynamic image transmission
• Future work
* Add more components
* Continue implementation
* Try new ideas (see thesis)
Outline
40