Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
INF SERV – Media Storage and Distribution Systems: User Modeling 13/9 – 2004 Why user modeling? Multimedia approach If you can’t make it, fake it Translation Present real-life quality If not possible, save resources where it is not recognizable Requirement Know content and environment Understand limitations to user perception If these limitations must be violated, know least disturbing saving options INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen User Modelling What? Formalized understanding of users’ awareness user behaviour Why? Achieve the best price/performance ratio Understand actual resource needs achieve higher compression using lossy compression potential of trading resources against each other potential of resource sharing relax relation between media INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Applications of User Modelling Encoding Formats Exploit limited awareness of users JPEG/MPEG video and image compression MP3 audio compression Based on medical and psychological models Quality Adaptation Adapt to changing resource availability no models - need experiments Synchronity Exploit limited awareness of users no models - need experiments Access Patterns When will users access a content? Which content will users access? How will they interact with the content? no models, insufficient experiments - need information from related sources INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen User Perception of Quality Changes Quality Changes Quality of a single stream Issue in Video-on-Demand, Music-on Demand, ... Not quality of an entire multimedia application Quality Changes Usually due to changes in resource availability overloaded server congested network overloaded client INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Kinds of Quality Changes Long-term change in resource availability Random Planned Short-term change in resource availability Random Planned INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Kinds of Quality Changes Long-term change in resource availability Random no back channel no content adaptivity continuous severe disruption Planned Short-term change in resource availability Random Planned INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Kinds of Quality Changes Long-term change in resource availability Random no back channel no content adaptivity continuous severe disruption Planned change to another encoding format change to another quality level requires mainly codec work Short-term change in resource availability Random Planned INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Kinds of Quality Changes Long-term change in resource availability Random Planned Short-term change in resource availability Random Planned INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Kinds of Quality Changes Long-term change in resource availability Random Planned Short-term change in resource availability Random packet loss frame drop alleviated by protocols and codecs Planned INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Kinds of Quality Changes Long-term change in resource availability Random Planned Short-term change in resource availability Random packet loss frame drop alleviated by protocols and codecs Planned scaling of data streams appropriate choices require user model INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Kinds of Quality Changes Long-term change in resource availability Random no back channel no content adaptivity continuous severe disruption Planned change to another encoding format change to another quality level requires mainly codec work Short-term change in resource availability Random packet loss frame drop alleviated by protocols and codecs Planned scaling of data streams appropriate choices require user model INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Planned quality changes Audio Lots of research in scalable audio No specific results for distribution systems Rule-of-thumb Always degrade video before audio Video Long-term changes Short-term changes INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Planned quality changes Audio Video Long-term changes Use separately encoded streams Switch between formats Non-scalable formats compress better than scalable ones (Source: Yuriy Reznik, RealNetworks) Short-term changes Switching between formats Needs no user modeling Is an architecture issue INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Planned quality changes Audio Video Long-term changes Short-term changes Use scalable encoding Reduce short-term fluctuation by prefetching and buffering Two kinds of scalable encoding schemes Non-hierarchical encodings are more error-resilient o fractal single image encoding Hierarchial encodings have better compression ratios Scalable encoding Support for prefetching and buffering is an architecture issue Choice of prefetched and buffered data is not INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Planned quality changes Audio Video Long-term changes Short-term changes Use scalable encoding Reduce short-term fluctuation by prefetching and buffering Short-term fluctuations Characterized by frequent quality changes small prefetching and buffering overhead Supposed to be very disruptive See for yourself: subjective assessment INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Subjective Assessment A test performed by the Multimedia Communications Group at TU Darmstadt Goal Predict the most appropriate way to change quality Approach Create artificial drop in layered video sequences Show pairs of video sequences to testers Ask which sequence is more acceptable Compare two means of prediction Peak signal-to-noise ratio (higher is better) compares degraded and original sequences per-frame ignores order Spectrum of layer changes (lower is better) takes number of layer changes into account ignores content and order INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Subjective Assessment INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Subjective Assessment layers layers Used SPEG (OGI) as layer encoded video format frames frames layers layers amplitude of layer variation frames frames frequency of layer variation INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Subjective Assessment layers layers What is better? frames frames layers layers First gap first or lowest gap first? frames frames Early or late high quality? INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Subjective Assessment How does the spectrum correspond with the results of the subjective assessment? Comparison with the peak signal-to-noise ratio # Metric ts 1 1 Subjective assessment 2 PSNR (higher is better) 3 Spectrum (lower is better) # Farm 1 Clip 1 Subjective assessment 2 PSNR (higher is better) 3 Spectrum (lower is better) ts 1 0.35 M&C1 ts 2 ts 1 0.55 ts 2 0.73 62.86 49.47 61.46 73.28 63.15 52.38 2 Clip Metric ts 2 Farm 2 2 6.86 M&C3 ts 1 4 2 M&C4 ts 2 ts 1 1.18 1 T-Tennis3 ts 2 1.02 ts 1 ts 2 2.18 48.01 25.08 49.40 26.95 66.02 63.28 2 0 2 0 0.5 0.5 According to the results of the subjective assessment the spectrum is a more suitable measure than the PSNR INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Subjective Assessment Conclusions Subjective assessment of variations in layer encoded videos Comparison of spectrum measure vs. PSNR measure Observing spectrum changes is easier to implement Spectrum changes indicate user perception better than PSNR Spectrum changes do not capture all situations Missing Subjective assessment of longer sequences Better heuristics "thickness" of layers order to quality changes target layer of changes INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen User Model for Synchronity Synchronization Content Relation se.g.: several views of the same data Spatial Relations Layout Temporal Relations Intra-object Synchronization Intra-object synchronization defines the time relation between various presentation units of one time-dependent media object Inter-object Synchronization Inter-object synchronization defines the synchronization between media objects Relevance Hardly relevant in current NVoD systems Somewhat relevant in conferencing systems Relevant in upcoming multi-object formats: MPEG-4, Quicktime INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Inter-object Synchronization Lip synchronization demands for a tight coupling of audio and video streams with a limited skew between the two media streams Slide show with audio comment Main problem of the user model permissible skew INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Inter-object Synchronization A lip synchronized audio video sequence (Audio1 and Video) is followed by a replay of a recorded user interaction (RI), a slide sequence (P1 - P3) and an animation (Animation) which is partially commented using an audio sequence (Audio2). Starting the animation presentation, a multiple choice question is presented to the user (Interaction). If the user has made a selection, a final picture (P4) is shown Main problem of the user model permissible latency analysing object sequence allow prefetching user interaction complicates prefetching INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Synchronization Requirements – Fundamentals 100% accuracy is not required, i.e., skew is allowed Skew depends on Media Applications Difference between Detection of skew Annoyance of skew Explicit knowledge on skew Alleviates implementation Allows for portability INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Experimental Set-Up Experiments at IBM ENC Heidelberg to quantify synchronization requirements for Audio/video synchronization Audio/pointer synchronization Selection of material Duration 30s in experiments 5s would have been sufficient Reuse of same material for all tests Introduction of artificial skew By media composition with professional video equipment With frame based granularity Experiments Large set of test candidates Professional: cutter at TV studios Casual: every day “user” Awareness of the synchronization issues Set of tests with different skews lasted 45 min INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Lip Synchronization: Major Influencing Factors Video Content Continuous (talking head) vs. discrete events (hammer and nails) Background (no distraction) Resolution and quality View mode (head view, shoulder view, body view) Audio Content Background noise or music Language and articulation INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Lip Synchronization: Level of Detection Areas In sync QoS: +/- 80 ms Transient Out of sync INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Lip Synch.: Level of Accuracy/Annoyance Some observations Asymmetry Additional tests with long movie +/- 80 ms: no distraction -240 ms, +160 ms: disturbing INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Pointer Synchronization Fundamental CSCW shared workspace issue Analysis of CSCW scenarios Discrete pointer movement (e.g. “technical sketch”) Continuous pointer movements (e.g. “route on map”) Most challenging probes Short audio Fast pointer movement INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Pointer Synchronization: Level of Detection Observations Difficult to detect “out of sync” i.e., other magnitude than lip sync Asymmetry According to every day experience INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Pointer Synchronization: Level of Annoyance Areas In sync: QoS -500 ms, +750 ms Transient Out of sync INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Quality of Service of Two Related Media Objects Expressed by a quality of service value for the skew Acceptable skew within the involved data streams Affordable synchronization boundaries Production level synchronization Data should be captured and recorded with no skew at all To be used if synchronized data will be further processed Presentation level synchronization Reasonable synchronization at the user interface To be used if synchronized data will not be further processed INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Quality of Service of Two Related Media Objects Media Video Mode, application QoS Animation Correlated +/- 120 ms Audio Lip synchronization +/- 80 ms Images Overlay +/- 240 ms No overlay +/- 500 ms Overlay +/- 240 ms No overlay +/- 500 ms Text INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Quality of Service of Two Related Media Objects Media Audio Mode, application QoS Animation Event colleration +/- 80 ms Audio Tightly coupled (stereo) +/- 11 μs Loosely coupled (dialog mode with various participants) +/- 120 ms Loosely coupled (background music) +/- 500 ms Tightly coupled (music with notes) +/- 5 ms Loosely coupled (slide show) +/- 500 ms Text Text annotation +/- 240 ms Pointer Audio related to shown item -500 - +750 ms Image INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen User Model for Access Patterns Modelling User behaviour The basis for simulation and emulation In turn allows performance tests Separation into Frequency of using the VoD system Selection of a movie User Interaction Models exist But are not verified Selection of a movie Dominated by the access probability Should be simulated by realistic access patterns INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Focus on Video-on-Demand Video-on-demand systems Objects are generally consumed from start to end Repeated consumption is rare Objects are read-only Hierarchical distribution system is the rule Caching approach Simple approach first Various existing algorithms Simulation approach No real-world systems exist Similar real-world situations can be adopted INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Using Existing Models Use of existing access models ? Some access models exist Most are used to investigate single server or cluster behavior Real-world data is necessary to verify existing models Optimistic model Cache hit probabilities are over-estimated Caches are under-dimensioned Network traffic is higher than expected Pessimistic model Cache hit probabilities are under-estimated Cache servers are too large or not used at all Networks are overly large INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Existing Data Sources for Video-on-Demand Movie magazines Data about average user behaviour Represents large user populations Small number of observation points (weekly) Movie rental shops Actual rental operations Serves only a small user population Initial peaks may be clipped Cinemas Actual viewing operations Serves only a small user population Few number of titles Short observation periods INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Model for Large User Populations Zipf Distribution N C z (i) , C 1 / 1 / j i j 1 Verified for VoD by A. Chervenak N - overall number of movies ξ – skew factor i - movie i in a list ordered by descreasing popularities z(i) - hit probability Many application contexts all kinds of product popularity investigations http://linkage.rockefeller.edu/wli/zipf/ collects applications of Zipf’s law natural languages, monkey-typing texts, web access statistics, Internet traffic, bibliometrics, informetrics, scientometrics, library science, finance, business, ecological systems, ... INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Verification: Movie Magazine Movie magazine Characteristics of observations on large user populations Smoothness Predictability of trends Sharp increase and slower decrease in popularities Highlander 3 Highlander 3 0 20 60000 top 100 ranking media control index 80000 40000 20000 40 60 80 0 100 0 5 10 15 weeks 20 INF5070 – media servers and distribution systems 25 0 5 10 15 weeks 20 25 2004 Carsten Griwodz & Pål Halvorsen Comparison with the Zipf Distribution probability curves for 250 movie titles 1 rental probability 0.9 0.8 0.7 0.6 4/3/96 0.5 z(i) 0.4 0.3 4/6/96 0.2 0.1 0 0 20 40 60 movie index 80 100 Well-known and accepted model Easily computable Compatible with the 90:10 rule-of-thumb INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Verification: Small and Large User Populations INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Verification: Small and Large User Populations Similarities Small populations follow the general trends Computing averages makes the trends better visible Time-scale of popularity changes is identical No decrease to a zero average popularity Differences Large differences in total numbers Large day-to-day fluctuations in the small populations Typical assumptions 90:10 rule Zipf distribution models real hit probability INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Problems of Zipf Does not work in distribution hierarchies Access to independent caches beyond first-level are not described Not easily extended to model day-to-day changes Is timeless Describes a snapshot situation Optimistic for the popularity of most popular titles Chris Hillman, bionet.info-theory, 1995 Any power law distribution for the frequency with which various combinations of ‘‘letters’’ appear in a sequence is due simply to a very general statistical phenomenon, and certainly does not indicate some deep underlying process or language. Rather, it says you probably aren’t looking at your problem the right way! INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Approaches to Long-term Development Model variations for long-term studies Static approach CD sales model Smooth curve with a single peak Models the increase and decrease in popularity Shifted Zipf distribution No long-term changes Movie are assumed to be distributed in off-peak hours Zipf distribution models the daily distribution Shift simulates daily shift of popularities Permutated Zipf distribution Zipf distribution models the daily distribution Permutation simulates daily shift of popularities INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Verification: Zipf Variations popularity index change relevance change 100 80 relevance change of a real movie 100 80 60 40 20 0 0 50 100 150 200 250 age in days 60 40 20 0 0 50 100 150 200 age in days 250 popularity index change popularity index change Rotation model for day-to-day relevance changes relevance change of a real movie 100 80 60 40 20 0 0 50 100 150 200 250 age in days INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Verification: Zipf Variations popularity index change relevance change 100 80 relevance change of a real movie 100 80 60 40 20 0 0 50 60 100 150 200 250 age in days 40 20 0 0 50 100 150 200 age in days 250 popularity index change popularity index change Permutation model for day-to-day relevance changes relevance change of a real movie 100 80 60 40 20 0 0 50 100 150 200 250 age in days INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Modelling: Requirements Model should represent movie life cycles To To To To reflect the aging of titles observe movement of movies through a hierarchy of servers make observations with respect to a single movie support the idea of pre-distribution Model should work for large and small user populations To allow variations in client numbers To prevent from built-in smoothing effects Model can not be trace-driven The number of movies is too small The observation time is too short The user population size is not variable One title can not be re-used without similarity effects INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen New Model: Movie Life Cycle Characteristics Quick popularity increase Various top popularities Various speeds in popularity decrease Various residual popularity INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen New Model: User Population Size 50 draws per day 500 draws per day 1.5 movie hits movie hits 2 1 0.5 0 0 50 100 150 200 250 9 8 7 6 5 4 3 2 1 0 0 50 100 200 250 days 50000 draws per day 70 700 60 600 movie hits movie hits days 5000 draws per day 150 50 40 30 20 10 500 400 300 200 100 0 0 0 50 100 150 days 200 250 0 50 100 150 200 250 days Smoothing effect of larger user populations Day-to-day relevance changes Probability distribution of all movies by „new releases“ INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Problems with Data Sources Lack of additional real-world data No verification data for medium-sized populations available Missing details Genres Single day probability variations Children´s choices at daytime, adults´ choices at night Regional popularity differences Popularity rise and decline depends on genres Single users´ behaviour can be predicted Ethnic groups Regional information Comebacks Sequels inspire comebacks Detail overload Simplifications are required for large simulations INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Video Access Modeling Simple Zipf models are not suited for simulation of server hierarchies Trace-driven simulation can not be used Our model is sufficient for general investigation on caching Long-term movie life cycles can be modeled nicely Optimistic assumptions due to smoothness are removed Variations in movie behavior are supported Day-to-day popularity changes are realistic It is not sufficient yet for advanced caching mechanisms Single-day variations are missing Genres are missing INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen Summary User modeling helps achieving a good price/performance ratio for multimedia systems User modeling allows cheating Examples seen: Modeling quality assessment of layered video Modeling audio/video synchronization Modeling video access probability INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen References Ann Chervenak: Tertiary Storage: An Evaluation of New Applications, PhD thesis, University of California, Berkeley, 1994 Carsten Griwodz, Michael Bär, Lars Wolf: Long-Movie Popularity Models in Video-on-Demand Systems, ACM Multimedia, Seattle, WA, USA, Nov. 1997 Charles Krasic, Jonathan Walpole: Priority-Progress Streaming for Quality-Adaptive Multimedia, ACM Multimedia Doctoral Symposium, Ottawa, Canada, Oct. 2001 Ralf Steinmetz, Klara Nahrstedt: Multimedia Fundamentals, Volume I: Media Coding and Content Processing (2nd Edition), Prentice Hall, 2002, ISBN 0130313998 Michael Zink, Oliver Künzel, Jens Schmitt, Ralf Steinmetz: Subjective Impression of Variations in LayerEncoded Videos, IWQoS, Monterey, CA, USA, Jun. 2003 Michael Zink, Jens Schmitt, and Carsten Griwodz. Layer-Encoded Video Streaming: A Proxy's Perspective. In IEEE Communications Magazine, Vol. 42, No. 8, August 2004 INF5070 – media servers and distribution systems 2004 Carsten Griwodz & Pål Halvorsen