Download domain-pmwmnc05 - UCSB Computer Science

Choosing an Accurate Network Model
using Domain Analysis
Almudena Konrad, Mills College
Ben Y. Zhao, UC Santa Barbara
Anthony Joseph, UC Berkeley
The First International Workshop on Performance Modelling
in Wired, Wireless, Mobile Networking and Computing
(PMW2MNC, July 2005)
Modeling Network Measurements
• Model-driven traces as experimental tool
– Real measurements costly to perform
– Produce models by extracting statistics from measurements
– Use model to generate traces w/ “similar” characteristics
• New networks (wireless) difficult to model
– Traditional models require stationary statistics
– Gilbert model, Higher order DTMC, Hidden Markov Models (HMM)
– Artifacts disrupt stationarity of wireless traces
• Bursty error or delays due to signal interference or loss
– Result: traditional models generate inaccurate results
• Our solution:
– Data preconditioning: preprocess traces before extracting model
– Result: MTA, MMTA accurately model wireless traces
A New Challenge
• Many mathematical models are available
– Gilbert, Higher order DTMC, HMM, MTA, MMTA…
– Each optimized for certain characteristics
– Each with different computational and memory costs
• Different networks exhibit different characteristics
– IP networks, wireless (802.11), sensor nets
• The challenge
– Match up network type with more accurate, lowest cost model
– No tools or techniques to do this
• Our solution:
– Domain analysis
Outline
• Motivation
• Data preconditioning and its models
– The Markov-based Trace Analysis model (MTA)
– The Multiple states MTA model (MMTA)
• Quantifying modeling accuracy
• Domain Analysis
• Conclusion
Modeling using Data Preconditioning
• Collect network characteristic trace
– Binary traces show occurrence of an event, e.g. lost or delayed
frame
• Identify data patterns or “states” (regions with stationary behavior)
• Precondition the data to fit traditional models
– Calculate probability distribution for each state
– Determine transition probabilities among states
Collected
Trace
Subtrace 1
Subtrace 2
Subtrace 3
Preconditioning Model (MTA)
• MTA: Markov-based Trace Analysis Algorithm
– Two states: lossy & error-free states
– Create two subtraces  lossy & error-free subtraces
– Algorithm to optimize the change-of-state variable, C
• For a given trace, executes stationary test for large range of
values C
• Choose the highest C that yields a stationary lossy states
– Model subtraces as Exponential distributions
States:
Trace:
Error-free sub-trace:
Lossy sub-trace:
Lossy
Error-free
Lossy
Error-free
C
C
…10001110011100….0 0000…0000 11001100…00 00000..000...
...10001110011100….0
11001100…00 ...
… 0000…0000
00000..000...
Preconditioning Model (MMTA)
• MMTA: Multiple states MTA Algorithm
– Allows for multiple states in the data
– Identifies states in original trace
– Concatenates similar states to create subtraces
– Uses a DTMC to model subtraces & transition between states
Network Path Traces
• A selection of end to end & wireless layer path traces
…0000000 10101111 00000000000…
– End-to-end IP traces (IP-1)
– End-to-end WLAN delay trace (WLAN-D)
– Wireless WLAN trace (WLAN-E)
– Wireless GSM trace (GSM-E)
• Trace collected at the Radio Link Protocol (RLP)
• Traces can be decomposed into two stationary states
– Lossy & error-free states
• Characterize traces with three parameters (Lexp, EFexp, Lden)
– Exponential length distributions of lossy and error-free states
– Lossy error density
Quantifying Accuracy of Models
•
Previously used metrics
– FER and simulated transmission time
– No actual correlation to error distribution
•
How accurate is a particular model?
– Synthetic traces comparison (Lexp, EFexp, Lden)
– Compute correlation coefficient between error-burst distributions
Reference trace:
(Lexp, EFexp, Lden)=
(0.006, 0.1, 1)
Relationship between cc
and accuracy:
CC > 0.96 indicates
an accurate model
Example of Model Accuracy:
GSM Error Burst Distributions
• GSM, MTA & MMTA experience similar error burst characteristics
• Gilbert, 3rd order and HMM don’t reproduce large error burst
Cumulative Density Function
1.0
0.9
0.8
0.7
0.6
GSM
Gilbert
MTA
3rd Markov
MMTA
HMM
0.5
CC values
Gilbert: 0.74
HMM : 0.89
MTA : 0.99
MMTA : 0.99
0.4
0.3
0.2
0.1
0.0
0
5
10
15
20
25
30
35
Error Burst Length (Frames)
40
45
50
Choosing the Right Network Model:
Domain Analysis
• Preconditioning models also work well for traditional networks
– There could be a “bursty” nature to congestion loss in IP networks
– Key is the presence of “bursty” behavior
• Need a way to choose optimal model for given network
– Solution: domain analysis
• Domain analysis
– Create Domain Analysis Plots (DAPs) for wide range of parameters
– Collect empirical packet trace: T= …00110110100…
• 1: corrupted/delayed packet, 0: normal packet
– Calculate parameters (Lexp, EFexp, Lden) for T
– Use DAPs to lookup optimal model
Domain Analysis Plots (DAPs)
• Generate reference traces (T1..Tx)
– Range of values (Lexp, EFexp, Lden)
• For each reference trace Ti create
– mathematical models
– artificial traces from models
real network
metric trace
trace analysis
algorithm
network
model
artificial network
metric trace
• Plot error and error-free distribution
• Calculate CC between reference and artificial distributions
• Optimal model for Ti => models that yield highest CC value
DAPs for (Lexp,EFexp) 0.001 -> 0.1
Error-free State Exp
Gilbert
HMM
MMTA
MTA
0.1
0.1
0.08
0.08
0.06
0.06
0.04
0.04
0.02
0.02
0
0
0
0.02
0.04
0.06
Lossy State Exp
0.08
0.1
0
0.02
0.04
0.06
0.08
Lossy State Exp
DAP 1 Lden=0.2
DAP 2 Lden = 0.4
• Gilbert region CC values
(Gilbert = 0.99)
• MMTA = 0.98
• MMTA region CC values
(MMTA =0.97)
• Gilbert = 0.96
• Gilbert region CC values
(Gilbert = 0.99)
• MTA = 0.97
• MTA region CC values (MTA = 0.98)
• MMTA region CC values (M3=0.97)
0.1
DAP for (Lexp,EFexp) 0.001 -> 0.1
Error-free State Exp
0.1
0.08
0.06
0.04
0.02
0
0
0.02
0.04
0.06
0.08
0.1
Lossy State Exp
DAP 3 Lden=0.7
• MMTA region CC values (MMTA=0.98)
• MTA=0.97
•MTA region CC values (MTA=0.99)
• MMTA=0.99
Conclusions
• The challenge
– How to choose the most accurate, least cost model for each
network type
• The solution
– Key metric for modeling accuracy by error burst distributions
– Domain analysis plots show optimal model for different networks
• Using domain analysis
– Take your own traces for network X
– Calculate the three parameter values
– Use DAP to determine optimal model for your trace
Thank you
Questions?
[email protected]