Download PowerPoint Designvorlage

Adaptable bandwidth planning for enhanced
QoS support in user-centric broadband
architectures
Dr. Ilka Milouchewa (FHG)
Dirk Hetzer (T-Systems, M&B)
Bandwidth on Demand Planning.
Topics






Background
Bandwidth planning based on reinforcement learning
Operations research for bandwidth scheduling
Combining reinforcement learning with scheduling
 Q-learning
 Informed learning
 Relational learning
QORE system for adaptable bandwidth planning
Integration of bandwidth planning
in research projects and Telecom networks
1
Bandwidth on Demand Planning.
Background (1)

need for planning of resources in new on-demand services
(rich-media, IPTV, VoD, gaming) on all IP core,
replacing ISDN, ATM etc.

high bandwidth demands, efficiency
achievable by bandwidth planning and resource reservation
Scenario: Planning based resource reservation in advance at aggregation points (DSLAM, BB-PoP)
2
Bandwidth on Demand Planning.
Background (2)
Environment state
Feedback
Network access
link
Scenario: Planning is
based on resource
reservation “in advance”
at access routers
considering learning of
performance data
Broadband
Infrastructure
Internet traffic
Traffic
Source / Sink
Network access
link
Network access
link
B
andwidth
Flexible
connectivity
broker
Scare resources
Advance resource reservation requests
Planning of
resource allocation
to applications
Immediate resource reservation
requests
Best effort traffic


Need for capacity planning based on ADVANCE resource reservations for
different kinds of QoS based applications and best effort traffic
Total resources (bandwidth) are always restricted, therefore advance
reservation could be used for planning and enhanced utilization



Learning of performance to predict resource requirements
Optimized resource allocations
adapting advance resource requests to predicted requests
Find compromise between different resource requests
and enhance QoS for all kinds of applications (traffic)
3
Convergence & Triple Play.
Challanging the way of Service Creation.
 Content delivery on demand
Access evolution path:
Triple Play evolution path:
•
•
•
•
•
•
Telephony (F/M)
Others
Radio Access
•
•
•
•
GSM
GPRS
EDGE
3G W-CDMA
HSDPA
Broadband wireless access
+ DSL-/ 3G-Internet
+ VoIP
+ TV
+ VoD
Microwave
Satellite
Laser/Optical
WiMax
UNIFIED USER
EXPERIENCE
Fixed Line Access
•
•
•
•
PSTN/ISDN
FTTP
Cable
Broadband
wireline access
+ Cable Internet
+ Communication
Television
Triple
Play
•h
+ DSL-Internet
+ VoIP
+ IPTV
+ Smart Home
Broadband
Internet
Communications Services evolution path:
Voice
SMS
MMS
Rich Media
Voice + Text
Voice + Text +
Picture + Sound
Voice + Text +
Picture + Sound + Video
4
User-centric approach for bandwidth planning.
Scenarios and applications Planning for Triple Play.
Example 1: TV-based Telecom Services.
 Integrates telephony service with a user’s television
 Supports the delivery of telephony services in
conjunction with cable, DSL, and IP-based video
services
 PSTN, mobile,
or VoIP phones
SIP Servlet
Video
Distribution
Network
Bill Smith
732-699-3232
• TV Calling Name
• User directed routing
• Click-to-dial
• Participation TV
- Voting
- Gaming
- Shopping
• Messaging
PSTN
/VoIP
- Picture Sharing
- Multimedia Message Display
- Content Services
- Voice Mail Screening
5
User-centric approach for bandwidth planning.
Scenarios and applications Planning for Triple Play.
Example 2: TV-Mobile Convergent Participation Service (Blogging).
 Participation TV as a Mobile Communication/
Broadcast Communication convergence case.
 Participation TV denotes the integration of user
feedback/interaction into TV-formats (such as game
shows).
 Traditionally very limited e.g. by calling into the game,
sending SMS, …

MMS diaries (blogging) – uploads from personal pictures,
texts via MMS

Virtual pubs/discos tour (spying on events)

Virtual classroom

Contests (e.g. Best amateur news report of the day/month)

MMS/iTV chat

Other services (personalized weather reports, group
contests, alerts,etc.)
6
Bandwidth on Demand Planning.
Bandwidth planning based on reinforcement learning (1)
Different “learning” approaches to optimize and plan bandwidth
Stochastic automata
- a stochastic policy by associating a
probability with each action, so that
actions are chosen at random according to
their probabilities
- the policy does not take into
consideration the current state of the
system, when choosing an action.
Reinforcement learning benefits
-> Rewards from interactions (action) with environment at each state (dynamical learning)
-> Adaptive control considering states and actions (change of bandwidth scheduling dependent on the
performance “feedback”)
7
Bandwidth on Demand Planning.
Bandwidth planning based on reinforcement learning (2)
Scenarios for usage of reinforcement learning for bandwidth planning
Learning performance
(delay, throughput) and
predict resource needs
-> In case of HDTV
Allocate resource in advance
for different traffic classes
-> Multimedia Conference
-> GRID Transfers
-Resource usage is derived in
interaction with environment
- Using reinforcements,
prediction
is done for the period T
- Optimal sharing of resources in
advance for traffic classes
considering reinforcements
for each traffic
Resource reservation for advance
and immediate resource reservation
requests
Dynamic sharing of resources
Best effort traffic
Adaptation of advance
reservations to resource
needs of traffic classes
-> On-demand service
-Different strategies for advance
allocation of resources for
on-demand traffic considering
predicted and reinforcements
of actual resource usage of
traffic classes
8
Bandwidth on Demand Planning.
Usage of QORE for bandwidth planning (1)
Reservation in advance parameters depend on applications and users
Reservation in advance
Parameter
Options
Example Application
Alternative Bandwidth
Dependent on application and
possibilities for varying QoS levels
Multimedia streaming,
content delivery, file download
Increment
Additional resource demand for
traffic aggregation
VoIP aggregate, content delivery
Duration
Dependent on application usage
GRID, multimedia, mission critical
Cost
Dependent on application users
(cost wanted to pay)
Tele-Radiology, VoIP, video
Time constraints
Flexibility of usage:
- Fixed start,
- Interval based (flexible start,
flexible end)
- Deadline
Distributed multimedia, Grid,
multimedia conferencing
Dependency
Order of allocations to applications
Grid
9
Bandwidth on Demand Planning.
Usage of QORE for bandwidth planning (2)
Problem: Reinforcement learning
based on rewards from environment (delay of best effort traffic)
-->finding optimal bandwidth allocation (optimal schedule)
which satisfies resource requirements in advance
of QoS based applications and enhancing QoS (delay) of best effort traffic
Reinforcement learning problems
for bandwidth planning
-> User interface
for bandwidth allocation in advance
considering traffic classes
-> Reinforcements: periodical
performance measurements for best
effort traffic
-> Value function cumulative rewards
evaluating the schedule for the
planning period
-> Interaction with Bandwidth Broker
Reward
( end-toedn
delay)
planning
Adaptable bandwith
agent interacting with environment
-> optimal resource allocation policies
for advance resource requests using
value functions
Reward
( end-toedn
delay)
Bandwidth broker
Network
Monitoring data base
(QoS, best effort traffic
)
Periodical Performance
measurements
10
Combining reinforcement learning methods and scheduling (1)
Obtaining optimal bandwidth schedules for proactive and reactive
planning
using combination of operation research & reinforcement learning
methods
Pattern based scheduling
Sim
pleQ
-Learning
A
(R) Set of
conflict freeschedules
basedonresource
restrictionsR
-nform
I
edQ
-Learning
Proactive
planning
A
Set of conflict free
pr(R) 
schedulesfor predictedpatterns
basedonresourcerestrictionsR
Relational Q
-Learning
Reactive
planning
A
Set of conflict free
rel (R) 
schedulesfoundinonline
operational netw
orksbasedon
resourcerestrictionsR
Conflict-free schedule
with minimum duration
Partial displacement scheduling
11
Combining reinforcement learning methods and scheduling (2).
Simple Q-Learning - Approach

A model-free RL approach combining conflict-free scheduling with
minimum duration
−

Selection of bandwidth schedule for evaluation
−

Random selection, e-greedy (the best Q-value)
Q-value of bandwidth schedule updated every time
at the end of planning period, for instance a day
−

Bandwidth schedules are characterized with Q-values
based on rewards and value function
(cumulative sum of rewards evaluating
exceeded end-to-end delay threshold for planning periods)
Update based on Learning rate (Recency Weighted Average..)
Pure “trial” and “error”
−
for practical usage not efficient,
because predictions of resource usage of applications
are not considered
12
Combining reinforcement learning methods and scheduling (3) .
Simple Q-Learning - Example
13
Combining reinforcement learning methods and scheduling (9).
Relational Q-Learning - Modified schedule based on patterns
When pattern is detected,
the planned allocation is
rescheduled
Using rewards from endto-end delay measurements
 for daily planning
period
14
Bandwidth on Demand Planning.
Usage of QORE for bandwidth planning (1)
Integration of patterns in scheduling algorithms
based on reinforcement learning
Patterns
- abstractions for structures
describing behaviour of
performance parameters for
network connections
Bandwidth planning focus
End
systems
- extracted from monitoring data
bases
Different kinds of patterns
considered for bandwidth
planning:
- Outliers
- Threshold overload patterns
- Patterns describing traffic,
QoS parameter behavior, routing
events and anomalies of
network connections
Routers
Network connections
QoS
parameter
Traffic
volume
Immediate
resource
request
Routing
events
Failure
events
Patterns describing “normal”and “abnormal” behavior
of multivariate time series data
15
Bandwidth on Demand Planning.
Usage of QORE for bandwidth planning (2)
QORE components
and interaction with monitoring and planning database
QORE: automated tool for adaptable
QoS-oriented proactive and reactive
bandwidth planning
Advance resource specifications
Application
QoSMonitoring
Components using common knowledge
database for monitoring & planning
- User interface for „advance“
reservation
- QoS parameter monitoring
- Scheduling algorithms
Simulator of
resource
constraints
for
connections
Effective bandwidth
estimation
Scenario manager for
bandwidth allocation
Knowledge database for bandwidth scheduling
- Application traffic definitions
- QoSmeasurement scenario and results
- Bandwidth estimation
- Connection resource simulations
- Scenario specification
- Bandwidth schedules and results
- Patterns
Pattern
analyser and
outlier
detection
- Resource constraints simulator
- Effective bandwidth estimation
- Pattern analyser
- Visual data mining for bandwidth
planning
Bandwidth scheduling
User interface
for bandwidth
scheduling
Constrained based
scheduling
algorithms
Visual data mining
for bandwidth
scheduling
16
Scenario for bandwidth planning in converged fixed / mobile
environment
Selection of optimal access network for content delivery
17
Summary.
Automated adaptable bandwidth planning in Internet
based on reinforcement learning,
QoS parameter patterns and scheduling heuristics

Novel management strategies
for an automated proactive and reactive bandwidth planning
in Internet using QoS monitoring data (QORE system)

Integration of data mining technologies
using patterns for bandwidth planning

Bridging a gap in operations research techniques
for bandwidth planning
integrating dynamic learning of QoS parameters (pattern detection)

Integrated architecture based on reinforcement learning,
operations research and data mining for bandwidth planning
18
Bandwidth on Demand Planning.
Outlook & ongoing work in EU projects



NETQOS

Application specific bandwidth planning concepts

Policy based bandwidth management and planning
DAIDALOS

Enhanced QoS brokerage architectures
based on resource reservation in advance

Advance resource reservation for mobile QoS based services
Practical integration of the adaptable bandwidth planning of QORE
in network management systems
19