Download ppt

CS7701: Research Seminar on Networking
http://arl.wustl.edu/~jst/cse/770/
Review of:
Operating System Support for
Planetary-Scale Network Services
• Paper by:
– Andy Bavier, Scott Karlin, Steve Muir, Larry Peterson, Tammo Spalink, Mike
Wawrzoniak (Princeton)
– Mic Bowman, Brent Chun, Timothy Roscoe (Intel)
– David Culler (Berkeley)
• Published in:
– First Symposium on Network Systems Design and Implementation (NSDI),
2004
• Additional presentation information:
– www.planet-lab.org
• Reviewed by:
– Chip Kastner
• Discussion Leader:
– Manfred Georg
1 - CS7701 – Fall 2004
Outline
• Introduction and Virtualization
• The Operating System
– Motivation and overview
– Virtualization
– Isolation / Resource Management
– Evolution
• Evaluation
• Conclusions
2 - CS7701 – Fall 2004
Introduction
• This paper is more or less an examination of
PlanetLab’s operating system
• What is PlanetLab?
– 441 machines at 201 sites in 25 countries
– Supporting 450 research projects
– Each machine runs a Linux-based OS and other tools
• Why does it exist?
– Mainly a testbed for planetary-scale network services
– Lets researchers test services in real-world conditions
at a large scale
– Also a deployment platform
• Who uses it?
– AT&T labs, HP labs, Intel
– Many prestigious universities – including Wash U
3 - CS7701 – Fall 2004
Introduction
• A PlanetLab user sees a private set of
machines (nodes) on which he can run his
applications
User 1
4 - CS7701 – Fall 2004
User 2
Introduction
• Reality is a bit different
• PlanetLab is an overlay network
Intel Labs
Interne
t
Princeton
5 - CS7701 – Fall 2004
Wash U
Tel-Aviv
University
Introduction
• Users are granted slices – Sets of nodes on
which users get a portion of the resources
Intel Labs
Interne
t
Princeton
6 - CS7701 – Fall 2004
Wash U
Tel-Aviv
University
Introduction
• So, a PlanetLab user is managing a set of
distributed Virtual Machines (VMs)
User 1
7 - CS7701 – Fall 2004
User 2
The Operating System
• Operating system design was designed
around two high-level goals
– Distributed virtualization: Each service runs in
an isolated section of PlanetLab’s resources
– Unbundled management: The OS is separate
from services that support the infrastructure
8 - CS7701 – Fall 2004
The Operating System
• Researchers wanted to use PlanetLab as
soon as the first machines were set up
• No time to build a new OS
• PlanetLab designers chose to deploy Linux
on the machines
– Linux acts as a Virtual Machine Monitor
– Designers slowly transform Linux via kernel
extensions
9 - CS7701 – Fall 2004
The Operating System
• A node manager runs on top of Linux
– A root VM that manages other VMs on a node
– Services create new VMs by calling the node
manager
– Services can directly call the only the local
node manager
• The node manager is hard-coded
– Local control can be added
• Infrastructure services can be given
privileges to perform special tasks
10 - CS7701 – Fall 2004
The Operating System
Unprivileged Slices
P2P Networks,
Embedded network
storage
Privileged Slices
Slice creation,
monitoring,
environment service
Node Manager
Resource allocation,
Auditing, Bootstrapping
Linux w/ kernel extensions
Virtual Machine
Monitor
11 - CS7701 – Fall 2004
The Operating System
• PlanetLab Central (PLC) is a service
responsible for creating slices
– It maintains a database of slice information on
a central server
– Users request slices through the PLC
• The PLC communicates with a resource
manager on each node to start up slices
and bind them to resources
12 - CS7701 – Fall 2004
Virtualization
• Virtualization can be done at several levels
• Virtualization of hardware would allow each
VM to run its own OS
– This is infeasible due to performance/space
constraints
• PlanetLab uses system-call level
virtualization
– Each VM sees itself as having exclusive access
to an OS
– All VMs on a node are actually making system
calls to the same OS
13 - CS7701 – Fall 2004
Virtualization
• How is virtualization maintained?
– The OS schedules clock cycles, bandwidth,
memory, and storage for VMs, and provide
performance guarantees
– It must separate name spaces – such as
network addresses and file names – so VMs
can’t access each other’s resources
– It must provide a stable base so that VMs can’t
affect each other (no root access)
14 - CS7701 – Fall 2004
Virtualization
• Virtualization is handled by a Linux utility
called vserver
• Vservers are given their own file system
and a root account that can customize that
file system
• Hardware resources, including network
addresses, are shared on a node
• Vservers are sufficiently isolated from one
another
• VMs are implemented as vservers
15 - CS7701 – Fall 2004
Isolation
• Buggy/malfunctioning or even malicious
software might be run on PlanetLab
• This can cause problems at many levels
Yahoo
Wash U
Google
Intel Labs
Princeton
Interne
t
SlashDot
Tel-Aviv U
AOL
16 - CS7701 – Fall 2004
Isolation
• The problem of isolation is solved in two ways
– Resource usage must be tracked and limited
– Resource usage must be able to be audited
later to see what actions were performed by
slices
17 - CS7701 – Fall 2004
Resource Management
• Non-Renewable Resources are monitored
and controlled
– System calls are wrapped and intercepted
– The OS keeps track of resources allocated to
VMs
– Resource requests are accepted or denied
based on a VM’s current resource usage
18 - CS7701 – Fall 2004
Resource Management
• Renewable Resources can be guaranteed
– A VM that requires a certain amount of a
renewable resource will receive it
– A fair best-effort method allocates resources to
remaining VMs
• If there are N VMs on a node, each VM receives at
least 1 / N of the available resources
19 - CS7701 – Fall 2004
Resource Management
• Scheduling & Management are performed
by Scout on Linux Kernel (SILK)
– Linux CPU scheduler cannot provide fairness
and guarantees between vservers
• Uses a Proportional Sharing method
– Each vserver receives a number of CPU shares
– Each share is a guarantee for a portion of the
CPU time
20 - CS7701 – Fall 2004
Auditing
• The PlanetLab OS provides “safe” sockets
– Each socket must be specified as TCP or UDP
– Each socket must be bound to a port
– Only one socket can send on each port
– Outgoing packets are filtered to ensure this
• Privileged slices can sniff packets sent
from each VM
– Anomalies can be caught before they become
disruptive
21 - CS7701 – Fall 2004
Evolution
• PlanetLab can be easily extended
• Researchers can create their own
infrastructure services for PlanetLab
– Services that help maintain PlanetLab itself
– Slice creation, performance monitoring,
software distribution, etc.
• It is possible for multiple researchers to
develop similar services in parallel
– A rather unique system evolution problem
22 - CS7701 – Fall 2004
Evolution
• What does this mean for PlanetLab’s OS?
– It must be responsible only for the basic tasks
– It must be fair in granting privileges to
infrastructure services
– It must provide an interface for creating a VM
23 - CS7701 – Fall 2004
Evolution
Unprivileged Slices
Privileged Slices
Node Manager
Linux w/ kernel extensions
24 - CS7701 – Fall 2004
• New services
should be
implemented at
the highest level
possible
• Services should
be given only the
privileges
necessary to
perform their
task
Evaluation
• Scalability
– A VM’s file system requires 29 MB
– 1,000 VMs have been created on a single node
• Slice creation
– Nodes look for new slice info every 10 minutes
– Creating a new VM from scratch takes over a
minute
• Service initialization
– Upgrading a package takes 25.9 sec on a
single node
– Slower when many nodes update a package at
once
25 - CS7701 – Fall 2004
Conclusion
• PlanetLab is an excellent option for
researchers to test new network services
• Virtualization provides an easy interface
• Services are reasonably well-protected
from one another
• Opportunities exist to expand PlanetLab’s
functionality
26 - CS7701 – Fall 2004