Download Cloud Project – Calibration

Academic Compute Cloud Provisioning
and Usage
AAA Project
Peter Kunszt
ETH/SystemsX.ch
2012, October 23
Project Goals
• How to extend current cluster services using cloud
technology?
• Support new application models (MapReduce, specialized
servers).
• Test real applications.
• Understand performance implications.
1. Define Service Models: How to move to cloud-like service
orientation models.
2. Define Business Models: How to accommodate pay-peruse, OpEx vs. CapEx, how to plan an academic private
cloud, and how to use and offer public clouds
3. Run real applications: Run a regular, a compute-intensive
and a data-intensive application on the cloud.
Project Goals
• How to extend current cluster services using cloud
Provide input to the mid- and longtechnology?
• Support
newstrategy
applicationfor
models
(MapReduce,
specialized
term
cluster
and
cloud
servers).
at ETH and UZH.
• Test realinfrastructure
applications.
• Understand performance implications.
1. Define Service Models: How to move to cloud-like service
orientation models.
Disseminate
results
Switzerland
2. Define
Business Models:
How toin
accommodate
pay-peruse,
OpEx vs.in
CapEx,
how to plan
an academic
private
broadly
academia
and
to
interested
cloud, and how to use and offer public clouds
(Workshop
at project
end)
3. Runparties
real applications:
Run a regular,
a compute-intensive
and a data-intensive application on the cloud.
Project Organization
SWITCH
AAA
Project Lead:
Peter Kunszt
UZH
Sergio Maffioletti
Riccardo Murri
Christian Panse
Tyanko Alekseyev
Antonio Messina
ETHZ
Olivier Byrde
+Brutus team members
Sandro Mathys
Software
Peter Kunszt
SyBIT team, FGCZ
Malmström group
Guido Capitani
others as needed
Business Model
Dean Flanders
Markus Eurich
Consultants
Project Organization
ETH Project Steering Board
Reto Gutmann
Olivier Byrde
Dordaneh Arangeh
SWITCH
AAA
Peter Kunszt
Dean Flanders
Michelle Binswanger
Project Lead:
Peter Kunszt
UZH
Sergio Maffioletti
Riccardo Murri
Christian Panse
Tyanko Alekseyev
Antonio Messina
ETHZ
Olivier Byrde
+Brutus team members
Sandro Mathys
Software
Peter Kunszt
SyBIT team, FGCZ
Malmström group
Guido Capitani
others as needed
Business Model
Dean Flanders
Markus Eurich
Consultants
Motivation
• Today : World of Products
– Hardware, Software to be bought as products
– Users buy, set up, install, configure and use
• Evolving into: World of Services
– Software and Services bought directly as Apps
– Users make use what they need immediately
Users will buy more services in the future, not just
products. These services will be often times in the cloud.
We too want to offer services, not just products.
DEFINITION
Cloud Attributes:
When do we talk about a cloud
• Self-service, On-demand, Cost transparency
– Access to immediately available resources, paying
for usage only. No long-term commitments. No
up-front investments needed. Operational
expenses only.
• Elasticity, Multi-tenancy, Scalability
– Grow and shrink size of resource on request.
Sharing with other users without impacting each
other. Economies of scale.
7
Definitions
• Self-service: A consumer can unilaterally provision
computing capabilities, such as server time and network
storage, without requiring human interaction.
• On-demand: As needed, at the time when needed,
automatic provisioning.
• Cost Transparency: Accounting of actual usage
transparent to user and service provider both,
measured in corresponding terms (Hours CPU time, GB
per Month, MB Transfer, etc)
Definitions
• Elastic: Capabilities can be elastically provisioned and
released, in some cases automatically, to scale
rapidly outward and inward commensurate with
demand.
• Multi-tenant: The provider’s computing resources
are pooled to serve multiple consumers, with
resources dynamically assigned and reassigned
according to consumer demand.
• Scalable: To the consumer, the capabilities available
for provisioning often appear to be unlimited and
can be appropriated in any quantity at any time.
http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
HPC Pyramid
Computing needs
CSCS
Local Cluster
(e.g. ETH Brutus)
Servers / Mini-clusters
Laptop, Desktop, iPad
Number of users
Relation to Cloud: As User (extension)
Computing needs
CSCS
Local Cluster
(e.g. ETH Brutus)
Cloud
Use
Servers / Mini-clusters
Laptop, Desktop, iPad
Number of users
Burst
Today, University Clusters do not make
use of the Cloud:
• Technical details to be investigated:
– Bursting the cluster into the cloud
• Networking?
• User Management?
• File System?
• Cloud-compatible licenses for commercial
products are often not available
• No billing mechanism to bill users of cluster
for pay-per-use services
Relation to Cloud: As Provider
Computing needs
CSCS
Account / charge usage
Local Cluster
(e.g. ETH Brutus)
Servers / Mini-clusters
Laptop, Desktop, iPad
Number of users
Cloud
Expose to
Not clear how to be a Cloud
Provider with a University Cluster
•
•
•
•
•
•
Univ. cluster is not self-service
Capital expenses, not just pay-per-use
Long-term commitment
Not extensible on-demand, not elastic
Sharing with others only according to policies
More stringent terms of use, needs account
• We have examples to look at:
– SDSC, Cornell, Oslo
Infrastructure and Platform as a
Service
Classic Approach Today
IaaS
.
PaaS
From www.cloudadoption.org
95%
time savings
SaaS
Infrastructure
START
Platform
Software
FINISH
Software & Apps run on
platforms,
NOT
infrastructure
www.cloudadoption.org
Cloud Stack
Users or Portals. Can directly use each stack.
CLIENTS
User
Interface
Machine
Interface
Software
Components
Services
Platform
Compute
DEFINITION
Storage
Network
Infrastructure
HARDWARE
SaaS = Software as a Service
• Scientific / office / business / etc. Software as a
service. Interactive or programmable.
PaaS = Platform as a Service
• Programming and deployment frameworks.
Integrated programmable high-level services for
composition.
IaaS = Infrastructure as a Service
• Virtual or hosted hardware: for HPC, compute,
storage, network, specialized servers (memory,
GPU, DB)
Any kind of infrastructure for any of the stacks.
Who can makes use of what
User Portal
SaaS
PaaS
IaaS
Hardware
• Users may use any
service
• Portals may use any
service
• SaaS may or may not be
built on top of PaaS or
IaaS
• PaaS may or may not be
built on top of IaaS
DEFINITION
Public, Private, Hybrid Clouds
Private Cloud
• Own infrastructure
only
• In-house or hosted
• Internal use or for
sale
Hybrid Cloud
Connect
• Offered by partner
organizations or
cloud providers
• Private Cloud connected
to Public Cloud
• Remote cloud resources
on-demand
• Only operational
expenses
Institutional boundary
• Constraints on own
• Full control on cloud
cloud stack: needs to
stack, accounting, etc
interoperate with public
cloud
Public Cloud
• No control on cloud
stack, dependency on
external partner
Goal: Understand the relationships..
• ..in terms of virtual Servers
• ..in terms of Storage
• ..in terms of Networking
Goal: How to evolve the HPC Service..
• ..to be able to offer a Platform as a Service.
• ..to be able to make use of public clouds
seamlessly (Hybrid model, CloudBursting)
Goal: New Software Services
Goal: New Business Models
• Cannot charge at full cost if we want to be the
service provider (competitive advantage)
• Internal and external views
• Efficient, fair, feasible and generally accepted
funding and charging model
• New opportunities should not require to change
existing business procedures for existing
infrastructure (evolution not revolution)
• Transparent Financial Accounting mechanism
Status: Information and Survey
• We collected a lot of information and
conducted a survey on existing solutions
• Choices (we need to limit ourselves):
– OpenStack
– VMWare
– HP Matrix
Cloud Stack Comparison Matrix
OpenStack Distribution comparison
Public IaaS Comparison
Status: Infrastructure 1
• ETH: HP CloudSystem Matrix Testbed
– Operational as of THIS WEEK
• 8 Intel, 8 AMD blades
• 128GB memory per blade
• 10TB storage 3PAR
• HP Matrix cloud software is fixed
• This is on RENT we have to give it back
Status: Infrastructure 2
• ETH: Build our own from new components.
– Standard cluster nodes x16, diskless
– 128GB RAM on each node
– Very fast storage (SSD based) for VM images
• Attach standard storage NAS from ETH
• Cloud Stack:
– OpenStack
– VMWare
• Will be here in 2 weeks
• This remains at ETH after the project
Status: Infrastructure 3
• University of Zurich: Recycle existing
components.
– Set of old cluster nodes, heterogeneous
– Cloud filesystem using local node storage
(technologies will be evaluated)
• GlusterFS
• Ceph
Status: Software
• Use Cases are defined and chosen
• MapReduce (Hadoop)
– Existing sw deployment: Crossbow (genomics)
– New development (proteomics)
• Compute intensive
– GAMESS
– Rosetta
• Data intensive
– HCS (High Content Screening) – image analysis
• Servers
– Matlab, R, CLC Bio, etc servers
Status: Business Model
• Several models are being worked out
– Shareholder model – one-time fee for TFLOPS or TB
– Subscription model – yearly fee
– Pay-per-use model
• Self service options
– Very detailed like Amazon
– High-level ‘virtual cluster’ or PaaS
– Top-level SaaS user gateways
Lots of Interactions
• With Cloud providers
– IBM, Amazon, CloudSigma, HP, Google
• Software providers
– VMWare, HP, Dell, OpenStack flavors (Piston, ..)
• Universities
– SWITCH, ZHAW, SDSC, Cornell, Imperial College, U
Oslo, Zaragoza
Next Steps
• Cloud Bursting of Cluster into our own cloud
and into Amazon (reproducing VM-MAD)
– Startup and teardown times
– Management tests
– Performance
• Use Cases are set up on the infrastructure
SDCD 2012: Supporting Science
with Cloud Computing
• November 19 2012, University of Bern,
http://www.swing-grid.ch/sdcd2012/
•
•
•
•
•
•
•
•
•
The EcoCloud Project [EPFL: Anne Wiggins]
Academic Compute Cloud Project at ETH [ETH/SystemsX: Peter Kunszt]
From Bare-Metal to Cloud [ZHAW/ICClab: Andrew Edmonds]
Review of CERN Data Center Infrastructure [CERN: Gavin McCance]
Big Science in the Public Clouds: Watching ATLAS proton collisions at CloudSigma
[CloudSigma: Michael Higgins]
Supporting Research with Flexible Computational Resources [University Oxford:
David Wallom]
The iPlant Collaborative: Science in the Cloud for Plant Biology [University of
Arizona/iPlant: Edwin Skidmore]
Tiny Particle within Huge Data [ETH: Christoph Grab]
Roundtable discussion: Cloud Strategies and thoughts for Researchers in
Switzerland