Download Grids, Clouds and the Community

Grids, Clouds and the Community.
Cloud Technology and the NGS
Steve Thorn
Edinburgh University
Matteo Turilli, Oxford University
Presented by David Fergusson
NeSC, University of
Edinburgh
• National eScience Centre
– Support and develop IT supported advanced
research
• University of Edinburgh
– College of Science and Engineering
– Coordination and cooperation to make best use of
leading research across research.
NGS
• JISC supported national service to:
– Provide a framework for institutions to cooperate
and share services.
– Help promote access to JISC services and synthesis
for researchers.
Views of the Clouds
•
•
•
•
Infrastructure as a Service
Platform as a Service
Service as a Service
Software as a Service
• Private Clouds
• Public Clouds
• Hybrid Clouds
• Cloud vs Web 2.0 (?)
Views of the Clouds
•
•
•
•
Infrastructure as a Service
Platform as a Service √
Service as a Service
Software as a Service
• Private Clouds
• Public Clouds
• Hybrid Clouds
• Cloud vs Web 2.0 (?)
Cloud computing for
NGS/NeSC (incomplete list)
• Dynamic provision of services
– Individualisation
– Peaky demand
• Extending virtualisation
• Packaging
– Service configuration
– Provenance for research
– New publishing models
Head in the clouds?
• Dynamic (service) provisioning
• How is it applicable to the NGS?
• Training
– Rapidly deploy NGS services for training
– Isolate training from production
• Other
– Specialised research environments
– Rapid deployment
• Identify use cases and gather requirements
NGS 3 EWP2
• “NGS Agile Deployment Environments”
• EPSRC funded, 2 years
• People
– Matteo Turilli (OeRC, Oxford) [0.75 FTE]
– Steve Thorn (NeSC, Edinburgh) [0.5 FTE]
– David Fergussion (NeSC, Edinburgh) [WP Leader]
Overview
• Agile service deployment
• Virtualization vs. Cloud?
• Use cases and requirements gathering
– Training
– Identify other (scientific) communities
• Create images
– NGS Services. Which ones?
Overview (cont.)
• Realistic usage
– Training event on virtualized infrastructure
• Hosting infrastructure?
– Amazon EC2 compatible
• De facto standard currently, with open source
implementation
– Ease of deployment
– Eucalyptus, Nimbus and others
• Hardware
– Edinburgh: 8 cores 16+ dual cores => 64 cores
– Oxford: 64 cores (older), new cluster => 64 cores
Eucalyptus
• “Elastic Utility Computing Architecture Linking
Your Programs To Useful Systems”
• Open source and Commercial
• Amazon Web Services API compatible
– EC2, storage - S3, Elastic Block Store (EBS)
• Easy to install
• Xen and KVM hypervisors
– Commercial version supports others (inc.
VMWare)
In the past
• We have worked with Xen in the past to have
Live CDs
• Virtualisation
• Works, but
– Issues with security setups
– networking
Eucalyptus networking
challenges
• Eucalyptus 'Managed' networking
– Different networking modes
– 'Managed' is the most flexible and feature rich – more complex
– Allows elastic IP pool and image isolation.
• VMs have private and public IPs
– Introspection issues
• Elastic IP
– User assignable (may be somewhat different from Amazon)
• X509 Service certificates (NGS Host)
• Switch configurations
cont.....
• Security Groups (EC2)
– Implemented in Eucalyptus
– isolate VMs
• VM public traffic routed through Cluster
controller
– Instance doesn't have knowledge of its public IP
– Bit like a NAT
• Implications for GSI: $GLOBUS_HOSTNAME
Eucalyptus architecture
• Cloud controller
– Entry point
– Gathers information
• Cluster controller
– Schedules VM execution
– Manages virtual network
• Node controller
– Controls VM execution
•
(Xen running on node)

Storage controller (Walrus)
implements Amazon’s S3 interface
cont.....
• Security Groups (EC2)
– Implemented in Eucalyptus
– isolate VMs
• VM public traffic routed through Cluster
controller
– Instance doesn't have knowledge of its public IP
– Bit like a NAT
• Implications for GSI: $GLOBUS_HOSTNAME
Clouds vs Virtualisation
• Similar security and networking issues in
Clouds and Virtualised instances
– Virtualisation – virtualise instance
– Clouds – virtualise the network (and other things)
too
• All arise from the requirements for rapid,
automated, dynamic, reliable, reproducible,
robust, provisioning
Progress
• Started with Eucalypus 1.4.2
• Eucalyptus 1.6.2 deployed at Oxford and
Edinburgh
• Existing Images:
– GSI-OpenSSH server
– 'Single node cluster': torque/maui + Globus GRAM
& GridFTP
• Next step – some real world testing of phase1
images.
• Image snapshots – not straightforward
Further work
• Re-evaluate hosting infrastructure
• Develop more images
– Distributed torque/maui cluster + GRAM &
GridFTP
– Condor & GRAM?
– 'Core site'
• Training event in near future
• Identify pilot community & gather
requirements
• Deploy fledgling cloud infrastructure
Further work 2
• Can snap shotting in Clouds side step
packaging issues (configuration) in
middleware?
– By automating the copying and re-deployment of
successful server installs.
– Not just having a machine but a set of services
which can be copied and deployed directly
Statement 1
• NRENs are still able to provide services that
are generally better or more economic than
those from commercial services providers.
• Commercial offerings are currently shaped to particular modes of usage
(due to their “by-product” heritage. This may limit utility for some users.
EBI have reported issues due to virtual network sharing & unpredictability.
• Data out costs are high
• Commercial providers can offer “infinite” resource rapidly – not easy for
NRENs
• Should NRENs own large computing centres?
• Payment models in institutions & research – FEC in the UK
Statement 2
• NRENs generally operate as a network for a
closed group of users who have advanced
requirements to support their research and
education users.
• There are certainly major issues with current commercial provision in
regards to data confidentiality, isolation of users, service/data guarantees,
internal/external access.
– Recent data losses from commercial services
• Commercial providers are almost certainly at an advantage in providing
simple generic computing
• NGS/UoE sees need to provide more complex research services.
Statement 3
• The NRENs do not compete with commercial
ISPs, but offer a different level of service in
parallel with them.
• This is necessarily so. NRENs and other academic providers
should find “blue water” for services.
–
–
–
–
–
Trust networks and security for public clouds ?
Service discovery ?
Data services ?
Accounting, monitoring ?
............. >
• Commercial providers will have no interest in some research
services, cf. current research computing software provision.