Download I-Cluster

I-Cluster
I-Cluster
ACM-SIGOPS
JTE Cluster Computing
Bruno Richard
Research Program Manager
HP Labs Grenoble
HP Labs
• Focus on infrastructure,
information appliances and eservices
• World’s second largest
computer research lab
• 750 researchers in 6 labs
globally
• Leading-edge collaborations
• Helps create HP’s IP portfolio
2October 2001
Page 2
HP Labs Grenoble
• Role
• Deliver technology related to
emerging Internet access devices
• Two main research areas:
• PPP
– personalization, profiling, privacy
– User-adapted computing environment
• LEA
– Local Environment of Access Devices
– Wireless communities
• I-Cluster
• Compute-intensive services
• Partnership with ID-IMAG, INRIA
2October 2001
Page 3
Project overview
What is I-Cluster?
A distributed framework of tools that
transparently takes advantage of
unused network resources and
transforms them into computeintensive services
Project rationale
Support distributed virtual functions
utilising unmodified, standard
hardware
Learn how cluster devices interact
with each other -- potential and
limitations
Create environment for
development and execution of
applications that will use enterprise
infrastructure or internet rather than
dedicated cluster
Apply knowledge gained from
I-Cluster to future products
2October 2001
Page 4
I-Cluster Value
• Supercomputing-enabled
• Not limited to metacomputing
– Cluster computing environment
• Fine-grained problems
– Network communications can be stressed
• User compliance
• Self-organized system
– Self sustaining
• From federative model to community model
–
–
–
–
Server-less
Service oriented
No administration
Peer-to-peer system
• Real-world conditions
–
–
–
–
Massively scales (10000 devices)
No specific hardware required
Heterogeneous environments
Roaming, disconnections
• Applications do not need to be rewritten
– But no shared memory
2October 2001
Page 5
Usage example
Step 1
Request a
computing
service from the
cloud.
“Render Star
Wars movie for
my PDA”
I-Cluster Cloud
Device
Device
Device
Device
Step 2
Identify cluster
aggregate that
fits the required
service
I-Cluster Cloud
Device
Device
Device
Device
Device
Device
Device
Device
Device
Device
Starwars.avi
Step 3
Efficiently
distribute the
job on identified
devices
I-Cluster Cloud
Device
Device
Step 4
My service is
executed on the
cluster within
seconds
I-Cluster Cloud
Device
Device
Device
Device
Device
Device
Device
Device
2October 2001
Device
Device
Device
Device
Page 6
Research areas
I-Cluster cloud
P2P community
Gathers resources
Discovers network topology
Mode switch
Any PC becomes a cluster machine
Use idle periods
No (lowest) user impact
Match finder
Instantiates a cluster on the cloud
Allocates devices to given jobs
Tetris
Inter-intra task scheduling
2October 2001
Page 7
I-Cluster cloud
• Devices operate in peer to peer:
• A server answers other’s requests
• A client actively polls others
Emitter
Receiver
Emitter
• Local database: a view of the cloud
• Incomplete
– Some elements are unknown
– Elements are forgotten
Receiver
Database
Database
Emitter
Receiver
Emitter
Receiver
Database
• Lazy consistency
Database
– Best effort consistency
• Network analysis
• Topology
• Bandwidth, latency
• Congestion analysis
• Very fast convergence
2October 2001
Page 8
Cloud operation: An example
Full network view
A
B
2October 2001
Blue: I-Cluster
devices
Yellow: Routers+other
Page 9
Cloud operation (cont’d)
View from device A
A
B
2October 2001
Page 10
Cloud operation (cont’d)
View from device B
A
B
2October 2001
Page 11
Cloud operation (cont’d)
A
B
Combined view
After synchronization
Each device will forget
some items
Based on relevance of
peers
2October 2001
Page 12
I-Cluster mode switch
Each I-Cluster PC has 2 modes
- User mode
- Standard Windows operation
- Cluster mode
- Cluster Linux distribution
- Sandbox for jobs execution
- Secure mode (no user data access)
Idle periods used for cluster computing
- Idleness detection
- Automatic switch between modes
Inactivity prediction
User mode
Job end
Lowest user impact
User request
- Easy installation
- No change in user partitions
Available
Switch off
Switch on
Tentative
I-Cluster mode
Allocated
RWU
Reserved
Allocation
- Low psychological impact
- Automatic transitions
Off
Ease of development
- Easier than other sandboxing technologies
2October 2001
Page 13
Hidden cluster distribution
• User hard disk partitions are kept without modification
2October 2001
Page 14
Hidden cluster distribution
• Then anchors are added
• Master Boot Record is changed to a new code
• A hidden zone is used for storing our mode control tools
2October 2001
Page 15
Hidden cluster distribution
• Then a fake partition is created
• Big file in user’s file system
• Contiguous, unfragmented
• System-protected, unwriteable
• This partition will be usable as a boot option by the switcher
2October 2001
Page 16
Match finder
• Called upon user request
• A service is invoked
Service request:
- 4 nodes required
- At least 256 MB
• User data attached to invocation
• Service requirements available
• Number of processing nodes
4 machines allocated
Job started there
• Minimum RAM
• Maximum Network latency
• …
• Allocates a cluster within the cloud
2October 2001
Page 17
Tetris
• Optimal use of computing
resources
• Inter/Intra job scheduling
• Use of job knowledge for
intelligent task/resource
assignment
• Use of past experiences to
improve future scheduling
2October 2001
Agreements between
HP-Labs and INRIA
Page 18
Tetris
duration
number of
processors
2
Page 19
Tetris
duration
number of
processors
2
Page 20
Backup slides
“It is all about power, space and time”
Chessmaster Savielly Grigorievitch Tartakower
2October 2001
Page 21
The experimental platform
2October 2001
Page 22
I-Cluster
TOP500 results
Supercomputing with a
mainstream cluster
A TOP500 cluster based
on 225 standard hp eVectra machines
Partnership with IMAG-ID, INRIA, Intel
225 HP e-Vectra:
- Pentium® III
- 733 MHz
- 256 MB
- Standard Ethernet (100 MBps)
76 Gflop/s * as of April 15th 2001
http://www.top500.org/
2
(*) Standard LINPACK benchmark
Page 23