Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
I-Cluster I-Cluster ACM-SIGOPS JTE Cluster Computing Bruno Richard Research Program Manager HP Labs Grenoble HP Labs • Focus on infrastructure, information appliances and eservices • World’s second largest computer research lab • 750 researchers in 6 labs globally • Leading-edge collaborations • Helps create HP’s IP portfolio 2October 2001 Page 2 HP Labs Grenoble • Role • Deliver technology related to emerging Internet access devices • Two main research areas: • PPP – personalization, profiling, privacy – User-adapted computing environment • LEA – Local Environment of Access Devices – Wireless communities • I-Cluster • Compute-intensive services • Partnership with ID-IMAG, INRIA 2October 2001 Page 3 Project overview What is I-Cluster? A distributed framework of tools that transparently takes advantage of unused network resources and transforms them into computeintensive services Project rationale Support distributed virtual functions utilising unmodified, standard hardware Learn how cluster devices interact with each other -- potential and limitations Create environment for development and execution of applications that will use enterprise infrastructure or internet rather than dedicated cluster Apply knowledge gained from I-Cluster to future products 2October 2001 Page 4 I-Cluster Value • Supercomputing-enabled • Not limited to metacomputing – Cluster computing environment • Fine-grained problems – Network communications can be stressed • User compliance • Self-organized system – Self sustaining • From federative model to community model – – – – Server-less Service oriented No administration Peer-to-peer system • Real-world conditions – – – – Massively scales (10000 devices) No specific hardware required Heterogeneous environments Roaming, disconnections • Applications do not need to be rewritten – But no shared memory 2October 2001 Page 5 Usage example Step 1 Request a computing service from the cloud. “Render Star Wars movie for my PDA” I-Cluster Cloud Device Device Device Device Step 2 Identify cluster aggregate that fits the required service I-Cluster Cloud Device Device Device Device Device Device Device Device Device Device Starwars.avi Step 3 Efficiently distribute the job on identified devices I-Cluster Cloud Device Device Step 4 My service is executed on the cluster within seconds I-Cluster Cloud Device Device Device Device Device Device Device Device 2October 2001 Device Device Device Device Page 6 Research areas I-Cluster cloud P2P community Gathers resources Discovers network topology Mode switch Any PC becomes a cluster machine Use idle periods No (lowest) user impact Match finder Instantiates a cluster on the cloud Allocates devices to given jobs Tetris Inter-intra task scheduling 2October 2001 Page 7 I-Cluster cloud • Devices operate in peer to peer: • A server answers other’s requests • A client actively polls others Emitter Receiver Emitter • Local database: a view of the cloud • Incomplete – Some elements are unknown – Elements are forgotten Receiver Database Database Emitter Receiver Emitter Receiver Database • Lazy consistency Database – Best effort consistency • Network analysis • Topology • Bandwidth, latency • Congestion analysis • Very fast convergence 2October 2001 Page 8 Cloud operation: An example Full network view A B 2October 2001 Blue: I-Cluster devices Yellow: Routers+other Page 9 Cloud operation (cont’d) View from device A A B 2October 2001 Page 10 Cloud operation (cont’d) View from device B A B 2October 2001 Page 11 Cloud operation (cont’d) A B Combined view After synchronization Each device will forget some items Based on relevance of peers 2October 2001 Page 12 I-Cluster mode switch Each I-Cluster PC has 2 modes - User mode - Standard Windows operation - Cluster mode - Cluster Linux distribution - Sandbox for jobs execution - Secure mode (no user data access) Idle periods used for cluster computing - Idleness detection - Automatic switch between modes Inactivity prediction User mode Job end Lowest user impact User request - Easy installation - No change in user partitions Available Switch off Switch on Tentative I-Cluster mode Allocated RWU Reserved Allocation - Low psychological impact - Automatic transitions Off Ease of development - Easier than other sandboxing technologies 2October 2001 Page 13 Hidden cluster distribution • User hard disk partitions are kept without modification 2October 2001 Page 14 Hidden cluster distribution • Then anchors are added • Master Boot Record is changed to a new code • A hidden zone is used for storing our mode control tools 2October 2001 Page 15 Hidden cluster distribution • Then a fake partition is created • Big file in user’s file system • Contiguous, unfragmented • System-protected, unwriteable • This partition will be usable as a boot option by the switcher 2October 2001 Page 16 Match finder • Called upon user request • A service is invoked Service request: - 4 nodes required - At least 256 MB • User data attached to invocation • Service requirements available • Number of processing nodes 4 machines allocated Job started there • Minimum RAM • Maximum Network latency • … • Allocates a cluster within the cloud 2October 2001 Page 17 Tetris • Optimal use of computing resources • Inter/Intra job scheduling • Use of job knowledge for intelligent task/resource assignment • Use of past experiences to improve future scheduling 2October 2001 Agreements between HP-Labs and INRIA Page 18 Tetris duration number of processors 2 Page 19 Tetris duration number of processors 2 Page 20 Backup slides “It is all about power, space and time” Chessmaster Savielly Grigorievitch Tartakower 2October 2001 Page 21 The experimental platform 2October 2001 Page 22 I-Cluster TOP500 results Supercomputing with a mainstream cluster A TOP500 cluster based on 225 standard hp eVectra machines Partnership with IMAG-ID, INRIA, Intel 225 HP e-Vectra: - Pentium® III - 733 MHz - 256 MB - Standard Ethernet (100 MBps) 76 Gflop/s * as of April 15th 2001 http://www.top500.org/ 2 (*) Standard LINPACK benchmark Page 23