Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Self-configuring Condor Virtual Machine Appliances for Ad-Hoc Grids Renato Figueiredo Arijit Ganguly, David Wolinsky, J. Rhett Aultman, P. Oscar Boykin, ACIS Lab, University of Florida http://wow.acis.ufl.edu Advanced Computing and Information Systems laboratory Outline Motivations Background Condor Virtual Appliance: features On-going and future work Advanced Computing and Information Systems laboratory 2 Motivations Goal: plug-and-play deployment of Condor grids • • High-throughput computing; LAN and WAN Collaboration: file systems, messaging, .. Synergistic approach: VM + virtual network + Condor • “WOWs” are wide-area NOWs, where: • Nodes are virtual machines • Network is virtual: IP-over-P2P (IPOP) overlay • VMs provide: • Sandboxing; software packaging; decoupling • Virtual network provides: • Virtual private LAN over WAN; self-configuring and capable of firewall/NAT traversal • Condor provides: • Match-making, reliable scheduling, … unmodified Advanced Computing and Information Systems laboratory 3 Condor WOWs - outlook 5. VMs obtain IP addresses from MyGrid Virtual DHCP server, join virtual IP network, discover available manager(s), and join pool 4. Download base and custom VM images; boot up 5b. VMs obtain IP addresses from OtherGrid Virtual DHCP server, join virtual IP network, discover available manager(s), and join pool 3. Create virtual IP namespace for pool: MyGrid:10.0.0.0/255.0.0.0 Prime custom image with virtual namespace, desired tools Bootstrap manager(s) 10.0.0.2 10.0.0.1 2. Download image; boot using free VM monitor (e.g. VMware Player or Server) 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.1 10.0.0.3 10.0.0.4 1. Prime base VM image with O/S, Condor, Virtual network; publish (Web/Torrent) Advanced Computing and Information Systems laboratory 4 Condor WOW snapshot Gainesville Zurich Long Beach Advanced Computing and Information Systems laboratory 5 Roadmap The basics: 1.1 VMs and appliances 1.2 IPOP: IP-over-P2P virtual network 1.3 Grid Appliance and Condor The details: 2.1 Customization, updates 2.2 User interface 2.3 Security 2.4 Performance Usage experience Advanced Computing and Information Systems laboratory 6 1.1: VMs and appliances System VMs: Homogenous system Sandboxing Co-exist with unmodified hosts Virtual appliances: • VMware, KVM, Xen • Hardware/software configuration packaged in • easy to deploy VM images Only dependences: ISA (x86), VMM Advanced Computing and Information Systems laboratory 7 1.2: IPOP virtual networking Key technique: IP-over-P2P tunneling • • Interconnect VM appliances WAN VMs perceive a virtual LAN environment IPOP is self-configuring • • Avoid administrative overhead of VPNs NAT and firewall traversal IPOP is scalable and robust IPOP networks are isolated • P2P routing deals with node joins and leaves • • One or more private IP address spaces Decentralized DHCP serves addresses for each space Advanced Computing and Information Systems laboratory 8 1.2: IPOP virtual networking Structured overlay network topology • • • Bootstrap 1-hop IP tunnels on demand Discover NAT mappings; decentralized hole punching VM keeps IPOP address even if it migrates on WAN • [Ganguly et al, IPDPS 2006, HPDC 2006] App IPOP Node A B IPOP Node B App A tap0 tap0 (10.0.0.3) (10.0.0.2) eth0 (128.227.136.244) eth0 P2P Overlay Advanced Computing and Information Systems laboratory (139.70.24.100) 9 1.3 Grid appliance and Condor Base: Debian Linux; Condor; IPOP • Works on x86 Linux/Windows/MacOS; • VMware, KVM/QEMU 157MB zipped Uses NAT and host-only NICs Managed negotiator/collector VMs Easy to deploy schedd/startd VMs • No need to get IP address on host network • Flocking is easy – virtual network is a LAN Advanced Computing and Information Systems laboratory 10 2.1: Customization and updates VM image: Virtual Disks Disks are logically stacked • Portable medium for data • Growable after distribution • Leverage UnionFS file system • Three stacks: • Base – O/S, Condor, IPOP • Module – site specific configuration (e.g. nanoHUB) • Home – user persistent data Major updates: replace base/module • Minor updates: automatic, apt-based Advanced Computing and Information Systems laboratory 11 2.2: User interface (Windows host) Host-mounted loop-back Samba folder VM console: X11 GUI Loopback SSH Advanced Computing and Information Systems laboratory 12 2.2: User interface (Mac host) Host-mounted loop-back Samba folder VM console: X11 GUI Loopback SSH Advanced Computing and Information Systems laboratory 13 2.2: User interface (Linux host) Host-mounted loop-back Samba folder VM console: X11 GUI Loopback SSH Advanced Computing and Information Systems laboratory 14 2.3 Security Appliance firewall • eth0: block all outgoing Internet packets • Except DHCP, DNS, IPOP’s UDP port • Only traffic within WOW allowed • eth1 (host-only): allow ssh, Samba IPsec • X.509 host certificates • Authentication and end-to-end encryption • VM joins WOW only with signed certificate bound to its virtual IP • Private net/netmask: ~10 lines of IPsec configuration for an entire class A network! Advanced Computing and Information Systems laboratory 15 2.4: Performance User-level C# IPOP implementation (UDP): • • Link bandwidth: 25-30Mbit/s Latency overhead: ~4ms Connection times: • ~5-10s to join P2P ring and obtain DHCP address • ~10s to create shortcuts, UDP hole-punching 89.35 100.00 79.92 80.18 90.00 80.00 70.00 Time 60.00 SimpleScalar 3.0 (cycle-accurate CPU simulator) 50.00 40.00 30.00 20.00 10.00 0.00 Physical VMWare Xen Advanced Computing and Information Systems laboratory 16 Experiences Bootstrap WOW with VMs at UF and partners • • Currently ~300 VMs, IPOP overlay routers (Planetlab) Exercised with 10,000s of Condor jobs from real users • nanoHUB: 3-week long, 9,000-job batch (BioMoca) • submitted via a Condor-G gateway P2Psim, CH3D, SimpleScalar Pursuing interactions with users and the Condor community for broader dissemination Advanced Computing and Information Systems laboratory 17 Time scales and expertise Development of baseline VM image: Development of custom module: Deployment of VM appliance: • VM/Condor/IPOP expertise; weeks/months • Domain-specific expertise; hours/days/weeks • No previous experience with VMs or Condor • 15-30 minutes to download and install VMM • 15-30 minutes to download and unzip • appliance 15-30 minutes to boot appliance, automatically connect to a Condor pool, run condor_status and a demo condor_submit job Advanced Computing and Information Systems laboratory 18 On-going and future work Enhancing self-organization at the Condor level: • Structured P2P for manager publish/discovery • Distributed hash table (DHT); primary and flocking • Condor integration via configuration files, DHT scripts • Unstructured P2P for matchmaking • Publish/replicate/cache classads on P2P overlay • Support for arbitrary queries • Condor integration: proxies for collector/negotiator Decentralized storage, cooperative caching • • • Virtual file systems (NFS proxies) Distribution of updates, read-only code repositories Caching and COW for diskless, net-boot appliances Advanced Computing and Information Systems laboratory 19 Acknowledgments National Science Foundation NMI, CI-TEAM SURA SCOOP (Coastal Ocean Observing and Prediction) http://wow.acis.ufl.edu Publications, Brunet/IPOP code (GPL’ed C#), Condor Grid appliance Advanced Computing and Information Systems laboratory 20 Questions? Advanced Computing and Information Systems laboratory 21 Self-organizing NAT traversal, shortcuts Sends CTM request Node A Node B CTM request: connect to me at my NAT IP:port - A starts exchanging IP packets with B - Traffic inspection triggers request to create shortcut - Connect-to-me (CTM) - “A” tells “B” its known address(es): - “A” had learned NATed public IP/port when it joined overlay Advanced Computing and Information Systems laboratory 22 Self-organizing NAT traversal, shortcuts Link request: NAT endpoint (IP:port)A Node A Node B CTM reply through overlay: send NAT (IP:port)B - “B” sends CTM reply – routed through overlay - “B” tells “A” its address(es) - “B” initiates linking protocol by attempting to connect to “A” directly Advanced Computing and Information Systems laboratory 23 Self-organizing NAT traversal, shortcuts A Gets CTM reply; initiates linking Node A Node B - B’s linking protocol message to A pokes hole on B’s NAT - A’s linking protocol message to B pokes hole on A’s NAT CTM protocol establishes direct shortcut Advanced Computing and Information Systems laboratory 24 Performance considerations CPU-intensive application, Condor • SimpleScalar 3.0d execution-driven computer architecture simulator 89.35 100.00 79.92 80.18 90.00 80.00 70.00 Time 60.00 50.00 40.00 30.00 20.00 10.00 0.00 Physical VMWare Advanced Computing and Information Systems laboratory Xen 25 Performance considerations I/O: PostMark 12 • Version 1.51 • Parameters: • Minimum file size: 500 bytes • Maximum file size: 4.77 MB • Transactions: 5,000 9.93 10 8 MBs 11.94 5.38 6 Read 4.47 4.28 Write 3.56 4 2 0 Advanced Computing and Information Systems laboratory Host Vmware Xen 26 Performance considerations User-level C# IPOP implementation (UDP): • • Link bandwidth: 25-30Mbit/s (LAN) Latency overhead: ~4ms Connection times: • (Fine-tuning has reduced mean acquire time to ~ 6-10s, with degree of redundancy n=8) Advanced Computing and Information Systems laboratory 27 Condor Appliance on a desktop VM Hardware configuration Swap User files Domainspecific tools Linux, Condor, IPOP Advanced Computing and Information Systems laboratory 28 Related Work Virtual Networking Internet Indirection Infrastructure (i3) IPv6 tunneling • VIOLIN • VNET; topology adaptation • ViNe • Support for mobility, multicast, anycast • Decouples packet sending from receiving • Based on Chord p2p protocol • IPv6 over UDP (Teredo protocol) • IPv6 over P2P (P6P) Advanced Computing and Information Systems laboratory 29