Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Forschungszentrum Karlsruhe in der Helmholtz-Gemeinschaft European and German Grid Computing Projects Marcel Kunze Forschungszentrum Karlsruhe [email protected] DESY Seminar December 2002 Grid Computing Potential 1980 2005 Marcel Kunze, DESY Seminar What is Grid Computing? Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations Marcel Kunze, DESY Seminar Grid Computing: Diverse Areas of Application • Computational Grid High Performance Computing (HPC) Perspective: Parallelization of programs, as fast as possible • Data Grid High Throughput Computing (HTC) Perspective : Parallelization of data, as much as possible • Gaming Grid Communication between players Perspective: LAN + WAN-Party, as entertaining as possible Marcel Kunze, DESY Seminar Grid Computing: Commercial Aspects • Huge potential of Grid Computing in the field of information technology: – Cost reduction through more efficient/diverse use of system resources – „Resource on demand“ Added flexibility, e.g. dynamically satisfy increased computing demand in bank accounting sector at the end of an accounting period – „Business continuity“ Proliferation of mission-critical IT-Services to ease disaster recovery – Possibility of system consolidation: Reduction of system complexity by standardization of services Marcel Kunze, DESY Seminar Grid Computing: Scientific Aspects • Enables formation of „critical mass“ of resources and expertise to allow for interdisciplinary projects in socalled “Virtual Organizations” (VO) – – – – Tele-Collaboration Instrument Sharing Computing Resource Sharing Data Sharing • Potential fields of application – – – – Bio-Informatics and Medicine Astronomy Earth Sciences High Energy Physics (HEP) Marcel Kunze, DESY Seminar Biomedical Applications • Data mining on genomic databases (exponential growth) • Indexing of medical databases (Tb/hospital/year) • Collaborative framework for large scale experiments (e.g. epidemiological studies) • Parallel processing for –Databases analysis –Complex 3D modelling Fabrizio Gagliardi Marcel Kunze, DESY Seminar Astronomy: Past and Future of the Universe New phase of astronomy, storing, searching and analysing Petabytes of data: • Virtual Observatories – GAVO, NVO, AVO, AstroGrid – Store all wavelengths, need distributed joins – NVO 500 TB/yr from 2004 • Grid Computing might help to – Master the data streams – Federate databases with different schema – Maintain meta data (information) and provenance data (history) Marcel Kunze, DESY Seminar Crab Nebula viewed At four different wavelengths: X-ray, optical, infrared, radio. Earth Observation ESA missions: • about 100 Gbytes of data per day (ERS 1/2) • 500 Gbytes, for the next ENVISAT mission (launched March 1st) EO requirements for the Grid: Marcel Kunze, DESY Seminar Fabrizio Gagliardi • enhance the ability to access high level products • allow reprocessing of large historical archives • improve Earth science complex applications (data fusion, data mining, modelling …) What is Grid Computing? Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations Marcel Kunze, DESY Seminar Elements of the Problem • Resource sharing – Computers, storage, sensors, networks, … – Heterogeneity of device, mechanism, policy – Sharing conditional: negotiation, payment, … • Coordinated problem solving – Integration of distributed resources – Compound quality of service requirements • Dynamic, multi-institutional virtual orgs – Dynamic overlays on classic org structures – Map to underlying control mechanisms Marcel Kunze, DESY Seminar The Grid World: Current Status • Dozens of major Grid projects in scientific & technical computing/research & education – Deployment, application, technology • Considerable consensus on key concepts and technologies – Open source Globus Toolkit™ a de facto standard for major protocols & services – Far from complete or perfect, but out there, evolving rapidly, and large tool/user base • Global Grid Forum a significant force • Industrial interest emerging rapidly Marcel Kunze, DESY Seminar “Web Services” • Increasingly popular standards-based framework for accessing network applications – W3C standardization; Microsoft, IBM, Sun, others • WSDL: Web Services Description Language – Interface Definition Language for Web services • SOAP: Simple Object Access Protocol – XML-based RPC protocol; common WSDL target • WS-Inspection – Conventions for locating service descriptions • UDDI: Universal Description, Discovery & Integration – Directory for Web services Marcel Kunze, DESY Seminar Open Grid Services Architecture (OGSA) • Platform independent industrial and scientific standard to construct Grid computing components • Service orientation to virtualize resources • From Web services: – Standard interface definition mechanisms: multiple protocol bindings, multiple implementations, local/remote transparency • Building on Globus Toolkit: – – – – Grid service: semantics for service interactions Management of transient instances (& state) Factory, Registry, Discovery, other services Reliable and secure transport • Multiple hosting targets: J2EE, .NET, “C”, … Marcel Kunze, DESY Seminar The Grid Service • A (potentially transient) Web service with specified interfaces & behaviors, including – – – – – – – – Creation (Factory) Global naming (GSH) & references (GSR) Lifetime management Registration & Discovery Authorization Notification Concurrency Manageability Marcel Kunze, DESY Seminar Grid Projects Marcel Kunze, DESY Seminar GRID Projects World Wide • EU – EDG (EU-IST) – R&D EU GRID project [ www.edg.org ] – CrossGRID QoS – Real Time apps. [ www.crossgrid.org ] – DataTAG GLUE (EU-USA) [ www.datatag.org ] – LCG The LHC Computing GRID – Deployment [ cern.ch/lcg ] – The new 16,2 B Euro EU VI Framework Prog. GEANT based GRID projects • USA – GriPhyN [ www.griphyn.org ] iVDGL-VDTv1 [ www.idvgl.org ] • Asia – ApGrid [ www.apgrid.org ] Pragma (USA-Asia) Marcel Kunze, DESY Seminar PPDG ( NSF, DoE ) [ www.ppdg.org ] IST Grid Project Space - Links with European National efforts - Links with US projects (GriPhyN, PPDG, iVDGL,…) Applications EGSO CROSSGRID GRIA GRIP Middleware & Tools Underlying Infrastructures EUROGRID GRIDLAB DAMIEN DATAGRID DATATAG Industry / business Science A. Baxevanidis Marcel Kunze, DESY Seminar The EU DataGrid Project • • • • • 21 Partners 9.8 M Euros EU funding over 3 years 90% for middleware and applications (Particle Physics, Earth Obs. and Biomedical) Three year phased developments & demos (2001-2003) Spin-off: – DataTAG (2002-2003) – CrossGrid (2002-2004) – GridStart (2002-2004) Fabrizio Gagliardi Marcel Kunze, DESY Seminar DataGrid Partners Main Partners •CERN – International (Switzerland/France) •CNRS - France •ESA/ESRIN – International (Italy) •INFN – Italy •NIKHEF – The Netherlands •PPARC - UK Industrial Partners •Datamat (Italy) •IBM-UK (UK) •CS-SI (France) Research and Academic Institutes Marcel Kunze, DESY Seminar Fabrizio Gagliardi •CESNET (Czech Republic) •Commissariat à l'énergie atomique (CEA) – France •Computer and Automation Research Institute, Hungarian Academy of Sciences (MTA SZTAKI) •Consiglio Nazionale delle Ricerche (Italy) •Helsinki Institute of Physics – Finland •Institut de Fisica d'Altes Energies (IFAE) - Spain •Istituto Trentino di Cultura (IRST) – Italy •Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany •Royal Netherlands Meteorological Institute (KNMI) •Ruprecht-Karls-Universität Heidelberg - Germany •Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands •Swedish Research Council - Sweden EU DataGrid Project Objectives • • Use Grid technology to develop a sustainable computing model for effective share of computing resources and data for large scientific communities Specific project objectives: – Middleware for fabric & Grid management – Large scale testbeds – Production quality demonstrations • Key products: – Resource broker (Compute elements, storage elements) – Replica manager – Virtual Organization manager Marcel Kunze, DESY Seminar EDG: Interaction with Sister Projects • CrossGrid • NorduGrid – Using the same security certs. – Testbed sites install EDG software • Extending it for needs of intensive interactive applications – Using the same security certs. – Involved in EDG architecture work • Good ideas for gatekeeper and MDS configuration • Helped develop GDMP and GSI extensions for Replica Catalog • Involved in GLUE schema work • Security policy – Participating in the EDG testing activities – Representatives in each projects architecture & management groups • DataTAG (EDT) • iVDGL/GriPhyN/PPDG – US members in EDG architecture group – Looking for common packaging and toolkit usage solutions Marcel Kunze, DESY Seminar Fabrizio Gagliardi – EDT is deploying EDG sw to investigate inter-operability with US projects (iVDGL, GriPhyN, PPDG) – Results feedback into EDG software releases (e.g. GLUE compatible information providers/consumers) – Middleware testing – Working in WP8 (HEP applications) CrossGrid: An EDG Partner • New category of Grid enabled applications – – – – • • • • Computing and data intensive Distributed Interactive, near real time response (a person in a loop) Layered New programming tools Grid more user friendly, secure and efficient Interoperability with other Grids Implementation of standards Marcel Kunze, DESY Seminar CrossGrid Layered Architecture Biomedical Flood Application Application Applications And Supporting Tools Applications Development Support HEP Data Mining on Grid Application MPI Verification Portal Weather Forecast application HEP High LevelTrigger Performance Analysis Metrics and Benchmarks MPICH-G Globus Replica Manager Grid Common Services Datagrid Job Manager GRAM Local Resources HEP Interactive Distributed Data Access Application Interactive Distributed Data Access DataGrid Replica Manager Replica Catalog Grid Visualisation Kernel Data Mining on Grid Grid Resource Management User Interaction Service GSI Globus-IO MDS Distributed Data Collection Roaming Access Grid Monitoring GridFTP GASS Resource Manager Resource Manager Resource Manager Resource Manager Resource Manager Resource Manager Resource Manager Secondary Storage CPU Optimization of Data Access Scientific Instruments (Medical Scaners, Satelites, Radars) Detector Local High Level Trigger VR systems (Caves, immerse desks) Visualization tools Tertiary Storage Marcel Kunze, DESY Seminar Interactive Treatment Planning The vascular geometry can be modified using a library of models Draw interactively Computational geometry Marcel Kunze, DESY Seminar Biomedical Application Sample pulse flow simulation Marcel Kunze, DESY Seminar Benefit of Standardization Marcel Kunze, DESY Seminar Fabrizio Gagliardi Through links with sister projects, there is the potential for a uniform global scientific applications grid Common WorldGrid Demo at IST2002 VO centric Ganglia monitor • WorldGrid Demo together with US and EU partners at IST2002 • Seamless interoperation of EDG with US Middleware (via GLUE) Marcel Kunze, DESY Seminar WorldGrid Lessons (+) • Advantages of the Grid: – Possibility to execute tasks and move files over a distributed computing infrastructure by using one single personal certificate (no need to memorize dozens of passwords) – Possibility do distribute the workload adequately and automatically, without logging in explicitly to each remote system – Possibility to do worldwide production in a perfectly coordinated way, using identical software (RPMs), scripts and databases • GLUE interfacing works to make the middleware worlds talk to each other. Better solution: Agree on open standards environment like OGSA ! Marcel Kunze, DESY Seminar WorldGrid Lessons (-) • EDG stability very much dependent on the quality of the basic building blocks (Globus, Condor, etc) • Globus support needs to be strengthened (formal contract, European support center, internal Globus support team) • Understand impact of OGSA and industrial involvement – EDG Toolkit might need to be refactored Marcel Kunze, DESY Seminar EDG Future Plans • Concentrate on production quality and real applications • Educate new users and disseminate results • Complete the program of work till end of 2003 • Port EDG to other platforms than Linux (Solaris) • Port EDG to Globus ToolKit version 3 (OGSA) • Make plans to conserve momentum and assets in the EU FP6 Marcel Kunze, DESY Seminar A Proposal to the EU • EDG has already demonstrated the viability of Grid technology • EU had a fast start in supporting 18 Grid projects in the last two years! • RN Geant offers an excellent basis for a large European Grid infrastructure • The EU FP6 program should encourage and support the deployment and production quality operation of a large international Grid infrastructure open to research and industry in Europe Fabrizio Gagliardi Marcel Kunze, DESY Seminar Enabling Grids and e-Science in Europe • EGEE: Integrated Infrastructure Initiative to support European Research Area • Vision: to create and deploy Grid technologies to enable the widespread uptake of e-Science applications throughout the European Research Area • Four key objectives: – integrating Grid technological developments from across Europe; – establishing a Europe-wide Grid infrastructure for science and industry with a focus on heterogeneity and interoperability; – enabling the creation of e-Science applications from across the scientific and industrial spectrum; – ensuring the timely delivery of the project’s programme of work, guided by the needs of academic and industrial partners. Marcel Kunze, DESY Seminar Fabrizio Gagliardi Start by integration of the national Grid initiatives GRID Computing Center GridKa GRID Computing Center: Infrastructure and Services Competence Center: Applications and e-Science Marcel Kunze, DESY Seminar Partners in the HGF 24.000 employees, 2.5 Billion Euro yearly budget Grid Computing perfectly well matches the HGF vision of “Konzertierte Forschung” • • • • • • • • • • • • • • • ALFRED-WEGENER-INSTITUT FÜR POLAR- UND MEERESFORSCHUNG AWI DEUTSCHES ELEKTRONEN-SYNCHROTRON DESY DEUTSCHES KREBSFORSCHUNGSZENTRUM DKFZ DEUTSCHES ZENTRUM FÜR LUFT- UND RAUMFAHRT DLR FORSCHUNGSZENTRUM JÜLICH FZJ FORSCHUNGSZENTRUM KARLSRUHE FZK GESELLSCHAFT FÜR BIOTECHNOLOGISCHE FORSCHUNG GBF GEOFORSCHUNGSZENTRUM POTSDAM GFZ GKSS-FORSCHUNGSZENTRUM GEESTHACHT GKSS GSF-FORSCHUNGSZENTRUM FÜR UMWELT UND GESUNDHEIT GSF GESELLSCHAFT FÜR SCHWERIONENFORSCHUNG GSI HAHN-MEITNER-INSTITUT BERLIN HMI MAX-PLANCK-INSTITUT FÜR PLASMAPHYSIK IPP MAX-DELBRÜCK-CENTRUM FÜR MOLEKULARE MEDIZIN MDC UFZ-UMWELTFORSCHUNGSZENTRUM LEIPZIG-HALLE UFZ Marcel Kunze, DESY Seminar A German Grid Initiative D-GRID • • • • • Rostock Kiel Hamburg Initially driven by the HGF Global Upstream Oldenburg centers and the DFN Braunschweig Hannover Berlin Magdeburg Open to accept further partners Bielefeld Essen Göttingen in academia and industry Leipzig St. Augustin Dresden Marburg Aim at a coordination of Grid Ilmenau Würzburg Aachen Frankfurt activities Erlangen Heidelberg Deployment of a new generation Karlsruhe Regensburg Kaiserslautern networking infrastructure Stuttgart Garching Augsburg (towards “TeraGrids”) Promotion of open standards for interfaces and protocols (GGF) 10 Gbit/s 2,4 Gbit/s 2,4 Gbit/s 622 Mbit/s Marcel Kunze, DESY Seminar Scientific Program • Installation of a coordination office for Grid Computing • Foster the enhanced application of IT in the field of scientific and engineering disciplines • Management of the relationship between the German Grid competence centers and representation of their common interests in the Global Grid Forum • Training and education in the field of Grid Computing and e-Science • Deployment of national Grid resources and development of a program for academic and industrial projects Marcel Kunze, DESY Seminar Technical Program • Goal: Integration of all kind of resources e.g. sensors, computing infrastructure, people,… • Deployment of a state-of-the-art production backbone (2003: 10 GBPS, 2006: 100 GBPS) • Discussion of safety and trust relationship • Management of German Grid certificates • Selection of middleware components and installation procedures (Globus, UNICORE) • Infrastructure: Find qualified manpower to offer and support the additional services • Definition of cost model (Grid-Accounting, credit points) • Collaboration with the German HPC centers => D-GRID Kickoff Workshop in January 2003 (Wissenschaftszentrum Bonn) Marcel Kunze, DESY Seminar Where do we go from here?? Marcel Kunze, DESY Seminar e-Science (enhanced Science) Konzertierte Forschung • e-Science is about more than networks, GRIDs, High Performance Computing, cluster computing ... • e-Science: „e-Science is about global collaboration in key areas of science, and the next generation of infrastructure that will enable it.“ (Dr. John Taylor, director of UK research council) • Foster the transition of data Grid to semantic Grid – Federation of resources – Federation of distinguished data sources – Knowledge retrieval and collaboratories Marcel Kunze, DESY Seminar Semantic Grid: Three Layer Grid Abstraction e-Science Information Grid Computation/Data Grid Automation Marcel Kunze, DESY Seminar Control Knowledge Grid The Future • Where do we need to get to ? – Applications to support an “e-society” (“Cyber-Infrastructure”) – An international Grid infrastructure which hides the complexities from the users (“Invisible Computing”) – A powerful and flexible network infrastructure • Grid-enabled Applications Where do we need to invest ? – Applications targeted at realistic problems in “e-science” – Prototypes of Grid infrastructures – Maintain and improve the GEANT network Marcel Kunze, DESY Seminar Prototype Grid Infrastructures Gèant: World Class Networking