Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Enabling Grids for E-sciencE The EGEE Infrastructure and Remote Instruments Erwin Laure EGEE-II Technical Director [email protected] www.eu-egee.org EGEE-II INFSO-RI-031688 EGEE and gLite are registered trademarks EGEE Enabling Grids for E-sciencE Flagship grid infrastructure project co-funded by the European Commission Now in 2nd phase with 91 partners in 32 countries Main Objectives • Operate a large-scale, production quality grid infrastructure for e-Science • Attract new resources and users from industry as well as sciences EGEE-II INFSO-RI-031688 RISGE - OGF22 2 EGEE – What do we deliver? Enabling Grids for E-sciencE • Infrastructure operation – Sites distributed across many countries Large quantity of CPUs and storage Continuous monitoring of grid services & automated site configuration/management Support multiple Virtual Organisations from diverse research disciplines • Access Middleware CLI – Production quality middleware distributed under open source licence Implements a service-oriented architecture that virtualises resources Adheres to recommendations on web service inter-operability and evolving towards emerging standards • Security API Information & Monitoring Authorization Information & Monitoring Auditing Authentication Data Management Metadata Catalog File & Replica Catalog Storage Element Data Movement Application Monitoring Workload Management Job Provenance Package Manager Computing Element Workload Management Accounting User Support Managed process from first contact through to production usage – – – – Training Expertise in grid-enabling applications Online helpdesk Networking events (User Forum, Conferences etc.) EGEE-II INFSO-RI-031688 RISGE - OGF22 3 Enabling Grids for E-sciencE Archeology Astronomy Astrophysics Civil Protection Comp. Chemistry Earth Sciences Finance Fusion Geophysics High Energy Physics Life Sciences Multimedia Material Sciences …INFSO-RI-031688 EGEE-II 250 sites 48 countries 50,000 CPUs 13 PetaBytes >5000 users >200 VOs >140,000 jobs/day 32% RISGE - OGF22 4 Application Examples Enabling Grids for E-sciencE • EGEE is used to analyze data coming from remote instruments • LHC • Medical Imaging • Earth Observation • and many others EGEE-II INFSO-RI-031688 RISGE - OGF22 5 Accelerating and colliding particles Enabling Grids for E-sciencE Large Hadron Collider • • • 27 km circumference tunnel Due to start up in 2008 40 Million Particle collisions per second – Online filter reduces to a few 100 “good” events per second recorded on disk and magnetic tape at 1001,000 MegaBytes/sec – ~15 PetaBytes per year for all four experiments • Data analyzed by 100s of research groups world wide Mont Blanc (4810 m) Downtown Geneva EGEE-II INFSO-RI-031688 RISGE - OGF22 6 The Data Acquisition Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 RISGE - OGF22 7 Acquisition, First pass reconstruction, Storage Distribution Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 RISGE - OGF22 8 Data Distribution on the Grid Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 RISGE - OGF22 9 Medical Data Manager Enabling Grids for E-sciencE • Objectives DICOM Interface SRM – Expose an standard grid interface (SRM) for medical image servers (DICOM) – Fulfil application security requirements without interfering with clinical practice DICOM server DICOM clients EGEE-II INFSO-RI-031688 Worker Node User Interface RISGE - OGF22 10 Medical Data Registration Enabling Grids for E-sciencE 1. Image is acquired 2. Image is stored in DICOM server 4. image metadata are registered 3. lcg-put AMGA Metadata gfal DICOM server 3a. Image is registered (a GUID is associated) 3b. Image key is produced and registered LFC Hydra Key store EGEE-II INFSO-RI-031688 RISGE - OGF22 11 Earth Science Applications in EGEE Enabling Grids for E-sciencE Flood of a Danube riverCascade of models (meteorology,hydraulic ,hydrodynamic….) UISAV(SK)ESA, UTV(IT), KNMI(NL), IPSL(FR)Production and validation of 7 years of Ozone profiles from GOME Rapid Earthquake analysis (mechanism and epicenter) 50- 100CPUs IPGP(FR) DKRZ(DE)- Data access studies, climate impacts on agriculture Mars atmosphere CETP( FR): EGEE-II INFSO-RI-031688 Specfem3D: Seismic application. Benchmark for MPI (2 to 2000 CPUs) (IPGP,FR) Geocluster for Academy and industry CGG(FR)Data mining Meteorology & Space Weather (GCRAS, RU) Air Pollution model- BAS(BG) Modelling seawater intrusion in costal aquifer (SWIMED) CRS4(IT),INAT(TU), Univ.Neuchâtel(CH)- RISGE - OGF22 12 GOME Enabling Grids for E-sciencE Raw satellite data from the GOME instrument (~75 GB - ~5000 orbits/y) Level 1 ESA(IT) – KNMI(NL) Processing of raw GOME data to ozone profiles. 2 alternative algorithms ~28000 profiles/day Level 2 Meta Database server PosgreSQL geospatial search EGEE-II INFSO-RI-031688 (example of 1 day total O3) IPSL(FR) Validate some of the GOME ozone profiles (~106/y) Coincident in space and time with Ground-Based measurements Visualization & Analyze EGEE environment RISGE - OGF22 13 Summary Enabling Grids for E-sciencE • EGEE provides a unique environment for storing, managing, and analyzing data from remote instruments • Data production, collection, and initial processing is typically out of band – Instruments are not (and will not) be integrated in EGEE – Data is initially stored at domain specific data stores not connected to the Grid • Sensor Grids are being established – E.g. LOFAR – Potential to be more directly integrated • EGEE provides mechanisms to connect data collections to the infrastructure such that they can be used on the infrastructure EGEE-II INFSO-RI-031688 RISGE - OGF22 14