Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
COMP_3:Grid Interoperability and Data Management CC-IN2P3 and KEK Computing Research Center FJPPL workshop@LAPP, Annecy June 15, 2010 Members (2010) Japan(KEK) M. Nozaki T. Sasaki Y. Watase G. Iwai Y. Kawai S.Yashiro Y. Iida France(CC-IN2P3) D. Boutigny G. Rahal S. Reynaud F. Hernandez J.Y. Nief Y. Cardenas P. Calvat FJPPL 2010 2 Activities in 2009-2010 • Cooperative development of SAGA and iRODS – Please see following slides • Workshop at Lyon on February 17 – Status report on each side – SAGA and iRODS development discussion – 3 Japanese visited Lyon • FJPPL+other KEK budget • One had to cancel the trip because the person was suspected to be influenced by a swein flu FJPPL 2010 3 Common concern • GRID interoperability – How we build the world wide distributed computing infrastructure in HEP and related fields? • Different middleware are deployed and operated in the different region • Data handling in smaller experiments – Should be simple, but efficient enough FJPPL 2010 4 Virtualization of Grid/cloud resources and Grid interoperability SAGA FJPPL 2010 5 International collaboration and computing resource e-Science infrastructures are developed and operated independently and not compatible each other Japan:NAREGI Europe: gLite(EGEE), ARC, UNICORE Local computing resource How they can share the resources? How they can develop the software together? United states: globus and VDT(OSG) Grid Interoperability and SAGA will be the key. FJPPL 2010 6 SAGA • Simple API for Grid Applications – The API to provide the single method to access the distributed computing infrastructure, such as “cloud”, GRID, local batch schedulers and independent local machines. • API definition itself is language independent – This is the technology for a world size collaboration, such as Belle-II or ILC • Different institutes depends on different technologies – There are implementations in two languages • JAVA – JSAGA: CC-IN2P3 – JAVA SAGA • C++ (SAGA-C++): KEK and others – Python and C languages bindings are also available FJPPL 2010 7 The aim of the project • Exchange the knowledge and information • Converge two implementations in the future FJPPL 2010 8 Converging JSAGA and SAGA-C++ a user application another user application the most used SAGA Python Binding Boost-based implementation PySAGA ? JySAGA C Python Jython SAGA C Binding SAGA-C++ SAGA Java Binding JSAGA Java SAGA Java GAT FJPPL 2010 9 Converging JSAGA and SAGA-C++ a user application another user application the most used SAGA Python Binding Boost-based implementation PySAGA JPySAGA JySAGA C Python Jython SAGA C Binding SAGA-C++ SAGA Java Binding JSAGA Java SAGA Java GAT FJPPL 2010 10 Converging all SAGA implementations a user application another user application common SAGA Python Binding (PySAGA ?) Boost-based implementation JPySAGA JySAGA C Python Jython SAGA C Binding SAGA-C++ SAGA Java Binding JSAGA Java SAGA Java GAT FJPPL 2010 11 JPySAGA • Developed by J. Devemy (CC-IN2P3) – based on • Compatible with reference implementations of… – Python (CPython) – SAGA (python binding of SAGA-C++) • First release available for download – namespace, file system and replica functional packages only – execution management functional package will come soon… – https://forge.in2p3.fr/projects/list_files/jpysaga • Will be used by – . to integrate JSAGA into (Distributed Infrastructure with Remote Agent Control) FJPPL 2010 12 Summary of KEK activities related SAGA • This activity is a part of the RENKEI project – RENKEI:Resource Linkage for e-Science – funded by MEXT during 2008-2011JFY • Job adaptors for NAREGI, PBSpro and Torque have been implemented • File adaptors for NAREGI(Gfarm v1 and v2) has been implemented also • File adaptors for RNS and iRODS are under development • Service Discovery for NAREGI will be implemented FJPPL 2010 13 Unified GRID Interface(UGI) Goal: •Hide the differences of underlying middleware from users •Single commands set will work for everything Python Interface (Unified GRID Interface ) SAGA-C++ SAGA File adaptors RNS OGF standards RNS iRODS SAGA Job adaptors gLite NAREGI PBSPro/torque FJPPL 2010 LSF cloud RENKEI-KEK globus 14 Summary of CC-IN2P3 activities related SAGA Latest developments • JSAGA plug-ins for – gLite-LFC, by – Globus GK with Condor for OSG, by – SSH with offline monitoring, by • JSAGA core engine – many improvements (scalability, features…) Next developments • JSAGA plug-ins for – ARC (NorduGrid) – DIET (Decrypton) – Grid Engine (next batch system at CC-IN2P3) • Service Discovery API (SAGA extension) • GridRPC SAGA package – needed for DIET FJPPL 2010 15 Data handling for small size projects IRODS FJPPL 2010 16 What is iRODS? • iRODS is the successor of SRB – Data management software • Meta data catalogue and rule based data management – Considered as a data Grid solution – The project is led by Prof. Reagan Moore of North Carolina University FJPPL 2010 17 iRODS service at KEK HPSS-VFS DB server (ICAT) Postgres server – – – – – – IBM x3650 QX5460 (4 core) Memory 8GB HDD 293.6GB RHEL 5.2 Postgres 8.2.5 iRODS server iRODS server × 4 IBM x3650 QX5460 (4 core) Memory 8GB HDD 293.6GB + 600GB RHEL 5.2 iRODS 2.1 HPSS-VFS client GPFS client HPSS (Tape library) HPSS – – – – – – – TS3500 HPSS 6.2.2.2p 3PB in maximum (3000 vols) 10TB cache disk 10 tape drives 5 movers 2 VFS servers Client tools Client tools i-commands JUX (GUI Application) Davis (Web Application) Client tools JUX (Java Universal eXplorer) Works on Linux, Windows and Mac Looks like windows explorer visually confirm the file structuring copy the files by drag and drop not able to recognize the replicated files not able to handle Japanese character Client tools Davis (A webDAV-iRODS/SRB) running Jetty and Apache on iRODS server Useful for a small laboratory in a university don’t need a special software at client side use only https port not able to upload/download some files at the same time not support parallel transfer KEK wiki page Wiki page in Japanese for end users what is iRODS how to use at KEK how to install how to make rule how to make MS … http://wiki.kek.jp/display/irods/ MLF@J-PARC MLF : Materials and Life Science Experimental Facility raw data 20~50TB/year raw rawdata data in each groups storage simulated raw data data raw data raw data raw rawdata data Use case Scenario Data preservation and distribution for MLF groups Storage @J-PARC Raw data is used once After processing, move to KEK storage Storage@KEK raw data raw data raw rawdata data simulated raw data data raw data simulated raw data data raw data Simulated data can be accessed from collaborators Replicate between J-PARC and KEK After a certain term, delete from J-PARC Keep it forever at KEK From J-PARC to Collaborators J-PARC (Tokai) iRODS Server Storage KEK (Tsukuba) iCAT Collaborators (Internet) Web Client iRODS Server iRODS Client iRODS Client Data Server HPSS Client(?) Storage HPSS Rules and Micro-services Main Rules 1. All created data should be replicated to the KEKCC storage 10 min later. 2. All row data older than 1 week should be removed from the JPARC storage, with checking the existence of their replicated data in the KEKCC storage before removing. 3. All simulated data should be removed in the same way but the period of time can be changed by each research group. Rules and Micro-services Created a new micro-service To detect the files matched with the specified age. Implemented by Adil Hasan at University of Liverpool Other experiments use the different rule Send the files for successful runs only Check file sizes and age Speed Performance Data transfer between KEK and J-PARC KEK J-PARC Client pftp put: 26MB/s pftp get: 33MB/s HPSS work server iRODS server HPSS scp: 24MB/s scp: 4MB/s ssh iput: 43MB/s iget: 40MB/s iRODS New server setup Set up parallel iRODS servers iRODS iRODS server server iRODS-A iRODS-B iRODS-C iRODS-D iRODS-B iRODS-A iRODS-D iRODS-C : : : : Before March: running 1 iRODS on 2 machine (active & standby) Now: running separate iRODS on each machine (backup each other) run the iRODS for each experiment in order to change the writing user to HPSS for each experiment group in order to avoid the influence of the congestion of other experiment groups iRODS setup @ CC-IN2P3 In production since early 2008. 9 servers: – 3 iCAT servers (metacatalog): Linux SL4, Linux SL5 – 6 data servers (200 TB): Sun Thor x4540, Solaris 10. Metacatalog on a dedicated Oracle 11g cluster. HPSS interface: rfio server (using universal MSS driver). Use of fuse-iRODS: – For Fedora-Commons. – For legacy web applications. TSM: backup of some stored data. Monitoring and restart of the services fully automated (crontab + Nagios + SMURF). Automatic weekly reindexing of the iCAT databases. Accounting: daily report on our web site. iRODS usage: prospects Starting: – Neuroscience: ~60 TB. – IMXGAM: ~ 15 TB ( X and gamma ray imagery). – dChooz (neutrino experiment): ~ 15 TB / year. Coming soon: LSST (astro): – For the IN2P3 electronic test-bed: ~ 10 TB. – For the DC3b data challenge: 100 TB ? Thinking about a replacement of light weight transfer tool (bbftp). communities: High Energy physics, astrophysics, biology, biomedical, Arts and Humanities. iRODS contributions Scripts: – Test of icommands functionnalities. icommand: – iscan (release 2.3): admin command. Micro-services: – Access control: flexible firewall. – Msi to tar/untar files and register them in iRODS. – Msi to set ACLs on objects/collections. Universal Mass Storage driver. Miscealeneous (related to the Resource Monitoring System): – Choose best resource based on the load. – Automatic setup of status for a server (up or down). JUX: Java Universal eXplorer Provide a single GUI for accessing the data on the GRID. JUX tries to be intuitive and easy to use for non-expert users: – use context menus, drag-and-drop… – close to widely used explorer (i.e. Windows explorer) Written in Java by Pascal Calvat. Based on the JSAGA API developed at ccin2p3 by Sylvain Reynaud. JSAGA provides the data management layer: – Protocols: srb, irods, gsiftp, srm, http, file, sftp, zip… – SRB and iRODS plugins are using Jargon. – Can add a plugin easily for a new protocol. JSAGA provides security mechanisms: – Globus proxy, VOMS proxy, Login/Password, X509 JUX: Java Universal eXplorer Download: https://forge.in2p3.fr/wiki/jux iRODS overall assessement iRODS is becoming more and more popular in IN2P3 community and beyond. Very flexible, large amount of functionnalities. Can be interfaced with many different technologies (no limit): – Cloud, Mass Storage, web services, databases, …. Able to answer a vast amount of needs for our users community. Lot of projects = lot of work for us ! Goal for this year: ~ x00 TB (guess: > 300 TBs). Should reach PB scale very quickly. FJPPL 2010 36 FJKPPL? • CC-IN2P3, KISTI Super Computing Center and KEK Computing Research Center are agreed to build the three points collaboration – We share the common interests on Grid computing – We will discuss what we will do together • The same effort is done in BIO_1 also FJPPL 2010 37 SUMMARY FJPPL 2010 38 Summary • CC-IN2P3 and KEK-CRC are working to solve the common problems in Grid computing mostly independently, but interactively and complementary – SAGA as the solution for Grid interoperability – iRODS as the solution for data management in smaller size projects • Long term collaboration has a benefit – For KEK. CC-IN2P3 is very strong partner who provides useful software tools FJPPL 2010 39