Download Document

ILDG Middleware Status
Chip Watson
ILDG-6 Workshop
May 12, 2005
Outline




Status: small changes from Dec 2005
Quick review of architecture
Minimal implementation facts
Next steps
Status (quick look)

Only a small amount of middleware work
has been done in the last 6 months
– development of new metadata catalog prototype
at Adelaide based on XML database
– modifications to metadata catalog prototype at
Fermilab to conform to new interface
– small amount of work on replica catalog
prototypes at several sites (JLab, Adelaide,
Fermilab)

Architecture remains unchanged
Architecture (review)

Web Services
– Metadata Catalog maps meta data to a
global name
– Replica Catalog maps a global name to one
or more instances
– Storage Resource Manager (optional)
manages a disk, or disk + tape resource
Draft schemas (WSDL) for these services exist
Architecture (review)

File based directories contain...
– Master directory of all collaborations’ MDC, RC
and membership lists, stored as XML files
– Distributed group membership lists (XML)
Initial version of schemas (XML) exist
Implementation View
Master Directory
http://www.lqcd.org/<tbd>.xml
contains for each collaboration:
metadata catalog
replica catalog
group membership
MDC for UKQCD
MDC for USQCD
MDC for Japan
RC for UKQCD
RC for USQCD
RC for Japan
Japan group file
UKQCD group file
USQCD group file
subgroup A file
subgroup B file
file X
MetaData Catalog

ILDG schema defines only a query interface
– multiple query languages (syntax) allowed for
now (no clear winner yet)
– queries map from physics metadata values to
Global File Name (GFN)
– proposed minor modification can also return the
full physics metadata
Minimal Implementation


Master XML directory to be held at
www.lqcd.org/<tbd>.xml
For each collaboration, need at least these:
– MetaDataCatalog (e.g. running at
www.usqcd.org/<tbd>)
– trivial Replica Catalog (does 1:1 name mapping)
– standard web or ftp server to serve files
Getting going...
(or, what must a collaboration do?)
First: Deploy a metadata catalog
1. choose an existing prototype & deploy
2. populate the catalog with qcdml v1.1
compliant documents, with ILDG compliant
GFN’s (global file names)
Note: names must have collaboration name as part of
the string; this name matches the entry name in the
master directory:
gfn://collaboration/local-name
3. request [email protected] to add your MDC to
the master directory on www.lqcd.org
Getting going...
(or, what must a collaboration do?)
Second: Deploy a replica catalog
1. (option 1) write a simple function which maps
your collaboration’s GFN naming convention
into a static URL pointing to the file
(i.e., no database, just string shuffling)
OR
2. (option 2) get / implement a true RC, with
multiple instance tracking (a database)
3. request [email protected] to add your RC to the
master directory on www.lqcd.org
Third: Serve the files (http, ftp, srm, ...)
Nice things to also do...




Deploy a real RC, which can track another
collaboration’s copies of your files
Populate a group membership file, to support group
read/write access (otherwise your collaboration is
relegated to “world” status)
Deploy an SRM (with protocol negotiation) and also
at least one file server that supports parallel
streams (gridftp, bbftp, ...) for higher performance
file retrieval
Implement a web interface to your metadata
catalog
Near Term Expectations



Adelaide will deploy an MDC, RC within the next
few months
USQCD will also try to match this within the next 6
months, but is currently distracted with getting
machines into production
others have not committed yet
Middleware Working Group
Near Term Task List



Approve minor changes to MDC interface
Decide on the URL for, and deploy:
master directory file
master membership file
Collect official CA certificates from all collaborations
and post at www.lqcd.org for all to easily retrieve
(for configuring servers for strongly authenticated operations)
Most Significant
Challenges

Get data into ILDG compliant format
– create or automate creation of metadata
compliant with qcdml1.1
– write files in ILDG format (or write translation
program for on-the-fly translation)
will LQCD application developers do this?
or will manpower need to be found for translation
programs?

Get the MDC operational and populated
(other tasks are comparatively easy)
Other Challenges

Manpower to implement a nice user
interface for browsing, and optionally
retrieving files
(once per collaboration, or shared, even
hosted at www.lqcd.org ?)

Manpower to write some simple command
line client tools to be used in workflow
scripting
Goal of reaching an operational status by
June 2006 is still feasible!