Download BaBar Tier A @ CC-IN2P3

BaBar data distribution using the Storage
Resource Broker (SRB)
Jean-Yves Nief, CC-IN2P3
Wilko Kroeger, SCCS/SLAC
Adil Hasan, CCLRC/RAL
HEPiX, SLAC
October 11th – 13th, 2005
BaBar overview.
• Large amount of data produced by BaBar: > 800 TB.
• Using the collaboration Computing Centres:
– Data (re)processing (prompt reco, skimming).
– Simulation production.
– Data analysis  allow foreign collaborators to access data.
• 6 Tier A:
– SLAC stores all the data + simulation: central repository.
– Computing tasks divided among all the Tier A sites.
HEPiX Conference, SLAC, October
11th-13th 2005
2
BaBar data organization.
SLAC
CC-IN2P3 Padova
RAL
CNAF
GridKa
Data
processing
Data
analysis
+ simulation production (~20 sites involved)
 Lots of data movement
HEPiX Conference, SLAC, October
11th-13th 2005
3
Requirements.
• Short turn-around between data production and
analysis.
• Data must be transfered efficiently and quickly to
Computing Centres.
• Data transfer tools must be:
–
–
–
–
–
Reliable.
Robust.
Automated.
Scalable.
Easy to use.
HEPiX Conference, SLAC, October
11th-13th 2005
4
SLAC & CC-IN2P3.
• First analysis centres opened to the BaBar
collaboration.
• Both held complete copies of data (Objectivity).
• Now only SLAC old a complete copy of the data.
• Natural candidates for testing and deployment of
grid middleware.
• SRB: good candidate for data distribution.
HEPiX Conference, SLAC, October
11th-13th 2005
5
What’s SRB ?
• Storage Resource Broker: developed by SDSC
(San Diego).
• Provides an uniform interface to heterogeneous
storage system (disk, tape, databases) for data
distributed in multiple sites.
• Who is using SRB ?
– Biology, biomedical applications (e.g: BIRN).
– Astrophysics, Earth Sciences (e.g: NASA).
– Digital libraries (e.g: NARA).
• Used world wide: USA, Europe, Asia, Australia.
HEPiX Conference, SLAC, October
11th-13th 2005
6
Some SRB features.
• Files organized in a logical space (directories,
subdirectories …): logical view totally decoupled
from the physical location of the files.
• Replica management.
• Search for files based on their attributes (metadata).
• Fine granularity for SRB users’ACLs.
• Interfaced with various Mass Storage System:
– Access to HPSS transparent to users.
– Small files management.
• And many more…
HEPiX Conference, SLAC, October
11th-13th 2005
7
SRB architecture.
• 1 zone:
– 1 SRB/MetaCatalog server: contains list of files, physical
resources, users registered.
– several SRB servers to access the data at their physical
location.
MCAT
SRB
SRB
(3)
Site 1
(4)
Site 3
(2)
(1)
SRB
test1.txt
Application
(asking for test1.txt)
Site 2
HEPiX Conference, SLAC, October
11th-13th 2005
8
SRB BaBar architecture (2001-2004).
1 Zone (SLAC)
(2)
(1)
SRB
(3)
client
HPSS/SLAC
SRB
SRB
MCAT
SLAC
(Stanford, CA)
HPSS/Lyon
client
CC-IN2P3 (Lyon)
SRB
HEPiX Conference, SLAC, October
11th-13th 2005
9
Extra details.
• Hardware:
– SUN servers (Solaris 5.8, 5.9): NetraT 1405, V440.
• Software:
– Oracle 8, 9 for the SLAC MCAT (530,000 files registered).
– Multithreaded client application (Python).
• Advantages:
– All the 3 steps controlled by the client application in //  no
latency due to prestaging.
– Small amount of cache needed on both sides (a few GBs).
• Drawbacks:
– No registration into the MCAT of the files stored in Lyon.
• Performance:
– Up to 3 TB/day from tape to tape.
HEPiX Conference, SLAC, October
11th-13th 2005
10
1 Terabyte/day
ESNET Traffic with one server on both sides (April 2004).
HEPiX Conference, SLAC, October
11th-13th 2005
11
SRB BaBar architecture (since 2005).
2 Zones (SLAC +
Lyon)
(2)
(1)
SRB
(3)
SRB
HPSS/SLAC
SRB
SRB
MCAT
SLAC
(Stanford, CA)
MCAT
SRB
HPSS/Lyon
SRB
CC-IN2P3 (Lyon)
HEPiX Conference, SLAC, October
11th-13th 2005
12
Extra details.
• Hardware:
– Still the same.
• Software:
– Oracle 10g for the SLAC MCAT.
– Oracle 9i for the Lyon MCAT (migration to 10g foreseen).
• MCATs synchronization: only users and physical
resources.
• Comparison of the MCATs contents to transfer the data.
• Advantage:
– External client can pick up data from SLAC or Lyon without
interacting with the other site.
HEPiX Conference, SLAC, October
11th-13th 2005
13
Overall assessment.
• A lot of time saved for developping applications thanks
to the SRB.
• Transparent access to data:
– Very useful in an hybrid environment (disk, tape).
– Easy to scale the service (adding new servers).
– Not dependent of physical locations changes in the
application.
•
•
•
•
Fully automated procedure.
Easy for SLAC to recover corrupted data.
132 TB (232,000 files) shipped to Lyon.
Up to 3 TB /day from tape to tape (minimum latency).
HEPiX Conference, SLAC, October
11th-13th 2005
14
Future plans.
• Possibly include RAL as part of the system:
– Would reduce the network activity on the SLAC &
ESNET.
• Some tests:
– Assessment of web services technologies.
RAL
SRB
HPSS/Lyon
HPSS/SLAC
CC-IN2P3
SLAC
SRB
SRB
HEPiX Conference, SLAC, October
11th-13th 2005
15
Summary and outlook.
•
•
•
•
•
SRB used by BaBar for more than 3 years.
Data shipment fully automated.
Lightweight development to make it work.
Easy administration.
More servers on both side:
– Increase peak transfer rate to 5 TB/day.
– Hundreds of TB foreseen.
• Scalability: the sky is the limit… well and the
money too…
• Aiming at including RAL.
HEPiX Conference, SLAC, October
11th-13th 2005
16
Never forget…
Cannot be achieved without the work of:
– The network and system admins.
– The database admins.
Many thanks to them!
HEPiX Conference, SLAC, October
11th-13th 2005
17