Download CHEP 2000 - 07 Feb 2000

The ALICE DAQ:
Current Status and Future Challenges
P. VANDE VYVRE
CERN-EP/AID
ALICE DAQ
•
•
•
The original and updated requirements
– The original requirements
– The updated requirements:
higher multiplicity, addition of detector
Future challenges
– The Region-Of-Interest readout
– Online filtering
– Enhanced data compression
– The new architecture
Current prototyping status
– The ALICE DATE
– Data transfer, Sub-event building and event building
– Mass Storage System and Permanent Data Storage
– The ALICE Data Challenge
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
2
P. Vande Vyvre CERN/EP
ALICE DAQ
•
•
•
The original and updated requirements
– The original requirements
– The updated requirements:
higher multiplicity, addition of detector
Future challenges
– The Region-Of-Interest readout
– Online filtering
– Enhanced data compression
– The new architecture
Current prototyping status
– The ALICE DATE
– Data transfer, Sub-event building and event building
– Mass Storage System and Permanent Data Storage
– The ALICE Data Challenge
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
3
P. Vande Vyvre CERN/EP
Original requirements: event size
Detector
Inner Tracking System
Time Project Chamber
Time-Of-Flight
Photon Spectrometer
High Momentum Particle Identification
Dimuon Forward Spectrometer
Photon Multiplicity Detector
Trigger System
Min. Event Size Max Event Size
0.14
0.28
1.50
1.50
0.16
0.16
30.00
36.00
0.18
0.18
0.02
0.02
0.12
0.12
0.15
0.15
0.03
0.12
0.12
0.12
ITS Pixel
ITS Drift
ITS Strips
TPC
TOF
PHOS
HMPID
MUON
PMD
TRG
Total
32.42
38.65
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
4
P. Vande Vyvre CERN/EP
Updated requirements: event size
Higher multiplicity: increased TPC event size
Transition Radiation Detector (TRD) added to ALICE
Detector
Inner Tracking System
Time Project Chamber
Transition Radiation Detector
Time-Of-Flight
Photon Spectrometer
High Momentum Particle Identification
Dimuon Forward Spectrometer
Photon Multiplicity Detector
Trigger System
ITS Pixel
ITS Drift
ITS Strips
TPC
TRD
TOF
PHOS
HMPID
MUON
PMD
TRG
Total
Min. Event Size
0.14
1.50
0.16
56.00
8.00
0.18
0.02
0.12
0.15
0.03
0.12
Max Event Size
0.28
1.50
0.16
75.90
8.00
0.18
0.02
0.12
0.15
0.12
0.12
66.42
86.55
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
5
P. Vande Vyvre CERN/EP
Original requirements: data throughput
Physics
Hadronic
Charm
Dimuon
Trigger
Central
Min. Bias
Central (C)
Min. Bias (MB)
C+Dimuon
Central
Total
Detectors
Max Event Event
Data
Size
Rate Throughput
(MBytes) (Event/s.) (MBytes/s)
All
87
2
All
22
2
All
39
40
1560
All
10
40
400
Pixel, Muon,PMD
0.6
1000
600
PHOS,TRG
Pixel, Muon,PMD
39
40
PHOS,TRG
2560
Conservative data compression to reduce the data throughput to 1.25 GBytes/s.
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
6
P. Vande Vyvre CERN/EP
Updated requirements: data throughput
Physics
Hadronic
Charm
Dielectron
Dimuon
Trigger
Central
Min. Bias
Central (C)
Min. Bias (MB)
C+Dielectron
MB+Dielectron
C+Dimuon
Central
Total
Detectors
Max Event Event
Data
Size
Rate
Throughput
(MBytes) (Event/s.) (MBytes/s)
All
87
2
All
22
2
All
87
20
1740
All
22
20
440
All
87
200
17400
All
22
200
4400
Pixel, Muon,PMD
0.6
1000
600
PHOS,TRG
Pixel, Muon,PMD
87
20
PHOS,TRG
24580
Conservative data compression and event rate reduction insufficient
The TRD allows new types of online processing
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
7
P. Vande Vyvre CERN/EP
ALICE DAQ
•
•
•
The original and updated requirements
– The original requirements
– The updated requirements:
higher multiplicity, addition of detector
Future challenges
– The Region-Of-Interest readout
– Online filtering
– Enhanced data compression
– The new architecture
Current prototyping status
– The ALICE DATE
– Data transfer, Sub-event building and event building
– Mass Storage System and Permanent Data Storage
– The ALICE Data Challenge
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
8
P. Vande Vyvre CERN/EP
Future Challenges 1
•
For dielectron events
– Region-Of-Interest identified by the TRD
– Could be used for
– Region-Of-Interest readout
Electron tracks in the TPC and TRD detectors
Target: Reduce the event size from 80 to 5 MBytes
– Online Filtering
Refine dielectron L1 trigger by a software filter
Target: Reduce the event rate from 200 to 20 Hz
•
Requires limited CPU power.
Current estimate done with STAR data: 40 kCU
Physics simulation and DAQ prototyping are starting
•
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
9
P. Vande Vyvre CERN/EP
Future Challenges 2
•
For central and min. bias events
Enhanced data compression for the TPC data
Data compressed by applying to the raw data the following conversion:
– Clusters finder
– Local tracking
– Raw data converted into:
• Parameters of a local track model
• Distances of the raw data clusters with the local track model
•
Requires massive CPU power.
Current estimate done with STAR data: 400 kCU
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
10
P. Vande Vyvre CERN/EP
Updated requirements: data throughput
Physics
Hadronic
Charm
Dielectron
Dimuon
Trigger
Central
Min. Bias
Central (C)
Min. Bias (MB)
C+Dielectron
MB+Dielectron
C+Dimuon
Central
Total
Detectors
Max Event Event
Data
Size
Rate Throughput
(MBytes) (Event/s.) (MBytes/s)
All
87
2
All
22
2
All
87
20
1740
All
22
20
440
Partial TPC/TRD
4.6
20
92
Partial TPC/TRD
1.2
20
24
Pixel, Muon,PMD
0.6
1000
600
PHOS,TRG
Pixel, Muon,PMD
87
20
PHOS,TRG
2896
Partial readout for dielectron triggers
Online filtering
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
11
P. Vande Vyvre CERN/EP
Architecture upgrade
Inner
Tracking
System
Particle
Identification
Photon
Spectrometer
Muon
Tracking
Chambers
FEE
FEE
Trigger Dete ctors: - Micro Channel Plate
- Zero-Degree Calorimeters
- Muon Trigger Chambers
Trigger Decisions
- Transition Radiation Dete ctor
Dete ctorbusy
Interaction rate
FEE
FEE
8 103 Hz Pb-Pb
FEE
105 Hz p-p
Trigger
Data
1300 Hz Pb-Pb
L0 Trigger
1200 Hz p-p
1.2 us
FEDC
Time Projection Chamber
FEE
FEE
FEE
DDL
FEDC
FEDC
RORC
RORC
RORC
FEDC
FEDC
RORC
RORC
RORC
RORC
RORC
RORC
RORC
RORC
RORC
FEDC
RORC
RORC
RORC
L3 Filter
& Partial
RORC
RORC
RORC
L1 Trigger
L3 Filter
& Partial
L2 Trigger
LDC
LDC
LDC
LDC
LDC
LDC
1100 Hz Pb-Pb
1000 Hz p-p
5.5 us
20 Hz central + 20 Min. Bias +
1000 Hz dimuon + 200 Hz diel.
500 Hz
5.5 - 100 us
Pb-Pb
p-p
EBL
EDM
Switch
L3 Global
Trigger
2.5-5 GByte s/sec. Pb-Pb run
500 MBytes/sec. p-p run
FCL
GDC
FEE: Front-End Electronic
DDL: Dete ctorData Link
RORC: Read-Out Receiver Card
FEDC: Front-End Digital Crate /Computer
EBL: Event Building Link
LDC: Local Data Concentrator
GDC: Global Data Collector
EDM: Event Destination Manager
TDL: Trigger Distribution Link
FCL: Flow Control Link
PDS: Permanent Data Storage
STL: Storage Link
GDC
GDC
GDC
L3
L3
L3
L3
L3
L3
L3
L3
L3
L3
L3
L3
L3
L3
L3
L3
STL
1250 MByte s/sec. Pb-Pb run
100 MByte s/sec. pp run
Switch
Nov-99
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
PDS
PDS
PDS
PDS
12
P. Vande Vyvre CERN/EP
ALICE DAQ
•
•
•
The original and updated requirements
– The original requirements
– The updated requirements:
higher multiplicity, addition of detector
Future challenges
– The Region-Of-Interest readout
– Online filtering
– Enhanced data compression
– The new architecture
Current prototyping status
– The ALICE DATE
– Data transfer, Sub-event building and event building
– Mass Storage System and Permanent Data Storage
– The ALICE Data Challenge
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
13
P. Vande Vyvre CERN/EP
The ALICE DATE
•
•
•
•
DATE: Data Acquisition and Test Environment
Software framework for the ALICE DAQ development & prototyping
Cover multiple needs with one common DAQ system
– Need for a system to develop the DAQ
– Need for a system for detector tests (lab and test beams)
– Need for a framework to develop readout and monitoring programs
ALICE DATE
– Data flow: multiple LDCs, multiple GDCs
– Run control, error reporting, bookeeping
– Common software interfaces for readout, online monitoring with ROOT
– Independent from physical layers: LDC I/O bus, event building network,
GDC machine
Used by ALICE test beams, NA57 and COMPASS
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
14
P. Vande Vyvre CERN/EP
Data transfer
Front-end electronics
Detector
Data
Links
Read
Out
Receiver
Card
Source
Interface
Unit
Destination
Interface
Unit
P2 Cavern
DDL SIU
Optical Fibre
200 meters
DDL DIU
RORC
LDC
Local
Data
Concentrator
P2 Access
shaft
FrontEnd
Digital
Crate/Computer
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
15
P. Vande Vyvre CERN/EP
Detector Data Link (DDL)
•
•
One-to-one data communication link (FEE and DAQ)
Main Requirements
– Common interface between detector front-end electronics and DAQ
• Single hardware & software to develop and maintain
• Define interface soon to allow all the teams to work in parallel
•
– Raw data transfer to DAQ
– Data blocks download to FEE
– Cover the distance from the detector in the cavern to the ALICE
computing room in the access shaft (200 m.)
Implementation
– Optical link
– Off-The-Shelf components Gbit/s opto-electronic
– Prototypes integrated with DATE.
– Tests with detectors will start this year
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
16
P. Vande Vyvre CERN/EP
DDL SIU prototype
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
17
P. Vande Vyvre CERN/EP
DDL DIU prototype
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
18
P. Vande Vyvre CERN/EP
RORC prototype
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
19
P. Vande Vyvre CERN/EP
Sub-event building
•
Many-to-one data collection inside a crate or a computer
– Collect data from several data sources over computer I/O bus
– Assemble these data as one sub-event from a fraction of detector
– Can work as a standalone DAQ system
•
Data sources
– Current data sources are electronics cards in VME or Camac
– Current sub-event building done in software by a DATE program
(Readout) running in on the processor in a VME board
– In the future: data sources will be DDL links
– First RORC prototypes done in VME form-factor (1 VME board)
– Second RORC prototype will be in PCI form-factor (1 PC adapter)
– Following closely the industry evolution (PCI, PCIX, NGIO, FIO, SIO)
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
20
P. Vande Vyvre CERN/EP
Event building
•
•
Event building network was initially a specially demanding application
Today’s dominant trends in computing and networking industry:
– Internet is the strongest incentive for always higher bandwidth
– Commodity computing and networking is driving the industry
– Switches replace shared media
– Ethernet is the standard LAN media, TCP/IP is the standard protocol
– Ethernet’s successors have the advantage of the existing installed base
•
Event building network is similar to the backbone of a site like CERN:
Ports: 15000 on Eth10, 2000 sur Eth 100, 30 on Eth 1000,
Switches: 100 Eth100 or Eth1000, central bw 60 Gbps
•
Work focus: can we use standard LAN media and protocol and how ?
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
21
P. Vande Vyvre CERN/EP
Mass Storage System
ALICE
DAQ
GDC
Disk
Arrays
MSS
core
server
MSS
GDC
GDC
Network
MSS
Meta
data
Tape
Arrays
Client
System
Client
System
NFS
Servers
DFS
Servers
Main
Data
Servers
Secondary
Data
Servers
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
22
P. Vande Vyvre CERN/EP
Mass Storage System
– Isolate the DAQ and computing architecture from
• the problems of physical data recording, CDR, volume handling etc
• the technology evolution in the storage area (magnetic/optical, robotic etc)
– Provide a logical structure to the storage infrastructure
• For example a file system:
/hpss/alice/2005/pbpb_run/run00001.raw
/hpss/alice/2005/pbpb_run/run00002.raw
...
– The MSS currently used by ALICE is HPSS
but HPSS expensive and supported on a limited set of platforms,
but MSS market is small
– Other systems used in future prototypes: CASTOR, EUROSTORE
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
23
P. Vande Vyvre CERN/EP
Permanent Data Storage
•
•
•
Multiple parallel streams of magnetic tapes
– By LHC startup:
• Standard drive should achieve 30-50 MBytes/s
• Standard capacity should be 100-200 GBytes
• 40 drives with 80 (dis)mounts/hour in total
Current CERN installation
– Drive bandwidth 10 MBytes/s
– Tape capacity 50 GBytes
– 45 drives
– 6 silos of 6000 cartridges of 50 GB: 1.8 PBytes capacity
Feasible but expensive solution.
Ratio disk storage cost/tape storage cost decreasing rapidly
By LHC time, online disk storage and offline archiving could be cost effective
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
24
P. Vande Vyvre CERN/EP
Prototyping
•
•
•
Development and prototyping progressing in parallel
Prototyping with the ALICE Data Challenge (ADC)
– Combined activity of the ALICE DAQ/ALICE Offline/IT teams
– ADC1: 6 days at 14 MB/s (7 TB ROOT dataset)
ADC2: Large DATE system
– Data sources (18 LDCs)
• 9 Motorola VME + 2 IBM WS
(Test beam area - Hall 887 on Prevessin site)
• 7 Motorola VME + DDL prototypes
(DAQ Lab - Bld 53 on Meyrin site)
– Network: Fast Ethernet switches, gigabit ethernet backbone
– Data destinations (computing center)
• 20 PC/Linux for event building, ROOT I/O formatting, L3 filter
• Central data recording (Target 100 MB/s)
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
25
P. Vande Vyvre CERN/EP
ALICE Data Challenge II
ALICE
Test Beam
Exp. Hall 887
ALICE
DAQ Lab
Bld 53
DATA
SOURCE
DATA
SOURCE
DDL
LDC: Local Data Concentrator
GDC: Global Data Collector
CDR: Central Data Recording
LDC
LDC
LDC
LDC
RORC
RORC
LDC
LDC
LDC
LDC
LDC
CONTROL
SWITCH
MONITOR
SWITCH
Fast Ethernet switch
(3COM3900)
24x100 Base-T
1x1000 Base-T uplink
Gbit Eth
1 Gbit/s
Gbit Eth
1 Gbit/s
Fast Ethernet switch
(3COM3900)
24x100 Base-T
1x1000 Base-T uplink
Computing
Center
Bld 513
SWITCH
SWITCH
GDC
GDC
GDC
GDC
CDR
GDC
GDC
GDC
GDC
GDC
GDC
GDC
GDC
CDR
GDC
GDC
GDC
GDC
CDR
GDC
GDC
GDC
GDC
CDR
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
26
P. Vande Vyvre CERN/EP
Conclusion
•
•
•
•
•
•
The requirements of the ALICE DAQ have evolved a lot
New ways to reduce the huge data volume will be investigated
– Region-Of-Interest readout
– Online filtering
– Enhanced data compression scheme
Development progressing (almost according to schedule)
The prototypes are tested during the ALICE Data Challenges
Future milestones:
– Integration of DDL with detectors
– ALICE Data Challenge II: from DDL to MSS @ 100MB/s
Computing and communication technology evolution positive
Area of concerns: Storage cost and Mass Storage System
The ALICE DAQ : Current Status and Future Challenges
07-Feb-2000
27
P. Vande Vyvre CERN/EP