Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Introduction to DAQ Architecture - Indico
Document related concepts
Transcript
Introduction to DAQ Architecture Niko Neufeld CERN / IPHE Lausanne Features • Two types of data flows on the same network • • • One shared farm • • • • Level 1: 5.5 kB @ 1 MHz HLT: 40 kB @ 40 kHz subdivided in ~ 90 sub-farms contains ~ 2000 CPU nodes (single CPU) Everything connected by Gigabit Ethernet (1000 BaseT) – use IP as transport protocol Pure push protocol, with back-pressure • local: Ethernet flow-control • global: throttle signal via TFC LHCb DAQ Architecture Level-1 Traffic 126-224 Links 44 kHz 5.5-11.0 GB/s Front-end Electronics FE FE FE FE FE FE FE FE FE FE FE FE TRM 62-87 Switches Switch Switch Switch Switch Switch Storage System L1-Decision SFC SFC 94-175 SFCs Switch Switch Switch Mixed Traffic Sorter TFC System 94-175 Links 5.5-10 GB/s SFC HLT Traffic 29 Switches 32 Links Readout Network Level-1 Traffic 323 Links 40 kHz 1.6 GB/s Multiplexing Layer 64-137 Links 88 kHz Gb Ethernet HLT Traffic CPU CPU CPU CPU CPU CPU CPU CPU CPU SFC SFC Switch Switch CPU ~2000 CPUs CPU CPU CPU CPU CPU CPU Farm Front-end board • • • • • • Most FE boards are of the same type: TELL1 connect to the detector receive L0 and L1 trigger from TTC format data for L1 and HLT and sends to DAQ send throttle signal to TFC if buffers threaten to overflow Some (e.g. RICH) do not participate in L1 FE FE A-RxCard A-RxCard PP-FPGA PP-FPGA L1B L1B FE FE O-RxCard PP-FPGA PP-FPGA L1B L1B Sync Li nk-FPGA ECS ECS TTCrx TTC FEM RO-Tx L1T HLT L0 andL 1 Thro ttl e Properties relevant for simulation • • • • • Latencies: zero suppression, … Adjustable water marks (for throttle signals) Separate data-paths for L1 and HLT Buffer sizes (L1-buffer, possibly HLT formatting buffer) should be adaptable to final specifications / functionality of TELL1 FE FE A-RxCard A-RxCard PP-FPGA PP-FPGA L1B L1B FE FE O-RxCard PP-FPGA PP-FPGA L1B L1B Sync Link-FPGA ECS ECS TTCrx TTC FEM RO-Tx L1T HLT L0 andL 1 Throttl e L0 Decision Uni/t Generator TFC (simplified) • Readout Supervisor TFC switch L0 throttle switch L1 Decision Sorter L1 throttle switch • TTCtx TTCtx TTCoc TTCoc TTCoc TTCrx TTCrx TTCrx TTCrx L1 FE VELO L1 FE L0 throttle OR TTCrx TTCrx L1 throttle OR L1 FE VELO L1 FE TTCoc L0 FE VELO VELO VELO L0 FE L0 L0FE FE L0 Throttle OR L0 FE VELO VELO VELO L0 FE L0 L0FE FE TTCrx TTCrx TTCtx TTCrx TTCrx TTCrx TTCrx L1 Throttle OR TTCtx Only blue parts need to be simulated Decision sorting and RS only complex devices with several delays, buffers, etc… L1 Decision Sorter • • • L1 decisions must be sorted when they arrive from the farm L1 Decision Sorter delivers sorted decisions to Readout Supervisor L1DS checks for time-outs of decisions Switching Network • • • • • • • • Parameterised switch model Nice to have parameters, which can be measured (in our test-bed) Mostly buffers and latency Possibly back-pressure (flow-control) on Ethernet links Support for various frame sizes (MTUs) Routing / switching on the basis of Ethernet or IP Much more detail in Jean-Pierre’s presentation Aggregation switches should be included in simulation they add a “hop”, possibly “smear” arrival times at the main switch Subfarm Controller • • Performs event-building, checks for corruption, time-outs, etc. Level-1 events: • if free worker CPU forward, set timer for maximal • • • • processing time if decision received in time forward to Level 1 decision sorter, else send default decision (programmable) if no free worker CPU, buffer until either • time-out (discard) • CPU free – send onwards with time-penalty events age in the SFC buffer HLT events: • send to free CPU nodes buffer several events Can issue throttle signal (via ECS slow!) Farm node • • • • • • • • Runs algorithms for L1 and HLT Generates time-out (when algorithm does not terminate) Buffers several HLT events Has at maximum one L1 event at any given time For HLT forwards events to SFC For L1 forwards decision to SFC Can fail, loop, time-out... Communicates with SFC with a token-scheme Summary • • • • • The DAQ system is designed to consist of simple components of well-defined functionality Most components will have a one-to-one correspondence in the simulation Details of TFC system irrelevant Details of latencies and buffers in FE-boards and farm very important The system is scalable. Upgrades will add data to the L1 stream (can double the total required band-width)
