Download Towards the FAIR Timing System using White Rabbits

Towards the FAIR Timing System using
White Rabbits
• Towards FAIR
• White Rabbit
• Timing System
17 January 2012
Dietrich Beck, EE(BEL)
Acknowledgements
• GSI Timing Team: Marcus Zweig, Stefan Rauch, Mathias Kreider,
Cesar Prados, Wesley Terpstra, Ralph Bär, Dietrich Beck
former members: Tibor Fleck, Sergio Mauro
• CERN Timing Team: Tomasz Włostowski, Javier Serrano, Maciej
Lipinski, Evangelia Gousiou, Erik van der Bij, Jean-Claude Bau,
Pablo Alvarez
• GSI/EE: Jan Hoffmann, Nikolaus Kurz, Holger Brand, …
17 January 2012
Dietrich Beck, EE(BEL)
Towards FAIR…
17 January 2012
Dietrich Beck, EE(BEL)
17 January 2012
Dietrich Beck, EE(BEL)
23 November 2011
17 January 2012
Dietrich Beck, EE(BEL)
8 December 2011
Bild von heute
17 January 2012
Dietrich Beck, EE(BEL)
12 January 2012
17 January 2012
Dietrich Beck, EE(BEL)
12 January 2012
General Machine Timing System @ FAIR
• parallel execution of beam production chains
• cycles: 20ms to hours
• trigger and sync. equipment actions
• 1 µs precision in 99% of all cases
• few ns precision for kickers
• (ps for rf-systems: BuTiS)
• many rings
• > 2000 devices connected to timing system
• large distances
• robustness: lose at most one message per year
17 January 2012
Dietrich Beck, EE(BEL)
Idea: Timing System Based on Time
Not:
Timing Master
do this
Equipment 1
do this
Equipment 2
1km distance:  5s propagation time
1s precision: requires compensation for cable length 
17 January 2012
Dietrich Beck, EE(BEL)
Idea: Timing System Based on Time
"Oh dear! Oh dear! I
shall be too late!"
Instead:
Timing Master
do this
@ 15:03
???
Equipment 1
do this
@ 15:03
•
•
•

•
do that
@ 15:01
Equipment 2
equipment pre-programmed for autonomous action at a given time
pre-programming via timing-events
required: clock synchronization ns required
distribution of information and execution of action are decoupled !!!
timing-events must be sent “early enough”
• upper bound latency for transmission (e.g. 100 s): real-time!
• lossless transmission: robustness!
17 January 2012
Dietrich Beck, EE(BEL)
White Rabbit Cooking Recipe I: Clocks
• network: Gigabit-Ethernet
• PTP (Precision Time Protocol) IEEE1588-2008
with value t1.. t4 one can
• calculate one-way link delay
• syntonize clock rate by tracking the value
• calculate clock offset
17 January 2012
Dietrich Beck, EE(BEL)
White Rabbit Cooking Recipe I: Clocks
• network: Gigabit-Ethernet
• PTP (Precision Time Protocol) IEEE1588-2008
– free-running oscillators on network nodes
– need to re-sync often: lot’s of traffic, bad for determinism
– only 1-100 s synchronization of clocks
• SyncE (Synchronous Ethernet)
– receiver’s clock recovered from 125MHz carrier
– 8ns precision
• precise phase measurement and adjustment
• clock synchronization with sub-ns precision and low-ps jitter
17 January 2012
Dietrich Beck, EE(BEL)
White Rabbit Cooking Recipe II: Network
multi-layered
WR Switches
• Clock propagation:
• Determinism
–
–
–
–
GPS
System
Timing
Master
“timing events shall arrive early enough”
upper bound latency from timing master to nodes
no handshake: use UDP or raw Ethernet (no TCP)
switches implement cut-through for high-priority messages
• Robustness
– bit errors: Hamming code
– packet loss: Reed-Solomon code
• Redundancy
– redundant links between switches
– link aggregation (in combination with Reed-Solomon)
17 January 2012
Dietrich Beck, EE(BEL)
WR
Nodes
WR Cooking Recipe III: Implementation…
• path asymmetry (two cables) impacts phase compensation
 White Rabbit PTP
– one bidirectional single-mode optical fiber
– different wave lengths for TX/RX: speed differs slightly (dispersion)
– link-delay-model
• relative delay coefficient α depends only on the type of fiber (off-line)
• fixed delays are measured on-line
•
White Rabbit node
VCO 125MHz
VCO 20MHz
FPGA
WR clock!!!
transceiver
host bus interface
17 January 2012
Dietrich Beck, EE(BEL)
Inside the FPGA:
FPGA
within FPGA: Wishbone “System-on-Chip” bus for interconnection of IP cores
17 January 2012
Dietrich Beck, EE(BEL)
Transmission of Timing-Events: Etherbone
• idea: Connect distant Wishbone buses via a network protocol
layer. Ethernet + Wishbone = Etherbone
• direct memory access to devices via White Rabbit network
• used for timing-events
17 January 2012
Dietrich Beck, EE(BEL)
The White Rabbit Project
• development in the frame of CERN’s renovation and the FAIR
projects
• collaboration: CERN, GSI, …, industry partners
• Open HardWare Repository: www.ohwr.org
– Open Source: Software and HDL
– “CERN Open Hardware License”: Hardware
• commercial production and support
“The White Rabbit (WR) project is a multi-laboratory and
multi-company effort to bring together the best of the data
transfer and timing worlds …” (Proc. ICALEPCS 2009)
17 January 2012
Dietrich Beck, EE(BEL)
Timing Systems @ FAIR
General Machine Timing System
• based on White Rabbit
• fairly cheap: € 600,- SPEC board
• sub-ns synchronization
• distribution of
• (clock)
• time-stamps
• timing-events
• BEL group
Bunch phase Timing System (BuTiS)
• one global timing master
• clock derived from BuTiS
Talk T. Fleck, FAIR Technikforum
• one global BuTiS center
Remark:
• invest € 15k per receiver station
• ps precision (100ps/km accuracy)
• distribution of
• clocks!!!
• NO time-stamps at FAIR
• NO timing-events
• HF group
Talk P. Moritz, FAIR Technikforum
If you are interested time-stamps, you need White Rabbit.
If you are interested in the most accurate clocks, you need BuTiS.
If you are interested in both, you need both.
17 January 2012
Dietrich Beck, EE(BEL)
The FAIR Control System (slightly simplified )
Settings
Management
1a: Data supply:
set-values
Front-end
Controller
1b: Data supply:
schedules,
alternatives
Front-end
Controller
Timing Master
2: broadcast timing-events
3. timing receiver
Timing
1.
settings management (via normal network)
a.
b.
2.
3.
preloads set-values to the FECs (set-value can also be a ramp…)
provides schedule to timing-master
according to schedule, timing-master broadcasts timing-events to FECs
(timing-events: time of execution and reference to a set-value)
timing-receiver of FEC: timely triggering of FEC, FEC applies set-value
17 January 2012
Dietrich Beck, EE(BEL)
The FAIR Control System (slightly simplified )
Settings
Management
Data supply:
set values
Front-end
Controller
Data supply:
schedules,
alternatives
Front-end
Controller
Timing Master
timing receiver
Timing
Analogy to performing a classical piece of music:
• composer: settings management
• conductor: timing master
• musicians: FECs
• move of conductor's baton: timing-events - one timing-event may trigger
multiple FECs
17 January 2012
Dietrich Beck, EE(BEL)
Control System (application layer)
API
Post Mortem System
BuTiS
Timing Master
Equipment 2
Equipment 3
Equipment 4
Disclaimer: This is my personal view and may be incorrect and not up-to-date
17 January 2012
Dietrich Beck, EE(BEL)
Beam Interlock System
another link
BuTiS
WR
a link
FESA
Equipment 1
API
API
LSA (settings management)
Equipment ..
Common
Systems
Alarm, Archive,
Logging,
Oscilloscope
Equipment N
BuTiS and WR
17 January 2012
Dietrich Beck, EE(BEL)
What the Timing System is About
• campus wide distribution system for
– timing-events
– time-stamps
– other machine or beam related telegrams
•
•
•
•
not limited to accelerator controls
available to “everyone”
typically, data is propagated top-down
other traffic only in exceptional cases
– kicker control
– bunch-to-bucket transfer
– beam interlock notification
17 January 2012
Dietrich Beck, EE(BEL)
What the Timing System is NOT About
•
•
•
•
general purpose communication
transmission of bulk data
personnel protection system
things not related to machine timing
Exceptions only in well justified cases
• FECs not connected to FESA
• Info-Service
• part of machine protection system
17 January 2012
Dietrich Beck, EE(BEL)
Next Steps…
@GSI: Set-up a first network 2012
•
•
•
•
usage in a field environment
investigation of stability and long-term effects
connection between new and existing timing system
first tests with existing machines at GSI
17 January 2012
Dietrich Beck, EE(BEL)
…Next Steps
@GSI/CERN: WR starter kit (timing system demo) spring 2012
@Saclay: Low energy part of p-linac: first WR based timing system provided
by GSI (early 2013)
@CERN:
• Installation of WR based timing system at AD ring (2013)
• Installation of WR link between CERN and Gran Sasso (2012)
17 January 2012
Dietrich Beck, EE(BEL)
Status
• Slovenia: In-Kind contribution timing receivers. Receivers and
FPGA building blocks for FAIR timing system specified.
• Germany/GSI: In-Kind contribution General Machine Timing
System: specification until March 2012.
•
•
•
•
•
•
White Rabbit PTP link between two nodes established
sub-ns synchronization with a jitter of hundred ps
propagation of WR PTP over three layers of switches
FPGA building blocks for simple timing system exists
integration work in progress
planning of WR network cabling in progress
• First WR nodes developed by EE
17 January 2012
Dietrich Beck, EE(BEL)
Further reading
• www.ohwr.org
• https://wwwacc.gsi.de/wiki/Timing/TimingSystemDocumentation
(search for “FAIR timing system”)
17 January 2012
Dietrich Beck, EE(BEL)
Moersbacher grund
17 January 2012
viewEE(BEL)
towards
Dietrich Beck,
SIS 100, 23 November 2011