Download 4 Power - RadioNet

UniBoard
Board Design
Organisatie / Organization
Datum / Date
Auteur(s) / Author(s):
Gijs Schoonderbeek
Sjouke Zwier
ASTRON
Controle / Checked:
ASTRON
Goedkeuring / Approval:
ASTRON
Autorisatie / Authorisation:
Handtekening / Signature
ASTRON
ASTRON-FO-017 2.0
© ASTRON 2009
All rights are reserved. Reproduction in whole or in part is
prohibited without written consent of the copyright owner.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
1 / 21
Distribution list:
Group:
Others:
Andre Gunst
Eric Kooistra
Arie Doorduin
Sjouke Zwier
Albert Jan Boonstra
Document history:
Revision
0.1
Date
Author
Modification / Change
2009-01-12
Schoonderbeek
Creation
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
2 / 21
Table of contents:
1
Introduction .................................................................................................................................................. 7
1.1
1.2
Scope..................................................................................................................................................... 7
System Functionality ............................................................................................................................. 7
2
Board Block Diagram .................................................................................................................................. 8
3
PCB ............................................................................................................................................................. 9
3.1
3.2
4
Power ........................................................................................................................................................11
4.1
4.2
5
Power Estimation .................................................................................................................................12
Implementation ....................................................................................................................................13
FPGA .........................................................................................................................................................15
5.1
5.2
6
Via and Board Thickness ...................................................................................................................... 9
Layer build up ......................................................................................................................................10
Remarks on FPGA selection ...............................................................................................................15
Configuration .......................................................................................................................................15
In and Output Interfaces ............................................................................................................................17
6.1
6.2
6.3
Remarks on optical selection ..............................................................................................................18
XAUI to SFI conversion .......................................................................................................................18
Clock ....................................................................................................................................................19
7
Memory .....................................................................................................................................................19
8
Clock and Control ......................................................................................................................................20
8.1
8.2
9
Clock ....................................................................................................................................................20
Control .................................................................................................................................................20
Test............................................................................................................................................................21
9.1
9.2
Boundary Scan ....................................................................................................................................21
Operation Status information ...............................................................................................................21
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
3 / 21
List of figures:
Figure 1 System Block Diagram ......................................................................................................................... 7
Figure 2 Board overview..................................................................................................................................... 8
Figure 3 UniBoard Block diagram ...................................................................................................................... 9
Figure 4 Via calculation ....................................................................................................................................10
Figure 5 Layer build up for 10 layer (left), 12 layer (mid) and 14 layer (right). .................................................10
Figure 6 Power Supply Block Diagram .............................................................................................................12
Figure 7 Block diagram optical interface ..........................................................................................................18
List of tables:
Table 1 Layer comparison ................................................................................................................................11
Table 2 Total Power Estimation .......................................................................................................................13
Table 3 Circuit protection possibilities ..............................................................................................................14
Table 4 UniBoard POLs ....................................................................................................................................14
Table 5 POL comparison ..................................................................................................................................14
Table 6 FPGA comparison ................................................................................ Error! Bookmark not defined.
Table 7 Optical interconnect options ................................................................................................................18
Table 8 XAUI to SFI selection ..........................................................................................................................19
Table 9 Memory comparison ............................................................................................................................19
Table 10 Ethernet Switch solutions ..................................................................................................................20
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
4 / 21
Terminology:
ADC
BF
bps
BW
DDR
EMI
Firmware
FPGA
GMII
Hardware
HDL
IO
IP
IPC
LOFAR
LUT
MAC
PCB
POL
Pps
PHY
RF
RSP
SGMII
SFP
SFP+
Subband
TBB
XAUI
XGMII
XFP
Analogue to Digital Converter
BeamFormer
Bits per second
BandWidth
Double Data Rate
Electro-Magnetic Interference
Embedded or real-time code that runs on a microprocessor (e.g. written in C)
Field Programmable Gate Array
GbE media independent Interface 8 bits @125MHz
Boards, subracks and COTS equipment
Hardware Description Language
Input Output
Intellectual Property
Association Connecting Electronics Industries ( formally Institute for Interconnecting and
Packaging Electronic Circuits)
LOw Frequency Array
Look Up Table
Multiply and Accumulate, Medium Access, Monitoring and Control, Media Excess Controller
(layer 2 of OSI model)
Printed Circuit Board
Point of Load
Pulse Per Second
physical interface (layer 1 of OSI model)
Radio Frequency
Remote Station Processing (in LOFAR)
Serial Gb Media Idependent Interface
Small From-factor Pluggable transceiver
SFP for 10GbE
Frequency band, unit output of the filterbank
Transient Buffer Board (in LOFAR)
10G attachment Unit Interface (4x 3.125Gbps) interface between MAC and PHY
10G Media Independent Interface
10G Small form-factor Pluggable transceiver
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
5 / 21
References:
[1] Quinten System document
[2] Xilinx White Paper WP285 V1.0, Virtex-5 FPGA System Power Design Considerations, February 14,
2008
[3] Micron Technology Inc., Rev B 8/07 EN, TN-41-01 Calculation Memory System Power for DDR3, 2007
[4] Micron Technology Inc., Rev A 4/08 EN, DDR3 SDRAM RDIMM MT36JSZS1G72PY-8GB Datasheet.
[5] Xilinx UG191 (v3.5) Virtex-5 FPGA Configuration User Guide, October 29, 2008
[6] Altera Volume I Stratix IV Device handbook ver 2.0, Chapter 10 Nov 2008
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
6 / 21
1 Introduction
1.1
Scope
Quinten is a system which can run stand alone and is able to filter, correlate and or beamform data. The
system will be developed based on the experiences with LOFAR and expand the processing capabilities
significantly. The aim is to integrate more bandwidth and processing power per cubic meter. To narrow the
scope of Quinten five key applications are used. One Quinten or multiple Quinten systems should be able to
handle the requirements these applications:
1.
2.
3.
4.
5.
Next generation EVN correlator
Apertif beamformer
Apertif correlator
LOFAR core beamformer
LOFAR core correlator
In this document the board design for the processing board of Quinten, UniBoard is described. More system
information can be found in [6]
1.2
System Functionality
In Figure 1 the block diagram of Quinten architecture is shown. This functionality can be mapped on single
UniBoards, or mapped to multiple boards on a backplane.
Delay
Compensation
Poly fase Filter
Bank
Correlator /
Beamformer
Transpose
Delay
Compensation
Poly fase Filter
Bank
Correlator /
Beamformer
Figure 1 System Block Diagram
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
7 / 21
2 Board Block Diagram
Input
10 GbE
Interfaces
Clock and Sync
Control
Supply
In Figure 2 a simplified block diagram of UniBoard showing the connections can be seen. UniBoard has
10GbE in- and outputs, an ADC input and control and power lines. The memory for the delay compensation
is not drawn in this figure.
UniBoard
10 GbE
Interfaces
Transpose
Parallel input
Figure 2 Board overview
This block diagram is worked out in more detail in Figure 3, only the interconnections from FPGA A0 to A1,
B1, C1 and D1 are draw.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
8 / 21
Parallel Input
Memory Bank
Optical Interface
FPGA
A0
Optical Interface
FPGA
A1
Parallel Input
Memory Bank
Optical Interface
FPGA
B0
Optical Interface
FPGA
B1
Parallel Input
Memory Bank
Optical Interface
FPGA
C0
Optical Interface
FPGA
C1
Parallel Input
Memory Bank
Optical Interface
FPGA
D0
Optical Interface
FPGA
D1
Power
Clock & Control
Mesh between all input and all output fpga’s
8 x Gb Transeiver fpga-to-fpga connection
4 x Gb Transeiver or 1 x 10GbE board-to-board connection
1 x 10GbE IO connection
Figure 3 UniBoard Block diagram
3 PCB
3.1
Via and Board Thickness
For a better estimate on the possibilities of the PCB, the IPC calculator has been used to estimate the
maximal PCB thickness. In Figure 4 the result is shown.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
9 / 21
Figure 4 Via calculation
In this calculation the aspect ration is 7.0:1. For UniBoard we will use a bit more challenging 8.0:1 aspect
ration. With this aspect ration the maximum board thickness is 2.4mm. As shown in the calculator the
maximal current is 0.85A. With 20W in the FPGA and 1V core supply at leased 23 via’s to the power plane
are needed.
A second argument to limit the board thickness to 2.4mm is the connectors used. The maximal board
thickness for a memory connectors is 2.5mm (this is for both SMT and through hole connectors, both have
metal Beveled Pins).
3.2
Layer build up
To analyse the maximum number of layers for UniBoard a 10, 12 and 14 layer build up is taken as starting
point, see Figure 5. When horizontal / vertical routing is used, two adjacent layers can be used for HighSpeed routing. Special care must be taken that not traces in these layers run in parallel, this will introduce
crosstalk, and thus limiting the links capabilities.
1
2
3
H.S
GND
1
2
3
4
Signal
GND
5
6
PWR
PWR
PWR
PWR
7
8
6
7
H.S
H.S
9
10
GND
Signal
8
9
10
11
12
Signal
GND
H.S
H.S
H.S
Prepreg 2116 (110um or 8.7 mil)
Core 10 mil 35/35
4
5
H.S
GND
Signal
GND
H.S
Prepreg 1080 (70um or 5.5 mil)
Prepreg 2166 (110um or 8.7 mil)
Core 10 mil
Core 6 mil
1
2
3
4
H.S
GND
H.S
H.S
GND
5
6
Signal
PWR
7
8
PWR
Signal
9
10
11
12
13
14
GND
H.S
H.S
GND
H.S
Prepreg 1080 (70um of 2.76mm)
Core 6mil
Figure 5 Layer build up for 10 layer (left), 12 layer (mid) and 14 layer (right).
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
10 / 21
In Table 1 a comparison for High-Speed traces for the three layers build-ups are shown. The calculation has
been done for the inner layer (I.L.) and outer layer (O.L.). For all situations the maximal board thickness is
2.4mm (95mil) the copper for all layers is 35um (1 oz, or 1.4mil).
The 10 layer is build up with: 8.7 / 10 / 8.7 / 10 / 8.7 / 10 / 8.7 / 10 / 8.7 (14 mil Cu)
The 12 layer is build up with: 5.5 / 10 / 8.7 / 6 / 5.5 / 6 / 5.5 / 6 / 8.7 / 10 / 5.5 (16.8 mil Cu)
The 14 layer is build up with: 5.5 / 6 / 5.5 / 6 / 5.5 / 6 / 5.5 / 6 / 5.5 / 6 / 5.5 / 6 / 5.5 mill (19mil Cu)
Table 1 Layer comparison
I.L Layer Spacing
I.L High-Speed trace
width
O.L Layer Spacing
O.L High-Speed trace
width
10 Layer
-
12 Layer
5.5 mil
W:7mil S:9mil
14 Layer
5.5 mil
W:5 mil S:8mil
Offset 10 – 19 mil
W:7mil S:10mil
10 mil – 8.7 mil
W:6 mil S: 12 mil
Offset 6 – 11.5 mil
W:4 mil S:8 mil
The 14 layer build-up needs for the high speed traces 4 mil. This trace width is not reliable to fabricate. For
both 10 and 12 layer, 2 layers for high-speed traces can be used and 2 layers for general purpose. The
benefit of the 12 layer with respect to the 10 layer is the better reference for the High-Speed traces. For the
10 layer X-Y routing is necessary, but changing from the horizontal to the vertical layer also changes from a
GND reference to a power reference.
3.3
Board setup
For getting an understanding on the board us two setups are made, one for the SPF+ IO and one for the
CX-4 IO. For the CX-4 situation 3 10GbE are taken as an examples. Both boards have an Ethernet switch
for control and readout of correlator results.
Memory Bank
POL CORE
XAUI
SFI
POL CORE
Memory Bank
XAUI
SFI
4 x SFP+
Connector
FPGA
A0
FPGA
A0
FPGA
A1
XAUI
SFI
FPGA
A1
FPGA
A0
Connector
Memory Bank
XAUI
SFI
4 x CX4
POL CORE
Connector
Memory Bank
POL CORE
Memory Bank
FPGA
A0
XAUI
SFI
Memory Bank
FPGA
A0
Connector
XAUI
SFI
FPGA
B0
4 x SFP+
FPGA
B0
FPGA
B1
XAUI
SFI
FPGA
B1
Connector
Memory Bank
XAUI
SFI
Connector
4 x CX4
POL CORE
Memory Bank
POL CORE
Memory Bank
XAUI
SFI
Connector
Memory Bank
XAUI
SFI
FPGA
C0
4 x SFP+
FPGA
C1
FPGA
C0
Connector
FPGA
C1
XAUI
SFI
FPGA
A0
Memory Bank
XAUI
SFI
RJ45
Connector
Memory Bank
POL CORE
XAUI
SFI
POL CORE
Memory Bank
Connector
4 x SFP+
FPGA
A0
XAUI
SFI
FPGA
D0
FPGA
A0
4 x CX4
Memory Bank
FPGA
D1
FPGA
D0
Connector
FPGA
D1
XAUI
SFI
Memory Bank
Ethernet switch
XAUI
SFI
Memory Bank
Ethernet switch
48V-12V
POL 3V3
RJ45
48V-12V
POL 3V3
Figure 6 Board Layout for SPF+ (left) and CX-4 (right)
4 Power
The input power of UniBoard will be the telecom standard -48V. After the circuit protections, with inrush
current control and short circuit protection, an isolated intermediate bus converter is used to convert the -48
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
11 / 21
input power to a intermediate power supply, e.g. 12 V. This intermediate supply is distributed over the board.
Local power supply converters, also known as POLs (Point Of Load), convert the power to the necessary
supplies. Depended on the used POLs the 12V intermediate bus can be changed to a different voltage.
Optionally a power manager can be used to control the power-up sequence of the POLs.
12V to 1.2V
12V intermediate power
FPGA
12 to 2.5V
-48V
Circuit
Protection
Power Manager
Intermediate Bus
Converter
Memory Power
Supply
Memory
Optical Interface
Power Supply
Optical
Interfaces
Power
Monitoring
Figure 7 Power Supply Block Diagram
To get a better understanding of the power needed for UniBoard, the power consumption is estimated in the
following subsection.
4.1 Power Estimation
The power estimation of UniBoard is split into four major parts:
 FPGA
 Memory
 Optical interface
 Power conversion
FPGA
For the power estimation of the FPGAs the LOFAR RSP and TBB boards are taken as reference. The Xilinx
FPGAs on RSP are 95um FPGA consuming approximately 8W each. Reference [2] has been used to
estimate the power consumption of the FPGA for UniBoard.
The clock speed for UniBoard will be twice the clock speed of RSP this means a factor 2 of power
consumption. For the FPGAs of UniBoard a new technology will be used, 65um in stead of 95um this means
40% power reductions. For UniBoard bigger FPGAs will be used, we estimate 100% logic and or multipliers
(given together with the double clock speed a factor of 4 more processing capabilities). This al gives a power
estimation of 20W per FPGA.
Memory
For the calculation of the power consumption for the memory banks a technical note from Micron [3] and the
Excel sheet from Micron have been used. The parameter in the excel sheet are taken from the 8GB module
MT36JSZS1G72PY [4] running at 800MT/s . The power for a single module is 3.8W. On UniBoard 4 modules
will be placed. This gives and power estimation for the memory modules of ~16W.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
12 / 21
For the LOFAR TBB board DDR2 memories have been used. The estimated power consumption of a 2GB
module running at 400MT/s (MT18HTF25672PD) is 0.8W. This is also measured. The measured power
consumption during write is approximately 1W.
In- and output interfaces
For UniBoard an optical interface will be used for the incoming data stream. For this interface an optical to
electrical converter a media-conversion from XAUI to SFI / XFI is needed. Typical values for these
components are:
 Optical module: 1.5W
 Media conversion: 1W
Eight interfaces will be placed on the board given a total of approximately 20W.
Power Supply
Although high efficient DC/DC power converters will be used, the power budget for these devices must not
be underestimated. As seen from the pervious subsection the expected total power for the FPGA is 160W
and for the IO-interfaces 20W. The local power has to be made in two steps, -48V-to12V and 12V to local
power (1, 2V5 or 3V3). Every step is approximately 90% efficiency, given an overall efficiency of 80%. This
gives a power loss of approximately 40W in the power supply chain.
Total Power
In Table 2 the total power estimations is summarized.
Table 2 Total Power Estimation
FPGA (8 FPGA’s)
Interfaces (8 interfaces)
Memory (4 modules)
Power supply
Total Power
4.2
Power each
20 W
2.5 W
4W
20%
Number
8
8
4
1
Total
160 W
20 W
16 W
40 W
236 W
Implementation
In this section the solution for the different block of Figure 7 are discussed in more detail.
4.2.1
Circuit protection
The tasks for the circuit protection is:
 Overvoltage protection (peak)
 Short circuit protection
 Inrush current limiter
For this input interface the products for ATCA are taken as an starting point. In .. the possible solutions are
described in more detail.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
13 / 21
Table 3 Circuit protection possibilities
LTC4260
Linear Technology
yes
LTC4356
Linear Technology
yes
LT1641
Linear Technology
yes
MAX8621
Maxim
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes I2C
No
No
No
Additional Features
I2C interface for
Supply monitoring
-
-
-
Package
Power
Distributor
Price indication
SOIC 24
10 MSOP
SOIC8
SOIC 8
Digikey
€7.25
ACAL
Digikey
€2.15
Digikey
€4.61
Manufacturer
Overvaltage
protection
Inrush current
protection
Short Circuit
protection
Remote Switch off
4.2.2
Point of Loads
In Table 4 the point of loads are summarised. In this overview the expected current is shown.
Table 4 Point Of Load (POL) requirements
Supply
FPGA core
FPGA IO
Memory bank
Voltage
1.2V
2.5V
1.5V
Current
100A (8 x 12.5A)
16A (8 x 2A)
13.3A (4 x 3.3A)
Power
120W (8 x 15W)
40W (8 x 5W)
20W (4 x 5W)
XAUI-to-SFI
SFP+
3V3 ??
2.5A (8 x 0.3A)
8W (8 x 1W)
Remark
Fast transients
Termination and Reference
supplies needed.
As can be seen from this table it is not feasible to make the FPGA core power with one POL, each FPGA will
have it own POL. The other supplies can be combined into one POL.
Table 5 POL comparison
Manufacterer
Molude / discrete
Input voltage
Output voltage range
Max. current
Efficiency
Features
Z-POL
Power one
SuperLynx
Lineage (Tyco)
module
8-13V
0.5-5.5V
60A
90%
I2C bus controller
Software control
Module
3-14V
0.75-5V
16A
92-95%
Overcurrent
protection, remote
sense, remote
on/off, over temp.
F-class
Emerson Network
Power (Artesyn)
Module
10.8-13.2V
1.2V
15A
86%
Fast Transient
Response
Doc.nr.:
DESP
Rev.:
Date:
Class.:
TPS51116
Texas Instruments
Discrete
3-28V
1.5V DDR3
Vtt and Vref
ASTRON-RP-303
0.1
12-01-09
Public
14 / 21
Size
32x14mm
protection
33x13.5mm
33x14mm
Distributer
Avnet Time
(Digikey)
~€13
Avnet Time
(Digikey)
€10
Avnet Time
(Digikey)
€30
Costs
Digikey
€2 (only IC)
Table 6 DDR3 power solutions
Part number
Manf
Input
VTT / VREF
TPS51116
Ti
yes
MAX17000
Maxim
26V
yes
Remark
Distributor
Price
Used on TBB
Digikey
€3.30
Digikey
€2.80
4.2.3
VNC60/70
muRata PS
12 V
Yes
????
Avnet Time
Sequencers and monitoring
The Altera/Xilinx FPGAs do not need a well defined sequence for the power up.
4.2.4
Heat
The first approach for heat management for UniBoard will be active heat sinks (heat sink with fan). For his is
chosen, because of the memory modules on the board. Given an 20W FPGA with a maximum junction
temperature of 85 Dec, and ambient temperature of 30 Dec and a typical thermal resistance between the
junction and the case of 0.2K/W, a heatsink with 2.5K/W will be needed. This can be Fischer Elektronik LA
ICK 17x17 of the LOFAR TBB with 1.6K/W, this heat sink use a 5V or 12V FAN.
5 FPGA
5.1
Remarks on FPGA selection
See Quinten Architecture document.
5.2
Configuration
The requirements for the configuration are:
 Remote configurations (downloading new images in the flash) must be possible.
 Configuration within 5 sec
In the following subsection the configuration for Xilinx and Altera are discussed in more detail.
5.2.1
Xilinx
In [5] the configuration for Xilinx devices is described. In this section a small summary is given.
For the Virtex5 devices Xilinx specific configuration devices can be used but also standard flash device. For
UniBoard these standard solutions are taken as a starting point.
SPI
Serial Peripheral Interface bus flash device are standard component on the marker. These devices use
simple protocol with 4 lines to communicate with the flash. The size of flash devices range from 4 till
128 Mbit.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
15 / 21
The top Xilinx Virtex-5 FPGA (XC5VLX330) needs 80Mit memory. This means that only one image can be
stored inside a flash device, for smalle FPGA multiple images can be stored in the flash. The expected
configuration time is 1-2 seconds (all FPGA are programmed in parallel)
Parallel Programming
The second option for configuration is standard NOR Flash memory devices. These devices have a parallel
data bus, thus reducing the configuration time. The size of these flash devices range from Mbit till Gigabit
memories. With these large devices multiple FPGA images can be stored in the flash, enabling all the
possibilities of remote configuration.
5.2.2
Altera
In this section a small summary the configuration for Altera devices as described in [6] is given. For the
Stratix IV devices both parallel and serial schemes can be used. For the parallel scheme a separate Altera
Max device must be used between the FPGA and the standard flash. For the serial configuration Altera has
EPCS devices, these devices range from 1 till 128 MBit. The image size for a GX230 is 104Mbit. There are
application notes with standard SPI flashes in stead of the Altera EPCS devices. With Serial configuration it
will not be possible to store multiple images into the flash device.
5.3
Pinning
In this section we want to give an overview of the pins needed for the FPGAs.
MOET NOG XAUI_SFI CONTROL BIJ
Table 7 FPGA pinning
Function
Per function
Total pins in EEF
Total pins in FEF
Input interface
(optic)
Output interface
(backplane)
FPGA
interconnect
Memory
ADC interface
16 H.S. 6
44 (2x or 8
transceivers+ctrl)
32 (2x or 8
transceivers)
56 ( 14
transceivers)
118 (1x)
19
88 (4x or 16
transceivers+ctrl)
64 (4x or 16
transceivers)
64 (4x or 16
transceivers)
236 (2x)
38 (2x)
4
4
4
4
4
4
4
4
6
15
30
6
15
48
6
15
32
6
298
519
417
180
Configuration
Control
Clock
Other
Total
Transceivers
total
16 H.S.
Eef:28
IOF:16 H.S.
118
19 (16 + clk
+ I2C)
4
4 (SGMII +
I2C)
6
Total pins in
input FIO
88 (4x or 16
transceivers+ctrl)
64 (4x or 16
transceivers)
236 (2x)
Total pins
output FIO
64 (4x or 16
transceivers)
64 (4x or 16
transceivers)
38 (2x)
32
For the number of configuration pins the Altera Fast Active Serial Configuration has been used. From this
table the IO is not extreme; however, a remark has to be made. Pining in FPGA are placed in banks. In
these banks the output power is set for all pins. This means that a bank used for memory can not be used for
other functions. In Figure 8 the banks for the Altera EP4GX230, taken from the Stratix IV Device handbook is
shown. Using all IO in a bank can introduce simultaneous switching noise, degrading the performance of the
device. This is especially the case for the memory interface, where the termination will be used on the FPGA
chip.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
16 / 21
Figure 8 Example of IO banks for Altera EP4SG230-FBGA1517
For the High-speed serial lines special care have to be taken with IO next to the High-Speed serial lines (this
is not always allowed). Instead of using pin’s banks are used to indicate what is needed. Not all banks are
equal, with the Altera device from Figure 8 the column banks (top and bottom) have higher data rates to
memory than the row banks (left and right) but column banks have limited LVDS support (no internal
termination resistor).
Table 8 Banks / FPGA side's
Function
Total pins in EEF
Total pins in EF
Total pins in
input FIO
3 GX banks
Input interface
(optic)
Output interface
(backplane)
FPGA
interconnect
Memory
ADC interface
Configuration
Control
Clock
Other
total
2 GX banks
3 GX banks
2 GX banks
3 GX banks
4 GX Banks
3 GX banks
3 GX banks
4-5 banks
1 bank
0.3 bank
0.3 bank
0.3 bank
7 banks 6+ GX
8-10 banks
2 banks
0.3 bank
0.3 bank
0.3 bank
13 banks+9
8-10 banks
Total pins
output FIO
3 GX banks
3 GX banks
0.3 bank
0.3 bank
0.3 bank
11 banks 6+ GX
2 banks
0.3 bank
0.3 bank
0.3 bank
3 bank 6+ GX
6 In and Output Interfaces
For the input interface, 10GbE interfaces will be used. These interfaces can also work in Infiniband mode or
with a custom build protocol. For the front side (the input) an optical ready interface will be made and for the
backside a copper interface will be made. This section will focus on the optical ready input (an optical
module can be placed or a cable with e.g. SFP+ Direct Attach).
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
17 / 21
12V intermediate power
Power
Optical Interface
XAUI to
XFI / SFI
Optical Interface
XAUI to FPGA
Low Jitter Clock
to FPGA
Clock
Figure 9 Block diagram optical interface
6.1
Remarks on optical selection
The options for the optical interconnect are described in Table 9. Not only optical interconnects are
described but some copper interconnects as well.
Table 9 Optical interconnect options
Size
width x depth
Power
Input
Range
Optical connector
type
External clock
Supplier
Maximal Distance
Cost overhead
Cost Connector /
Cage
Cost 3m cable
Cost module
XFP
18 x 71.12 mm
SFP+
14.8 x 44.95
SFP+ Cu
14.8 x 44.95
CX4_optic
CX-4_Cu
1.5 W + 1.3W
XFI
Long / short
Duplex LC
1 W + 1.3W
SFI
Short
Duplex LC
1.3W
0
XAUI
0
XAUI
short
Ref Clock
Finisar
80km
€60
€12 (with
heatsink)
€56
€170
Not needed
Finisar
10km
€60
€5 (€20 for
2x2)
€56
€106
<25m
Cable on SFP+
connector
Gore / Tyco
10-15m
€60
€5 (€20 for 2x2)
€36
0
Parallel optic
Fujitsu
Gore
10-15m
€8
0
€74
0
From the table it can be seen that SFP+ cage is the most promising solution. With SFP+ a cable can be
used (SFP+ Direct Attach), or an optical module can be plugged into the cage for an optical interconnect of
up to 10km.
6.2
XAUI to SFI conversion
For the interface between the FPGA and the SPF+, a media converter is needed to convert from the FPGA
output XAUI to the optical module input SFI. In Table 10 an overview is of the different suppliers is given.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
18 / 21
Table 10 XAUI to SFI selection
Manf
Pins (size)
Power
Supplies
Clock
Boundary Scan
Test facilities
Sales
Price
Typical lead time
6.3
BCM8706
Broadcom
256 FBGA (13x13mm)
<1.5W
1.0V / 3.3V
25MHz or 156.25MHz
VSC8486
Vitesse
256 FCBGA (17x17mm)
0.75 W
1.2V
yes
BIST, loopback
Atlantic
yes
BIST, loopback
ACAL / Avnet Memec
~$ 57
16 wks
16 wks
QT2035
AMCC
BGA (15x15mm)
0.9W
1.2V
DATA TAKEN FROM
QT2032 XAUI - XFI
Clock
The clock requirements for the 10GbE links are:

< 28ps rms

45-55% Duty Cycle
7 Memory
The memory requirements for UniBoard are:

1 second of data storage

FIFO (write and read within sample rate)
Given the FIFO nature, read and write have to be done during the data flow. With a data input of 20Gbps,
read and write have to be done with 40Gbps or 5Gbyte per second. With a clock of 400MHz and double data
rate, the burst (peak) data transfer rate to a standard DDR2/3 module is 6.4Gbyte per second. This means
that there is little room left for overhead. In other words, with one slot is not sufficient to handle the worst
case 2x10Gbps data rate. This can be solved by reducing the input data rate of placing two slots to the
FPGA.
For the memory, DDR2 and the newer DDR3 can be used. In Table 11 the two types are discussed in more
detail.
Table 11 Memory comparison
Mechanical size non buffered
Mechanical size buffered /
registered
Max speed (RDIMM)
Supply voltage
Max size (available in RDIMM)
DDR2
Both DIMM and SODIMM
Both RDIMM and SORDIMM
DDR3
Both DIMM and SODIMM
Only RDIMM
800MT/s (PC2 6400)
1.8 V
8 GByte (3.2sec for 2x10GbE)
1333MT/s (PC3-10600)
1.5 V
8 GByte (3.2sec for 2x10GbE)
From this table it can be seen that DDR2 memory are just fast enough for a UniBoard running at 400MHz.
For reliable data transfer, faster memory is needed. Special care must be taken with the transfer rates, these
are peak rates, the latency (number of clock cycles) for selecting the address is increased for DDR3
modules.
Although multiple manufactures for memory modules are available, Micron modules will be used for the
prototypes due to the availability of datasheets and modules for Xilinx.
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
19 / 21
8 Clock and Control
8.1
Clock
UniBoard will have a central clock system distributed on the board among the FPGAs. For the high-speed
interfaces a more dedicated clock tree will be used. For the 10GbE interfaces a 156.25MHz clock will be
needed. This clock will be located near the FPGAs, this will reduce jitter. Instead of using multiple Xo’s a
clock distribution can be used. The added noise of the clock distribution has to be investigated (e.g. the
onsemi M100LVEP111FATW add 0.2ps rms jitter).
8.2
Control
Options:
 Every FPGA it’s own 1000Base-T interface
o The same link can be used to read out correlator results. High output bandwidth is available
o More connector space is needed. This can be reduced by the use of a MRJ21 connector, 6
1000BASE-T connections can be combined into one.
o More components are needed when 10/100/1000BASE-T is used (every link an PHY-device)
 Single 1000Base-T to UniBoard with a switch on the board to distribute to the FPGAs.
o Using an onboard switch reduces the number of connectors and components on the board.
 Use SFP for control connection (direct from FPGA to SFP module)
Comparison LC/LC cable cost: 5m optic ~€30.49, 5m copper €5.5, Optical module 0.8W.
Optionally a SFP-RJ45 module can be placed.
 Special FPGA for control.
In Table 12 a selection of possible Ethernet switches has been made.
Table 12 Ethernet Switch solutions
Vendor
Ports
Type of ports
managing
Pins (size)
PHY needed for
1000BASE-T ?
Power
Distributor
Price estimate
VSC7389
Vitesse
8+8
8 x SGMII +
8 x Copper
Unmanaged +
Web-Managed
BGA 596 (35 x
35mm)
no
BCM56228
Broadcom
8+4
8 x SGMII
4x
SGMII/2.5GbE
BCM53716
Broadcom
16
16 x SGMII
88E6182 / 85
Marvell
10
10 x SGMII
FM3104
Fulcrum
10
2 x 10G
8 x SGMII
Unmanaged
1152 FCBGA
yes
Yes
yes
6.8W
1W
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
20 / 21
Table 13 Ethernet PHY solutions
Part number
Verdor
Ports
Pins (size)
Input interface
10/100/1000Base-T
VSC8234
Vitesse
4
(19x19mm)
SGMII
yes
Remarks
Distributor
Costs
BCM5461S
Broadcom
1
GMII / SGMII
yes
88E1112
Marvell
1
64QFN (10x10)
SGMII
Yes
Used on RSP
9 Test
From the start of the design in both the firmware and in the hardware test facilities will be implemented. On
the hardware these test facilities range from the production of the board, like flying probe test points and
boundary scan, till debug facilities for the firmware, LEDs and jumpers and operation status information like
voltage and temperatures.
9.1
Boundary Scan
On the board high pin count components are placed. To enable interconnections tests during production,
boundary scan will be implemented on the board. Due to the large number of components and the different
types of components (FPGAs, optical transceivers and Ethernet devices) a bridge will be placed on the
board. This bridge will have an input via the backplane to enable system level boundary scan.
For the FPGA TAPs bypass possibilities will be implemented to skip devices.
9.2
Operation Status information
On UniBoard monitoring will be placed on:
 FPGA temperature
 Supply voltages
 Power supply status
Doc.nr.:
DESP
Rev.:
Date:
Class.:
ASTRON-RP-303
0.1
12-01-09
Public
21 / 21