Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Buck converter wikipedia , lookup
Electric power system wikipedia , lookup
Audio power wikipedia , lookup
Mains electricity wikipedia , lookup
Power engineering wikipedia , lookup
Optical rectenna wikipedia , lookup
Power electronics wikipedia , lookup
Power over Ethernet wikipedia , lookup
Opto-isolator wikipedia , lookup
Immunity-aware programming wikipedia , lookup
UniBoard Board Design Organisatie / Organization Datum / Date Auteur(s) / Author(s): Gijs Schoonderbeek Sjouke Zwier ASTRON Controle / Checked: ASTRON Goedkeuring / Approval: ASTRON Autorisatie / Authorisation: Handtekening / Signature ASTRON ASTRON-FO-017 2.0 © ASTRON 2009 All rights are reserved. Reproduction in whole or in part is prohibited without written consent of the copyright owner. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 1 / 21 Distribution list: Group: Others: Andre Gunst Eric Kooistra Arie Doorduin Sjouke Zwier Albert Jan Boonstra Document history: Revision 0.1 Date Author Modification / Change 2009-01-12 Schoonderbeek Creation Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 2 / 21 Table of contents: 1 Introduction .................................................................................................................................................. 7 1.1 1.2 Scope..................................................................................................................................................... 7 System Functionality ............................................................................................................................. 7 2 Board Block Diagram .................................................................................................................................. 8 3 PCB ............................................................................................................................................................. 9 3.1 3.2 4 Power ........................................................................................................................................................11 4.1 4.2 5 Power Estimation .................................................................................................................................12 Implementation ....................................................................................................................................13 FPGA .........................................................................................................................................................15 5.1 5.2 6 Via and Board Thickness ...................................................................................................................... 9 Layer build up ......................................................................................................................................10 Remarks on FPGA selection ...............................................................................................................15 Configuration .......................................................................................................................................15 In and Output Interfaces ............................................................................................................................17 6.1 6.2 6.3 Remarks on optical selection ..............................................................................................................18 XAUI to SFI conversion .......................................................................................................................18 Clock ....................................................................................................................................................19 7 Memory .....................................................................................................................................................19 8 Clock and Control ......................................................................................................................................20 8.1 8.2 9 Clock ....................................................................................................................................................20 Control .................................................................................................................................................20 Test............................................................................................................................................................21 9.1 9.2 Boundary Scan ....................................................................................................................................21 Operation Status information ...............................................................................................................21 Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 3 / 21 List of figures: Figure 1 System Block Diagram ......................................................................................................................... 7 Figure 2 Board overview..................................................................................................................................... 8 Figure 3 UniBoard Block diagram ...................................................................................................................... 9 Figure 4 Via calculation ....................................................................................................................................10 Figure 5 Layer build up for 10 layer (left), 12 layer (mid) and 14 layer (right). .................................................10 Figure 6 Power Supply Block Diagram .............................................................................................................12 Figure 7 Block diagram optical interface ..........................................................................................................18 List of tables: Table 1 Layer comparison ................................................................................................................................11 Table 2 Total Power Estimation .......................................................................................................................13 Table 3 Circuit protection possibilities ..............................................................................................................14 Table 4 UniBoard POLs ....................................................................................................................................14 Table 5 POL comparison ..................................................................................................................................14 Table 6 FPGA comparison ................................................................................ Error! Bookmark not defined. Table 7 Optical interconnect options ................................................................................................................18 Table 8 XAUI to SFI selection ..........................................................................................................................19 Table 9 Memory comparison ............................................................................................................................19 Table 10 Ethernet Switch solutions ..................................................................................................................20 Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 4 / 21 Terminology: ADC BF bps BW DDR EMI Firmware FPGA GMII Hardware HDL IO IP IPC LOFAR LUT MAC PCB POL Pps PHY RF RSP SGMII SFP SFP+ Subband TBB XAUI XGMII XFP Analogue to Digital Converter BeamFormer Bits per second BandWidth Double Data Rate Electro-Magnetic Interference Embedded or real-time code that runs on a microprocessor (e.g. written in C) Field Programmable Gate Array GbE media independent Interface 8 bits @125MHz Boards, subracks and COTS equipment Hardware Description Language Input Output Intellectual Property Association Connecting Electronics Industries ( formally Institute for Interconnecting and Packaging Electronic Circuits) LOw Frequency Array Look Up Table Multiply and Accumulate, Medium Access, Monitoring and Control, Media Excess Controller (layer 2 of OSI model) Printed Circuit Board Point of Load Pulse Per Second physical interface (layer 1 of OSI model) Radio Frequency Remote Station Processing (in LOFAR) Serial Gb Media Idependent Interface Small From-factor Pluggable transceiver SFP for 10GbE Frequency band, unit output of the filterbank Transient Buffer Board (in LOFAR) 10G attachment Unit Interface (4x 3.125Gbps) interface between MAC and PHY 10G Media Independent Interface 10G Small form-factor Pluggable transceiver Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 5 / 21 References: [1] Quinten System document [2] Xilinx White Paper WP285 V1.0, Virtex-5 FPGA System Power Design Considerations, February 14, 2008 [3] Micron Technology Inc., Rev B 8/07 EN, TN-41-01 Calculation Memory System Power for DDR3, 2007 [4] Micron Technology Inc., Rev A 4/08 EN, DDR3 SDRAM RDIMM MT36JSZS1G72PY-8GB Datasheet. [5] Xilinx UG191 (v3.5) Virtex-5 FPGA Configuration User Guide, October 29, 2008 [6] Altera Volume I Stratix IV Device handbook ver 2.0, Chapter 10 Nov 2008 Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 6 / 21 1 Introduction 1.1 Scope Quinten is a system which can run stand alone and is able to filter, correlate and or beamform data. The system will be developed based on the experiences with LOFAR and expand the processing capabilities significantly. The aim is to integrate more bandwidth and processing power per cubic meter. To narrow the scope of Quinten five key applications are used. One Quinten or multiple Quinten systems should be able to handle the requirements these applications: 1. 2. 3. 4. 5. Next generation EVN correlator Apertif beamformer Apertif correlator LOFAR core beamformer LOFAR core correlator In this document the board design for the processing board of Quinten, UniBoard is described. More system information can be found in [6] 1.2 System Functionality In Figure 1 the block diagram of Quinten architecture is shown. This functionality can be mapped on single UniBoards, or mapped to multiple boards on a backplane. Delay Compensation Poly fase Filter Bank Correlator / Beamformer Transpose Delay Compensation Poly fase Filter Bank Correlator / Beamformer Figure 1 System Block Diagram Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 7 / 21 2 Board Block Diagram Input 10 GbE Interfaces Clock and Sync Control Supply In Figure 2 a simplified block diagram of UniBoard showing the connections can be seen. UniBoard has 10GbE in- and outputs, an ADC input and control and power lines. The memory for the delay compensation is not drawn in this figure. UniBoard 10 GbE Interfaces Transpose Parallel input Figure 2 Board overview This block diagram is worked out in more detail in Figure 3, only the interconnections from FPGA A0 to A1, B1, C1 and D1 are draw. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 8 / 21 Parallel Input Memory Bank Optical Interface FPGA A0 Optical Interface FPGA A1 Parallel Input Memory Bank Optical Interface FPGA B0 Optical Interface FPGA B1 Parallel Input Memory Bank Optical Interface FPGA C0 Optical Interface FPGA C1 Parallel Input Memory Bank Optical Interface FPGA D0 Optical Interface FPGA D1 Power Clock & Control Mesh between all input and all output fpga’s 8 x Gb Transeiver fpga-to-fpga connection 4 x Gb Transeiver or 1 x 10GbE board-to-board connection 1 x 10GbE IO connection Figure 3 UniBoard Block diagram 3 PCB 3.1 Via and Board Thickness For a better estimate on the possibilities of the PCB, the IPC calculator has been used to estimate the maximal PCB thickness. In Figure 4 the result is shown. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 9 / 21 Figure 4 Via calculation In this calculation the aspect ration is 7.0:1. For UniBoard we will use a bit more challenging 8.0:1 aspect ration. With this aspect ration the maximum board thickness is 2.4mm. As shown in the calculator the maximal current is 0.85A. With 20W in the FPGA and 1V core supply at leased 23 via’s to the power plane are needed. A second argument to limit the board thickness to 2.4mm is the connectors used. The maximal board thickness for a memory connectors is 2.5mm (this is for both SMT and through hole connectors, both have metal Beveled Pins). 3.2 Layer build up To analyse the maximum number of layers for UniBoard a 10, 12 and 14 layer build up is taken as starting point, see Figure 5. When horizontal / vertical routing is used, two adjacent layers can be used for HighSpeed routing. Special care must be taken that not traces in these layers run in parallel, this will introduce crosstalk, and thus limiting the links capabilities. 1 2 3 H.S GND 1 2 3 4 Signal GND 5 6 PWR PWR PWR PWR 7 8 6 7 H.S H.S 9 10 GND Signal 8 9 10 11 12 Signal GND H.S H.S H.S Prepreg 2116 (110um or 8.7 mil) Core 10 mil 35/35 4 5 H.S GND Signal GND H.S Prepreg 1080 (70um or 5.5 mil) Prepreg 2166 (110um or 8.7 mil) Core 10 mil Core 6 mil 1 2 3 4 H.S GND H.S H.S GND 5 6 Signal PWR 7 8 PWR Signal 9 10 11 12 13 14 GND H.S H.S GND H.S Prepreg 1080 (70um of 2.76mm) Core 6mil Figure 5 Layer build up for 10 layer (left), 12 layer (mid) and 14 layer (right). Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 10 / 21 In Table 1 a comparison for High-Speed traces for the three layers build-ups are shown. The calculation has been done for the inner layer (I.L.) and outer layer (O.L.). For all situations the maximal board thickness is 2.4mm (95mil) the copper for all layers is 35um (1 oz, or 1.4mil). The 10 layer is build up with: 8.7 / 10 / 8.7 / 10 / 8.7 / 10 / 8.7 / 10 / 8.7 (14 mil Cu) The 12 layer is build up with: 5.5 / 10 / 8.7 / 6 / 5.5 / 6 / 5.5 / 6 / 8.7 / 10 / 5.5 (16.8 mil Cu) The 14 layer is build up with: 5.5 / 6 / 5.5 / 6 / 5.5 / 6 / 5.5 / 6 / 5.5 / 6 / 5.5 / 6 / 5.5 mill (19mil Cu) Table 1 Layer comparison I.L Layer Spacing I.L High-Speed trace width O.L Layer Spacing O.L High-Speed trace width 10 Layer - 12 Layer 5.5 mil W:7mil S:9mil 14 Layer 5.5 mil W:5 mil S:8mil Offset 10 – 19 mil W:7mil S:10mil 10 mil – 8.7 mil W:6 mil S: 12 mil Offset 6 – 11.5 mil W:4 mil S:8 mil The 14 layer build-up needs for the high speed traces 4 mil. This trace width is not reliable to fabricate. For both 10 and 12 layer, 2 layers for high-speed traces can be used and 2 layers for general purpose. The benefit of the 12 layer with respect to the 10 layer is the better reference for the High-Speed traces. For the 10 layer X-Y routing is necessary, but changing from the horizontal to the vertical layer also changes from a GND reference to a power reference. 3.3 Board setup For getting an understanding on the board us two setups are made, one for the SPF+ IO and one for the CX-4 IO. For the CX-4 situation 3 10GbE are taken as an examples. Both boards have an Ethernet switch for control and readout of correlator results. Memory Bank POL CORE XAUI SFI POL CORE Memory Bank XAUI SFI 4 x SFP+ Connector FPGA A0 FPGA A0 FPGA A1 XAUI SFI FPGA A1 FPGA A0 Connector Memory Bank XAUI SFI 4 x CX4 POL CORE Connector Memory Bank POL CORE Memory Bank FPGA A0 XAUI SFI Memory Bank FPGA A0 Connector XAUI SFI FPGA B0 4 x SFP+ FPGA B0 FPGA B1 XAUI SFI FPGA B1 Connector Memory Bank XAUI SFI Connector 4 x CX4 POL CORE Memory Bank POL CORE Memory Bank XAUI SFI Connector Memory Bank XAUI SFI FPGA C0 4 x SFP+ FPGA C1 FPGA C0 Connector FPGA C1 XAUI SFI FPGA A0 Memory Bank XAUI SFI RJ45 Connector Memory Bank POL CORE XAUI SFI POL CORE Memory Bank Connector 4 x SFP+ FPGA A0 XAUI SFI FPGA D0 FPGA A0 4 x CX4 Memory Bank FPGA D1 FPGA D0 Connector FPGA D1 XAUI SFI Memory Bank Ethernet switch XAUI SFI Memory Bank Ethernet switch 48V-12V POL 3V3 RJ45 48V-12V POL 3V3 Figure 6 Board Layout for SPF+ (left) and CX-4 (right) 4 Power The input power of UniBoard will be the telecom standard -48V. After the circuit protections, with inrush current control and short circuit protection, an isolated intermediate bus converter is used to convert the -48 Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 11 / 21 input power to a intermediate power supply, e.g. 12 V. This intermediate supply is distributed over the board. Local power supply converters, also known as POLs (Point Of Load), convert the power to the necessary supplies. Depended on the used POLs the 12V intermediate bus can be changed to a different voltage. Optionally a power manager can be used to control the power-up sequence of the POLs. 12V to 1.2V 12V intermediate power FPGA 12 to 2.5V -48V Circuit Protection Power Manager Intermediate Bus Converter Memory Power Supply Memory Optical Interface Power Supply Optical Interfaces Power Monitoring Figure 7 Power Supply Block Diagram To get a better understanding of the power needed for UniBoard, the power consumption is estimated in the following subsection. 4.1 Power Estimation The power estimation of UniBoard is split into four major parts: FPGA Memory Optical interface Power conversion FPGA For the power estimation of the FPGAs the LOFAR RSP and TBB boards are taken as reference. The Xilinx FPGAs on RSP are 95um FPGA consuming approximately 8W each. Reference [2] has been used to estimate the power consumption of the FPGA for UniBoard. The clock speed for UniBoard will be twice the clock speed of RSP this means a factor 2 of power consumption. For the FPGAs of UniBoard a new technology will be used, 65um in stead of 95um this means 40% power reductions. For UniBoard bigger FPGAs will be used, we estimate 100% logic and or multipliers (given together with the double clock speed a factor of 4 more processing capabilities). This al gives a power estimation of 20W per FPGA. Memory For the calculation of the power consumption for the memory banks a technical note from Micron [3] and the Excel sheet from Micron have been used. The parameter in the excel sheet are taken from the 8GB module MT36JSZS1G72PY [4] running at 800MT/s . The power for a single module is 3.8W. On UniBoard 4 modules will be placed. This gives and power estimation for the memory modules of ~16W. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 12 / 21 For the LOFAR TBB board DDR2 memories have been used. The estimated power consumption of a 2GB module running at 400MT/s (MT18HTF25672PD) is 0.8W. This is also measured. The measured power consumption during write is approximately 1W. In- and output interfaces For UniBoard an optical interface will be used for the incoming data stream. For this interface an optical to electrical converter a media-conversion from XAUI to SFI / XFI is needed. Typical values for these components are: Optical module: 1.5W Media conversion: 1W Eight interfaces will be placed on the board given a total of approximately 20W. Power Supply Although high efficient DC/DC power converters will be used, the power budget for these devices must not be underestimated. As seen from the pervious subsection the expected total power for the FPGA is 160W and for the IO-interfaces 20W. The local power has to be made in two steps, -48V-to12V and 12V to local power (1, 2V5 or 3V3). Every step is approximately 90% efficiency, given an overall efficiency of 80%. This gives a power loss of approximately 40W in the power supply chain. Total Power In Table 2 the total power estimations is summarized. Table 2 Total Power Estimation FPGA (8 FPGA’s) Interfaces (8 interfaces) Memory (4 modules) Power supply Total Power 4.2 Power each 20 W 2.5 W 4W 20% Number 8 8 4 1 Total 160 W 20 W 16 W 40 W 236 W Implementation In this section the solution for the different block of Figure 7 are discussed in more detail. 4.2.1 Circuit protection The tasks for the circuit protection is: Overvoltage protection (peak) Short circuit protection Inrush current limiter For this input interface the products for ATCA are taken as an starting point. In .. the possible solutions are described in more detail. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 13 / 21 Table 3 Circuit protection possibilities LTC4260 Linear Technology yes LTC4356 Linear Technology yes LT1641 Linear Technology yes MAX8621 Maxim yes yes yes yes yes yes yes yes yes yes I2C No No No Additional Features I2C interface for Supply monitoring - - - Package Power Distributor Price indication SOIC 24 10 MSOP SOIC8 SOIC 8 Digikey €7.25 ACAL Digikey €2.15 Digikey €4.61 Manufacturer Overvaltage protection Inrush current protection Short Circuit protection Remote Switch off 4.2.2 Point of Loads In Table 4 the point of loads are summarised. In this overview the expected current is shown. Table 4 Point Of Load (POL) requirements Supply FPGA core FPGA IO Memory bank Voltage 1.2V 2.5V 1.5V Current 100A (8 x 12.5A) 16A (8 x 2A) 13.3A (4 x 3.3A) Power 120W (8 x 15W) 40W (8 x 5W) 20W (4 x 5W) XAUI-to-SFI SFP+ 3V3 ?? 2.5A (8 x 0.3A) 8W (8 x 1W) Remark Fast transients Termination and Reference supplies needed. As can be seen from this table it is not feasible to make the FPGA core power with one POL, each FPGA will have it own POL. The other supplies can be combined into one POL. Table 5 POL comparison Manufacterer Molude / discrete Input voltage Output voltage range Max. current Efficiency Features Z-POL Power one SuperLynx Lineage (Tyco) module 8-13V 0.5-5.5V 60A 90% I2C bus controller Software control Module 3-14V 0.75-5V 16A 92-95% Overcurrent protection, remote sense, remote on/off, over temp. F-class Emerson Network Power (Artesyn) Module 10.8-13.2V 1.2V 15A 86% Fast Transient Response Doc.nr.: DESP Rev.: Date: Class.: TPS51116 Texas Instruments Discrete 3-28V 1.5V DDR3 Vtt and Vref ASTRON-RP-303 0.1 12-01-09 Public 14 / 21 Size 32x14mm protection 33x13.5mm 33x14mm Distributer Avnet Time (Digikey) ~€13 Avnet Time (Digikey) €10 Avnet Time (Digikey) €30 Costs Digikey €2 (only IC) Table 6 DDR3 power solutions Part number Manf Input VTT / VREF TPS51116 Ti yes MAX17000 Maxim 26V yes Remark Distributor Price Used on TBB Digikey €3.30 Digikey €2.80 4.2.3 VNC60/70 muRata PS 12 V Yes ???? Avnet Time Sequencers and monitoring The Altera/Xilinx FPGAs do not need a well defined sequence for the power up. 4.2.4 Heat The first approach for heat management for UniBoard will be active heat sinks (heat sink with fan). For his is chosen, because of the memory modules on the board. Given an 20W FPGA with a maximum junction temperature of 85 Dec, and ambient temperature of 30 Dec and a typical thermal resistance between the junction and the case of 0.2K/W, a heatsink with 2.5K/W will be needed. This can be Fischer Elektronik LA ICK 17x17 of the LOFAR TBB with 1.6K/W, this heat sink use a 5V or 12V FAN. 5 FPGA 5.1 Remarks on FPGA selection See Quinten Architecture document. 5.2 Configuration The requirements for the configuration are: Remote configurations (downloading new images in the flash) must be possible. Configuration within 5 sec In the following subsection the configuration for Xilinx and Altera are discussed in more detail. 5.2.1 Xilinx In [5] the configuration for Xilinx devices is described. In this section a small summary is given. For the Virtex5 devices Xilinx specific configuration devices can be used but also standard flash device. For UniBoard these standard solutions are taken as a starting point. SPI Serial Peripheral Interface bus flash device are standard component on the marker. These devices use simple protocol with 4 lines to communicate with the flash. The size of flash devices range from 4 till 128 Mbit. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 15 / 21 The top Xilinx Virtex-5 FPGA (XC5VLX330) needs 80Mit memory. This means that only one image can be stored inside a flash device, for smalle FPGA multiple images can be stored in the flash. The expected configuration time is 1-2 seconds (all FPGA are programmed in parallel) Parallel Programming The second option for configuration is standard NOR Flash memory devices. These devices have a parallel data bus, thus reducing the configuration time. The size of these flash devices range from Mbit till Gigabit memories. With these large devices multiple FPGA images can be stored in the flash, enabling all the possibilities of remote configuration. 5.2.2 Altera In this section a small summary the configuration for Altera devices as described in [6] is given. For the Stratix IV devices both parallel and serial schemes can be used. For the parallel scheme a separate Altera Max device must be used between the FPGA and the standard flash. For the serial configuration Altera has EPCS devices, these devices range from 1 till 128 MBit. The image size for a GX230 is 104Mbit. There are application notes with standard SPI flashes in stead of the Altera EPCS devices. With Serial configuration it will not be possible to store multiple images into the flash device. 5.3 Pinning In this section we want to give an overview of the pins needed for the FPGAs. MOET NOG XAUI_SFI CONTROL BIJ Table 7 FPGA pinning Function Per function Total pins in EEF Total pins in FEF Input interface (optic) Output interface (backplane) FPGA interconnect Memory ADC interface 16 H.S. 6 44 (2x or 8 transceivers+ctrl) 32 (2x or 8 transceivers) 56 ( 14 transceivers) 118 (1x) 19 88 (4x or 16 transceivers+ctrl) 64 (4x or 16 transceivers) 64 (4x or 16 transceivers) 236 (2x) 38 (2x) 4 4 4 4 4 4 4 4 6 15 30 6 15 48 6 15 32 6 298 519 417 180 Configuration Control Clock Other Total Transceivers total 16 H.S. Eef:28 IOF:16 H.S. 118 19 (16 + clk + I2C) 4 4 (SGMII + I2C) 6 Total pins in input FIO 88 (4x or 16 transceivers+ctrl) 64 (4x or 16 transceivers) 236 (2x) Total pins output FIO 64 (4x or 16 transceivers) 64 (4x or 16 transceivers) 38 (2x) 32 For the number of configuration pins the Altera Fast Active Serial Configuration has been used. From this table the IO is not extreme; however, a remark has to be made. Pining in FPGA are placed in banks. In these banks the output power is set for all pins. This means that a bank used for memory can not be used for other functions. In Figure 8 the banks for the Altera EP4GX230, taken from the Stratix IV Device handbook is shown. Using all IO in a bank can introduce simultaneous switching noise, degrading the performance of the device. This is especially the case for the memory interface, where the termination will be used on the FPGA chip. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 16 / 21 Figure 8 Example of IO banks for Altera EP4SG230-FBGA1517 For the High-speed serial lines special care have to be taken with IO next to the High-Speed serial lines (this is not always allowed). Instead of using pin’s banks are used to indicate what is needed. Not all banks are equal, with the Altera device from Figure 8 the column banks (top and bottom) have higher data rates to memory than the row banks (left and right) but column banks have limited LVDS support (no internal termination resistor). Table 8 Banks / FPGA side's Function Total pins in EEF Total pins in EF Total pins in input FIO 3 GX banks Input interface (optic) Output interface (backplane) FPGA interconnect Memory ADC interface Configuration Control Clock Other total 2 GX banks 3 GX banks 2 GX banks 3 GX banks 4 GX Banks 3 GX banks 3 GX banks 4-5 banks 1 bank 0.3 bank 0.3 bank 0.3 bank 7 banks 6+ GX 8-10 banks 2 banks 0.3 bank 0.3 bank 0.3 bank 13 banks+9 8-10 banks Total pins output FIO 3 GX banks 3 GX banks 0.3 bank 0.3 bank 0.3 bank 11 banks 6+ GX 2 banks 0.3 bank 0.3 bank 0.3 bank 3 bank 6+ GX 6 In and Output Interfaces For the input interface, 10GbE interfaces will be used. These interfaces can also work in Infiniband mode or with a custom build protocol. For the front side (the input) an optical ready interface will be made and for the backside a copper interface will be made. This section will focus on the optical ready input (an optical module can be placed or a cable with e.g. SFP+ Direct Attach). Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 17 / 21 12V intermediate power Power Optical Interface XAUI to XFI / SFI Optical Interface XAUI to FPGA Low Jitter Clock to FPGA Clock Figure 9 Block diagram optical interface 6.1 Remarks on optical selection The options for the optical interconnect are described in Table 9. Not only optical interconnects are described but some copper interconnects as well. Table 9 Optical interconnect options Size width x depth Power Input Range Optical connector type External clock Supplier Maximal Distance Cost overhead Cost Connector / Cage Cost 3m cable Cost module XFP 18 x 71.12 mm SFP+ 14.8 x 44.95 SFP+ Cu 14.8 x 44.95 CX4_optic CX-4_Cu 1.5 W + 1.3W XFI Long / short Duplex LC 1 W + 1.3W SFI Short Duplex LC 1.3W 0 XAUI 0 XAUI short Ref Clock Finisar 80km €60 €12 (with heatsink) €56 €170 Not needed Finisar 10km €60 €5 (€20 for 2x2) €56 €106 <25m Cable on SFP+ connector Gore / Tyco 10-15m €60 €5 (€20 for 2x2) €36 0 Parallel optic Fujitsu Gore 10-15m €8 0 €74 0 From the table it can be seen that SFP+ cage is the most promising solution. With SFP+ a cable can be used (SFP+ Direct Attach), or an optical module can be plugged into the cage for an optical interconnect of up to 10km. 6.2 XAUI to SFI conversion For the interface between the FPGA and the SPF+, a media converter is needed to convert from the FPGA output XAUI to the optical module input SFI. In Table 10 an overview is of the different suppliers is given. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 18 / 21 Table 10 XAUI to SFI selection Manf Pins (size) Power Supplies Clock Boundary Scan Test facilities Sales Price Typical lead time 6.3 BCM8706 Broadcom 256 FBGA (13x13mm) <1.5W 1.0V / 3.3V 25MHz or 156.25MHz VSC8486 Vitesse 256 FCBGA (17x17mm) 0.75 W 1.2V yes BIST, loopback Atlantic yes BIST, loopback ACAL / Avnet Memec ~$ 57 16 wks 16 wks QT2035 AMCC BGA (15x15mm) 0.9W 1.2V DATA TAKEN FROM QT2032 XAUI - XFI Clock The clock requirements for the 10GbE links are: < 28ps rms 45-55% Duty Cycle 7 Memory The memory requirements for UniBoard are: 1 second of data storage FIFO (write and read within sample rate) Given the FIFO nature, read and write have to be done during the data flow. With a data input of 20Gbps, read and write have to be done with 40Gbps or 5Gbyte per second. With a clock of 400MHz and double data rate, the burst (peak) data transfer rate to a standard DDR2/3 module is 6.4Gbyte per second. This means that there is little room left for overhead. In other words, with one slot is not sufficient to handle the worst case 2x10Gbps data rate. This can be solved by reducing the input data rate of placing two slots to the FPGA. For the memory, DDR2 and the newer DDR3 can be used. In Table 11 the two types are discussed in more detail. Table 11 Memory comparison Mechanical size non buffered Mechanical size buffered / registered Max speed (RDIMM) Supply voltage Max size (available in RDIMM) DDR2 Both DIMM and SODIMM Both RDIMM and SORDIMM DDR3 Both DIMM and SODIMM Only RDIMM 800MT/s (PC2 6400) 1.8 V 8 GByte (3.2sec for 2x10GbE) 1333MT/s (PC3-10600) 1.5 V 8 GByte (3.2sec for 2x10GbE) From this table it can be seen that DDR2 memory are just fast enough for a UniBoard running at 400MHz. For reliable data transfer, faster memory is needed. Special care must be taken with the transfer rates, these are peak rates, the latency (number of clock cycles) for selecting the address is increased for DDR3 modules. Although multiple manufactures for memory modules are available, Micron modules will be used for the prototypes due to the availability of datasheets and modules for Xilinx. Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 19 / 21 8 Clock and Control 8.1 Clock UniBoard will have a central clock system distributed on the board among the FPGAs. For the high-speed interfaces a more dedicated clock tree will be used. For the 10GbE interfaces a 156.25MHz clock will be needed. This clock will be located near the FPGAs, this will reduce jitter. Instead of using multiple Xo’s a clock distribution can be used. The added noise of the clock distribution has to be investigated (e.g. the onsemi M100LVEP111FATW add 0.2ps rms jitter). 8.2 Control Options: Every FPGA it’s own 1000Base-T interface o The same link can be used to read out correlator results. High output bandwidth is available o More connector space is needed. This can be reduced by the use of a MRJ21 connector, 6 1000BASE-T connections can be combined into one. o More components are needed when 10/100/1000BASE-T is used (every link an PHY-device) Single 1000Base-T to UniBoard with a switch on the board to distribute to the FPGAs. o Using an onboard switch reduces the number of connectors and components on the board. Use SFP for control connection (direct from FPGA to SFP module) Comparison LC/LC cable cost: 5m optic ~€30.49, 5m copper €5.5, Optical module 0.8W. Optionally a SFP-RJ45 module can be placed. Special FPGA for control. In Table 12 a selection of possible Ethernet switches has been made. Table 12 Ethernet Switch solutions Vendor Ports Type of ports managing Pins (size) PHY needed for 1000BASE-T ? Power Distributor Price estimate VSC7389 Vitesse 8+8 8 x SGMII + 8 x Copper Unmanaged + Web-Managed BGA 596 (35 x 35mm) no BCM56228 Broadcom 8+4 8 x SGMII 4x SGMII/2.5GbE BCM53716 Broadcom 16 16 x SGMII 88E6182 / 85 Marvell 10 10 x SGMII FM3104 Fulcrum 10 2 x 10G 8 x SGMII Unmanaged 1152 FCBGA yes Yes yes 6.8W 1W Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 20 / 21 Table 13 Ethernet PHY solutions Part number Verdor Ports Pins (size) Input interface 10/100/1000Base-T VSC8234 Vitesse 4 (19x19mm) SGMII yes Remarks Distributor Costs BCM5461S Broadcom 1 GMII / SGMII yes 88E1112 Marvell 1 64QFN (10x10) SGMII Yes Used on RSP 9 Test From the start of the design in both the firmware and in the hardware test facilities will be implemented. On the hardware these test facilities range from the production of the board, like flying probe test points and boundary scan, till debug facilities for the firmware, LEDs and jumpers and operation status information like voltage and temperatures. 9.1 Boundary Scan On the board high pin count components are placed. To enable interconnections tests during production, boundary scan will be implemented on the board. Due to the large number of components and the different types of components (FPGAs, optical transceivers and Ethernet devices) a bridge will be placed on the board. This bridge will have an input via the backplane to enable system level boundary scan. For the FPGA TAPs bypass possibilities will be implemented to skip devices. 9.2 Operation Status information On UniBoard monitoring will be placed on: FPGA temperature Supply voltages Power supply status Doc.nr.: DESP Rev.: Date: Class.: ASTRON-RP-303 0.1 12-01-09 Public 21 / 21