Download Microserver design and testing

September 2016
IBM Research - Zurich
Bringup Testing with Python and PSoC
Andreas Döring, IBM Zurich Research Laboratory
1
© 2016 IBM Corporation
© 2015 International Business Machines Corporation
1
IBM Confidential
Prototype-Testing and In-System-Test have many
Resources in Common
© 2014 International Business Machines Corporation
IBM Research - Zurich
Modular, high-density Microserver
Compute node
DOME 8-way u-server cluster
u-switch
module
© 2016 International Business Machines Corporation
Next gen 32node back
plane with
Storage and
Management
3
Overview
§ Complex System consisting several, interdependent components
§ Commercial subsystems with only partial documentation included (e.g. SSD)
§ New, series of operations needed for startup
– Many steps gets “automated” later, but during first tests needs to be done explicitely
© 2016 International Business Machines Corporation
4
Prototype Test, Production Test, Self-Test in Final Product
Design
Prototype
Production
Prototype
Test
Series
Production
Assembly-Test/
In-System Test
© 2016 International Business Machines Corporation
5
Target System
Final System:
128 compute modules
Current Test system:
32 compute modules
1 Switch
2 Power Converters
2 Storage Modules
(8 mSATA disks)
Nodes are very dense,
no PCB space for test points,
hard to reach pins
© 2016 International Business Machines Corporation
6
Topology of One (Example) Module
T4240 SoC
Temp. Sense
UART
I2C
JTAG
Support
Controller
(PSoC)
USB
PMBus
Power
Converter
© 2016 International Business Machines Corporation
Power
Switches
7
Example: T4240v2 startup -- Snapshot
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
Enable Pull-up for I2C on PSoC
Test PMBus to power converter
Check, if power converter is setup correctly, otherwise needs to be programmed
Configure the current sense
Enable 1.8V supply
Enable Power Stages (of Core Power Converter)
Turn on Core Power (PMBus command)
Enable DRAM power
Fill EEPROM emulation memory with Reset-Configuration-Word
Reset JTAG
Deactivate Reset-signals (DRAM, Core-reset, SoC-Reset)
(optional) download u-boot (“BIOS”) binary
© 2016 International Business Machines Corporation
8
Challenges
§ Complex system with high-speed signals (DDR3, DDR4 memory, SATA, Ethernet)
§ High signal density on connector and module (3 72-bit-wide memory channels on
35x5cm), 4x10Gbit Ethernet + 2xSATA+PCIe per module, Switch with 64 ports)
§ High power density (around 2kW for one BB2)
§ Heterogeneous system with many aspects (OS-level management, network
management, power, temperature, etc.)
© 2016 International Business Machines Corporation
9
Approach
§ USB-microcontroller on each module
–
–
–
–
–
–
Firmware update through USB
Minimal requirements for operation (3.3V power, only few decouple caps, usb-wires)
Visibility of many board aspects (voltages, currents, status lines)
Module firmware (feature matrix defines for each module which functions are supported)
In all current modules: “Cypress PSoC”, allows programmable analog and digital functions, both used
JTAG controller (FTDI’s MPSSE protocol, compatible with many test software packages, e.g. UrJTAG)
§ Host:
–
–
–
–
–
–
Linux/Windows (also Mac tested)
First C, now mainly Python code
Modular, object oriented
Python provides an interactive shell
Some boards have a GUI
IPMI fully supported
© 2016 International Business Machines Corporation
10
Typical development/testing sequence
§ Starting with low level python commands, e.g. directly writing into port register to
enable a power converter.
§ Combining several such commands into a python function
§ (Optionally): if faster or autonomous operation required: turn into C-code on controller
© 2016 International Business Machines Corporation
11
Automatic Test Sequences
§ Bootloader uses checksum of firmware code
§ Firmware checks raw supply voltage (on some modules, e.g. power converter)
§ Firmware startup behavior can be configured in EEPROM
§ Die Temperature is sensed through diode, at over-temperature, the power is turned off
§ Temperature is collected over all modules
§ For example, we found one module which poor thermal contact as it’s temperature rose
quicker than in all other modules
© 2016 International Business Machines Corporation
12
Convenience of Python
§ Class SenseResult
§ Each module has a function voltages()
§ To display a summary:
h.voltages()
§ To display the temperature of all compute modules in a system:
for hi in y.h:
hi.voltages().data[“Die Temperature”]
§ As easy, writing selected data to a file, e.g. for calibration
§ Though python is a “managed” language (with garbage collection, etc.) it runs fast
enough on the host for all our purposes
© 2016 International Business Machines Corporation
13
Observations
§ Restricitions of tools through licenses/confidentiality
– E.g. NXP tool to check reset state, tune DRAM-timing, etc. requires NXP’s JTAG tap
– Connectivity testing of DRAM would require third-part-tool with third-party JTAG adapter
– Need JTAG tap on board to configure Boot-flash without processor
(using Open Source tools)
– Non-recommended structure of Scan-chain
§ Availability of on-board monitor very useful
© 2016 International Business Machines Corporation
14
Conclusion
§ Prototype testing is a versatile task
§ Need to “build” setups for individual tests
§ Current components require software for most aspects
§ The combination of a microcontroller on the board (programmed in C)
with Python code on a host provides a flexible, agile work environment
§ After refinement the code becomes part of the final system
§ Provides In-System test capabilities
© 2016 International Business Machines Corporation
15