Download The OCP 700W-SH AC/DC power converter, a single voltage 12.5

8 Ways the Ideas at OpenCompute.org
will Change the Data Center Industry
Forever
Kun Liang
A20297859
ITM-535
Contents
1.
Open Rack ........................................................................................................................................... 4
2.
Electrical and Mechanical Design ..................................................................................................... 5
3.
Battery Cabinet ................................................................................................................................... 7
4.
Virtual IO .......................................................................................................................................... 10
5.
Intel and AMD Motherboard........................................................................................................... 10
6.
Power Supply..................................................................................................................................... 12
7.
Open Hardware Management ......................................................................................................... 14
8.
Open Vault Storage........................................................................................................................... 15
Reference ................................................................................................................................................... 17
A small team of Facebook engineers spent the almost two years tackling a big challenge:
how to scale the computing infrastructure in the most efficient and economical way possible.
They used:
-
Use a 480-volt electrical distribution system to reduce energy loss.
-
Remove anything in our servers that didn’t contribute to efficiency.
-
Reuse hot aisle air in winter to both heat the offices and the outside air flowing into
the data center.
-
Eliminate the need for a central uninterruptible power supply.
The result is that the trail data center uses 38 percent less energy to do the same work as
Facebook’s existing facilities, while costing 24 percent less.
Inspired by the model of open source software, they shared the innovations in their data
center for the entire industry to use and improve upon. They announced the formation of
the Open Compute Project, an industry-wide initiative to share specifications and best practices
for creating the most energy efficient and economical data centers.
My research paper is discussing about 8 ideas mentioned in OpenCompute.org will
change the Data Center industry forever. The 8 ideas are Open Rack,
1. Open Rack
The Open Rack can be configured as a server rack, storage box, and more. The
Open Rack uses an all-encompassing design to accommodate compatible Open Compute
Project chassis components, and include the power solution as well as input and output
voltage distribution. This Open Rack implementation is a single column rack in a single
cabinet. The rack is divided into one or more power zones. A power zone comprises an
equipment bay for the compute, storage, or other components and a power shelf, which
powers the computing components in the equipment bay. The power shelf includes
backup power capability using an external Open Compute Project Battery Cabinet, or it
comprises a power shelf and a Battery Backup Unit that embeds batteries. If there are
multiple power zones, they are stacked one above another. The power shelf is described
in Power Shelf Specifications. In this implementation, the rack has two Ethernet
switches that are installed at the top of the rack, above the topmost power zone.
This Open Rack implementation also include detail specification about:
-
Chassis Dimension Guidelines
-
Equipment Bay Options
-
Suggested Rack Heights
-
Prescribed Materials
-
Thermal Specifications
-
Power Shelf Specifications
-
Powering the IT Compute Chassis
-
Power Shelf and Battery Backup Options
-
Power Distribution Units (PDU)
-
Suggested PDU Dimensions and Configurations
Open Rack implementation example:
2. Electrical and Mechanical Design
The electrical system uses 480/277VAC distribution with a 48VDC UPS system
integrated with a server power supply. The systems are designed to meet or exceed these
standards: NEC, NFPA, ANSI, SFM, IEEE, NEMA, UL, IES, Local and State Building
Code.
Featured Electrical Systems
In addition to standard data center electrical design elements (from grounding to
transformers), the Facebook trail facility features these electrical and related systems:
• 115 kV substation
• Diesel backup generators
• Main switchboards (MSB)
• Battery monitoring
• 480/277VAC distribution switchboards
• Server battery backup
• Power busduct system
• Nitrogen-based fire sprinkler system eliminates pipe corrosion
• LED lighting systems
• Reactor power panels
• Cam lock connections
They designed a system with very low operating cost and a relatively low
installed cost when compared to a conventional data center. Energy intensive cooling
systems are replaced with far simpler and lighter technologies that allow for a ductless
overhead air distribution that can operate in an expanded temperature and humidity range
beyond 2008 ASHRAE TC9.9 guidelines.
3. Battery Cabinet
One component of Open Compute Project is the battery cabinet. The battery
cabinet is a standalone independent cabinet that provides backup power at 48VDC
nominal to an Open Compute Project server triplet in the event of an AC outage in the
data center. The batteries are a sealed VRLA 12V nominal, high-rate discharge type with
a 10 year lifespan, commonly used in UPS systems, connected in a series of four
elements for each group, for a nominal string voltage of 48V DC. There are five strings in
parallel in the cabinet.
The backup power requirement for the principal application is in the range of
42KW (6 x 7KW) to 72KW (6 x 12KW). The preliminary power level for all calculations
is 56.6KW (6 x 9KW + 2.6KW) for the whole cabinet.
The principal application is a system load composed of six independent main
loads (9KW each nominal) and one secondary load (2.6KW nominal). The main load is
the server load (offline backup power) while the secondary load is the IT switches load
(online power and backup power). All loads are connected in parallel to the battery
cabinet for the related backup functions. However:
while it is powered by a separate AC/DC source when AC is present.
batteries, whether or not AC is present. Any controllers needed in this
application may also run on batteries if they need to be functional all the
time.
There are three rectifiers in the cabinet fed directly by 277VAC (phase and
neutral from the 480VAC three-phase power distribution) implementing the functions of
charger, trickle charge, and supply for the SCL. During the trickle-charge phase, the
rectifier station supplies just enough current to compensate for the battery's internal
leakage plus feeding the SCL that is always present (2.6KW maximum).
The rectifiers work in conjunction with their controller (which may or may not be
included within the rectifiers' shelf itself) for the charging process and control.
The same controller may also handle the real-time impedance battery monitoring
process, in addition to the charging process and control. Otherwise, a second "slave"
controller may be used.
The controller continuously monitors the health of the batteries (reading of V, I,
T, Z, or a subset, including remote data accessibility, EOL approaching status_alarm, and
so forth).
The cabinet includes also several breakers, quick fuse disconnects, Hall sensors, a
high current DC bus bar, and an LVD device.
4. Virtual IO
The Open Compute Project seeks to provide scalable compute resources and
infrastructure in an open manner, aligning with open source tenets.
In a scale compute environment there is a tipping point where the demands of
“scaling” introduce multifaceted problems with supply chain activities that require
substantial time and investment to manage. Operational costs and component life cycles
have posed barriers to operational efficiency and capital costs; the approach of tailoring
hardware designs to reduce complexity and maximize efficiency is a viable means of
offsetting some of this cost.
The Virtual IO group believes that being able to combine and re-combine
resources such as compute, memory, networking and storage (to name the most pressing)
to fit a required need offers a new path to obtain greater efficiency. In short, Virtual IO
has the ability to reduce the support burden, reduce the time to deliver a usable IT
framework, and reduce the financial overhead of operating at scale; and do so utilizing
open standards that are consistent in approach and application.
5. Intel and AMD Motherboard
The Facebook engineers designed two different motherboards for the Open
Compute Project: one for AMD CPUs and the other for Intel CPUs. Both motherboards
have similar feature sets, including a direct interface with the power supply. The boards
are bare-bone designs; they removed many common features that didn't need, like
multiple expansion slots. In some cases they designed workarounds — like reboot over
LAN — to maintain the functionality of removed components without adding costs.
-Intel Motherboard
The OCP specification for the Intel v2.0 motherboard uses two Intel® Xeon® E52600 processors and 16 DIMM slots per board. The boards are power-optimized,
barebones designs that provide the lowest capital and operating costs. Many features
found in traditional motherboards have been removed from the design. The v1.0 Intel
motherboard is a dual Intel Xeon® 5500 or Intel Xeon® 5600 socket motherboard with
18 DIMM slots.
-AMD Motherboard
The OCP specification for the AMD v2.0 motherboard is a dual AMD G34 socket
motherboard with 16 DIMM slots. The boards are power-optimized barebones designs
that provide the lowest capital and operating costs. Many features found in traditional
motherboards have been removed from the design. The v1.0 AMD motherboard is a dual
AMD Opteron® 6100 Series socket motherboard with 24 DIMM slots.
6. Power Supply
One of the most challenging pieces of hardware to develop for the Open Compute
server was the power supply. They decided to push the limits of efficiency and at the
same time create a novel backup system to replace the traditional data center UPS.
The OCP 700W-SH AC/DC power converter, a single voltage 12.5Vdc, closed
frame, self-cooled power supply used in high efficiency IT applications. The supply is
configurable to a 450W-SH power rating (like the Open Compute Project 450W power
supply), as both models use the same PCBs, with just pin-to-pin component
replacements.This device works in conjunction with the Open Compute Project battery
backup cabinet and custom power strips.
The original OCP 450W power supply is an AC/DC power converter, single
voltage 12.5VDC, closed frame, self-cooled power supply used in high-efficiency
applications. The power converter includes independent AC input and DC output
connectors, plus a DC input connector for backup voltage. Current sharing and parallel
operations are not required, while the main focus is a design with very high electrical
efficiency.
7. Open Hardware Management
Scale computing requires a small and stable set of tools to remotely management
machines. The strategy is to define these tools by looking at the needs of a scaleout/stateless machine system administrator to trouble shoot and provision machines.
These use cases tease out the actual requirements and prevents features non-essential
feature out of the specification which divided into three sections:
1.
Approach
2.
Scale-out / Stateless Machine System Administrator Tasks and
recommended best practices
3.
Remote Machine Management Specification
Examples of the scale-out SA tasks are:
•
Initial machine provisioning
•
Reset machine BMC to defaults
•
Wrong boot order
•
Odd machine behavior
•
Hung Machine
•
Understanding power and cooling
•
Security and authorizing systems admins
•
Inventory
•
Machine crash analysis
For each scale-out task, there are recommended tool/functional best practices that
will be used with existing standards. As well, there are some infrastructure requirements
implicit in these best practices. When matching the recommendations, some of the
aspects are minor features that may be dropped, or some functions may have more
features than required.
8. Open Vault Storage
The Open Vault storage unit is a 2U-30HDD storage enclosure, consisting of two
identical 1U high HDD trays with 15 x 3.5" HDDs and two SAS expander boards on
each, one fan control board, and six redundant fan modules mounted externally in the rear
of the chassis. An Open Vault storage server fits into the Open Compute Project Open
Rack.
Each HDD tray contains one drive plane board and two SAS expander boards,
interfaces external to one or more separate server(s) via x4 SAS 6G link(s). Each SAS
expander board and HDD tray can be extracted and serviced independently without
impact to the other connected trays. This provides for the easy replacement of one SAS
expander board or the replacement of a single HDD while keeping the whole system
running. Each fan module is hot pluggable and field replaceable from the rear of the
chassis. Also there are power connecter(s) to the Open Rack bus bars so that the 12V
main power from the Open Rack can be fed into the Open Vault enclosure.
Reference
https://www.facebook.com/notes/facebook-engineering/building-efficient-datacenters-with-the-open-compute-project/10150144039563920
http://hardware.slashdot.org/story/12/09/19/2123219/open-compute-projectpublishes-final-open-rack-spec
http://opencompute.org/