Download 091-93 IntelPCI.qxd

091-93 IntelPCI.qxd
6/16/04
3:03 PM
Page 91
SYSTEM ARCHITECTURE
Exploring the Advantages
of PCI Express
Parallel encoding schemes such as parallel ATA, SCSI, Peripheral Component Interconnect (PCI), and Peripheral Component Interconnect Extended (PCI-X) are being replaced
by high-speed serial alternatives based on Peripheral Component Interconnect Express
(PCI Express™) architecture. Designed to meet the requirements of today’s high-bandwidth
applications, PCI Express creates a point-to-point serial architecture that offers key performance, cost, and scalability advantages.
BY CHRIS CROTEAU AND PAUL LUSE
T
o enable quicker response to business needs, enter-
Overcoming the limitations of parallel encoding
prises demand that the IT infrastructure process infor-
Serial protocols replace parallel encoding schemes at the
mation faster, access shared data more efficiently, and
physical layer with high-speed serial encoding. Already
improve end-to-end throughput. Such requirements are
fundamental to certain data communications protocols—
driving developments in processor, storage, and network
such as Fibre Channel, Asynchronous Transfer Mode
architectures that can help create more scalable infra-
(ATM), and Ethernet—serial architecture has recently
structures by overcoming the inherent limitations of
emerged in areas where parallel encoding schemes have
parallel protocols.
been prevalent for the past 15 years. The protocols for
Serial architecture can help administrators accom-
parallel ATA, SCSI, and Peripheral Component Inter-
plish business objectives by improving the performance
connect (PCI ®)—all of which are based on parallel encod-
and expanding the capabilities of network connections,
ing schemes—share many of the same characteristics and
drive interconnect protocols, and primary I/O buses for both
limitations, including high pin count, low frequencies,
server and desktop computing platforms. The Peripheral
and shared buses. Serial technology can help mitigate
Component Interconnect Express (PCI Express™) specifi-
each of these issues:
cation helps pave the way by providing a system interconnect for computing and communications platforms.
•
High pin count: Serial protocols do not need to
This article explores the concept of serialization, focus-
transfer a large amount of data in each clock cycle,
ing on the PCI Express architecture and its potential to
so they require fewer data lines than parallel
enable next-generation computing platforms for high-
schemes. Fewer data lines fit into smaller cables,
bandwidth enterprise applications.
which potentially can lower costs.
www.dell.com/powersolutions
POWER SOLUTIONS
91
091-93 IntelPCI.qxd
6/16/04
3:03 PM
Page 92
SYSTEM ARCHITECTURE
•
•
Parallel PCI has served as the dominant I/O bus for more than
Low frequencies: Unlike serial data transfers, parallel transfers
are frequency limited because of skew factors. Skew results
a decade. However, high-bandwidth applications—some available
from the requirement to synchronize all arriving data lines
today and many anticipated in the future—and usage models place
with one another at the receiving end of a parallel transfer.
demands on CPUs, I/O devices, and the I/O bus that neither PCI nor
Lower frequencies translate into lower performance when the
Peripheral Component Interconnect Extended (PCI-X®) is equipped
bus width is limited by physical and electrical considerations.
to meet, largely because of their parallel encoding architectures. For
Shared buses: Serial encoding schemes employ a point-to-
example, a high-volume, low-cost server is often used to support an
point topology, whereas parallel encoding usually requires
I/O-intensive video-on-demand system. Such systems are required to
multiple devices to access a shared set of data lines. In the
serve multiple streams of time-dependent data concurrently. Specifi-
latter case, resource contention can lead to suboptimal
cations to provide this type of capability are built into PCI Express.
performance.
Besides using serial encoding to help overcome the limitations of a
parallel protocol, PCI Express offers the following features:
Examining the move toward serial architecture for ATA and SCSI
With the emergence of Serial ATA and Serial Attached SCSI, the
•
transition of disk protocols from parallel to serial architecture is
well under way. Serial ATA, a high-speed point-to-point link for
Cost-effectiveness: Works with high-volume, low-cost
components such as standard chips and connectors
•
Performance across market segments: Meets the require-
ATA, was created to address performance-limiting design issues
ments of mobile, desktop, server, and communications
in the ATA specification. Serial ATA increases bandwidth and sim-
applications
plifies topologies. Originally conceived as a desktop architecture,
•
Scalability and life span: Scales to accommodate future
Serial ATA helps to improve performance and promises economic
applications through its capability to aggregate links to gain
benefits that have prompted storage designers to consider it for
additional bandwidth, and is designed with a life span to
enterprise applications.1
rival that of PCI
Serial ATA enhances the ATA specification with capabilities
•
such as hot-plug drive swapping and native command queuing,
which were not included in the parallel specification. By overcom-
Compatibility: Offers compatibility with previous PCI
and PCI-X specifications and programming models
•
ing the signal timing issues inherent in parallel ATA, Serial ATA allows
Data protection: Helps provide improved data integrity
and error handling
for longer cable lengths—and for narrower cable widths, which
can help improve airflow within a system enclosure.
Overview of PCI Express architecture
SCSI also is undergoing a parallel-to-serial conversion that prom-
More than an evolution of the PCI and PCI-X bus interface, PCI
ises new capabilities in addition to performance gains. Serial Attached
Express is a new, layered architecture that retains PCI and PCI-X
SCSI, the latest advance in SCSI technology following Ultra320 SCSI,
usage and software programming models. PCI Express architecture
offers serial point-to-point links that replace the parallel bus—and
is a point-to-point serial interconnect that uses low-voltage differ-
thus reduce the overhead—of today’s parallel SCSI topologies. Serial
ential signaling (LVDS). One new component in the architecture is
Attached SCSI is expected to be scalable to more than 16,000 phys-
a switch that replaces the parallel I/O bus of PCI and PCI-X. Devices
ical devices in a single domain. In addition, it offers backward com-
connect to this switch through point-to-point connections called links,
patibility with legacy SCSI drivers and software by maintaining
which consist of one or more lanes. A lane is a set of differential
compatibility with the SCSI protocol. Because Serial Attached SCSI
signals used to transmit data. PCI Express bridges to PCI and to
interoperates with Serial ATA devices, administrators can choose from
PCI-X are other critical components that will help facilitate the
a variety of drive types.
adoption of PCI Express.
The PCI Express architecture design offers the flexibility to
Migrating the I/O bus from PCI and PCI-X to PCI Express
aggregate lanes into higher-bandwidth links. A single lane, referred
The primary I/O bus is currently transitioning from a parallel to
to as a ×1 link (pronounced “by one link”), can help support up to
a serial architecture that is similar to ATA and SCSI. The PCI
250 MB/sec in each direction. For example, a ×4 link can help sup-
Express specification was developed to help minimize I/O bus
port up to 1 GB/sec in each direction (simplex) or 2 GB/sec in both
bottlenecks within systems and to provide the necessary bandwidth
directions (dual simplex). Figure 1 outlines the potential peak band-
for high-speed, chip-to-chip, and board-to-board communications
width capabilities projected in the PCI Express specification as lane
within a system.
widths are widened.
1 The Serial ATA II Working Group is developing a specification to double overall bandwidth for Serial ATA to 3 Gbps.
92
POWER SOLUTIONS
June 2004
091-93 IntelPCI.qxd
6/16/04
3:03 PM
Page 93
SYSTEM ARCHITECTURE
Lane width
Peak simplex bandwidth
Peak dual-simplex bandwidth
×¥ 1
250 MB/sec
500 MB/sec
×¥ 2
500 MB/sec
1 GB/sec
to
leverage
features
unique to PCI Express.
The transaction layer
Software
Transaction
PCI software and driver model
Packet-based protocol
of the PCI Express archi-
×¥ 4
1 GB/sec
2 GB/sec
×¥8
2 GB/sec
4 GB/sec
the creation of outgoing
×¥ 16
4 GB/sec
8 GB/sec
request packets and for
tecture is responsible for
Data link
Data integrity
Physical
Electrical signaling
the completion of incoming packets. In addition,
Figure 1. Potential bandwidth comparisons for PCI Express architecture
Figure 3. PCI Express layered architecture
the transaction layer hanFigure 2 shows a simple PCI Express topology, identifying key
dles some aspects of flow control and power management. For a trans-
elements within the architecture: the root complex and the end point,
mit operation, the transaction layer would field a read or a write—from
which are roughly equivalent to a PCI or PCI-X host bridge and a PCI
the software layer, for example—then create the corresponding packet
or PCI-X device, respectively. Bridges can accommodate interaction
and pass it to the data link layer. For a receive operation, the trans-
of PCI and PCI-X devices with PCI Express devices in the same system.
action layer would accept data from the data link layer and complete
PCI Express architecture also delivers features—such as advanced
the request in the operating system or an application in the software
error logging, advanced reporting, power management, and qual-
layer. Although all transactions in PCI Express are split transactions—
ity of service—that system architects require to meet the demands
transactions that are completed in multiple phases, which enables
of an enterprise environment. Because it requires fewer pins and
multiple transactions to be open or outstanding at one time—some
signals than PCI and PCI-X to support each link, PCI Express archi-
do not require a response. Split transactions are one of the key fea-
tecture provides a smaller slot connector and smaller form factor
tures contributing to the efficiency of PCI Express.
that can help make system layout simpler and more cost-effective.
The data link layer plays a critical role in PCI Express: it helps
ensure that data is properly ordered across each link. A link cycli-
Role of PCI Express architecture layers
cal redundancy check (LCRC) ensures data integrity, and sequence
PCI Express is a layered architecture (see Figure 3). The software layer
numbers handle the ordering of packets in the data link layer.
is responsible for the creation of a stable operating environment. This
The physical layer comprises physical components required to
layer includes services such as enumeration and configuration of PCI
configure and maintain communications across a link. These include
Express devices, and the allocation of resources such as memory and
mechanisms for link training, data scrambling, 8B/10B encoding,
interrupts. These types of services remain unchanged from those
packet framing, and signaling the data onto the link.
defined in PCI and PCI-X. Software compatibility with the PCI and
PCI-X models is a key feature of PCI Express because it helps enable
Taking IT infrastructure to the next level
existing operating systems to boot PCI Express without software mod-
Serial technology already has become rooted in enterprise storage
ifications. In addition, the runtime software model is distinguishable
and networking architectures, and the current wave of migration
from PCI and PCI-X only in that future enhancements will be designed
from parallel to serial encoding schemes is gaining momentum
throughout the IT infrastructure. Using PCI Express as a serial
I/O interconnect can help reduce the overhead for enterprise sys-
CPU
CPU
tems and bus bandwidths by using low-cost components, while
helping protect legacy software investments through backward
compatibility.
PCI Express
end point
PCI Express
PCI Express
Root
complex
Memory
PCI Express
PCI Express
PCI Express to
PCI or PCI-X
host bridge
Switch
PCI Express
PCI Express
PCI Express
PCI or PCI-X bus
Legacy
end point
PCI Express
end point
PCI Express
end point
Chris Croteau ([email protected]) is a market development director at the
Intel Communications Group, Storage Components Division. He is responsible for market creation and the promotion of Intel building blocks within server and storage market segments.
Paul Luse ([email protected]) is a senior technical marketing engineer specializing in
silicon storage components at the Intel Communications Group, Storage Components Division.
He has eight RAID-related U.S. patents pending.
Figure 2. Simple PCI Express topology
www.dell.com/powersolutions
POWER SOLUTIONS
93