Download Technology - Lyle School of Engineering

Technology
The IEEE802.3ae 10-Gigabit Ethernet has been developed from earlier Ethernet standards.
The original Ethernet standards worked over a shared medium, i.e., a single cable connecting all
the nodes together. Starting with 10Base-T, the network changed from a single cable to a star
topology with a repeater at the center. Although there were now two wires between each node
and the repeater, the network was still a shared medium and the total network bandwidth was
only 10 Mbit/s. By replacing the repeater with a switch, 10Base-T can support full duplex traffic –
10 Mbit/s in each direction between each node and the central switch. The total network
bandwidth is now the sum of the links to and from the switch, although the bandwidth may be
limited by the switch’s internal architecture. With 100-Mbit/s Ethernet and Gigabit Ethernet, the
bandwidth on each port of a switch is set following auto-negotiation between the node and switch.
Some ports support 10-Mbit/s, 100-Mbit/s or 1-Gbit/s Ethernet.
A key part of the Ethernet protocol has been the CSMA/CD collision detection mechanism used
to arbitrate for the shared media. Ten-Gigabit Ethernet is the first Ethernet technology to become
entirely full duplex and therefore does not support CSMA/CD.
To accelerate time-to-market for Gigabit Ethernet the Gigabit Fibre Channel PMD (Physical
Media-Dependent) layer was used for the optical interfaces. The 10-Gig Ethernet standard has
now leapfrogged over Fibre Channel development, and therefore several new optical PMDs have
been developed. Now the 10-Gig Fibre Channel standard is expected to use a 10-Gigabit
Ethernet PMD.
Gigabit Ethernet uses the 8B/10B coding to include additional symbols at the PMD layer.
Unfortunately, this increases the signal frequency to 12.5 GHz. To achieve the 10-Gbit/s
bandwidth and limit the signal frequencies as close to 10 GHz as possible, a new, more efficient
64B/66B coding was developed. Ten-Gigabit was originally defined as an optical-only network,
but there is now work in the Institute of Electrical and Electronics Engineers Inc. (IEEE) to support
copper media (see Startups Move 10-Gig to Copper ).
A significant goal for 10-Gigabit Ethernet has been to stretch the range from the 5 kilometers in
Gigabit Ethernet to 10 and 40 km. There are now a number of companies developing proprietary
optics to take Ethernet to 80 km, 100 km, and beyond.
There has been some deployment of Gigabit Ethernet in the WAN. A significant hurdle, however,
has been the different data rates supported by Gigabit Ethernet and 622-Mbit/s or 2.5-Gbit/s
Sonet/SDH. Unlike all other Ethernet standards, 10-Gig Ethernet is close to a Sonet/SDH data
rate (OC192/STM64). To improve compatibility further, the IEEE has defined the WAN PHY,
which will interface directly with 10-Gbit/s Sonet/SDH networks. The WAN PHY includes a WAN
interface sublayer (WIS), which is a cut-down Sonet framer.
Table 1 shows the main differences between 10-Gigabit Ethernet and Gigabit Ethernet.
Table 1: Gigabit vs 10-Gigabit Ethernet
Gigabit Ethernet
CSMA/CD + full duplex
10-Gigabit Ethernet
Full duplex only
Leveraged Fibre Channel PMDs New optical PMDs
Reused 8B/10B coding
New coding schemes 64B/66B
Optical/copper media
Optical media only
(copper in development)
Support LAN to 5 km
Support LAN to 40 km
Carrier extension
Throttle MAC speed for WAN
- Use Sonet/SDH as Layer 1 transport
Ten-Gigabit Ethernet has the same layers as other Ethernet technologies: Media Access Control
(MAC), Physical (PHY), and Physical Media-Dependent (PMD). Ten-Gig Ethernet also supports
all the existing Ethernet technologies, such as group trunking, and is therefore simple to introduce
into an existing Ethernet-based network.
Interfaces between the layers have been standardized. The 10-Gig Ethernet MAC is connected to
the PHY through either the XGMII interface or the low pin-count alternative, XAUI. The XAUI
interface has 4x 3.125-Gbit/s channels with 8B/10B encoding. XAUI can also be used to connect
the PHY to the PMD.
The IEEE has defined a total of seven optical port types for 10-Gig Ethernet; these are shown in
Table 2, together with two proposed standards, for copper cable, shown in italics.
Table 2: IEEE 802.3ae Port Types
Device
Application
10GBase-LX4 LAN
Range
Optics
Cable
300m MMF/ 1310nm Multimode or
10km SMF
WWDM singlemode
10GBase-SR
LAN
300 m
850nm
10GBase-LR
LAN
10 km
1310nm Singlemode
10GBase-ER
LAN
40 km
1550nm Singlemode
10GBase-SW
WAN
300 m
850nm
10GBase-LW WAN
10 km
1310nm Singlemode
10GBase-EW WAN
40 km
1550nm Singlemode
10GBase-CX4 LAN
15m
-
4 x Twinax
10GBase-T
25-100m
-
Twisted pair
LAN
Multimode
Multimode
The 10GBase-LX4 interface has four low-cost lasers and supports both multimode (MMF) and
singlemode (SMF) fiber. There are three versions of both the LAN and WAN interfaces supporting
multimode fiber for short distances and singlemode fiber for longer distances. The 10-Gigabit
Ethernet distances are defined as 300 meters for short reach (SR), 10 km for long reach (LR),
and 40 km for extended reach (ER).
“Most enterprises have pulled mixed cable plant,” says Force10’s Quiros. “So they have both
multimode and singlemode fiber in their backbone. That has given them a lot of flexibility in the
way they can deploy 10-Gigabit Ethernet.”
The majority of 10-Gig Ethernet deployment so far has been in the campus backbone and
between campuses. The 10Gbase-LR and 10GBase-ER are the most suitable interfaces for
these applications. Pre-standard 10-Gig systems from Cisco featured a proprietary interface
called 10GBase-EX, which supports a 50km range.
Within the IEEE there is now work to develop two copper interfaces. The first, 10GBase-CX4,
uses existing InfiniBand IB4X connectors and four Twinax cables. The second, 10GBase-T, is a
totally new development to support 10-Gigabit Ethernet over 25 to 100 meters of standard
twisted-pair cable.
10GBase-CX4 borrows much from Infiniband and is designed for short reach, data center
applications. By using existing connectors and cables, the standards group expects to achieve full
ratification by the end of 2003.
Components
During 2001 semiconductor companies started rolling out 10-Gigabit Ethernet products. Devices
available now include:




Discrete media access controllers (MAC)
Physical layer devices (PHY)
Sonet/SDH framer/mapper devices
Gigabit Ethernet switch devices with integrated 10-Gig Ethernet MACs
We are now seeing the introduction of second-generation silicon products with lower cost and
power together with higher integration. A major focus for semiconductor companies during 2003
will be the introduction of devices to support the next generation of optical transponder and the
integration of MAC and PHY in a single device.
The first 10-Gigabit Ethernet systems used either proprietary optical modules or transponders
conforming to the 300-pin MSA (multisource agreement). These are relatively large and
expensive modules initially developed to support Sonet/SDH.
The recent introduction of Xenpak transponders is a significant step towards expanding the 10Gig Ethernet market. There are seven companies now shipping, or planning to ship these
transponders:
In production
o
o
o
Agilent Technologies Inc. (NYSE: A - message board)
JDS Uniphase Corp. (Nasdaq: JDSU - message board; Toronto: JDU)
Molex Inc. (Nasdaq: MOLX/MOLXA)
Sampling, possibly in production
o
Intel Corp. (Nasdaq: INTC - message board)
o
o
Optillion AB
OpNext Inc.
Sampling
o
Mitsubishi Electric Corp.
Vendors that no longer list Xenpak modules
o
o
Infineon Technologies AG (NYSE/Frankfurt: IFX - message board)
TriQuint Semiconductor Inc. (Nasdaq: TQNT - message board)
Xenpak transponders are significantly smaller and cheaper, enabling more cost-effective
solutions with up to four 10-Gbit/s ports per line card. They are hot-pluggable, allowing a “pay as
you go” approach, with the expensive transponder modules being added as additional ports are
required.
The first systems with Xenpak transponders are now entering the market. Future cost reductions
will be driven by the introduction of even smaller transponders based on the XPAK, X2, and XFP
MSAs.
XPAK and X2 transponders use the same XAUI interface as Xenpak, but will also support 10Gigabit Fiber Channel. XPAK and X2 transponders are 40 percent smaller than Xenpak
transponders. The XPAK and X2 MSAs are very similar, and it is unclear at the moment which
will be more widely used (see The X-Wars: Agilent Strikes First ).
XFP transceivers, that integrate a new serial interface (XFI), will further reduce system cost. They
have a very small footprint and will support 10-Gigabit Fiber Channel and 10-Gbit/s Sonet/SDH
as well as 10-Gigabit Ethernet (see XFP Module Group Debuts Spec ).
Typical Systems
This section will look at several products from a few of the many companies now delivering 10Gigabit Ethernet enabled systems. These include core routers, enterprise and metro switches,
and an NIC. These products are summarized in Table 3:
Table 3: Typical Systems
Company
Cisco
Chassis
12400
Series
Catalyst
Max
Number
Max
of
Max I/O
Switching
Blades Bandwidth
Bandwidth
per
Chassis
15
12
150Gbit/s
120Gbit/s
160Gbit/s
128Gbit/s
10GE
Blade
10GE
Ports
per
Blade
Optical
Module
10GE
Port
Types
Switch
10GE
Bandwidth Ports
per 10GE
per
port
Chassis
10GE
10km
1
Proprietary 10GBase- 10Gbit/s
LR
15
10GE
40km
1
Proprietary 10GBase- 10Gbit/s
ER
15
10GE
10km
1
Proprietary 10GBase- 10Gbit/s
LR or
10GBaseER
11
Extreme
Black
Diamond
Force10
E Series
Networks
Foundry
Intel
BigIron
NetIron
FastIron
16
160Gbit/s
128Gbit/s
10GLRi
1
300 pin
10GBase- 8 Gbit/s
LR
16
14
280Gbit/s
640Gbit/s
2-port
LAN
1310
2
300 pin
10GBase- 40 Gbit/s
LR
28
2-port
LAN
1550
2
300 pin
10GBase- 40 Gbit/s
ER
28
2-port
WAN
1310
2
300 pin
10GBase- 40 Gbit/s
LW
28
LAN 850 1
300 pin
10GBase- 8 Gbit/s
SR
15
LAN
1310
1
300 pin
10GBase- 8 Gbit/s
LR
15
LAN
1550
1
300 pin
10GBase- 8 Gbit/s
ER
15
2 Port
2
XENPAK
XENPAK
10GBase- 4 Gbit/s
LR or
10GBaseER
30
PCI-X
Card
XENPAK
10GBase- 4 Gbit/s
LR
1
15
PRO/10GbE 1
LR Server
Adapter
300Gbit/s
10Gbit/s
120Gbit/s
-
1
In most cases there are a range of systems available, typically with four, eight, or 16 I/O slots.
For each family, the maximum number of I/O slots is shown. This capacity is assuming no
redundancy. Some systems, such as the Cisco 12400, use one of the slots to support a 1:1
redundant switch fabric, reducing the number of I/O slots, in this case, to 14. Other systems, such
as the Extreme Networks Black Diamond, support graceful degradation of performance if a switch
card fails but do not support 1:1 redundancy.
The fourth column shows the maximum I/O bandwidth with 10-Gigabit Ethernet blades fitted in
each slot. For systems, like the Foundry Networks BigIron family, that can support either a single
or dual 10-Gig Ethernet blade, it is assumed that dual 10-Gig Ethernet blades are fitted in all
available slots. All of the products listed here, except the Intel Server card, can support Gigabit
Ethernet ports as well, and most support a range of ports including Sonet/SDH and TDM.
The fifth column shows the maximum switching capacity of a fully loaded system. Some systems,
such as the Extreme Black Diamond and Foundry BigIron, only support 8-Gbit/s switching
bandwidth per I/O slot. The Force10 Networks E Series is the only system with enough raw
switching bandwidth to support 40-Gbit/s per I/O slot.
“We have a next-generation general purpose platform,” says Force10’s Quiros. “We have
targeted the group of users that need next-generation capability and full line-rate 10-Gigabit
Ethernet and Gigabit Ethernet with advanced capabilities like access lists and QOS, running
simultaneously without impact on performance.”
The second half of the table shows the type of blade and the number of 10-Gig Ethernet ports per
blade. Included are details of the type of optical module fitted and the 10-Gig Ethernet Port types
supported. The Foundry 2 Port Xenpak is the only announced blade supporting Xenpak
transponder modules. These blades, like the Cisco Catalyst blades, can be shipped with one of
two types of optical module.
“Our first 10-Gigabit Ethernet products started shipping in Q4 of 2001 for revenue and was the
industry's first 10-Gigabit Ethernet product to ship,” says Foundry’s Kopparapu. “We are now
introducing a second-generation 10-Gigabit Ethernet module, which is based on a Xenpak optics
and supports both 10GBase-LR and 10GBase-ER.”
Finally, for each blade type, the table shows the maximum, switched bandwidth available for each
10-Gig Ethernet port and the maximum number of 10-Gig Ethernet ports that can be supported in
a single chassis.
The Foundry 2 Port Xenpak blade has a switched bandwidth of only 4 Gbit/s – however, this
matches the maximum bandwidth supported by the Intel PCI-X server card. The PCI-X bus limits
the Intel server card bandwidth, and any other similar card, to 4 Gbit/s.
Market Overview
The first 10-Gigabit Ethernet equipment was announced at the end of 2001. In 2002, more than
22 vendors took part in interoperability demonstrations at Networld+Interop in Las Vegas and at
Supercomm in Atlanta (see Vendors Show Off 10-GigE at N+I and 10-GigE Vendors Get Cold
Feet ).
According to In-Stat/MDR, 10-Gig Ethernet only shipped about 500 ports in the first half of 2002.
This initial equipment deployment was primarily to educational and research establishments. Now
that 10-Gigabit Ethernet has been proven in the field, more extensive rollouts are underway.
“If you lease a dark fiber, you can simply light it up with 10-Gigabit Ethernet,” says Foundry’s
Kopparapu, ”and then connect your campuses in a ring or star or whatever topology you choose.”
In-Stat/MDR forecasts core LAN switch revenues to reach nearly $25 billion by 2006, with a 43
percent CAGR (compound annual growth rate). With port costs continuing to come down, InStat/MDR is forecasting a significant increase in the number of Ethernet ports – especially for
Gigabit and, to a lesser extent, 10-Gigabit Ethernet.
This growth in gigabit connectivity within the enterprise is building a huge latent demand for 10Gig Ethernet in the backbone. The key to turning this latent demand into system shipments is the
reduction in 10-Gig Ethernet systems cost that is now just starting to hit the street, as companies
introduce second-generation products.
Ten-Gigabit Ethernet already has significant cost advantages over competing technologies such
as OC192 (10 Gbit/s) Sonet and a one-year advantage over 10-Gigabit Fibre Channel for storage
networks. The cost of a 10-Gigabit Ethernet port today can be as low as $17,000 – or $35,000
with the chassis included – making this relatively cost-effective bandwidth.
Switches and routers with 10-Gigabit line cards are available from a large number of companies
including Cisco Systems Inc. (Nasdaq: CSCO - message board), Extreme Networks Inc. (Nasdaq:
EXTR - message board), Force10, and Foundry. Intel is the first company to introduce a 10Gigabit Ethernet NIC. (Products from these companies are examined in Part 6.)
Applications
Ten-Gigabit Ethernet has been developed to support a wide range of applications – from the
enterprise network, through the edge and metro, into the wide area.
The Enterprise: Data Centers and Backbones
The figure above shows 10-Gigabit Ethernet in an enterprise network. Ten-Gig Ethernet is used
for the enterprise backbone, both within a campus and between campuses, and to connect the
server farm in the corporate data center.
The hottest market today for 10-Gigabit Ethernet is in the data center. The performance of
servers has been increasing significantly with clock rates up to 3 GHz and faster storage
technology. To move data in and out of these high-performance servers, Gigabit Ethernet
network interface cards (NICs) are being fitted as standard. By moving to these high-performance
servers and connecting them together with 10-Gigabit Ethernet, companies can consolidate their
file servers into a small number of high-capacity data centers. This consolidation can yield
significant cost savings and higher system throughput. The introduction of a 10-Gigabit Ethernet
NIC from Intel Corp. (Nasdaq: INTC - message board) will further extend the benefits of 10-Gig
Ethernet in this application.
The biggest market in the future for 10-Gigabit Ethernet is likely to be in the corporate backbone.
The cost of Gigabit Ethernet has dropped significantly over the last 12 months, and most 10/100Ethernet workgroup switches now have Gigabit uplinks. High-end PCs are now becoming
available with integrated Gigabit Ethernet connectivity. The range of 5 kilometers limited the use
of Gigabit Ethernet between campuses. With 10-Gig Ethernet transponders capable of up to 40
km, the range is no longer an issue for Ethernet and this market.
The Metro: Lighting Up the Fiber
There is a major shift in the access market away from TDM towards 10/100 Ethernet, and even
Gigabit Ethernet, for business connectivity. This is opening up new demand for Ethernet in the
metro. The main deployment so far has been in the Asia/Pacific region; however it is likely that
this will grow in the U.S. and Europe during next year.
Using existing dark fiber, service providers can extend their Gigabit Ethernet networks into the
metro edge. As demand grows, the bandwidth through a single fiber is restricted to one gigabit,
unless expensive DWDM equipment is used to multiplex multiple gigabit feeds. By using 10Gigabit Ethernet, this multigigabit bandwidth can be achieved at significantly lower cost.
In the metro area, 10-Gigabit Ethernet can be deployed in either a star or ring topology. Unlike
Resilient Packet Ring (RPR), these networks use the standard Ethernet MAC protocol. The latest
10-Gigabit Ethernet metro switches can provide network reliability similar to those based on
Sonet/SDH rings.
“At Foundry Networks we support the Metro Ring Protocol (MRP), which is designed to work with
standards based on 10-Gigabit Ethernet and Gigabit Ethernet and offers sub-second protection,”
says Chandra Kopparapu, director of product marketing at Foundry Networks Inc. (Nasdaq:
FDRY - message board).
The WAN: Limited Deployment... So Far
Ten-Gigabit Ethernet has been designed to support the wide-area network with WAN-specific
physical layers. The combination of a very significant installed base of Sonet/SDH and the lack of
major infrastructure investment by carriers since its introduction has limited the deployment of 10Gig Ethernet in the WAN.
One important application for 10-Gigabit Ethernet in the WAN is grid computing – the subject of a
Light Reading Webinar in February 2003. Huge server farms spread across multiple locations
across the wide area are interconnected using Gigabit and 10-Gigabit Ethernet to form massive
virtual supercomputers. One example of grid computing is the TeraGrid project launched by the
National Science Foundation (NSF) in August 2001.
Storage: Fibre Channel or iSCSI
The storage market is seen as a likely killer application for 10-Gigabit Ethernet. The ratification of
the iSCSI standard last month by the Internet Engineering Task Force (IETF) is key to supporting
storage services over IP and 10-Gigabit Ethernet (see iSCSI Gets Go-Ahead ). Most storage
networks today are based on dedicated storage technologies such as Fibre Channel, though
there are already significant deployments of 10-Gig Ethernet in the data center.
“We see demand for 10-Gigabit Ethernet in storage applications that are based on IP, not
necessarily iSCSI at the moment,” notes Foundry’s Kopparapu.
For the next-generation 10-Gigabit storage network there is now a choice. On the one hand, there
is 10-Gigabit Fibre Channel; on the other, there is iSCSI running over 10-Gigabit Ethernet. Fibre
Channel is a well understood technology for this application. However, 10-Gig Ethernet and iSCSI
could bring significant cost savings – both through less costly equipment and the integration of
storage and corporate networks.
“People are hanging on to their SANS right now. It is technology they are comfortable with,” says
Rob Quiros, director of product marketing at Force10 Networks Inc. “People are essentially
running two parallel networks. They are running their server farms with Gig Ethernet or 10-Gigabit
Ethernet connecting out to the LAN or the metro area; and they are using SAN and Fiber Channel
on the back end to connect to the storage. Ten-Gigabit Ethernet gives them the ability to collapse
these two networks into one. That is when the performance is going to be crucial.”