Download OIF CEI-56G Application Note

OIFCEI-56GApplicationNote
CommonElectricalInterfaceat56Gb/s
Abstract:
TheOIFiswellalonginthedevelopmentoftheCEI-56Gsuiteofhighspeedinterconnect
ImplementationAgreements(IA)s.TheOIFisdevelopingIAsfornine56Gclauses,
spanningfivereachesandthreemodulationtechniques.
TheOIFhasalonglegacyofdevelopingIAsforserialelectricalinterfaces,withtheCEI
familybeingtheprincipalindustrydefinitionforSerDesforthe6Gb/s,11Gb/sand28
Gb/sgenerations.InterfacesthatleveragetheCEIfamilyofspecificationshavebeenbuilt
inmostofthemajorASICflowsandFPGAdevices.CEIhasbeenadoptedoradaptedinto
multipleinterfacesbyIEEE802.3,InfiniBandandFibreChannel.
AbouttheOIF:
TheOIFfacilitatesthedevelopmentanddeploymentofinteroperablenetworkingsolutions
andservices.MemberscollaboratetodriveImplementationAgreements(IAs)and
interoperabilitydemonstrationstoaccelerateandmaximizemarketadoptionofadvanced
internetworkingtechnologies.OIFworkappliestoopticalandelectricalinterconnects,
opticalcomponentandnetworkprocessingtechnologies,andtonetworkcontroland
operationsincludingsoftwaredefinednetworksandnetworkfunctionvirtualization.The
OIFactivelysupportsandextendstheworkofnationalandinternationalstandardsbodies.
Launchedin1998,theOIFistheonlyindustrygroupunitingrepresentativesfromacross
thespectrumofnetworking,includingmanyoftheworld’sleadingserviceproviders,
systemvendors,componentmanufacturers,softwareandtestingvendors.Informationon
theOIFcanbefoundathttp://www.oiforum.com.
Foradditionalinformationcontact:
TheOpticalInternetworkingForum,48377FremontBlvd,Suite117,
Fremont,CA94538USA
[email protected]
OFC2016
OpticalInternetworkingForum
1
TableofContents
Abstract:.......................................................................................................................................................................1
AbouttheOIF:.......................................................................................................................................................1
Glossary†.....................................................................................................................................................................4
TheOIFCEILegacy..................................................................................................................................................7
Introduction................................................................................................................................................................8
MotivationforCEI-56G..........................................................................................................................................9
CEI-56GChallenges...............................................................................................................................................12
DieArea.................................................................................................................................................................12
ChipPowerDissipation..................................................................................................................................12
PackagePowerDissipation...........................................................................................................................12
ChipPowerWiring............................................................................................................................................12
SystemPowerDissipation–Mid-planes.................................................................................................12
SystemPowerDissipation–Mid-boardoptics.....................................................................................13
I/ODensitiesonChipsandConnectors...................................................................................................13
ChannelMaterialsandCharacteristics....................................................................................................13
ChannelReach....................................................................................................................................................14
Cables.....................................................................................................................................................................14
Latency...................................................................................................................................................................14
Summaryofchallenges...................................................................................................................................14
ModulationTechniques.......................................................................................................................................16
OFC2016
OpticalInternetworkingForum
2
NRZ..........................................................................................................................................................................16
ENRZ.......................................................................................................................................................................16
PAM-4.....................................................................................................................................................................17
InterconnectReachesandApplicationSpaces..........................................................................................19
USR-DietoDieInterconnectWithinAPackage.................................................................................20
USR-DietoOpticalEnginewithinaPackage.......................................................................................21
XSR-ChiptoNearbyOpticalEngine.........................................................................................................21
XSR-CPUtoCPU...............................................................................................................................................22
XSR–DSPArrays...............................................................................................................................................22
XSR-CPUtoMemoryStack..........................................................................................................................22
VSR-ChiptoModule.......................................................................................................................................22
MR-ChiptoChipwithinaPCBA................................................................................................................23
LR–ChiptoChipacrossaBackplane/Midplane.................................................................................23
LR–ChiptoChipacrossaCable.................................................................................................................24
InteroperabilityPoints........................................................................................................................................24
DieBump...............................................................................................................................................................24
PackageBall.........................................................................................................................................................24
ModuleInterconnect........................................................................................................................................24
Summary....................................................................................................................................................................25
TableofFigures
Figure1InterconnectApplicationSpaces....................................................................................................8
Figure2ScalingofGates,Bumps,PinsandI/O(Ref.Xilinx,I/OLineadded)............................10
Figure3InterconnectChallenges..................................................................................................................11
OFC2016
OpticalInternetworkingForum
3
Figure4TypicalNRZEyeDiagramandStatisticalEyePlot...............................................................16
Figure5ENRZTypicalStatisticalEyes........................................................................................................17
Figure6TypicalPAM-4Eyes...........................................................................................................................18
Figure7CEI-56GReaches.................................................................................................................................19
Figure8UseofCEI-56GReaches...................................................................................................................20
Figure9USR-DietoDie....................................................................................................................................20
Figure10USR-DietoOpticalEngine..........................................................................................................21
Figure11XSRChiptoOpticalEngine..........................................................................................................21
Figure13MR-ChiptoChip..............................................................................................................................23
Figure14LRChiptoChipacrossabackplane..........................................................................................23
Figure15PackageBumpInteroperabilityPoints...................................................................................24
Figure16ChiptoModuleInteroperabilityPoint....................................................................................25
Glossary†
2.5D:Referstoatypeofdie-to-dieintegrationviaasiliconinterposerhavingthroughsiliconvias(TSVs)connectingitstopandbottommetallayers
3D:Referstoathree-dimensional(3D)integrateddeviceinwhichtwoormorelayersof
activeelectroniccomponents(e.g.,integratedcircuitdies)areintegratedverticallyintoa
singlecircuitwherethrough-siliconvias(TSVs)arecommonlyusedfordie-to-die
connection.
ApplicationSpaces:Portionsofequipmentornetworkarchitecturethatcouldbenefit
fromhavingadefinedsetofinterconnectionparameters.
ASIC:Anapplication-specificintegratedcircuitisanintegratedcircuit(IC)customizedfora
particularuse,ratherthanintendedforgeneral-purposeuse.
BCH:Bose,Ray-Chaudhuri,Hocquenghemforwarderrorcorrection(FEC)codesareaclass
ofadvancedcyclicerror-correctingcodesthatareconstructedusingfinitefields.
OFC2016
OpticalInternetworkingForum
4
BER:BitErrorRatioisthenumberofbiterrorsdividedbythetotalnumberoftransferred
bitsduringastudiedtimeinterval.
BGA:BallGridArray,apackagetype
CDR:Clockanddatarecovery,ablockinareceiverthatrecreatesaclockorclocksina
receiverbylookingatthestatisticaledgeinformationinthereceiveddataandthenuses
thatclockorclockstosamplethereceiveddata.
CEI:CommonElectricalInterface,anOIFImplementationAgreementcontainingclauses
definingelectricalinterfacespecifications.
ClockForwarding:Aclockcanbeforwardedinparallelwiththedatatoallowa
transceivertoavoidusingCDRinordertosavepower.ThisistypicallydoneinUSR
applications.
EMB:Effectivemodalbandwidth,seeTIA-492AAAD.
ENRZ:EnsembleNonReturntoZero,amulti-wirecodeinwhich3bitsaremodulatedwith
theHadamardTransformontofourwires.
FEC:Forwarderrorcorrectiongivesareceivertheabilitytocorrecterrorswithoutneeding
areversechanneltorequestretransmissionofdata.
FPGA:FieldProgrammableGateArray–areprogrammablelogicdevicethatoftenincludes
SerDes.
FR4:Agradedesignationassignedtoglass-reinforcedepoxyprintedcircuitboards(PCB).
Gb/s:Gigabitspersecond.Thestatedthroughputordatarateofaportorpieceof
equipment.Gb/sis1x109bitspersecond.
GBd:Thebaudrateistheactualnumberofelectricaltransitionspersecond,alsocalled
symbolrate.OneGigaBaudis1x109symbolspersecond.
IA:ImplementationAgreements,whattheOIFnamestheirdefinedinterfacespecifications.
IC:IntegratedCircuit
I/O:InputOutput,acommonnamefordescribingaportorportsonequipment
ISI:Inter-SymbolInterference,theeyeclosurecausedbytheenergyremainingonthe
channelfrompreviousunitintervals,whichtypicallyimpactsthehorizontaleyeopening.
ISI-Ratio:Theratioofthelargestcodeeyetothesmallestcodeeye,whichisasimple
metrictoevaluatetheimpactofISIonhorizontaleyeclosureforagivencode.
OFC2016
OpticalInternetworkingForum
5
LR:LongReach–longenoughtoreachachiplocatedacrossabackplane
MCM:Multichipmodule,aspecializedelectronicpackagewheremultipleintegrated
circuits(ICs),semiconductordiesorotherdiscretecomponentsarepackagedontoa
unifyingsubstrate,facilitatingtheiruseasasinglecomponent(asthoughalargerIC).
Mid-boardoptics:anopticaltransceiverthatismountedonaPCBAawayfromthePCBA
edge,closetoaswitchASICtoreducetheamountofPCBAtracelossbetweenanASICand
theopticaltransceiver.Thisisincontrasttothecommonpracticetodayoflocatingoptical
transceiversatthePCBAedge.
MUX/DEMUX:Multiplex/demultiplex,amultiplexer(ormux)isadevicethatselectsone
ofseveralanalogordigitalinputsignalsandforwardstheselectedinputintoasingleline,
Conversely,ademultiplexer(ordemux)isadevicetakingasingleinputsignalandselecting
oneofmanydata-output-lines,whichisconnectedtothesingleinput.
MR:MediumReach–longenoughtoreachachiplocatedacrossaPCBAincludingona
daughter-card
NRZ:NonReturntoZero,abinarycodeinwhich1sarerepresentedbyonesignificant
condition(usuallyapositivevoltage)and0sarerepresentedbysomeothersignificant
condition(usuallyanegativevoltage),withnootherneutralorrestcondition.
PAM:Pulseamplitudemodulation,aformofsignalmodulationwherethemessage
informationisencodedintheamplitudeofaseriesofsignalpulses.
PAM-4:Pulseamplitudemodulation-4isatwo-bitmodulationthatwilltaketwobitsata
timeandwillmapthesignalamplitudetooneoffourpossiblelevels.
PCBA:Printedcircuitboard(PCB)assembly,anassemblyofelectricalcomponentsbuilton
arigidglass-reinforcedepoxybasedboard.
RS:ReedSolomonFECcoding,thisisatypeofblockcode.Blockcodesworkonfixed-size
blocks(packets)ofbitsorsymbolsofpredeterminedsize.Itcandetectandcorrectmultiple
randomandbursterrors.
SerDes:Serializer/Deserializer
Tb/s:Terabitspersecond.Thestatedthroughputordatarateofaportorpieceof
equipment.Tb/sis1x1012bitspersecond
USR:Ultra-ShortReach–justlongenoughtoreachanotherdiewithinthesamepackage
VSR:Very-ShortReach–longenoughtoreachatransceiverinafront-panelopticalmodule
OFC2016
OpticalInternetworkingForum
6
XSR:eXtra-ShortReach–longenoughtoreachanearbydeviceormid-boardoptical
module
†Somedefinitionsincludecontentfromwww.wikipedia.com
TheOIFCEILegacy
TheOIFhasalonglegacyofdevelopingIAsforhighspeedelectricalinterfaces.TheCEI
familyhasbeentheprincipalindustrydefinitionforSerDesforthe6Gb/s,11Gb/sand28
Gb/sgenerations.TheOIFSxIinterfacewasCEI’simmediatepredecessorandservedthe
3Gb/sgeneration.TheOIFSPI/SFI3and4electricalinterfacesservedthe800Mb/sand
1.6Gb/sgenerations.Thislonghistoryof7generationsandsixdoublingsgivestheOIFa
uniqueviewpointontheneedsoftheindustryfortodayandtomorrow.
InterfacesthatsupporttheCEIfamilyofspecificationshavebeenbuiltinmostofthemajor
ASICflowsandFPGAdevices.CEIhasbeenadopted,adapted,orinfluencedthe
developmentofmultipleinterfacesincludingIEEE802.3,InfiniBandandFibreChannel,
SATA,SAS,RapidIOandHyperTransportinterfacesaswellasnumerousproprietary
protocols.
Name
Rateperpair
Year
Adopted,AdaptedorInfluenced
CEI-56G
56Gb/s
2016
Thefutureisbright
CEI-28G
28Gb/s
2011
InfiniBandEDR,32GFC,SATA3.2,
100GBASE-KR4and100GBASECR4,CAUI4,SAS-4
CEI-11G
11
2008
InfiniBandQDR,10GBASE-KR,
10GFC,16GFC,SAS-3,RapidIOv3
CEI-6G
6
2004
4GFC,8GFC,InfiniBandDDR,SATA
3.0,SAS-2,RapidIOv2,
HyperTransport3.1
OFC2016
OpticalInternetworkingForum
7
SxI5
3.125
1.6
2SPI4,SFI4
Introduction
SPI3,SFI3
0.800
2.1 Purpose
2002-3
Interlaken,FC2G,InfiniBandSDR,
XAUI,10GBASE-KX4,10GBASECX4,SATA2.0,SAS-1,RapidIOv1
2001-2
SPI-4.2,HyperTransport1.03
2000
(fromPL3)
The OIF Next Generation Interconnect Framework identifies application spaces for next
generation systems and identifies areas for future work by the OIF and other standards
bodies. The scope of this document explores Next Generation (NG) Interconnects that are
limited to data center or intra-office applications which are generally less than 2km from
both an electrical and optical perspective. Virtual Platforms in the cloud is an example of
just one of the applications that will take advantage of NG Interconnect technology to
achieve higher data bandwidths in a smaller footprint with better energy efficiency.
Introduction
Figure 1 Interconnect Application Spaces
Figure1InterconnectApplicationSpaces
As shown in Figure 1, interconnection interfaces in a typical system are needed for chip-tochip within a module, chip to chip within a PCBA (printed circuit board assembly), between
two PCBAs over a backplane/midplane, or between two chassis’. These interfaces may be
unidirectional or bi-directional,
optical or electrical, and may support a range of data rates.
OFC2016
OpticalInternetworkingForum
For each application space, the IAs that follow from this framework should identify
requirements to support interoperability across the various application spaces for optical
and electrical links. They may include, but not be limited to:
8
AsshowninFigure1,interconnectioninterfacesinatypicalsystemareneededforchip-tochipwithinamodule,chiptochipwithinaPCBA(printedcircuitboardassembly),between
twoPCBAsoverabackplane/midplane,orbetweentwochassis’.Theseinterfacesmaybe
unidirectionalorbi-directional,opticalorelectrical,andmaysupportarangeofdatarates.
Considerationsfortheselinksmayinclude:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Cost
LinkPerformance
PowerConsumption
Channellossbudgets
ChoiceofPCBmaterial
Modulationtechniqueandsignallevels
Numberoflanes,channelconfigurationandgeneralcharacteristics
Referenceclockjitter
ForwardedClockorCDR
Latency
Connectorperformance
Reliability
Size
OperatingTemperature
MotivationforCEI-56G
Nextgenerationsystemsarebeingdrivenbytheneedtohandletheincreasingvolumeof
datatraffic.Atthesametime,thenextgenerationsystemsareconstrainedbylimitson
powerconsumption,bylimitsonthesizeofasystem,andbytheneedtoprovideacost
effectivesolution.
Theseneedsdrivenextgenerationsystemstoeverincreasingcommunicationport
densities.Theincreaseddensityleadstosmallersurfaceareasavailabletodissipatethe
heatgeneratedandthereforerequiresdecreasedpowerconsumptionforaport.
Theindustrycurrentlyhaselectricalinterfacesfor10Gb/s(OIF’sCEI-11G),25Gb/s&
28Gb/s(OIF’sCEI-25/28G)inproduction,andworkiscomingalongon56Gb/s(OIF’sCEI56G).However,channelsproducedwithcopperPCBtracesareseverelybandwidthlimited,
andbecauseofthisitisincreasinglydifficulttoachievethesamelinkdistancesusing
highersignalingrates.
ImprovementsinICintegration,whichhasbeenrelentlesslydrivenbyMoore’sLawover
pastdecades,haveenabledhigherdensitiesoflogicgatesatescalatinghigherclockratesto
OFC2016
OpticalInternetworkingForum
9
beusedinICdesigns.Thesetrendshaveallowedtheindustrytodeployincreasinglymore
complexcommunicationsystemsateachgenerationtomeettheinfrastructureneeds.
However,asonedigsdeeper,thefutureappearstobechallenging.Manydifferent
technologiesneedtoconvergetoimprovethethroughput.ThesecomplexICshave
increasinggatecounts,butthepowerdissipationpergateandI/Ospeedhavenolonger
scaledatthesamerateasthegatecount.Inaddition,thenumbersofelectricalconnections
(bumpsandpackagepins)arealsonotscalingatthesamerate-leadingtoapower,
capacityandportcountgapforthenextgenerationinterconnectinterfaces.
Figure2ScalingofGates,Bumps,PinsandI/O(Ref.Xilinx,I/OLineadded)
Thepredominantinterconnectchallengestoovercomearepresentedbelowinasolution
spacediagram,andarediscussedingreaterdetailinsubsequentsub-sections.
OFC2016
OpticalInternetworkingForum
10
Figure3InterconnectChallenges
OFC2016
OpticalInternetworkingForum
11
CEI-56GChallenges
DieArea
SerDesimplementingCEI-56GareexpectedtobelargerthanSerDessupportingCEI-28G
interfacesbecauseoftheneedformoreadvancedequalization.Additionallythesizeofthe
analogportionoftheinterfacesdonotscaleaswellasthedigitalportions.
Thisareasituationhasrenderedsomepreviouschiparchitecturesunuseable.Thishasled
toatrendofSerDes“out-boarding”whereanUltra-ShortReach(USR)oreXtraShortReach
(XSR)interfaceisputonthemainASICandthelongerreachSerDesareputinthesame
packageornearby.
ChipPowerDissipation
SerDesimplementingCEI-56GareexpectedtobehigherpowerthanSerDessupporting
CEI-28Ginterfacesbecauseoftheneedformoreadvancedequalization.Additionallythe
powerdissipationoftheanalogportionoftheinterfacesdonotscaleaswellasthedigital
portions.
ThishasfurtheredthetrendmentionedaboveofSerDes“out-boarding”whereanUltraShortReach(USR)interfaceisputonthemainASICandthelongerreachSerDesareputin
thesamepackage.
PackagePowerDissipation
Whenpackagepoweristhekeyconstraint,theuseofanXSRinterfacecanhelpmovesome
ofthepowerdissipationoutsideofthemainASICandontootherchipsnearbyonthePCBA.
ChipPowerWiring
Lowervoltagepowersourcesareusedwithsmallertechnologynodes.Theneteffectof
higherintegrationoflowvoltagesemiconductorsisasignificantincreaseintotaldevice
current,whichrequireshighcurrentsourcepowersupplieswhichmustbecontrolled
withinseveralmVtolerances,whichinturnrequiresadditionalpowerpinsperdeviceto
accommodatethehighelectricalcurrents.
SystemPowerDissipation–Mid-planes
Becausesystemcoolingcanescalatebeyondphysicallimits,multiplepowersaving
strategiesmaybeneeded.
OFC2016
OpticalInternetworkingForum
12
SincetheI/Opowerisrelatedtothedistancethattheelectricalsignalstravelforagiven
channel’sproperties,reducingthedistancethattheelectricalsignalmustbedrivencan
reducepowerdissipation.Onetechniquethathasbeenusedistousemid-planes.
Withthisphysicalarchitecture,aswitchboardismountedhorizontallybehindthe
verticallymountedfrontboards.Withthisarchitecture,theelectricallinksareshorterand
onlyhavetotraverseoneconnector.
SystemPowerDissipation–Mid-boardoptics
Insomecases,itmaymakesensetointegratetheopticsclosetotheASICpackage,thereby
minimizingtheneedtodriveelectricalsignalsfar.Thesearesometimescalledoptical
enginesandthetechniqueisreferredtoasusingmid-boardoptics.
I/ODensitiesonChipsandConnectors
ThemaximumnumberofusefulI/Osforhighspeedseriallinksperdeviceisnotonly
limitedbytheavailablepackagetechnologyitself,butalsobytheabilitytoroutethedevice
onthePCBA.
Inordertomaintainsignalintegrityforahighspeedseriallinkdesign,itisrequiredtobe
abletorouteadifferentialpairbetweentwopackageballswhenescapingfromtheinner
ballrowsofaballgridarray(BGA),anditthereforemaybecomemorecostlytouse
packageswithaballpitchbelow1.0mm.
Inaddition,foreveryballrowfromtheedgeofthepackageonwhichdifferentialpairshave
beenplaced,aseparatecircuitpackagelayerhastobeused.Differentialpairsintheouter4
rowsoftheBGArequires4signallayersonthePCBA,whilethe6outerrowswouldrequire
6layers,andthusthePCBlayerstackgrowswitheveryinnerBGArowtoberouted.
ChannelMaterialsandCharacteristics
Linkapplicationsarecharacterizedbythesupportedlossbudgetandsignalimpairments.
Lossisdeterminedbylinklength,boardmaterials,back-drilling,viatypeusage,the
numberofconnectors,connectortypes,andpackagematerials.Increasesinsignalingrate
causemorefrequency-dependentlossandthusalowerSignaltoNoiseRatio(SNR).
Thedeviationofthelossfromasmoothcurveisalsoimportantastheripplesinaloss
curvearenoteasilyequalizable.ThisiscalledInsertionLossDeviation(ILD).
Ingreenfieldapplications,advancedmaterialsforthechannel(PCBandconnector)can
resultinandecreasedlossandcanthussupporteitheranincreasedelectricaldatarateora
decreasedamountofrequiredequalization.
OFC2016
OpticalInternetworkingForum
13
ChannelReach
Incommunicationsystemsusuallythefrontplateareaisoccupiedbypluggablemodules
forinter-systemcommunications,whiletheintra-systemtrafficbetweenthePCBAsis
connectedoverthebackplane/midplane.Tosupportreasonablesystemdimensions,
systembackplane/midplaneconnectionstypicallyneedtobridgedistancesofupto70–100
cm.Insomemorerecentapplications,alternativechannelarchitecturesreplacethe
backplanewithdirectplugorthogonal(DPO)structureswhereI/Olinecardsonthefront
sideoftheequipmentdirectlyplugintoswitchfabriclinecardsontherearsideofthe
equipment,thusdirectlyconnectingthelinecardswithoutthelossofthebackplane
betweenthem
Theintroductionofrepeaterdevicesintothedatapathcanincreasetheeffectivereachat
theexpenseofpowerandboardarea.
Cables
Twinax,micro-coaxand/orflexcircuitsarealsoanoptiontoreducelossandextendreach
forthehighspeedlinks,particularlyinsystemswithfewerhighspeedlinks.Insome
circumstances,thiscanallowthemainPCBAtobebuiltfromlowcostmaterial.Inanother
casecableassembliesarenowbeingintegratedontobackplaneconnectorandusedinlieu
ofPCBbackplanesduetotheirabilitytoreducelossandextendreach.Theuseofcables
bringstheirownshareofchallenges.
Latency
CPU-to-CPUandCPU-to-MemoryapplicationsofCEIareoftenintolerantoflatency.This
oftenprecludestheuseofheavyweightFECintheseapplications.Thecharacteristic
methodthatisintolerantoflatencyistheuseofcredit-basedflowcontrolwhereareceiver
grantsthetransmittertherighttosendacertainamountofdata.Onlyamoderateamount
ofbufferingistypicallyprovidedinthesesystems,mostofteninmorecostlySRAM.Mostof
themajorCPUtoCPUprotocolsusecredit-basedflowcontrol.
Summaryofchallenges
Astimeproceeds,ICswillbecomefasteranddenser.Tocopewiththeissueofinterconnect
capacityanddensityoffuturesystems,photonicinterconnectswillbecomeanevenmore
importantconnectiontechnology.
Theimplementationof56Gb/sInterconnectstechnologyposesseveralchallenges
especiallyinrelationto:diearea,chippowerdissipation,packagepowerdissipation,
systempowerdissipation,limitedI/Odensity,channelloss,channelreach,andlatency.
Highlightedweretheside-effectsofsomesolutions,whicharearesultofthecomplexinterOFC2016
OpticalInternetworkingForum
14
dependenciesof:higherintegration,complexmodulationschemes,chipbreak-outand
routing,signalconditioning,thermal&powerissues,andpackagefootprint.
OFC2016
OpticalInternetworkingForum
15
ModulationTechniques
AdvancedmodulationtechniquescanbeusedtolowertheNyquistfrequencyrequirementfora
linkusingabandwidthlimitedelectricalchannel.ThesetechniquesrequireahigherSNRfor
equivalentBER.Theprovisionofchannelswithmorecorrelationstomeasureagainstcanalsohelp.
Forwarderrorcorrection(FEC)canincreasetheeffectiveSNRandthereforethesupportedloss
budgetattheexpenseofhigherpowerdissipation,complexity,andlatency.Bandwidthlimitations
andincreasesinsignalingrateresultinimpairmentsthatmaybecompensatedbyusing
equalizationtechniques,whichthemselvesalsoimpactpowerconsumptionandcomplexity.
NRZ
NRZ(NonReturntoZero)isthemodulationtechniqueusedbymostcurrentspeedlinks.It
requiresachannelwithtwocorrelatedconductors.Itusestwosignallevelsandconveysonebit
perbaud.Tosupport56Gb/sperpair,NRZrunsatasymbolrateof56GBaudandhasaNyquist
frequencyof28GHz.IthasanISI-Ratioofone,meaningtheratioofthelargesteyetothesmallest
eyeisalwaysthesame.
At56Gb/s,NRZisusefulinapplicationswherethechannel’sfrequencydependentlossisnot
excessive.Thatlosshasbeenfoundtobeexcessiveinmanylongreachapplications.
in a
there
ulation
aches.
hnical
Figure4TypicalNRZEyeDiagramandStatisticalEyePlot
The use of two level signaling (0 and 1) provides a 9dB
SNR advantage over PAM4 (4 levels). This provides a
ENRZ
general
rule of thumb that the insertion loss of a given
channel measured at the PAM4 nyquist frequency should
be EnsembleNRZ(ENRZ)requiresachannelwithfourcorrelatedconductors.Usefulchannelscanbe
9dB less than the insertion loss at the NRZ Nyquist
constructedwithtwocorrelatedlooselycoupledpairs.Itusesfoursignallevelsandconveysthree
frequency
(2x the PAM4 Nyquist frequency) before PAM4
bitsperbaudoverthefourconductorsbymodulatingthethreesub-channelscollectivelyusingthe
should
be considered. Using this rule of thumb on the 28G
Hadamardmatrix.ItTosupporta56Gb/sperpaireffectiverate,ENRZrunsatasymbolrateof
VSR
insertion loss curves extended to 56G (See Figure X)
shows NRZ to be the best choice.
OFC2016
Why
PAM4?
OpticalInternetworkingForum
As Common Electrical Interfaces have increased in
data rates system vendors have always requested support
to the same printed circuit board trace distances. This has
16
Comparison
Channel for
opening analysis
Eye
comparison
Diagrams
ach!
ning vs. modulation technique
37.3GBaudandhasaNyquistfrequencyof18.6GHz.IthasanISI-Ratioofone,meaningtheratioof
thelargesteyetothesmallesteyeisalwaysthesame.ENRZcanoftenbeusedwithoutForward
ErrorCorrection,evenonsomelongreachchannels.
Ata56Gb/seffectiverate,ENRZisusefulinlongerreachapplications,particularlywhentheuseof
FECisnotdesired.
wires for each
throughput
per-wire
st
similar,
ENRZ(purple)
@ 18.66 GHz
ENRZ
is Nyquist
al
ISI-ratio = 1!
Figure5ENRZTypicalStatisticalEyes
reasonable frequency!
PAM-4balanced eye shape
PAM-4requiresachannelwithtwocorrelatedconductors.Itusesfoursignallevelsandconveys
twobitsperbaud.Tosupport56Gb/sperpair,PAM-4runsatasymbolrateof28GBaudandhasa
Nyquistfrequencyof14GHz.IthasanIntersymbolInterferenceRatio(ISI-Ratio)ofthree,meaning
thelargestcodeeyeisthreetimesaslargeasthesmallestcodeeye.Ithasaverticalimpairmentof
Source: TE Connectivity
9dBascomparedtoNRZandhasahorizontalimpairmentduetoitshighISI-Ratio.PAM-4is
usuallypairedwithequalizationandForwardErrorCorrection.
25
At56Gb/sPAM-4isusefulinapplicationswherethechannel’sfrequencydependentlosswould
otherwisebeexcessive.
OFC2016
OpticalInternetworkingForum
17
Demo 1: CEI-56G VSR NRZ QSFP channel demo
This demonstration shows a working VSR channel
using Test equipment from Tektronix to generate a
50Gbps NRZ signal which is sent thru commercially
availableQSFP connectors on test boards from Multilane
and Yamaichi. (See Figure ?)The receiver is a
commercially available NRZ receiver from Credo. The
BER of the system is shown on a PC using the BER tester
in the Credo receiver.
Figure6TypicalPAM-4Eyes
PAM modulation has been used for many years as the
solution of higher data rates over Category 3 and 5 twisted
pair wiring
for Ethernet. Figure Y shows an example of a
28G VSR compliant channel that would benefit from the
use of PAM4
modulation to double the data rate with no
changes in PCB materials. Note that the loss difference at
the NRZ Nyquist frequency vs the PAM4 Nyquist
is well beyond 9dB.
frequency
Demo 2: CEI-56G-VSR PAM4 QSFP Compliance
board
The ability to transmit and receive PAM4 signals across
a commercially available connectors is shown in Figure 14
using test equipment from Tektronix, Multilane and
Anritsu. One demonstration uses a Tektronix PAM4 signal
generator to transmit PAM4 formatted data over a QSFP
connector using test boards from Molex and TE
Connectivity. The PAM4 receiver is provided by Multilane.
An additional demonstration uses an Anritsu PAM4 data
generator sending signals through a CFP connector using
test boards from Yamaichi and Multilane. The PAM4
receiver is provided by Tektronix.
The CEI 56G Technology Showcase
In an effort to highlight the progress of the OIF in the
CEI 56G projects a technology showcase was developed
with the intent of demonstrating technical feasibility of the
different reaches. There are 6 demonstrations from
various members of the OIF.
OFC2016
OpticalInternetworkingForum
18
InterconnectReachesandApplicationSpaces
ThefiveCEI-56Greachescanbesummarizedinthefollowingtable:
Figure7CEI-56GReaches
Theuseofthesereachesforinterconnectapplicationspacescanbebrokendownas
follows.
OFC2016
OpticalInternetworkingForum
19
Figure8UseofCEI-56GReaches
USR-DietoDieInterconnectWithinAPackage
Figure9USR-DietoDie
Itmaybenecessarytousemultipledieswithinamulti-chipmodule(MCM)toachievethe
powerobjectives.Theseco-packagedsolutionscancommunicatewithlesspowersincethe
substrateprovidesahighqualitycommunicationchannel.
Thecommunicationchannelislessthan10mm.Thisshortelectricallinkmayallowfora
muchsimplerinterfaceandrequirelesspowerthananexistingstandardelectrical
interface.Forexample,equalizationisunlikelytobeneededanditmaybepossibleto
assumesuchshortlinksaresynchronous(singlereferenceclockgoingtoallchips),
removingtheneedforafrequencytrackingCDR.
OFC2016
OpticalInternetworkingForum
20
Atypicaluseforthisistooff-boardtheSerDesfromaswitchASIC.
ClockforwardingcanbeusedintheplaceofCDRforUSRlinks.
USR-DietoOpticalEnginewithinaPackage
Figure10USR-DietoOpticalEngine
Itmaybenecessarytouseadieandanopticalenginewithinamulti-chipmodule(MCM)to
achievetheindustry’sobjectives.Theseco-packagedsolutionscancommunicatewithlow
powersincethesubstrateprovidesahighqualitycommunicationchannel.
Thecommunicationchannelwouldtypicallybelessthan10mm.Thisshortelectricallink
mayallowforamuchsimplerinterfaceandrequirelesspowerthananexistingstandard
electricalinterface.
XSR-ChiptoNearbyOpticalEngine
Figure11XSRChiptoOpticalEngine
Itmaybeusefultoplaceanopticalinterfaceveryclosetothehostchip.Someoptical
devicescannotsitwithinahostMCMduetoheatrestrictionsoftheopticalcomponents.In
thiscase,ashortelectricallinkoflessthan50mmisanticipated.Theshortreachofthis
channelallowspowertobesaved.Theseopticalmodulesareoftencalledmid-board
optics.
OFC2016
OpticalInternetworkingForum
21
XSR-CPUtoCPU
CPUscanbeconnectedviaashortconnectionoverhighrateSerDes.SomeCPUsusea
coherencyprotocoltomakethedifferentprocessorsappeartobeinthesamecoherency
domain.
Itisgenerallynotpossibletouseheavy-weightForwardErrorCorrectioninCPU
applicationsduetothelatencyrequirementsoftheseinterfaces,whichtypicallyusecreditbasedflowcontrol.
XSR–DSPArrays
DSPsaresometimesconnectedinarraystoprocesshighrateinformationsuchasfroma
RADARoraLIDAR.Anautonomousvehicleisanexampleofsuchanapplication.
Sometimestheinterfacesareunidirectionalandothertimes,bidirectional.Bothare
typicallylatencysensitivesimilartoCPUsaseventheunidirectionalapplicationsoften
havecomplex,timesensitivecontrolflows.
XSR-CPUtoMemoryStack
CPUscanbeconnectedviaashortconnectiontoamemorystack.Aclassicwayto
accomplishthisistohaveadevicemadewithalogicprocessthatformsthebaselayer
devicefortheactuallymemorydies.
Itisnotpossibletouseheavy-weightForwardErrorCorrectioninthisapplicationdueto
thelatencyrequirementsofmemoryinterfaces.CPUthreadsorstatemachinesareoften
waitingfortheresultsofthememoryaccess.Memorycacheshelp,butdonoteliminatethe
significantperformancehitsthatCPUsseewhenlatencyisadded.
VSR-ChiptoModule
Itiscommoninmoderncommunicationsystemstosupportpluggablemodulesatthefront
faceplateoftheequipment.Thisfacilitateslowcostinitialdeploymentoftheequipmentif
someportsareleftunpopulated.Apayasyougopolicyisthenuseduntiltheentire
faceplateispopulatedwithpluggablemodules.Theelectricallinkusedtoconnectthese
pluggablemodulescanextendtobeyond30cm.Athigherdataratesthischallengesthe
abilityofthehostchiptodrivetheselongtracelengthswithinthepowerconstraintsof
largeswitchchips.Placingretimingdevicesinsidethepluggablemoduleprovidessupport
forlongerhosttracesbuttheinclusionofcomplexequalizationfeaturescanoverburden
thelimitedpowerbudgetsofthepluggablemodule.Advancedmodulationformats(suchas
PAMorDMTschemes),ForwardErrorCorrection(FEC)andequalizationfeaturesareall
possiblesolutionsforthechiptomoduleinterconnect.
OFC2016
OpticalInternetworkingForum
22
Onetechniquethathasbeenusedistoengineertheelectricallinkstohavealowernative
BERthanthatoftheopticallink.Inthisway,asingleFECcanbeusedtoprotectaCEI-56G
linkateachendandanopticallinkinthemiddle.
MR-ChiptoChipwithinaPCBA
Figure12MR-ChiptoChip
AninterconnectioninterfacemaybeneededbetweentwochipsonthesamePCBAorona
daughtercardorshortermid-plane.Bydefinition,thisinterfaceisrelativelyshortranging
upto50cm.Thisinterfacecouldincludeasingleconnector.
LR–ChiptoChipacrossaBackplane/Midplane
Figure13LRChiptoChipacrossabackplane
Thisinterfacecommunicatesbetweentwocardsacrossabackplane/midplanewithina
chassisandislessthan1mwithupto2connectors.
FECmaybearequirementtomeettheBER–howeverthechoiceoftheFECmustbe
consideredcarefullytoaddressbothlatencyandpowerconcerns.PossibleFEC
implementationsincludeRSorBCH.
OFC2016
OpticalInternetworkingForum
23
LR–ChiptoChipacrossaCable
WhilenotanexplicitgoaloftheCEI-56Gproject,cabledversionsofCEI-28Ghavebeen
usedtosupportbothbackplaneandchassistochassisinterconnects.
InteroperabilityPoints
CEI-56Gdefinesthreedifferentinteroperabilitypointsforthevariousreaches.Theseare
thepointsatwhichcompliancethetheOIFIAscanbedeterminedbymeasuringthe
propertiesofanimplementationandcomparingthoseresultstotheparametersintheIA.
DieBump
CEI-56G-USRdefinesinteroperabilityatthediebumpofaflip-chipsemiconductordevice
PackageBall
CEI-56G-XSR,CEI-56G-MRandCEI-56G-LRalldefinesinteroperabilityatthepackageball.
Figure14PackageBumpInteroperabilityPoints
OIFchosepackageballsastheinteroperabilitypointsforthesethreespecificationsbecause
theentirechannelistypicallytheresponsibilityofthesamecompany.XSRdoesnothavea
connectorandthereforenootherinteroperabilitypointisavailable.ForMRandLR,which
dohaveconnectors,thesystemdesignerstillownstheentirechannelandthereforeitdoes
notmakesensetoconstrainhowthesystemdesignerallocatesthelinkbudget.
ModuleInterconnect
CEI-56G-VSRdefinesinteroperabilityatthemoduleinterconnect.
OFC2016
OpticalInternetworkingForum
24
Figure15ChiptoModuleInteroperabilityPoint
Hostcomplianceboards(HCB)andmodulecomplianceboards(MCB)aredefinedto
measurecompliance.
Summary
TheCEI-56Gfamilyofclausesdefinesasetofinterfacesthatcanbeusedtosolvemosthigh
speedinterconnectneeds.Fiveseparatereachesandthreeseparatemodulation
techniquescansupporttheneedsofmostapplications.
OFC2016
OpticalInternetworkingForum
25