* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Cross-Domain and Cross-Layer Coarse Grained Quality of
Survey
Document related concepts
Piggybacking (Internet access) wikipedia , lookup
Point-to-Point Protocol over Ethernet wikipedia , lookup
Internet protocol suite wikipedia , lookup
Computer network wikipedia , lookup
Network tap wikipedia , lookup
Airborne Networking wikipedia , lookup
Zero-configuration networking wikipedia , lookup
Asynchronous Transfer Mode wikipedia , lookup
Wake-on-LAN wikipedia , lookup
List of wireless community networks by region wikipedia , lookup
Cracking of wireless networks wikipedia , lookup
IEEE 802.1aq wikipedia , lookup
Recursive InterNetwork Architecture (RINA) wikipedia , lookup
Multiprotocol Label Switching wikipedia , lookup
Deep packet inspection wikipedia , lookup
Transcript
Cross-Domain and Cross-Layer Coarse Grained Quality of Service Support in IP-based Networks von der Fakultät für Elektrotechnik und Informationstechnik der Technischen Universität Chemnitz genehmigte Dissertation zur Erlangung des akademischen Grades Doktoringenieur (Dr.-Ing.) vorgelegt von Dipl.-Ing. Thomas Martin Knoll geboren am 10. Januar 1973 in Reichenbach eingereicht am 27.7.2009 Gutachter: Univ.-Prof. Dr.-Ing. Thomas Bauschert Univ.-Prof. Dr.-Ing. Jörg Eberspächer Univ.-Prof. Dr.-Ing. habil. Klaus Franke Tag der Verteidigung: 11.11.2009 Verfügbar im MONARCH der TU Chemnitz: http://archiv.tu-chemnitz.de/pub/2009/0165 17.11.2009 Bibliographische Beschreibung Thomas Martin Knoll Cross-Domain and Cross-Layer Coarse Grained Quality of Service Support in IP-based Networks Dissertation (in englischer Sprache) 166 Seiten, 155 Abbildungen, 21 Tabellen, 185 Literaturverweise Referat Mit der zunehmenden Popularität des Internets steigt die Anzahl der Nutzer und vor allem die Anzahl zeit- und verlustkritische Dienste – wie zum Beispiel „Voice over IP“, Videoübertragungen und netzbasierte Spiele. Das Internet ist dabei der Zusammenschluss von ca. 30.000 Betreibernetzen, die mit Hilfe des „Internet Protocol (IP)“ derzeit ohne jede Dienstgüteunterstützung den Datenverkehraustausch realisieren. Massive Überdimensionierung der Netzkapazitäten führen zu einer Netzauslastung von nur ca. 10% und entsprechend guter Übertragungsqualität. Mit steigendem Verkehrsaufkommen wird in dieser Dissertation erwartet, das die Netzbetreiber infolge des Kostendrucks nicht schritthaltend den überhöhten Netzausbau aufrechterhalten können und somit Qualitätseinbußen zu erwarten sind. Innerhalb der Betreiber wird bereits jetzt Verkehrstrennung betrieben, jedoch am Übergabepunkt verworfen und im besten Fall im Nachbarnetz durch aufwendige Analyse erneut vorgenommen. Im Rahmen dieser Arbeit wurde deshalb ein domänen- und schichtenübergreifendes Konzept zur Realisierung grob-granularer Dienstgüte in IP-Netzen entworfen, zur Standardisierung bei der „Internet Engineering Task Force (IETF)“ vorgeschlagen, implementiert und in Auszügen simuliert und getestet. Dabei werden die Verkehrsklasseninformationen mehrere Netzschichten in transitiven Nachrichtenelementen des „Border Gateway Protocol (BGP)“ signalisiert und schichtenübergreifend assoziiert. Die vorliegende Dissertation beinhaltet im wesentlichen drei Teile: 1. Eine umfassende Zusammenstellung von vorhandenen Dienstgütekonzepten einschließlich der bereits existierenden QoS-Funktionselemente in verfügbaren Netzelementen, 2. Die detaillierte Spezifikation des neuen Konzeptes und 3. den Ergebnissen der Simulations- und Implementierungsaktivitäten zum Nachweis der Funktion und Skalierbarkeit des Entwurfes. Zwei wesentliche Erkenntnisse und Forderungen sind durch die Bearbeitung des Themas erwachsen. Die Einfachheit der Konzeptstruktur und die Einfachheit der angestrebten Dienstgüteunterstützung. Die angestrebte Dienstgüte beschränkt sich deshalb auf die primitive Verkehrstrennung in mehrere Klassen, die in den Weiterleitungsknoten getrennt abgelegt und mit verschiedenem Vorrang behandelt werden. Schlagwörter Quality of Service (QoS), Class of Service (CoS), Cross-Domain, Cross-Layer, Inter-AS, Marking Signalling, Ingress limitation Signalling, BGP, Extended Community Attribute ii 17.11.2009 Abstract The increasingly popular Internet with a steadily growing user base, the resulting traffic load and its rising usage for time and loss critical services, such as voice over IP, video streaming and gaming, consists of about 30,000 interconnected service provider networks. Those interconnections are based on the Internet Protocol (IP) and do not distinguish the mixed traffic types within the transported traffic load. The currently observed and mostly sufficient service quality can only be achieved by network internal and inter-domain link capacity over-provisioning. Resource utilization of about 10% is commonly applied to achieve stable and un-congested network operation. However, service providers are increasingly deploying Quality of Service (QoS) support mechanisms within their network domain in order to provide traffic separation and differentiated forwarding. Not only IP QoS, but also underlying link layer QoS mechanisms are applied. Such QoS support is currently removed at the interconnection link and possibly reapplied in an independent and uncoordinated fashion in the neighbouring domain. A new cross-domain and cross-layer coarse grained Quality of Service support concept has therefore been drafted, which allows for the automated inter-domain class of service (CoS) support information exchange about the distinguished traffic classes at different networking layers. The concept is based on the standard inter-domain signalling protocol, the Border Gateway Protocol (BGP) version 4. Transitive BGPbased cross-domain signalling and cross-layer CoS mapping is a novel contribution. The cross-domain signalling of cross-layer mapped class set information has been submitted for standardization within the Internet Engineering Task Force (IETF). This includes a class overload prevention signalling by means of applied token bucket based ingress limitations. Global scale usage and omnipresent traffic class of service support is targeted with the proposed and implemented concept. It is likely, that service providers might be tempted to misuse offered service classes, hence the overload limitation. Three major contributions are documented within this thesis: 1. A comprehensive compilation of QoS support concepts with detailed network and node internal building block descriptions has been arranged, which proves the technical readiness of currently deployed devices for an inter-domain CoS based interconnection. 2. The drafted specification of the new inter-domain CoS concept including the CoS marking and class overload limitation signalling is detailed herein. 3. Simulations and implementations of vital building blocks of the concept have been made to underline its functionality and technical feasibility. Resource estimates and successful field trials provide evidence for its scalable and functioning design. The thesis’ work identified two fundamental design requirements for the concept. They are simplicity in design and QoS support. QoS in this approach therefore refers to primitive traffic separation into several classes, which will experience differently prioritized forwarding behaviour in relaying nodes. Enqueueing in separate queues is thereby aspired to. iii 17.11.2009 Contents 1 Introduction ____________________________________________________________ 3 2 Fundamentals of IP routing and forwarding __________________________________ 4 2.1 IP datagram structure and addressing___________________________________________4 2.2 Routing basics _______________________________________________________________7 2.2.1 2.2.2 2.3 Router architecture _________________________________________________________19 2.3.1 2.3.2 2.3.3 3 Routing protocols and hierarchy _____________________________________ 7 Inter-domain routing using BGP ____________________________________ 12 Router control plane structure ______________________________________ 19 Router internal interconnection structure _____________________________ 20 Router internal queuing structure ___________________________________ 21 Basic QoS aspects ______________________________________________________ 23 3.1 Overview __________________________________________________________________23 3.1.1 3.1.2 3.2 QoS treatment scope_________________________________________________________37 3.2.1 3.2.2 3.2.3 3.3 QoS-based forwarding ____________________________________________ 38 QoS-based routing________________________________________________ 39 QoS-based tunnelling _____________________________________________ 41 Architectural scope__________________________________________________________44 3.3.1 3.3.2 4 relative vs. absolute vs. coarse-grained QoS _________________________ 23 QoS building blocks_______________________________________________ 25 Cross-layer QoS__________________________________________________ 44 Cross-domain QoS _______________________________________________ 45 State of the art QoS Concepts _____________________________________________ 46 4.1 IP QoS ____________________________________________________________________46 4.1.1 4.1.2 4.1.3 4.1.4 DiffServ _________________________________________________________ 47 IntServ __________________________________________________________ 52 IntServ / DiffServ combination ______________________________________ 54 ITU-T IP QoS concept_____________________________________________ 55 4.2 Ethernet QoS_______________________________________________________________56 4.3 MPLS QoS_________________________________________________________________61 4.4 QoS in access networks ______________________________________________________65 4.5 Summary of expected Class of Service support ___________________________________69 5 State of the art AS interconnection _________________________________________ 71 5.1 IP transit __________________________________________________________________74 5.2 IP peering _________________________________________________________________75 5.3 Internet Routing Registry - IRR _______________________________________________77 6 Related work___________________________________________________________ 78 iv 17.11.2009 7 New (coarse grained) CoS concept _________________________________________ 86 7.1 Motivation and target________________________________________________________86 7.2 Usage of BGP for QoS signalling ______________________________________________88 7.3 Definitions and information processing _________________________________________89 7.3.1 7.3.2 8 BGP extended community attribute for CoS marking __________________ 89 BGP class of service interconnection ________________________________ 96 Mapping strategies_____________________________________________________ 101 8.1 Problem statement _________________________________________________________101 8.1.1 8.1.2 mapping between different class sets of the same layer_______________ 101 mapping between different class sets of different layers_______________ 103 8.2 Existing recommendations___________________________________________________104 8.3 Coarse grained CoS mapping recommendations ________________________________113 9 Simulation results _____________________________________________________ 115 9.1 Setup selection for QoS marking and forwarding ________________________________115 9.2 Simulation results for QoS marking and forwarding _____________________________117 9.2.1 9.2.2 9.2.3 9.2.4 9.2.5 9.2.6 9.2.7 Scenario 1: single node interconnection ____________________________ 117 Scenario 2: AS interconnection – Single AS _________________________ 120 Scenario 3: AS interconnection – Multi-AS __________________________ 121 Scenario 4: AS interconnection – Multi-AS 2 ________________________ 122 Scenario 5: AS interconnection – Multi-AS 3 ________________________ 123 Scenario 6: AS interconnection – Multi-AS 4 ________________________ 124 Scenario 7: AS interconnection – Cross-Layer _______________________ 126 9.3 Setup selection for token bucket ingress filtering ________________________________127 9.4 Simulation results for token bucket ingress filtering _____________________________128 9.5 Summary of simulation results _______________________________________________130 10 Concept implementation_______________________________________________ 132 10.1 Linux implementation ____________________________________________________132 10.2 Wireshark implementation ________________________________________________136 10.3 Online debug form _______________________________________________________137 11 Implementation test __________________________________________________ 138 11.1 Test setup _______________________________________________________________138 11.2 Test result and observations _______________________________________________139 11.3 Ethernet QoS support test at IXPs __________________________________________142 11.4 Resource usage estimates __________________________________________________143 11.4.1 11.4.2 12 Increase in routing update information size ________________________ 145 Increase in memory consumption with routers _____________________ 148 Summary and outlook ________________________________________________ 152 12.1 Contributions and results__________________________________________________152 12.2 Practical usage___________________________________________________________153 12.3 Outlook ________________________________________________________________153 v 17.11.2009 Titel Domänen- und schichtenübergreifendes Konzept zur Realisierung grob-granularer Dienstgüte in IP-Netzen Inhaltsverzeichnis 1 Einleitung______________________________________________________________ 3 2 Grundlagen des IP Routing und Forwarding _________________________________ 4 2.1 IP Datagramstruktur und Adressierung _________________________________________4 2.2 Grundlagen des Routings______________________________________________________7 2.2.1 2.2.2 2.3 Router-Architektur _________________________________________________________19 2.3.1 2.3.2 2.3.3 3 Routing-Protokolle und -hierarchien __________________________________ 7 Inter-Domän-Routing mittels BGP___________________________________ 12 Struktur der Router-Steuerungsschicht ______________________________ 19 Struktur Router-internen Verbindungen ______________________________ 20 Struktur der Router-internen Warteschlangen ________________________ 21 Grundlegende Aspekte der Dienstgüte ______________________________________ 23 3.1 Überblick __________________________________________________________________23 3.1.1 3.1.2 3.2 Ausdehnungsbereich von QoS-Mechanismen ____________________________________37 3.2.1 3.2.2 3.2.3 3.3 QoS-basiertes Weiterleiten ________________________________________ 38 QoS-basierte Wegewahl___________________________________________ 39 QoS-basiertes Tunneln ____________________________________________ 41 Einflußbereiche der Konzept-Architektur_______________________________________44 3.3.1 3.3.2 4 Relative vs. absolute vs. grob-granulare QoS_________________________ 23 QoS-Bausteine ___________________________________________________ 25 Schichtenübergreifende QoS_______________________________________ 44 Domänübergreifende QoS _________________________________________ 45 Aktuelle QoS-Konzepte __________________________________________________ 46 4.1 IP QoS ____________________________________________________________________46 4.1.1 4.1.2 4.1.3 4.1.4 DiffServ _________________________________________________________ 47 IntServ __________________________________________________________ 52 Kombination von IntServ und DiffServ _______________________________ 54 IP QoS Konzept der ITU-T _________________________________________ 55 4.2 Ethernet QoS_______________________________________________________________56 4.3 MPLS QoS_________________________________________________________________61 4.4 QoS in Zugangsnetzen _______________________________________________________65 4.5 Zusammenfassung der zu erwartenden Dienstklassenunterstützung _________________69 5 Derzeitige AS-Kopplung _________________________________________________ 71 5.1 IP Transit _________________________________________________________________74 5.2 IP Peering _________________________________________________________________75 5.3 Internet Routing Registratur - IRR ____________________________________________77 6 Bisherige Arbeiten auf dem Gebiet _________________________________________ 78 vi 17.11.2009 7 Das neue (grob-granulare) CoS-Konzept ____________________________________ 86 7.1 Motivation und Zielsetzung ___________________________________________________86 7.2 Nutzung von BGP zur QoS-Signalisierung ______________________________________88 7.3 Definitionen und Informationsverarbeitung _____________________________________89 7.3.1 7.3.2 8 BGP Extended Community Attribut zur CoS-Markierung _______________ 89 Dienstklassen-basierte Kopplung mittels BGP ________________________ 96 Zuordnungsstrategien __________________________________________________ 101 8.1 Problembeschreibung_______________________________________________________101 8.1.1 8.1.2 Dienstklassenabbildungen innerhalb einer Schicht ___________________ 101 Dienstklassenabbildungen zwischen verschiedenen Schichten ________ 103 8.2 Vorhandene Empfehlungen__________________________________________________104 8.3 Empfehlungen zu grob-granularen CoS-Abbildungen ____________________________113 9 Simulationsergebnisse __________________________________________________ 115 9.1 Simulationsplanung für QoS-Markierungen und QoS-Weiterleitung _______________115 9.2 Simulationsergebnisse für QoS-Markierungen und QoS-Weiterleitung______________117 9.2.1 9.2.2 9.2.3 9.2.4 9.2.5 9.2.6 9.2.7 Szenario 1: Einzelknotenkopplung _________________________________ 117 Szenario 2: AS-Kopplung – Einzel-AS ______________________________ 120 Szenario 3: AS-Kopplung – Multi-AS _______________________________ 121 Szenario 4: AS-Kopplung – 2 AS __________________________________ 122 Szenario 5: AS-Kopplung – 3 AS __________________________________ 123 Szenario 6: AS-Kopplung – 4 AS __________________________________ 124 Szenario 7: Schichtenübergreifende AS-Kopplung ___________________ 126 9.3 Simulationsplanung für Token Bucket-Filterung ________________________________127 9.4 Simulationsergebnisse für Token Bucket Filterung ______________________________128 9.5 Zusammenfassung der Simulationsergebnisse __________________________________130 10 Implementierung des Konzeptes ________________________________________ 132 10.1 Linux-Implementierung ___________________________________________________132 10.2 Wireshark-Implementierung _______________________________________________136 10.3 Online-Formular zur Dekodierung __________________________________________137 11 Implementierungstest _________________________________________________ 138 11.1 Testaufbau ______________________________________________________________138 11.2 Testergebnisse und Beobachtungen _________________________________________139 11.3 Tests zur Ethernet-QoS Unterstützung bei IXPs _______________________________142 11.4 Abschätzung des Resourcenverbrauchs ______________________________________143 11.4.1 11.4.2 12 Anstieg der UPDATE-Größe_____________________________________ 145 Anstieg des Speicherbedarfs ____________________________________ 148 Zusammenfassung und Ausblick________________________________________ 152 12.1 Beitrag und Ergebnisse ___________________________________________________152 12.3 Praxisanwendung ________________________________________________________153 12.3 Ausblick ________________________________________________________________153 vii 17.11.2009 Einleitung Die Vernetzung aktueller IP-basierter Datennetze bildet zwar eine moderne Kommunikationstechnologie, besitzt jedoch einige Unzulänglichkeiten in der Netzkopplung. Die nachfolgende geschichtliche Analogie zeigt genau diese Schwachstellen des Internets auf, welche zugleich in dieser Arbeit aufgegriffen und verbessert werden. Im 19. Jahrhundert wurde die Kommunikation zwischen den Kolonien Südaustralien und Westaustralien durch Dampfschiffe realisiert, was durchaus Wochen für den Transport dauern konnte. Damals entschied man, die Kommunikation auf Telegraphie umzustellen. 1874 begann man deshalb mit dem Bau der Telegraphenleitung. Südaustralien trieb die Leitung von Port Augusta westwärts bis zur Grenze und Westaustralien begann mit dem Bau in Albany in Richtung Osten. An der Telegraphenstation in der kleinen Grenzsiedlung Eucla ([145], [166]) wurde 1877 die Verbindung beider Leitungsabschnitte erreicht. Die Station wurde zu gleichen Teilen mit Mitarbeitern betrieben, die entlang eines langen Nord-Süd ausgerichteten Tisches sich gegenüber saßen. Die Grenze war dabei die Mitte des Hauses und die Mitte des Tisches. Nachrichten, die zwischen den Staaten ausgetauscht werden sollten wurden somit vom jeweiligen Personal empfangen, manuell zur anderen Seite des Tisches gereicht und dort erneut als Telegraphennachricht gesendet. Grund dafür waren verschiedene Zeichenkodierungen, die auf beiden Seiten verwendet wurden. Südaustralien verwendete den amerikanischen Morse-Code und Westaustralien den internationalen. Die Ähnlichkeit besteht darin, dass das heutige Internet aus etwa 30000 unabhängig voneinander betriebener IP-Netze, so genannter Autonomer Systeme (AS), besteht, die in unkoordinierter Weise Dienstgütekonzepte verfolgen und auf einfachstem Niveau privat oder öffentlich vernetzt sind. Trotz dessen, dass diese ASse oft intern frei gewählte Verkehrstrennung und –priorisierung anwenden, wird bei deren Zusammenschluss die Trennung entfernt und ohne Verkehrstrennung und vorrangige Behandlung die Verkehrsübergabe vorgenommen. Einige Eintrittsvermittlungen der ASse betreiben dann aufwendige Klassifizierung anhand der gekapselten Empfangsdaten, um eine möglichst gute Schätzung der empfangenen Verkehrsart zu treffen und erneut die passende interne Verkehrstrennung und –priorisierung anzuwenden. Deshalb wurde in dieser Arbeit die Signalisierung und direkte Verkehrsklassen-basierte Kopplung Autonomer Systeme untersucht, dokumentiert und implementiert. viii 17.11.2009 Zusammenfassung und Ausblick Diese Dissertation betrachtet den Zusammenschluss von so genannten „Autonomen Systemen“, die derzeit keinerlei Dienstgüteunterstützung bieten. Die erbrachten Beiträge dieser Arbeit sind in wesentlichen in drei Teile gegliedert. Den ersten Teil bildet eine umfassende Zusammenstellung von vorhandenen Dienstgütekonzepten einschließlich der bereits existierenden QoS-Funktionselemente in verfügbaren Netzen und Geräten zur Netzkopplung. Diese Geräte sind nachweislich für die Unterstützung von domänenübergreifender, klassenbasierter Dienstgüte geeignet. Aus diesen Erkenntnissen und zusammen mit den mündlichen Aussagen führender Europäischer und Amerikanischer Netzbetreiber und Betreibern aus dem Nahen Osten über die akzeptable Komplexität solcher Dienstgütevorhaben entstand die vordringliche Forderung nach einem einfachen, leicht fassbaren und handhabbaren Dienstgütekonzept. In einem zweiten Teil wurde das angestrebte domänenübergreifende Dienstgütekonzept spezifiziert und zur Standardisierung bei der IETF eingereicht. Im dritten Teil wird durch Simulation und Implementierung wesentlicher Konzeptbestandteile deren Funktion und technische Machbarkeit dargelegt. Die Skalierbarkeit und Funktionalität des Konzeptes wurde durch Feldtests und durch Abschätzungen des Ressourcenverbrauchs nachgewiesen. Beitrag und Ergebnisse Folgende Erkenntnisse und Beiträge wurde in der Arbeit erbracht: • Der Zusammenschuss von autonomen Systemen zum globalen Internet stellt aus technischer und ökonomischer Sicht eine neuralgische Schnittstelle zwischen Netzbetreibern dar. Derzeitige Zusammenschlüsse basieren ausschließlich auf dem Austausch von IP-Nachrichten ohne Dienstgüteunterstützung. Überdimensionierung und netzinterne Dienstgüteunterstützung werden derzeit vorgenommen. Durch das anhaltende Wachstum des Internetverkehrs wird in der Dissertation ein Anstieg an Netzausbaukosten und zunehmender Verkehrsstau auf den Kopplungsleitungen erwartet. Eine neues klassenbasiertes Kopplungskonzept wurde deshalb entwickelt, das für globale Anwendung geeignet ist. • Die Einfachheit eines Entwurfes wurde als entscheidendes Entwurfskriterium für die Akzeptanz des Konzeptes in der Internet-Gemeinde erkannt. Es erstreckt sich dabei sowohl auf die Signalisierungsstrukturen als auch das tatsächliche Ausmaß der Klassenunterstützung. • Die Wichtigkeit der Unterstützung von mindestens zwei oder besser 4 Dienstklassen wurde mit Hilfe von Simulationen untermauert. • Im Gegensatz zu existierenden komplexen Dienstgütekonzepten, die Garantien zu Verzögerungen, Verzögerungsschwankungen und Verlustraten anstreben, wird aus Kosten- und Akzeptanzgründen im vorliegenden Konzept nur einfache Verkehrstrennung gefordert. • Der erreichte Grad an Einfachheit durch Wegfall von Dienstgütegarantien ist eine zentrale Voraussetzung für die globale Anwendbarkeit. • Die Entscheidung zur Verwendung von BGP für die Signalisierung wurde auf Basis der Betrachtungen zu bereits existierenden und emporkommenden Signalisierungsprotokolle getroffen. • Im BGP wurden neue so genannte “Extended Communities” und ein neues Pfadattribut definiert, die zur Signalisierung der erforderlichen domänen- und schichtenübergreifenden Klasseninformation verwendet werden. ix 17.11.2009 • • • • • • Das neuartige Prinzip der transitiven Weiterleitung von Dienstklasseninformationen mittels der “Extended Communities” und der vom Betreiber festlegbaren Zuordnung der Dienstgüteeinstellungen verschiedener Netzschichten innerhalb der Signalisierung stellt eine grundlegende Errungenschaft dar. Die Ergebnisse aufwendiger Einzelknoten-Simulationen und Simulationen auf AS-Niveau wurden auszugsweise in dieser Dissertation dokumentiert und sind auf Anfrage vollständig verfügbar. Der Nachweis der Anwendbarkeit des Konzeptes und der Interoperabilität mit vorhandenen Netzelementen wurde durch Tests mit der LinuxImplementierung erbracht. Abschätzungen zum Ressourcenverbrauch wurden vorgenommen, die einen vernachlässigbar kleinen Einfluss des zusätzlichen Signalisierens von Dienstklasseninformationen auf die Größe der BGP-UPDATE-Nachrichten aufzeigten. Ein maßvoller Verbrauch an Speicherressourcen wurde ebenfalls ermittelt. Dabei wurde unter der Annahme von realistischen Szenarien die Anwendbarkeit der Konzepten auch für große Netzausmaße nachgewiesen. Die Gestaltung des Konzeptes behindert nicht den zusätzlichen gezielten Einsatz komplexer Dienstgütemechanismen mit garantierter Dienstgüte. In der Tat wird der universelle Einsatz des hiesigen Konzeptes und der selektive Einsatz höherwertiger Konzepte an ausgewählten Kopplungen oder TransitPfaden unterstützt. Auf der Basis des Konzeptes wird die Umwandlung des heutigen Internets hin zu einem 2- oder besser 4-Klassen unterstützenden Internet. Praxisanwendung Besonderes Augenmerk wurde auf die praktische Nutzung des Konzeptes gelegt. Die folgenden Punkte listen wichtige Meilensteine für die Anwendbarkeit. • • • • Mit der Übertragung der Konzeptspezifikation an die IETF Standardisierung wurde praktisch eine lizenzfreie Nachnutzung ohne patentrechtliche Einschränkungen ermöglicht. Die globale Anwendung des Konzeptes ist angestrebt und mögliche Kosteneinsparungen auf Betreiberseite tragen zum durch das Konzept erreichbaren Gewinn bei. Die Implementierungen in der Linux Routing-Software, Quagga, und dem Netzanalysewerkzeug, Wireshark, sind frei verfügbar. Die WiresharkErgänzung ist dabei bereits von den Entwicklern akzeptiert und in die aktuelle Softwareversion integriert worden. Gleiches ist für die Quagga-Erweiterung geplant. Ein Online-Dienst wurde eingerichtet der die Dekodierung von signalisierten Klasseninformationen im Rohdatenformat akzeptiert. Er ist unter folgender Adresse zu finden: http://www.bgp-qos.org/draft-knoll/decode_attributes.php . Die Nummernvergabestelle, IANA, hat bereits Typnummern für die “QoS Marking” und “CoS Capabilities” Elemente zugeteilt, so dass diese offiziell in den Produktionsnetzen der Betreiber verwendet werden können. Damit hat das Konzept bereits die Schwelle vom Laboraufbau hin zum öffentlichen Einsatz überschritten. x 17.11.2009 Ausblick Derzeit ist die Anwendung des neuen domänen- und schichten-übergreifenden Konzept zur Realisierung grob-granularer Dienstgüte auf Linux-basierte Netzelemente beschränkt. Laufende Gespräche mit Netzbetreibern und Router-Herstellern zielen jedoch auf die generelle Unterstützung des Konzeptes in kommerziellen Routern ab. Die technische Machbarkeit wurde dabei bestätigt und Interesse daran wurde von Europäischen Betreibern bekundet. Zukünftige Praxiserfahrungen und Änderungswünschen werden dabei zur Verfeinerung des Konzeptes führen. Um die Anwendung des Konzeptes zu fördern, wird derzeit an der Ergänzung der herkömmlichen kommerziellen Router um eine interaktive Linux-basierte Fernsteuerung gearbeitet. Fig. 155 zeigt dabei den verdeckten Steuermechanismus des kommerziellen Routers durch einen internen Linux-PC. Dadurch, dass die Signalisierungselemente transitiv definiert wurden, kann der Router mit passivem bidirektionalem Durchleiten die Verarbeitung und Generierung von Dienstklasseninformationen an das Linux-System deligieren. Mit Hilfe einer zweiten Verbindung kann nun der Linux-PC die Steuerschnittstelle des Routers erreichen und die notwenigen Kommandos zur Konfiguration und Aktivierung der vorhandenen Router internen QoS Funktionen absetzen. Dies Übergangslösung erlaubt den Netzbetreibern ohne kostspielige Software- oder Hardwareaktualisierungen eine klassenbasierte Netzkopplung anzubieten. Fig. 153 Steuerung eines kommerziellen Routers durch einen Linux-PC Eine derzeitige Diskussion über „Netzneutralität“ beeinflusst die Bereitschaft von Netzbetreibern und Herstellern, domänenübergreifende Dienstgütemechanismen zu unterstützen. Dabei steht der neutrale Netzbetrieb ohne Dienstlimitierungen, Inhaltsfilter, und ohne jegliche Bevorzugung einzelner Nutzer im Vordergrund. Entsprechende Gespräche mit Netzbetreibern und verschiedener staatlicher Netzagenturen haben ergeben, dass das vorgeschlagene Dienstgütekonzept mit seiner einfachen und allgemein anwendbaren Struktur womöglich als nicht diskriminierende und flächendeckend einsetzbare Verbesserung des Internets angesehen würde. Zusätzliche techno-ökonomische Studien zu erreichbaren Kosteneinsparungen werden von Nöten sein, um die Entscheidungsprozesse der Betreiber hinsichtlich Geräteaktualisierungen und der Einführung von klassenbasierter Dienstgüte zu unterstützen. xi 17.11.2009 In Kapitel 5.2 wurde bereits kurz ein von der Firma Google vorgeschlagener Unterschriftsprozess beschrieben, der mit Hilfe von so genannten BGP „Communities“ die Teilnahme an neuen Diensten und Konzepten besiegelt. Je nach Erfolg dieses Vorhabens kann es dazu führen, dass das vorgeschlagene Dienstgütekonzept als Vertragsbasis für die Vereinbarung von klassenbasierter Dienstgüte zwischen Betreibern genutzt wird. xii 17.11.2009 Acronyms ABR ABR AD ADSL AFI ARP ASBR ASN ATM B-ISDN BA BGP BGRP BRAS CAC CAPEX CBR CBWFQ CIDR CIR CLI CLP COPS CR-LDP CS DE DFZ DiffServ DMA DNS DRR DS DSCP DSL DV E-LSP eBGP ECN EF ECN EGP EIGRP FCFS FIB Area Border Router Available Bit Rate Administrative Distance Asymmetric DSL Address Family Identifier Address Resolution Protocol Autonomous System Border Router Autonomous System Number Asynchronous Transfer Mode Broadband ISDN Behaviour Aggregate Border Gateway Protocol Border Gateway Reservation Protocol Broadband Remote Access Server Call Admission Control Capital Expenditure Constant Bit Rate Class-Based Weighted Fair Queueing Classless Inter-Domain Routing Committed Information Rate Command Line Interface Cell Loss Priority (CLP) bit Common Open Policy Service Constraint-based Routed LDP Class Selector Discard Eligibility bit in frame relay Default Free Zone Differentiated Services Direct Memory Access Domain Name System Deficit Round-Robin Differentiated Services DiffServ Code Point Digital Subscriber Line Distance Vector EXP-Inferred-PSC LSP / now: Explicitly TC-encoded-PSC LSP external Border Gateway Protocol Explicit Congestion Notification Expedited Forwarding Explicit Congestion Notification Exterior Gateway Protocol Enhanced Interior Gateway Routing Protocol First Come First Served Forwarding Information Base xiii 17.11.2009 FIFO First In First Out FR Frame Relay FSM Finite State Machine FTP File Transfer Protocol GbE Gigabit Ethernet GBR Guaranteed Bit Rate GCRA Generic Cell Rate Algorithm GIST General Internet Signalling Transport GMPLS Generalized MPLS GPS Generalized Processor Sharing GRE Generic Routing Encapsulation HDLC High Level Data Link Control HOLB Head of Line Blocking IANA Internet Assigned Numbers Authority iBGP internal Border Gateway Protocol ICMP Internet Control Message Protocol IETF Internet Engineering Task Force IESG Internet Engineering Steering Group IGP Interior Gateway Protocol IGRP Interior Gateway Routing Protocol IntServ Integrated Services IP Internet Protocol IPv4 Internet Protocol version 4 IPv6 Internet Protocol version 6 IRR Internet Routing Registry IS-IS Intermediate System to Intermediate System ISDN Integrated Services Digital Network ISO International Organization for Standardization ISP Internet Service Provider IXP Internet Exchange Point L-LSP Label-only-Inferred-PSC LSP LAN Local Area Network LDP Label Distribution Protocol LIB Label Information Base LIFO Last In First Out Loc-RIB Local RIB LQD Longest Queue Drop LS Link State LSDB Link State Database LSP Label Switched Path MAC Media Access Control MAC-in-MACEncapsulation of Ethernet frames in Ethernet frames MED Multiple Exit Discriminator MESCAL Management of End-to-end Quality of Service Across the Internet at Large MPLS Multi Protocol Label Switching MSS Maximum Segment Size MTU Maximum Transmission Unit NGN Next Generation Network NLRI Network Layer Reachability Information NSIS Next Steps In Signalling NSLP NSIS Signalling Layer Protocol NTLP NSIS Transport Layer Protocol xiv 17.11.2009 OPEX OS OSI OSPF PBB PBT PC PCN PCP PDB PDP PDU PFC PGPS PHB POTS PS PSTN PT q-BGP Q-in-Q QoS QoE RAM ReaSE RED RFD RIB RIP RPSL RPSLng RR RR RS RSVP RSVP-TE SAFI SDH SDU SLA SONET SP SPF SPI TC TCA TCP TOS TTL UBR UDP UMTS Operational Expenditure Operating System Open Systems Interconnection Open Shortest Path First Provider Backbone Bridges Provider Backbone Transport Personal Computer Pre-Congestion Notification Priority Code Point Per Domain Behaviour Policy Decision Point Protocol Data Unit Priority-based Flow Control Packet-by-packet Generalized Processor Sharing Per Hop Behaviour Plain Old Telephone Service Processor Sharing Public Switched Telephone Network Packet Type QoS enhanced BGP 802.1q in 802.1q encapsulation Quality of Service Quality of Experience Random Access Memory Realistic Simulation Environments for IP-based Networks Random Early Detection Route Flap Damping Routing Information Base Routing Information Protocol Routing Policy Specification Language Routing Policy Specification Language next generation Round Robin Route Reflector Router Server Resource Reservation Protocol RSVP-Traffic Engineering Subsequent Address Family Identifier Synchronous Digital Hierarchy Service Data Unit Service Level Agreement Synchronous Optical NETwork Strict Priority Shortest Path First System Packet Interface Traffic Class Traffic Conditioning Agreement Transmission Control Protocol Type of Service Time To Live Unspecified Bit Rate User Datagram Protocol Universal Mobile Telecommunications System xv 17.11.2009 URL VBR VC VLAN VLSM VoIP VOQ VTYSH WAN WDRR WiMAX WRED WRR WLAN WLL Uniform Resource Locator Variable Bit Rate Virtual channel Virtual LAN Variable Length Subnet Mask Voice over IP Virtual Output Queues Virtual TeletYpe shell Wide Area Network Weighted Deficit Round-Robin Worldwide Interoperability for Microwave Access Weighted Random Early Detection Weighted Round Robin Wireless LAN Wireless Local Loop xvi 17.11.2009 Acknowledgments The work presented in this thesis was done at Chemnitz University of Technology in Chemnitz, Germany. The interest for the topic and the idea for the proposed concept arose through the lecturing work at the Chair of Communication Networks. I would like to express my deep thanks to the current and the former head of chair, Prof. Thomas Bauschert and Prof. Klaus Franke, respectively, for their support during the last years and for invaluable discussions and comments on my work. I am very grateful to Prof. Jörg Eberspächer for his offer to act as a co-examiner of my thesis and for the chance to present this work at his institute. A special thanks goes to David Ward, Dr. Yakov Rekhter, Robert Raszuk and Jie Dong for their support with IANA’s number assignment, fruitful discussions and detailed feedback on the concept. I am very grateful to Arnold Nipper and Wolfgang Tremmel from DE-CIX as well as Jens Wengenmayr and Frank Benndorf from envia TEL GmbH for their technical feedback and support. Furthermore, I wish to thank Simon Ehnert for the programming support with the Quagga routing suite, my co-worker Daniel Manns for his support in the work with OMNET++ , Uwe Steglich for challenging hours with NS2 and the other co-workers and students at the chair of Communications Networks for their helpful comments and reflections. My thanks is due as well, to Brian Schaefer, who has helped me with correcting my writing. Finally, I would like to thank my family for their support, patience, and understanding during these challenging years. Thomas Martin Knoll Chemnitz, July 2009 2 17.11.2009 1 Introduction The internetworking of current IP-based data networks is a modern communication technology with some major interconnection drawbacks. The following historical allegory depicts the weak spot of the widely used Internet, that is addressed in this work. Back in the 19th century, the two colonies of South Australia and Western Australia decided to communicate between each other via telegraph, rather than steamship, which took weeks. In 1874 both colonies started to erect a new telegraph line to interconnect their independently operating telegraph systems. South Australia, started its line from Port Augusta towards the border in the west and Western Australia erected its line from Albany towards its eastern border. In 1877, the interconnection was established at the Eucla Telegraph Station ([158], [179]), a small settlement near the border between the colonies. The station was equally staffed and the telegraphists of both colonies sat along a north to south oriented table. In fact the technical border divided the building and the operators’ table in half. The West Australian operators received their inter-state messages at the western half of the table and pushed the message across it towards their respective South Australian colleague. From there, the message was again telegraphed into South Australia and vice versa. The reason for this manual repeater station was the different character encoding used on either side. South Australia used the American Morse code and Western Australia the International one. The similarity lies in the fact that the current Internet consists of about 30.000 independently operated IP networks, called Autonomous Systems (AS), which run uncoordinated quality of service concepts and are in a very basic manner privately or publicly interconnected. Despite the fact, that ASes often apply some sort of independently chosen traffic separation and prioritization within the respective network cloud, their interconnection removes all such separation and handles the exchange traditionally without any separation or prioritization. Some AS ingress routers in turn apply multi-layer ingress classification methods in order to make a good guess on what traffic enters the network and should be separated and or prioritized. The signalling and direct traffic class based interconnection of Autonomous Systems has therefore been investigated, documented and implemented. 3 17.11.2009 2 Fundamentals of IP routing and forwarding The robust and inexpensive exchange of information between end systems in global scale is the major achievement of the current Internet. Many networking technologies exist, which allow for the networking of electronic devices using different layer two technologies. However, such local area networks make use of several, independently chosen technologies, which require interworking functions for an internetworking between them. This barrier is removed with the introduction of the commonly used Internet Protocol (IP) as least common denominator regarding the very basic requirements for a primitive datagram based information exchange. The Internet is therefore a patchwork of many networking clouds, which all provide the means for an end-to-end IP-based datagram transmission service. 2.1 IP datagram structure and addressing In order to understand the capabilities of the globally available IP datagram service, it is best to review the protocol’s control information exchange, which is carried within the header structure of each single protocol data unit. Fig. 1 depicts the datagram structure of the currently predominantly used version four of the Internet protocol. Its original structure was defined in RFC791 [153]. Fig. 1 IP version 4 datagram structure The most important elements of the header are the destination and the source IP address, which are used for a hop-by-hop relay process towards the destination and for backward error reporting in case of delivery failures, respectively. IP addresses used to be grouped into address classes – A, B, C, D, E – following the structure given in Fig. 2. Each node belonging to a network cloud was assigned an IP address containing the same network part within the 32 bit number. A router would therefore decide by the destination address of the datagram as well as of the network 4 17.11.2009 number its receiving interface belongs to, whether the datagram is destined for the originating cloud or needs to be relayed towards a next hop router. Fig. 2 IPv4 address class system - [22] The stiff address class regime, as well as the huge and small network clouds for class A and C type networks, respectively, led to a revised scheme for network/host differentiation allowing any bit position within the 32 bit field as network address boundary. The scheme is called “Classless Inter-Domain Routing (CIDR)” [81], [82] and introduces a network mask field of 32 bit to support “variable length subnet masks (VLSM)”. Combined with the traditional address classes, it now allows the creation of subnets out of one larger network and supernets out of several consecutive smaller networks. Fig. 3 gives a subnetting example for the creation of 128 subnets out of one class B network. 10 Network Subnet Host . 11111111111111111111111 000000000 Fig. 3 CIDR example network mask Routers in CIDR networks now compare the network part of their interface address with the network part of the currently processed IP destination address using a simple AND operation with the network mask applied on both addresses. The major advantage of CIDR in global scale routing lies on the field of route aggregation. IP address ranges (so called prefix blocks) of Internet service providers or some large scale companies tend to have fine grained address allocations with network masks in their twenties. However, routers in the core regions of the Internet might see a number of consecutive address blocks in their routing tables, which all resolve towards the same next hop neighbour. Summarizing those table entries into just one bigger address block with a shorter network mask saves on table storage, table lookup delay and route advertisement messages. Such prefix aggregation by means of CIDR is therefore heavily used in today’s Internet routing. Further work on IP addressing was performed with the introduction of IP version 6 [63], [64], [5][1]. This new version extends the IP addresses to 128 bit fields and specifies a fixed size basic header structure of 40 bytes length. The new scheme of header extensions allows for a dynamic incorporation of additional header information. Fig. 4 depicts the version 6 datagram structure. 5 17.11.2009 Fig. 4 IP version 6 datagram structure The CIDR concept of address and netmask is continued, but the IP address classes vanished. IP version 6 introduces address types instead [92]. The following types have been defined: • Unicast Addresses, o Interface Identifiers, o The Unspecified Address, o The Loopback Address, o Global Unicast Addresses, o IPv6 Addresses with Embedded IPv4 Addresses, o Link-Local IPv6 Unicast Addresses, o Site-Local IPv6 Unicast Addresses (deprecated), • Anycast Addresses and • Multicast Addresses. Under the light of QoS support in IP-based networks, the differentiation of datagrams during the hop-by-hop relay process needs to be made. One approach could be to reserve address blocks for certain forwarding treatments and to enumerate those end devices, which have certain quality of service requirements, with such IP addresses. Relaying nodes could react to such special destination addresses within the datagram header and might even provide different routing decisions in their relay process. However, this puts an unnecessary burden on the globally arranged IP address planning and prevents end devices to support several services with possibly differing QoS requirements concurrently. The original IPv4 “Type of Service (ToS)” as well as the IPv6 “Traffic Class (TC)” header field both provide 8 bits for quality of service datagram marking information, but with different encodings. In the course of “Differentiated Services (DiffServ)” - a quality of 6 17.11.2009 service concept described in section 4.1.1 - the redefinition of both fields by RFC 2474 [142] into a so called “Differentiated Services field (DS field)” was the decisive step forward to achieve a common encoding scheme independent of IP protocol versions. Six bit “Differentiated Services Code Points (DSCP)” were specified for DiffServ purposes. Since DSCP occupies only 6 out of the redefined 8 bits of both fields, a second mechanism is incorporated in the remaining two bits. It is called “Explicit Congestion Notification (ECN)” and allows for forward congestion notification by relaying nodes – RFC 3168 [156]. The combination of both definitions and some clarification on the wording and meaning of the specifications is given in RFC 3260 [85]. Fig. 5 references the major redefinition RFCs as well as the four common differentiated services “Per Hop Behaviour (PHB)“ encodings. Fig. 5 Differentiated Services (DS) field in IPv4 and IPv6 datagram headers 2.2 Routing basics 2.2.1 Routing protocols and hierarchy The relay of IP datagrams is performed in a hop-by-hop manner, which transports the packetized information solely based on the destination address field contained in the IP packet’s header information. The interworking functions only relay the datagram out of the layer two networking cloud, if the IP destination address belongs to a different IP network than the one it originated from. The interfaces of such interworking devices are each members of the respective networking cloud and relay the datagram on behalf of the original source to the neighbouring relaying node, which they believe to be closer to the datagram’s destination. Those layer three interworking devices are called “routers” and can be equipped with different relaying capabilities depending on their positioning within the global hierarchically organized patchwork of networking clouds. The relay process of IP datagrams within a router consists of three major steps: 7 17.11.2009 1. IP lookup of the datagram’s IP destination address in a forwarding table to find the best matching relay path towards the correct next hop router together with the respective output interface connecting to that next hop neighbour, 2. IP header field processing to decrease the time-to-live value and to update the header check sum accordingly and 3. internal transfer of the processed datagram to the output interface for transmission towards the layer two address of the next hop router. The described action points relate to the lower half of the depicted functionality in Fig. 6 – the functions of the so called “forwarding plane”. The upper half controls the setup of the vital routing information within the mentioned forwarding table. The control plane functionality is based on dynamic reachability information exchanges using specialized routing protocols. Each routing protocol instance of a router performs some sort of neighbour discovery and advertises the known IP prefixes to them. Fig. 6 IP routing and forwarding functionality A router typically maintains two “routing tables” internally. The so called “Routing Information Base (RIB)” stores all valid routes the update process has learned locally or dynamically from other routers. In a second step the best routes to each advertised prefix are selected out of the RIB and installed into the so called “Forwarding Information Base (FIB)” used by the forwarding plane. If an IP prefix has been learned in several best path granularities (prefix lengths), then all of them are stored in the FIB. During IP lookup, a so called “longest prefix match” is performed in order to find the most specific best route towards the currently processed IP destination address. The dynamic routing process with the exchange of reachability information for IP prefixes is organized in a hierarchical fashion. In a flat routing structure, every newly established or on the contrary lost connectivity to an IP prefix needs to be communicated to all participating routers. This is not feasible in a global scale Internet, but is used in small portions of the internetworked clouds. 8 17.11.2009 The routing hierarchy is comprised of routing areas, routing domains and autonomous systems (AS) as shown in Fig. 7. Fig. 7 Internet routing hierarchy Each hierarchy level summarises its internal reachability changes and communicates those changes in summary routes to the next upper level. This way of operation reduces the size and frequency of routing updates between the routing hierarchy levels and uses CIDR with VLSM aggregation in large scale. The lowest level of the hierarchy is routing areas, which are solely used to encapsulate routing changes and errors into confined regions. This limits the reach of flooded information, reduces convergence time after route changes and covers (dampens) routing changes, which have only area local significance. Routing areas are run by the same authority and operate identical routing protocols and policies. Inter-area routing information exchange of summarized routes is performed by so called “area border routers (ABR)” or more general, “gateways”. The typical topology is a hub and spoke approach, which consists of one backbone area connecting all other areas within a routing domain. Routing domains are consistently operated by a single authority and normally rely on a single internal routing protocol with the same set of metrics and routing decision policies. They can also be regarded as single routing domains in the case of different internal routing protocols being used, as long as the domain provides a single and consistent routing behaviour to the outside network. An authority can operate one or more routing domains and request a so called “autonomous system number (ASN)” for registration in the global Internet. Each autonomous routing domain, which was assigned an ASN, turns into an “Autonomous System (AS)”. ASes are therefore characterized by a 16 bit (currently being transitioned to 32 bit [176]) AS number and a unified administrative routing policy internally. RFC 1930 [86] gives guidelines for creation, selection, and registration of an AS. AS-internal routing protocols are generally referred to as “Interior Gateway Protocols (IGP)” and the ones interconnecting ASes are called “External Gateway Protocols (EGP)”, respectively. Edge routers at AS interconnection points are referred to as “Autonomous System Border Routers (ASBR)”. Fig. 8 gives an example of a typical Internet routing architecture. 9 17.11.2009 Fig. 8 Internet routing architecture Commonly found routing protocols in today’s networks are RIP (Routing Information Protocol – version 1 [87] and 2 [130]), OSPF (Open Shortest Path First [140]), IS-IS (Intermediate System to Intermediate System [106], [42]) and BGP (Border Gateway Protocol version 4 [157]). Two proprietary protocols, IGRP (Interior Gateway Routing Protocol [48] - obsolete) and EIGRP (Enhanced Interior Gateway Routing Protocol [50]) are also in use, but are limited to networks, which solely deploy Cisco routers. The first inter-domain routing protocol, EGP (Exterior Gateway Protocol [138]), was limited to a tree topology and is no longer in use. IP routing protocols – applicability Intra-domain routing Inter-domain routing Interior Gateway Protocols (IGP) Exterior Gateway Protocols (EGP) - RIPv1 (obsolete) RIPv2 IGRP (obsolete) EIGRP OSPF IS-IS iBGP (version 4) - EGPv3 (obsolete) - eBGP (version 4) Fig. 9 IP routing protocols – classified by applicability 10 17.11.2009 The exchange of reachability information together with path characteristics can be classified in two major working principles: “distance vector routing (DV)” and “link state routing (LS)”. Smaller networks with less stringent convergence requirements and with low processing power / low energy consuming routers will opt for DV protocols. Otherwise, link state protocols are required. A third principle, “path vector routing”, is a modified distance vector principle, which is currently only used with the border gateway protocol. Fig. 10 gives an overview about the principle classification and typical examples. Distance vector routing The advertisement of all known routes and their associated cost (hop count) is periodically sent out to all neighbours within a broadcast/multicast domain. Each router will in turn incorporate the reachable prefixes and adopted costs into its routing table and send this new table out to its topological neighbours. Update processing makes use of the Bellman-Ford algorithm ([22], [78]) , which minimizes the hop count within the route selection. The DV principle incurs a low processing load, considerable network traffic and the evolving update dissemination is vulnerable to routing loops long convergence times. Since routers along the advertisement track can not work out the original source of the information, this principle is also referred to as “routing by rumour”. Path vector routing The major characteristic of this working principle is the recording of the advertisement relay trail in the exchange prefix reachability update information. The trail record in the inter-domain case is the path of AS numbers, the advertisement passed through. Path vector routing can be combined with distance vector or other path selection mechanisms. The prominent example for path vector routing is the border gateway protocol. The advertisement and router selection process is governed by policy (filter) rules and does cover numerous criteria. Reachable IP networks are announced to carefully selected neighbours and might be selectively and neighbouring based filtered out by the above mentioned policy rules. The knowledge about the advertisement trail mitigates routing loops and enables route selection beyond the hop-by-hop scope. However, it still does not disclose precise network topology information. The processing load depends on the filter complexity. Since path vector routing is mainly used in inter-domain setups, routing stability is preferred before fast restoration times in case of resource failures. Hence, a slow convergence time is regarded as less critical. Link state routing Link state routers inform all such other routers in the respective routing areas about their status knowledge of connected links as well as the flooded link state information of the other neighbouring routers. LS routers maintain neighbouring sessions with each other and after an initial phase, where all link states are exchanged, only changes are flooded within the area. This way, each router incrementally receives a complete status of the network’s topology and can work out the shortest path routing table from its point of network view. This increased computational effort uses the “Shortest Path First (SPF)” algorithm of Dijkstra [67]. The higher processing load saves on network traffic and leads to faster convergence times. The flooded link state updates contain the information about the information originator. This routing principle is therefore referred to as “routing by propaganda.” 11 17.11.2009 IP routing protocols – working principle Distance Vector - RIPv1, RIPv2 - IGRP - EIGRP (hybrid) Path Vector - eBGP / iBGP “routing by rumour” Fig. 10 Link State - OSPF - IS-IS - EIGRP (hybrid) “routing by propaganda” IP routing protocols – classified by working principle The routing in the global Internet relies on the meshed interconnection of autonomous systems. So far, point-to-point connections are obvious solutions for the interconnection of ASBRs. However, the vast majority of public interconnections are accomplished by means of Ethernet based “Internet Exchange Points (IXP)”. As Fig. 11 indicates, hierarchical, redundant and mostly distributed switching clusters are common realizations for those neuralgic peering hubs with several hundred interconnected ASes. Fig. 11 Internet Exchange Point - IXP 2.2.2 Inter-domain routing using BGP The Border Gateway Protocol is explained in more detail due to its exclusive usage for inter-domain routing as well as to the importance for the proposed Inter-AS BGP-based signalling concept of this thesis. 12 17.11.2009 BGP is a so called Path-Vector protocol and distributes the reachability information of network prefixes together with associated attributes. An outstanding characteristic is the AS_PATH attribute, which records a list of relaying ASes for the respective reachability information. This way, not only the neighbouring AS for a specific prefix is known to the recipient, but the whole AS path that needed to be traversed in order to reach the announced network(s). The AS path is therefore used as an important metric for path selection (optimized for minimal path length) and loop detection. Only advertised network prefixes, which do not include their own AS number in the path list are accepted as valid route updates. BGP relies on TCP for reliable message exchange and sets up so called “BGP sessions” between interconnected AS border routers. Each end point of the session is called BGP peer and the BGP neighbour session establishment will only be successful, if the parties configure the IP address and AS number of the respective peer in their router internal BGP process. The border gateway protocol distinguishes two horizons, the internal (iBGP) – BGP peerings between edge routers within an AS and the external (eBGP) – BGP peerings between edge routers of adjacent ASes. Since one AS needs to have a consistent knowledge of reachable prefixes at its edges, internal peers need to establish and maintain a full mesh of peering sessions. AS confederations and the concept n ⋅ (n − 1) scalability problem of the full mesh in large of route reflectors to circumvent the 2 ASes are explained below. The exchanged reachability information is flooded across all BGP sessions, however they are filtered (in each ingress and egress) to apply strategic routing policy decisions. The border gateway protocol is therefore a global interconnection protocol with routing policy enforcements. Each AS decides, which of their own prefixes are advertised to which peering partner, which prefixes are accepted from external peers (ingress filter) and which selected best paths are advertised to which external peer (egress filter). This best path selection procedure is vital to understand BGP’s selection decisions of active paths out of the received available paths. The algorithm is applied for the case, when the same prefix is received several times. The decision points are processed in the given order and the first differing criterion will yield the decision. One optional route processing extension is BGP Multipath, which allows it to commit several paths to a prefix in the local forwarding table. The best path will still be worked out and announced to BGP peers, but multiple active paths will be installed and used for nodelocal packet forwarding. 13 17.11.2009 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. Fig. 12 Prefer the path with the highest WEIGHT. (Cisco proprietary) Prefer the path with the highest LOCAL_PREF. Prefer the path that was locally originated via a network or aggregate BGP subcommand or through redistribution from an IGP. Prefer the path with the shortest AS_PATH. Prefer the path with the lowest origin type. Prefer the path with the lowest multi-exit discriminator (MED). Prefer eBGP over iBGP paths. If bestpath is selected, go to Step 9 (multipath). Prefer the path with the lowest IGP metric to the BGP next hop. Continue, even if bestpath is already selected. Determine if multiple paths require installation in the routing table for BGP Multipath. Continue, if bestpath is not yet selected. When both paths are external, prefer the path that was received first (the oldest one). Prefer the route that comes from the BGP router with the lowest router ID. If the originator or router ID is the same for multiple paths, prefer the path with the minimum cluster list length. (for Route Reflector environment) Prefer the path that comes from the lowest neighbour session IP address. BGP Best Path Selection Algorithm - [49] Five types of BGP messages are exchanged between peers, which are: • OPEN, (Initial setup of the peering session and exchange of protocol timer settings) • UPDATE, (Reachability and withdraw advertisement of network prefixes combined with path attributes / complete update during initialization and triggered updates later on) • NOTIFICATION, (Closing message of the BGP session) • KEEPALIVE and (periodic handshake message, if no UPDATE is sent) • ROUTE-REFRESH [47]. (Request message for a complete reachability information exchange (refresh) e.g. for non-disruptive policy change enforcements ) All messages share the same structure, which is depicted in Fig. 13. 14 17.11.2009 Fig. 13 BGP message structure The UPDATE message structure (see Fig. 15) in particular consists of a fixed length message header, the variable length withdrawn route section, a variable length path attribute section, and the variable length “network layer reachability information (NLRI)” section. All advertised prefixes within the NLRI are “labelled” with the signalled attributes. Prefixes, which require different attribute association, need to be sent in a separate UPDATE message. New UPDATE messages for the same previously advertised prefixes will override the stored NLRI and attribute information at the receiving end. All attributes are classified by type numbers, the same type must not be included several times in a message and the type number is used as ordering criteria. There are different types of path attributes defined as shown in Fig. 14. Path Attributes optional wellknown mandatory discretionary - Origin - AS-Path - Next Hop - Local Fig. 14 transitive - Aggregator - Community - Extended Preference - Atomic Aggregate non-transitive - Multi-Exit- Discriminator (MED) Community BGP path attribute classification [46], [161] It is important to note, that “transitive” and “non-transitive” in the attributes’ context relates to the attribute signalling. The optional attribute is either relayed across the AS towards the next neighbour or it is terminated within the peering AS. A non-transitive attribute might be sent out to a different AS, but will terminate there. 15 17.11.2009 Fig. 15 BGP UPDATE message structure – after [157] The concept of optional transitive community [46] and extended community [161] attributes has been added to BGP for the purpose of rather free-style signalling of e.g. policy rule triggers or other mutually agreed on activities. Fixed size community and extended community structures have been defined, which are carried within the respective attribute type. Several communities are therefore consecutively embedded within just a single (extended) community attribute. Of particular interest for this thesis are extended communities, which are further detailed as follows. Extended communities are of fixed 8 byte size and consist of a type field followed by the remaining community bytes. Extended community types are divided into “regular types” (1 byte type field) and “extended types” (2 byte type field). The remaining community bytes are therefore either 7 or 6 byte. A type number registry for extended community types is administered by the Internet Assigned Numbers Authority (IANA) [95], which ensures, that no assigned regular type number is the high byte part of an assigned extended type number. Extended communities are contained in the transitive optional extended community path attribute. However, the communities themselves are classified as “transitive” and “nontransitive”. It is important to note, that “transitive” and “non-transitive” in the communities’ context relates to the AS border crossing nature of the community. That is, transitive extended communities are relayed internally and externally. Non-transitive extended 16 17.11.2009 communities, however, must not cross any AS border. Despite the fact, that they are embedded in a transitive attribute, they are by definition confined within the iBGP of a single AS. Fig. 16 depicts the UPDATE message structure with optional transitive extended community path attribute. Fig. 16 BGP UPDATE message structure with Extended Community attribute The above mentioned scalability problem for fully meshed internal BGP sessions between border routers is addressed in two ways: “Router Reflector” and “Confederations”. BGP Route Reflector BGP route reflector has been defined in RFC 4456 [20]. If fully meshed BGP speakers are grouped into clusters, a hierarchy of route reflectors and route clients can be setup. That is, only route reflectors still need to be fully meshed and they serve their clients with incoming UPDATE as a relay node and speak on behalf of the clients for UPDATEs raised within the cluster. 17 17.11.2009 Fig. 17 BGP Route Reflector topology BGP Confederation Autonomous System Confederations for BGP have been defined in RFC 5065 [172]. The main idea of the concept is to divide an AS internally into several confederation ASes. Such internal ASes are not seen externally, so that the overall behaviour of the original AS will not change. Internally, routing convergence and scalability is greatly enhanced due to the route confinement within such artificial (private) ASes. Some special rules on advertisement and attribute handling were to be specified in order to establish the right procedures for confederation internal eBGP and confederation external AS representation. One example would be the AS path handling, which will record private AS numbers during the confederation internal signalling. Such private AS numbers, however shall not cross the original AS border and need to be stripped off outside the confederation. Fig. 18 depicts the resulting AS internal topology of e.g. AS 4321. Fig. 18 Autonomous System Confederations for BGP 18 17.11.2009 2.3 Router architecture A router provides interconnection and relaying functionality between several inputs and typically the same number of outputs. Major characteristics are the number of supported ports and the achievable throughput. Fig. 19 depicts the general block diagram structure of a router. Fig. 19 IP router block diagram 2.3.1 Router control plane structure The control plane of an IP router is equipped with one or more IP routing protocol instances and provides the routing update generation and processing functionality accordingly. Command line and web based interfaces allow the direct access to the control instance for configuration and monitoring. Each enabled routing protocol allocates storage and processing power resources according to its working principle. Fig. 20 depicts the typical block structure for the control plane part. Each protocol maintains its specific update information storage (e.g. link state database (LSDB)) and performs protocol local route selection algorithms. The resulting routing table information is stored in protocol local routing information bases (e.g. display with “command line interface (CLI)” command: “show ospf route”). The route redistribution manager is a central building block, which controls the route selection for the node local routing table as well as the mutual routing information exchange between routing protocol instances. For instance, external BGP learned routes can be redistributed into OSPF to announce global connectivity and vice versa for prefixes originating inside that routing domain. The filtered out routes are stored in the node’s local IP routing information base (display with CLI: “show ip route”). A protocol precedence, called “administrative distance (AD)”, has been defined, which decides on the precedence order of the protocol RIBs. The lower the AD value, the more important the information source is. In a last step, the condensed forwarding relevant information is installed in the forwarding information base (display with CLI: “show ip cef”). This FIB is often replicated in the input port units for fast interface local lookups. 19 17.11.2009 Fig. 20 IP router internal structure -> route processing 2.3.2 Router internal interconnection structure This section looks more closely into the internal structure of the generalized block diagram as of Fig. 19. The internal relaying of IP packets between input and output port units can be implemented using three major concepts for the interconnection: - shared memory, - bus interconnection and - crossbar interconnection. The shared memory concept is the implementation variant, where received packets at the input port are stored in the node local shared main memory and read out again from the output port unit for transmission. Input and output port units use direct memory access (DMA) methods for a fast copy transaction to and from the shared memory. A central processing unit performs the IP lookup operation, the IP header update operations and informs the respective output port about the lined up sending task. The intermediate copy phase with the write and read access of the shared memory limits the achievable throughput, which in turn limits the number of ports that can be served with such a structure. 20 17.11.2009 A common bus infrastructure between the port units is used in the second implementation variant, where the packet data transfer is arranged directly between the input and output units. Each input port unit needs to be equipped with the forwarding table and the route processing unit, which performs the lookup and header update operations. It independently determines the output port unit and initiates the bus transfer. Hybrid solutions with bus interconnection for frequently served prefixes and shared memory operation for error handling and complex routing operations are possible. The saved copy operation as well as the concurrently performed packet processing speeds up the router’s throughput and increases device scalability. The commonly used implementation variant in commercial routers is the crossbar interconnection between the input and output port units. A switching matrix – often referred to as “switching fabric” – provides dynamically arranged interconnections between the respective input and output unit at any given point in time. Such switching fabrics are classified into blocking and non-blocking cross-connect types. Given that two or more input port units independently work out to forward a packet towards different output port units, a non-blocking switching fabric ensures the concurrent transfer of all packets across the crossbar. Such non-blocking operation of the switching fabric can be achieved using Clos’ concept [57] of a multi-stage switching network. The clocked relay of variable length packets across the switching fabric is cumbersome. High speed routers therefore split up the packets into small fixed length packets in the input port unit, which are again reassembled in the output port unit. Such small junks are called “cells” and have a size of e.g. 64 bytes. The direction of the cells across the fabric can either be centrally organised by the switch fabric controller or more frequently used in high speed routers, by so called “self-routed” cells. The latter solution adds a short header – so called “routing tag” – to each cell, which carries the short fabric local addressing of the respective output port unit. 2.3.3 Router internal queuing structure In routers with blocking switching fabrics as well as in non- blocking switching fabrics with certain constellations of packet arrivals and common relay destinations, it is necessary to provide queuing memory in the input port units for tentative package storage. Whenever a packet can not traverse the fabric straight away, it’s servicing is delayed and the input port queue starts to build up. Given that two or more input port units independently work out to forward a packet towards one and the same output port unit at a given point in time, the path across the fabric is blocked towards this free output. If the arrival rate outweighs the servicing rate, dropping mechanisms must be in place that handle the queue’s overflow events. Output port units also need to provide an output queue for line rate adaptation, if the incoming rate of all traffic that is routed out this interface exceeds the available sending rate. This is particularly important for traffic bursts within the multiplexed streams. Given the situation, that a blocked and currently postponed packet is followed by a consecutive packet that could be served instead of the blocked one, this situation is called “Head of Line Blocking (HOLB)”. The vendor Cisco describes in its 12000 series Internet router architecture documentation [51] a multi-queue solution – called “virtual output queues (VOQ)”. The queue within each input port unit is replicated times the number of output port units (plus one for multicast). This way, each postponed packet transfer can be queued in the separate queue towards 21 17.11.2009 the currently blocked output destination and gives way to the servicing of the adjacent queue of the input unit, which is not blocked in that switching time slot. Such an optimized high speed router structure is depicted in Fig. 21. Fig. 21 IP router with non-blocking fabric and virtual output queues 22 17.11.2009 3 Basic QoS aspects 3.1 Overview The abbreviation “QoS” stands for “Quality of Service” and is extensively used in the context of telecommunication systems. This wide applicability and usage leads to a lack of understanding, what the respective QoS is. The term “Quality” as well as “Service” is not standardized and needs to be discussed in each case. In terms of data packet-switched networks, the QoS term refers to packet forwarding characteristics (transmission speed, delay, jitter, packet dropping probability and packet distortion rate). If such forwarding parameters are quantified in fixed values, this is referred to as “absolute QoS”. On the other hand, “relative QoS” distinguishes different kinds of packets or packet flows, which are treated differently in order to achieve prioritized forwarding characteristics. No absolute parameter values are applied, but priorities are assigned to the distinguished kinds of packetized traffic. “Coarse-grained QoS” combines both QoS types in a way, where not single packets or packets flows are associated with fixed characteristic parameters, but groups/classes of traffic are infolded into a fixed characteristic parameter set. Traffic separation and tunnelling techniques are common ways to support coarse-grained QoS. The ability of a packet data networks to deliver data packets in time to the right destination, with acceptable loss and distortion rates and small transfer delay variations are the main technical criteria to judge about the Quality of (transmission) Service. These technically measurable parameters, however, do not easily relate to the experienced application service quality as seen by the users at both end points of the networking system. The term “Quality of Experience (QoE)” addresses the end user’s quality experience of the services, which rely on the underlying interconnected networks and their QoS in terms of transmission quality. The mapping rules between measurable transfer QoS parameters and the resulting QoE are application specific and still in development. Fast evolving application services and QoS adaptive application implementations create a closed control loop, which obfuscates the mapping. Quality of Experience is therefore indirectly addressed but not focussed on in this work. The following sections give an overview of the QoS types, their requirements, building blocks and scopes. 3.1.1 relative vs. absolute vs. coarse-grained QoS There are three general ways to support a distinguished and sufficiently good quality of transmission. The term “sufficiently” is in place since quality requirements are application service specific and a networking system that just fulfils those requirements is “good enough” for the particular service. Any transmission quality provided in excess of those requirements might be regarded as wastage. However, such wastage is justified, if the 23 17.11.2009 capital and operational expenditures (“CAPEX” and “OPEX”) for the QoS control outweighs the economical gain of a higher utilized network. Over-provisioning The first and easiest way to achieve a sufficiently good quality of transmission is overprovisioning, which is also referred to as “over-engineering”. In that case, the transmission capacity – often falsely referred to as “bandwidth” – far exceeds the sustainable transmission requirements of any given service. Link capacity utilization of less than 40% are common in over-provisioned networks, which ensures lightly loaded network components, hardly filled input and output queues in relaying nodes and thus, fast and low latency and low drop rate transmission for all services. As long as technical solutions and the economical trade-off between service revenue and transmission equipment capital and operational expenditures allow for an over-provisioning business case, it is the easiest way to achieve sufficiently good transmission quality. The ease of planning and network operation (configuration, debugging etc.) as well as the resulting network stability are major arguments of support for this approach. Over-provisioning functions well in both cases, either the connection-oriented network operation, where flows of packets are signalled to the network beforehand, or the connection-less transmission of datagrams. Relative QoS through Prioritization As outlined before, the aspired sufficiently good quality of transmission is application service specific and therefore allows for a differentiated quality of transmission solution. As long as each application service receives the sufficiently good transmission quality, it is a completely equivalent QoS solution. This differentiation, however, requires control overhead, which results in packet classification, marking and forwarding treatment rules in each node. The term “relative” relates to the fact, that application service packets no longer receive the same relaying treatment, but some are preferably forwarded and others are delayed or even dropped. Networks with differentiated forwarding treatment support both, connection-oriented and connection-less type of operation as long as the association to a traffic class can be derived from the information carried along in the packets/datagrams. The packet marking approach is common for networks with relative QoS support, but does not preclude the same forwarding treatment as derived from some sort of association between the possible combination of packet header data with the required treatment. The relative high transmission quality of the prioritized application services comes at the expense of transmission quality discrimination for low-priority services, which ideally suffice with the resulting quality. The applicability of relative QoS solutions is only justifiable, if capacity limitations or the economical trade-off between service revenue and CAPEX and OPEX require selective service discrimination. Absolute QoS through Reservation Transmission quality in terms of guaranteed quantified parameter limits for throughput, loss, delay and delay variation can only be safely achieved by means of resource reservation with admission control and limited overbooking. The prerequisites for reservation approaches are as follows. Flows of packets need to be identified as to belong to a pre-established reservation state. All relaying nodes need to be signalled about resource requests and have to admit and reserve reservation grants. They need to keep track of the flows’ resource usage in order to detect excess traffic and to make informed decisions about incoming new reservation requests. Furthermore, the edge 24 17.11.2009 nodes or even every node need to implement admission control functions in order to mitigate excess traffic requests and to screen the granted reservations. The usage metering needs to account for average rates as well as instant peak limitations. In order to prevent false alarms through the packet multiplex in relaying nodes, traffic shapers can smooth out bursts at the expense of increased transfer delay and delay variation. Networks with resource reservation in the forwarding path require a connection-oriented type of operation. The signalling of resource requests, the resource grant along the route, the setup of admission control, traffic metering and possibly shaping units needs to be arranged during connection setup prior to the actual packet transfer phase. The guaranteed high transmission quality with absolute parameter limitations for selected packet flows of application services comes at the expense of service discrimination for low-priority – non-reserved - services, which ideally suffice with the resulting quality. The applicability of absolute QoS solutions is only justifiable, if application services or the economical trade-off between service revenue and CAPEX and OPEX require service guarantees and make up for the high control overhead in every node. Guaranteeing QoS parameters can even be targeted in over-provisioned networks. At the operator’s risk, such commitments are made and abided by, because of the lightly loaded network. All three general QoS approaches are often used in combination. Depending on the carried traffic type and the available link capacity, traffic separation with prioritization, trunk reservation and flow reservation can be applied concurrently. Coarse-grained QoS The level of granularity for absolute or relative QoS is addressed with the “coarse-grained” QoS approach. Single flow reservations with absolute QoS guarantees are targeted with the Resource Reservation Protocol as described in 4.1.2. This is clearly a fine-grained approach and far too detailed in core network nodes with thousands of traffic flows being aggregated in the stream of IP packets going through. The Differentiated Services approach with traffic being classified into traffic classes is a feasible QoS concept in large scale networking. However, up to 64 classes can be distinguished with the 6 bit DSCP marking codepoints. This still appears to be too sophisticated for global scale internetworking tasks, where currently no traffic differentiation is used. The traditional IP precedence approach provided 8 traffic markings within the 3 precedence bits of the TOS field (see Fig. 1). Although the traffic separation in two, three or four classes is expected to be a sufficient level of differentiation, the term “coarse-grained QoS” is said to comprise no more than 8 traffic classes. This coarse traffic separation is easily configured in relaying nodes, enables class based tunnelling as described in 3.2.3 and allows for bundle reservations onto the few separated (and possibly tunnelled) traffic classes. 3.1.2 QoS building blocks The quality of service that a packet perceives during the forwarding process along a given path from one communication end point to the other depends on a number of forwarding decisions along the way. The major decision points reside on the path the packet will take through the networking cloud(s), the mixture of equivalently forwarded packets as well as the per hop forwarding behaviour in each relaying node. Since the QoS treatment scope is addressed in chapter 3.2, the following paragraphs will address the node local QoS-related treatment mechanisms in detail. 25 17.11.2009 The forwarding of packets in a single node is shown in Fig. 22 and characterized by: - (1) the classification, - (2) the routing decision, - (3) the input enqueuing with (4) dropping, - (5) the input queue scheduling, - (6) the fabric transmission, - (7) the output enqueuing and - (8) the output queue scheduling. Fig. 22 Router internal forwarding path per hop behaviour Each of the decision and treatment steps will be looked at more closely in the following paragraphs. Fabric transmission The node internal relay of packets (packet cells) is predominantly performed in nonblocking crossbar structures in a contention free way of operation. It does therefore not negatively influence the overall QoS characteristic of the router. Routing strategies Differentiated forwarding of packets along differing routes is a major QoS treatment, which is addressed in chapter 3.2.2. It is the result of collaborative routing decisions and FIB setups among a set of routers and does not directly relate to node internal QoS strategies. Enqueuing strategies (input and output) The tentative storage of packets after arrival and before send out is the first major internal QoS treatment strategy, which is based on the number of queues supported in input and output port units as well as the sorting criteria for them. This work considers the availability of separated queues for distinguished packet types, the very basic prerequisite for QoS support. Whether the queues are implemented as separate hardware components or virtually distinguished in software within one hardware component is irrelevant for the experienced separation behaviour. The number of queues and the possibly varying queue lengths are important QoS parameters. The enqueuing itself is a mapping operation between the distinguished types of traffic and the available set of queues. Routers will ideally match the classification of incoming traffic with the number of available queues for separate enqueuing. Output enqueuing will follow this decision and decide the mapping based on either the packet header or routing tag carried classification information. Classification strategies If QoS is supported in a node, there needs to be a classification block in the input port unit, which sorts the incoming traffic into distinguished traffic classes. The easiest way of operation would be based on the DSCP class markings in the IP header as depicted in Fig. 5. It will, however require a class mapping (grouping) operation between the poten- 26 17.11.2009 tially 64 available DSCP classes and the actually provided enqueuing classes. Such class mappings are addressed in chapter 8. If DSCP markings are missing or not trusted, the classification can be based on any combination of IP packet header information. However, since traffic separation is most likely be performed following application requirements, a more sophisticated classification based on multi-layer classification is performed. Mainly port information of Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) packets reveal the type of traffic being transported within the packet payload. It is at the operator’s discretion to define the level of deep packet inspection and to configure the resulting classification rules. Current commercial router equipment provides input port units (line cards), which allow for hardware based line rate deep packet inspection even within the TCP and UDP payload. Dropping strategies Because of the burstiness of multiplexed packets, queues can quickly fill up and might flow over. Normally, packets are dropped at the end of the queue as shown in Fig. 23. It is called “Drop-Tail queuing”. Fig. 23 Drop-Tail queue dropping strategy However, two optimization strategies are common, which aim for early dropping of packets as congestion indication for responsive transport protocols such as TCP or selective (preferred) dropping of less important packets, if traffic separation is in place. TCP interprets any lost packet as congestion indication and will reduce the sending rate accordingly. The respective congestion window mechanism therefore tries to mitigate queue overflows in bottleneck nodes. However, packet loss indication with tail drop far too late informs the sender about the congestion, which leads to slow TCP reaction times. Sally Floyd invented the so called “Random Early Detection (RED)” mechanism [77], which randomly selects packets from a filling up queue, in order to give an early congestion indication. However, the randomness is staggered into three sections as depicted in Fig. 24. As soon as the minimum threshold is crossed, the dropping probability of each enqueued packet rises as a function of the calculated average occupied queue size. Based in this, the congestion indication to each sender is roughly proportional to the sender’s share of the transmission capacity. There are several flavours of RED available, which will be briefly mentioned here. “Dropping from the front”: Since congestion indication is time critical, the randomized dropping decision should target packets at the queue’s head instead of the tail. This is particularly important, if traffic flows are accounted for within the queue management. That is, if a packet is worked out to be dropped, flow optimized congestion indication picks an earlier enqueued packet of the same flow for the dropping instead. “Congestion marking instead of dropping”: TCP’s deficiently congestion detection has been tackled by the “Explicit Congestion Notification (ECN)” and “Pre-Congestion Notification (PCN)” extensions, which are optionally available in IP networks. End points 27 17.11.2009 signal their ECN support in one of the two ECN bits in the IP header (see Fig. 5) and relaying routers might indicate with the second bit the upcoming congestion. This forward congestion notification saves the actual packet drop (and the resulting resending of it) and with backward signalling by the receiver achieves exactly the same sending rate reduction effect as before. Enqueued packets crossing the maximum threshold in Fig. 24 are therefore all ECN marked. Fig. 24 Random Early Detection (RED) for congestion avoidance The Pre-Congestion Notification is currently defined at the IETF pcn working group and reuses ECN bits under certain conditions. PCN markings are only applied for IP packets, which are marked with the so called “DSCP for Capacity-Admitted Traffic” value [17]. Those markings are used for domain internal admission and flow control between the ingress and egress routers of a PCN domain. That is, PCN enabled network domains differentiate DSCP marked traffic flows and ensure a smooth transport of admitted flows even under high load conditions. This is achieved by blocking of new incoming flows traversing the domain or even flow termination of some existing ones. A plethora of other dropping strategies are available to operators to choose from for their per hop dropping behaviour configuration. A simple but efficient flow based dropping scheme, called “Longest Queue Drop (LQD)” [170] can be applied, if flows are recorded, equally weighted and virtually queued in a separate queue. The respective queue length related to the flow’s usage of the link capacity and the simple scheme of dropping front end packets out the longest flow queue will actually degrade each flow according to its link share. 28 17.11.2009 Fig. 25 Longest Queue Drop (LQD) of virtually separated flows Lastly, the “Weighted Random Early Detection (WRED)” strategy is widely used for differently prioritized packet dropping behaviour (e.g. in [54]). Weighted RED combines standard RED detection with prioritized dropping behaviour based on DSCP class markings. The vendor Cisco thereby concentrates on the leading three DSCP bits and performs TOS conformant IP precedence prioritization (see Fig. 5). As a result, the RED minimum and maximum thresholds are replicated and differently configured for the distinguished classes of packet traffic. Low priority packets therefore experience an early threshold setting with resulting higher dropping probability. Congestion indication via dropping is only fully effective for responsive applications like TCP – otherwise, it only throttles the respective link share usage. Scheduling strategies (input and output) Stage (5) and (8) in Fig. 22 consider servicing decisions in the view of several queues waiting for transmission. Output port units only provide a set of queues, if they distinguish between traffic classes with associated forwarding priorities. Otherwise, there is only a single queue in place for incoming and outgoing rate adaptation with simple “First In, First Out (FIFO)” scheduling strategy. Input port units provide several queues for the available N output paths to prevent HOLB and might additionally distinguish C traffic classes in separate queues per path resulting to a maximum queue set of size C x N. There are several strategies available, which decide on the servicing order of the queue set. The easiest algorithm is called “Round Robin (RR)”, which services the queues in turns, each enqueued element at a time. 29 17.11.2009 Fig. 26 Round Robin scheduling If packets are prioritised and require differentiated scheduling behaviour, the most stringent algorithm is “Strict Priority (SP)” servicing. Here, queues of lower priority are only then serviced, if all higher priority queues are empty at that time (see Fig. 27). Fig. 27 Strict Priority scheduling The strict servicing paradigm can, however, starve out low priority queues completely, which is not intended. A plethora of altered round robin and priority scheduling strategies has been developed, which all aim for some fairness in the scheduling process. The major flavours called “Weighted Round Robin”, “Deficit Round-Robin”, “Weighted Fair Queuing” and “Self Clocked Fair Queuing” will be briefly described below. “Weighted Round Robin (WRR)” is the simplest approximation of the so called “generalized processor sharing (GPS)” model and aims to grant each scheduled queue the link share according to its priority. Since WRR can only service packets (more often cells), the priority guided fairness granularity of the link share depends on the served packet sizes. The major difference to strict priority scheduling is, that WRR does not necessarily empty the high priority queue before advancing to the lower one, but rather calculates the packets of a certain queue that should be serviced this round depending on the queue’s link share priority. The number of dequeued packets at a time is the queue’s priority divided by the lowest priority of any queue. 30 17.11.2009 Fig. 28 Weighted Round Robin scheduling “Deficit Round-Robin (DRR)” as well as “Weighted Deficit Round-Robin (WDRR)” are both modifications to the relating aforementioned two strategies. The dequeuing granularity is statistically stretched across multiple dequeuing phases and accounts for the difference between the calculated dequeuing share in bytes and the actually used up share in dequeued packet bytes. Unused granted bytes are added to the next round share, which eventually dequeues the delayer packet due to the accumulated granted byte count. The accumulated difference count is called “deficit count.” The number of granted bytes served in each round is called “quantum” and is either defined fixed or weighted by the queues priority. All fair queuing approaches follow the idea of an ideally fair mixing of traffic streams with infinitesimal fine granularity. This scheme is called “Generalized Processor Sharing (GPS)” and can be symbolized as in Fig. 29. Fig. 29 Symbolized fair queuing in an idealized GPS = Fluid-Flow Queuing “Weighted Fair Queuing (WFQ)” [66], [149] is a Packet-by-packet Generalized Processor Sharing (PGPS) approach, which takes into account, that the entire unit of traffic (packet) must be served in a dequeue process. The fluid-flow queuing ideal is approximated by means of an internal simulation of the theoretical service finishing time of each scheduled queue and ordering the actually performed servicing of the packets according this finishing time order. The difference between simulated finishing time and actual finishing time is a metric for the experienced service degradation (unfairness) for the respective queue. 31 17.11.2009 Fig. 30 Fluid-flow approximated queuing in WFQ The scheduling strategy called “Class-Based Weighted Fair Queuing (CBWFQ)” is a modified WFQ in the sense, that it specifically assigns traffic classes to queues and configures different weights and queue lengths to the set of serviced classes. The calculation of the finishing time in fluid-flow simulations is based on actual time steps and independent from enqueuing or dequeuing events processed in a scheduler. This absolute time scale is therefore replaced by an internally generated virtual time scale. The packet stream becomes “self-clocked”, hence the name “Self Clocked Fair Queuing (SCFQ)” [84]. The calculation of the self-clocked virtual finishing time is a function of the following parameters. * packet length, virtual finishing time = f * virt. finishing time of the previous packet of the same queue, * system’s virtual time at packet arrival In practice this can be implemented by stamping an arriving packet with a service tag equal to the corresponding virtual finishing time and servicing the packets in ascending order of service tags. Linking the virtual time to the work progress in the actual packet-based queuing system is a fast and computational simple operation, which yields the same packet ordering result as the original WFQ approach. SCFQ is therefore the preferred implementation variant of any WFQ inclined scheduling strategy. Given the described variety of building blocks along the router internal forwarding path, a basic QoS supporting router with non-blocking switching behaviour, virtual output queues and eight supported QoS classes is sketched in Fig. 31. Cisco’s 12000 series routers [52] apply weighted random early detection for class-based dropping (congestion avoidance) as well as deficit round robin scheduling within the input port unit and an proprietary modified deficit round robin for the output port scheduling. 32 17.11.2009 Fig. 31 VoQ with 8 classes CoS support (scheduling and dropping) As depicted in Fig. 32, the overall per hop forwarding behaviour experienced by any traffic being routed across a single router node is comprised of all router internal building blocks and their configuration. Per hop treatment Separate enqueueing Queue Scheduling Packet Dropping Number of queues 1:1 enqueueing M:N enqueueing Round Robin WRR/WDRR WFQ CBFQ Drop Tail RED WRED Combination = Per hop forwarding behaviour Fig. 32 Per hop forwarding behaviour composition in relaying nodes Additional building blocks are required, if absolute QoS parameters are to be guaranteed. This encompasses metering, rate based remarking, possibly shaping and admission control. All of those blocks rely on measurement algorithms at the input or output of the system, which either directly influence the relaying and sending behaviour (packet dropping, delaying or downgrading) or take the measurement thresholds as indicators for accounting or other traffic related configuration decisions. Generally speaking, a policing block is added to the input port unit and a shaping block to the output port unit. “Policing” is controlling the general access of a certain traffic type for the node or less stringent, a rate limitation element. The functionality can be packet based or flow based, depending on the ability to associate single packets into flows by indication or by classification outside the policing block. Traffic measurements for the respective packets or flows are taken and compared with a configured threshold in order to judge about rate conform- 33 17.11.2009 ing (“in”) traffic or excess (“out”) traffic. Policing of excess traffic can either result in packet drops or in downgraded markings, if traffic differentiation by packet markings is in place. A special policing operation is “admission control”. It is the strict dropping operation of packets either based on the excess traffic measurement for previously agreed on sending rates or based on a flow indication and a list of previously granted or refused flows. This essential part on agreed rates or flows can either be implemented with signalling protocols for rate or flow reservations or be part of a written agreement between interconnected parties, called “Service Level Agreement (SLA)”. “Shaping” controls the burstiness of transmitted streams of packets. A measurement based delaying of the enqueued outgoing packets assures a smoothed out packet sending rate. Traffic aggregation points are typical causes for traffic bursts due to the statistical multiplex of the independent packet streams. The required measurements are generally based on two measurement analogies, “leaky bucket” and “token bucket”. Both algorithms consider the filling and draining of an imagined bucket with controlled throughput characteristics. It is important to mention, that both algorithms come up with filling level and timing values, which can in turn control the policing or shaping action. “Leaky bucket” measurement model (see Fig. 33): This simple model emulates the filling of a bucket with elements and concurrently the dripping out of elements at a fixed rate. Two sources of events (and consecutively followed actions) can be taken out of this model. The first is the occurrence of overflowing elements, if more than b elements have been queued up in the buckets at any point in time. This happens, if the average filling rate of the variable filling process exceeds the constant drainage of the bucket. The size b of the bucket thereby represents the averaging time interval. Elements that are arriving in excess of the bucket’s capacity are dropped. The second source of events is the sending time of dripped out elements. The model is most often used for traffic shaping due to its applicability to smooth out bursty incoming traffic into constant rate outgoing traffic. Traffic peaks are eliminated by the delayed relay procedure. However, the overflow events can also be used in policing blocks, which detect excess traffic this way and predominantly drop or possibly remark the excess elements. Leaky bucket implementations are mostly used in “Asynchronous Transfer Mode (ATM)” networks and named “Generic Cell Rate Algorithm (GCRA)” there. 34 17.11.2009 Fig. 33 Leaky bucket algorithm “Token bucket” measurement model (see Fig. 34): This most commonly used model also fills and drains a bucket, but with a token controlled variable drainage rate. This rate variation is bounded by the bucket size b and leads to a constant average sending rate. Essential parameters of the model are the “Token Bucket Rate – r” and the “Token Bucket Size – b”. Tokens are released into the bucket at a constant rate r and will be used up for each drained element. Incoming elements leave the bucket, if enough tokens are available for the drainage. Otherwise, the elements are stored in the bucket until enough tokens have been released by the token rate. If b elements are stored in the bucket when a new element arrives, this element flows over and is classified as excess traffic. Useful additional parameters for a token bucket model are “Peak Data Rate – p”, “Minimum Policed Unit – m” and “Maximum Packet Size – M” (see e.g. [182]). The peak rate is limited by either the interface line rate or the maximum data generation rate at the sender. The maximum packet size is a fixed statement about the largest packet size being processed and must be smaller than the link’s maximum transmission unit (MTU). The minimum policed unit is the minimum granularity of the decision process. Elements of smaller size are set to have the minimum size in the calculations. Three sources of events (and consecutively followed actions) can be taken out of this model. The first is the occurrence of overflowing elements, if more than b elements have been queued up in the buckets at any point in time. The second source of events is the sending time of dripped out elements. A bucket filled with tokens that is seen by a newly arrived packet will immediately send out any incoming packet at the full incoming rate. However, if the token reserve has been used up, the dripping will follow the token rate. The third source of events is the first moment, where an incoming packet can not be sent out instantly, due to a lack of tokens for it. 35 17.11.2009 Fig. 34 Token bucket algorithm The token bucket metering is used within the integrated services architecture (IntServ) (see 4.1.2) and is addressed in detail in RFC 2210 [182]. Two widely used marking schemes make use of token buckets: - RFC 2697 - A Single Rate Three Colour Marker [89] and - RFC 2698 - A Two Rate Three Colour Marker [90]. They use one or two token buckets to determine the degree of conformity of traffic to several marking levels. Each token bucket can be used to compare the stream of packets against a certain token rate and bucket size parameter set. However, if the stream needs to be classified in several traffic classes, a number of token buckets with differing parameter sets can be used. The single rate marker uses two token buckets with the same token rate, but different sizes, in order to base a three colour marking on the differing burst sizes. The two rate marker also uses two token buckets, however with differing token rates and bucket sizes. The three colour markers are often used in differentiated services (DiffServ) (see 4.1.1) setups. The above mentioned congestion notification scheme PCN makes also use of token bucket metering for its signalling decisions. However, it introduces a filling threshold below the actual bucket size. Early warning signals can now be triggered, if the bucket filling crosses that warning threshold. The algorithm is therefore called “threshold marking algorithm” [70]. Both metering algorithms, leaky bucket and token bucket, are clearly understood, if equally sized elements are used. However, the transition to IP networks reveals a level of uncertainty or unfairness, when variable length packets are measured. If the bucket calculations would be based on packets as a whole, the decision taking remains easy and clean. However, the metering outcome is distorted by unequal packet sizes and thus not useful. 36 17.11.2009 In practise, both models revert to byte or even bit counts for bucket sizes, packet sizes and tokens in order to accommodate the unequal resource shares of the varying packets. However, packets can now partially overflow, find partially available tokens and introduce varying times for sending, threshold crossing and overflow events. This phenomenon is addressed in RFC 3290 [28] with the definition of loose and strict mode of metering operation. Strict conformance “Packets of length L bytes are considered conforming only if there are sufficient tokens available in the bucket at the time of packet arrival for the complete packet (i.e., the current depth is greater than or equal to L): no tokens may be borrowed from future token allocations.” Loose conformance “Packets of length L bytes are considered conforming if any tokens are available in the bucket at the time of packet arrival: up to L bytes may then be borrowed from future token allocations.” The strict mode of operation is commonly used, since it avoids negative parameter values. 3.2 QoS treatment scope Quality of Service differentiation of packet streams can be applied in different levels of granularity as well as treatment scope. The QoS per hop treatment options have been described in the preceding chapters and will be referred to as “QoS-based forwarding”. Outside the single node behaviour, the path - a packet stream takes through a meshed network - can also be guided by QoS differentiation, which will be called “QoS-based routing”. Lastly, groups of packets belonging to a traffic class can be encapsulated in tunnelling technologies either within the IP layer (e.g. Generic Routing Encapsulation (GRE) [72]) or below (e.g. MPLS LSP see 4.3, Ethernet VLANs see 4.2, etc.). Such QoSbased tunnelling will be titled as “QoS-based tunnelling”. Fig. 35 depicts the differences in the control plane routing processing as well as the resulting FIB differences. Fig. 35 QoS-based IP lookup variants As a starting point Fig. 36 depicts a simple inter-domain network setup, that will be used as an example for the following three sections. It assumes three interconnected ASes, which internally perform traffic separation in certain granularities (AS 1 with 4 classes in layer 2 and 3 / AS 2 without separation / AS 3 with 4 classes in layer 2 and 3). The interconnecting AS border routers remove any separation and provide no QoS-based (i.e. 37 17.11.2009 “best effort” only) traffic exchange. From the point of view of AS1 and AS3, AS2 is a transit provider, which also offers best effort only transit service. Fig. 36 Best Effort interconnection example Customer traffic from AS1 towards a prefix originating from AS3 will find AS2BR_1 as next hop and the respective output interface for the interconnection link during the IP lookup procedure. Packet markings within the IP header of the traffic have no influence in this lookup and will be removed or ignored by AS1BR and AS2BR_1. The packet relay within the transit AS is shortest path per-hop forwarding without traffic class separations. AS2BR_2 in turn will find AS3BR as next hop and the respective output interface for the interconnection link during the IP lookup procedure. Packet markings within the IP header of the traffic have no influence in this lookup and will be removed or ignored by AS2BR_2 and AS3BR. The traffic entering AS3 might either be relayed internally as a best effort traffic class toward the destination or AS3 might perform costly multi-layer ingress classification in order to guess the most suitable traffic class out of the supported three class set. The three treatment scope variants are described below. 3.2.1 QoS-based forwarding The simplest quality of service inferred packet transmission behaviour is achieved, if conventional routing (and thus path selection) remains unchanged and the traffic differentiation is made hop-by-hop within relaying nodes. All QoS building blocks as of chapter 3.1.2 are applicable. Packets will traverse the network along the same paths as if no QoS support were enabled. However, the experienced per hop treatment of each traffic class will reveal the differentiated forwarding and dropping behaviour, which results in measurable QoS improvements for higher prioritised classes in terms of delay, loss and throughput compared to the common unclassified best effort network behaviour. It is at the operator’s choice to select and apply a combination of QoS building blocks for forwarding in every node along the chosen route. The standard shortest path first routing behaviour of IP networks remains unchanged, which normally results in unbalanced network load distributions. The exchange of reachability information within the router control plane does not necessarily signal QoS information for advertised IP prefixes and thus allows for no QoS specific best path selection. RIB and FIB contain just one routing entry per IP prefix and point to the relevant next hop and port information. Remapping information might additionally be 38 17.11.2009 stored, if the next hop network requires a different DSCP marking in the IP header for the same traffic class. The quality of service in QoS-based forwarding solely relies on node internal QoS means based on the QoS information carried within the IP header. Fig. 37 QoS-based forwarding interconnection example In the example of Fig. 37, customer traffic from AS1 towards a prefix originating from AS3 will find AS2BR_1 as next hop and the respective output interface for the interconnection link during the IP lookup procedure. Packet markings within the IP header of the traffic are used in AS1BR for queue selection, scheduling and dropping decisions. Both interconnected border routers will respect the QoS packet markings. Since different class sets are supported in AS1 and AS2, either AS1BR or AS2BR_1 is responsible for class mapping and possibly packet remarking. The packet relay within the transit AS is shortest path perhop forwarding with traffic class separation based on the offered QoS-based transit class set as of AS2. AS2BR_2 in turn will find AS3BR as next hop and the respective output interface for the interconnection link during the IP lookup procedure. Packet markings within the IP header of the traffic are used in AS2BR_2 for queue selection, scheduling and dropping decisions. Both interconnected border routers will respect the QoS packet markings. Since AS2 and AS3 both support a three class setup, there might not necessarily be any class mapping and remarking in place. Whether or not identical class sets and markings are used in both systems needs to be checked during separate QoS signalling information exchange. There is no longer a necessity to perform multi-layer ingress classification due to the QoS-based forwarding and consistent marking procedure. 3.2.2 QoS-based routing Shortest path first routing tends to unbalance network loads due to the simple preference of the shortest path algorithm. The resulting routing along congested links can improve the experienced QoS for prioritized classes by means of QoS-based forwarding. However reducing the links’ load by diverting the network traversal for differentiated streams of traffic is far more effective. Routers would ideally support multi-path routing entries in their RIBs and FIBs and direct traffic of different classes along different next hop routes. If multipath routing is not supported, the selection of the shortest path could be augmented by QoS-based conditions. That is, paths with signalled QoS support should be preferred over others, even if the non-QoS paths would be shorter. If multiplex QoS-supporting paths are discovered towards the same prefix, the selection should prefer the one with the best matching QoS class set. This would lead to a best QoS path selection rather than a shortest path selection. A similar best path selection is already used in inter-domain 39 17.11.2009 routing with BGP (see Fig. 12). Applying QoS-based routing to BGP would therefore either require multi-path BGP and/or a modified best path selection with added QoS match checking (see Fig. 38). QoS based path selection Best path selection process Multi-path traffic assignment “QoS based load balancing” Additional selection condition for (extent of) QoS support Available in meshed setups, where multiple (instead of “best”) paths are selected for a prefix Best path selection modification ! Fig. 38 Multi-path support required QoS-based path selection in BGP The exchange of reachability information within the router control plane would ideally comprise the signalling of QoS information for advertised IP prefixes and thus allow for QoS specific best path selection. RIB and FIB either contain the best QoS matching routing entry per IP prefix or even multiple entries for the same IP prefix with associated QoS marking selections. Remapping information might additionally be stored, if the next hop network requires a different DSCP marking in the IP header for the same traffic class. The quality of service in QoS-based routing relies on QoS-based route selection possibly combined with node internal QoS means based on the QoS information carried within the IP header. Fig. 39 QoS-based routing interconnection example In the example of Fig. 39, customer traffic from AS1 towards a prefix originating from AS3 will either find AS2BR_1 or AS4BR_1 as next hop and the respective output interface for the interconnection link during the IP lookup procedure. This route selection would be based on the DSCP marking of the packets and sort out priority traffic using a different transit route than ordinary traffic. In either case, packet markings within the IP header of 40 17.11.2009 the traffic are used in AS1BR for queue selection, scheduling and dropping decisions. Each interconnected border router will respect the QoS packet markings. Since different class sets are supported in AS1, AS2 and AS4, either AS1BR, AS2BR_1 or AS4BR_1 is responsible for class mapping and possibly packet remarking. The packet relay within the transit AS2 or AS4 might or might not apply QoS-based routing internally as well. That is, shortest path per-hop forwarding or best QoS path per-hop forwarding with transit QoS traffic class separation is applied. Either AS2BR_2 or AS4BR_2 will eventually be reached by the relayed packet and in both cases will find AS3BR as next hop and the respective output interface for the interconnection link during the IP lookup procedure. Packet markings within the IP header of the traffic are used in AS2BR_2 or AS4BR_2 for queue selection, scheduling and dropping decisions. Each interconnected border router will respect the QoS packet markings. Remapping and packet header remarking will be performed by either AS4BR_2 or AS3BR in order to match the available class sets. Whether or not identical class sets and markings are used in the involved autonomous systems needs to be checked during the separate QoS signalling information exchange. There is no longer a necessity to perform multi-layer ingress classification due to the QoSbased forwarding and consistent marking procedure. 3.2.3 QoS-based tunnelling Offering network services to customers and service providers increasingly calls for tunnelled traffic transport. Two major reasons can be observed for the current trend towards encapsulated and route pinned IP packet transport. The first reason is traffic engineering and enhanced control over IP packet transmissions. Packet forwarding within tunnels enables operators to skirt around shortest path first routing. Tunnels can be planned, setup with fixed traversal nodes and dynamically switched around without IP forwarding and routing interference. Maintenance operations and network resilience characteristics make use of backup tunnel setups and fast switching operations between the active and the standby tunnels. The second reason for increased tunnelling usage is transparency. Tunnelled customer traffic does not interfere with network local addressing or QoS setup peculiarities. Virtual private networking services are one major application for tunnelled customer traffic transport. Tunnelling can either be realized within the IP layer, e.g. using GRE [72], or below the IP layer, most importantly with MPLS (see 4.3) and Ethernet VLANs (see 4.2). If customer IP traffic is encapsulated in transit provider IP-based GRE, then the DSCP of the outer IP header becomes the tunnel QoS marking. Similarly the 3 bit QoS markings of MPLS (traffic class (TC) marking) and Ethernet VLANs (“Priority Code Point (PCP)” marking) in each respective header format become the 8 class limited QoS marking of those lower layer tunnelling technologies. From the QoS perspective, two major distinctions should be made for QoS-based tunnelled transport. Either there is just one tunnel available with QoS markings in the tunnel header information or several tunnels are setup, which each represent a certain QoS class. The latter is generally available in any “virtual channel (VC)” based networking technologies. With the advent of “Generalized MPLS (GMPLS)”, any time slot, wavelength, fibre etc. can establish a separated channel, which can in turn stand for QoS-based traffic separation. Hence, a channel per traffic class option is expected to be increasingly used in future Internet setups. Applying the described tunnelled transport to IP forwarding is easily setup within an autonomous system. However, the AS interconnections are IP based only and any tunnelled peering needs mutual agreements of the interconnected peers. IP-based GRE encapsulation is always a valid option. However, inter-AS MPLS tunnels and inter-AS 41 17.11.2009 VLANs are more appealing. Point-to-point interconnections might be able to support both lower layer tunnelling options. The predominantly used Internet Exchange Points, however are neuralgic intersections, where Ethernet VLAN tunnelling would make a major difference for transparent QoS-based customer traffic transport. Fig. 40 depicts the described tunnelling options. The E-LSP and L-LSP abbreviations will be explained in detail in chapter 4.3. Tunnelling options Inter-AS tunnelling Intra-AS tunnelling - Inter-AS E-LSPs … - “tunnelling” through L2 marking (VLAN, PCP) - Unlikely: layered peering - E-LSPs, Carrier Eth.+PCP - L-LSPs, VCs, λs, fibres etc. Tunnelled forwarding strongly recommended Operator’s choice Fig. 40 L2 marking most likely Mutual agreement Tunnelling scope The exchange of reachability information within the router control plane would ideally comprise the signalling of IP layer QoS information for advertised IP prefixes as well as the supported tunnel QoS information. Routing is then augmented into a best tunnel selection process together with the instalment of respective tunnel mapping information in the router’s RIB and FIB. Depending on the 1:1 or n:1 class-to-tunnel mapping, there is either one best QoS matching routing entry per IP prefix with tunnel selection and tunnel marking adoption or even multiple entries for the same IP prefix with class-based tunnel selection. Due to the tunnelled transport, customer traffic no longer needs to be remarked in the header, but the tunnel selection and tunnel marking cares for the correct forwarding path and per hop treatment. QoS-based tunnelling relies solely on the tunnel “header” information to select the appropriate QoS building blocks within relaying nodes. Tunnelled transport can therefore provide several transit QoS class sets to external customers and does not need to change customer header information. It is the preferred type of transit service in the future. Major drawbacks of this approach are the currently missing Inter-AS tunnel support as well as the missing standardized mapping between encapsulated IP QoS and outer tunnel QoS markings. This is one of the addressed improvements of this thesis. 42 17.11.2009 Fig. 41 QoS-based tunnelling interconnection example In the example of Fig. 41, customer traffic from AS1 towards a prefix originating from AS3 will find AS2BR_1 as next hop and the respective output interface for the interconnection link during the IP lookup procedure. The packet markings within the IP header of the traffic are used in AS1BR for queue selection, scheduling and dropping decisions. Both interconnected border routers will respect the QoS packet markings. Since different class sets are supported in AS1 and AS2, either AS1BR or AS2BR_1 is responsible for class mapping and possibly packet remarking. The packet relay within the transit AS2 is chosen to be tunnelled in two provided edge-to-edge tunnels of the AS. Two classes are set to be provided for transit QoS, each class per tunnel. That is, incoming packets at the border router AS2BR_1 will find two routing table entries for all reachable IP prefixes originating from AS3, with differing outgoing interface mappings. Depending on the packet’s DSCP marking, either of the two QoS-based transit tunnels will be selected. This includes IP packet encapsulation. In case of single tunnel transport and tunnel QoS support, an additional mapping of IP QoS markings onto the tunnel header QoS marking would be performed. The encapsulated packets are now transferred across AS2 applying per hop behaviour in each relaying node as derived from the tunnel associated QoS building block behaviour. AS2BR_2 will eventually be reached by the relayed packet. The encapsulation is removed and normal IP lookup will work out AS3BR as next hop together with the respective output interface for the interconnection link. Packet markings within the IP header of the traffic are again used in AS2BR_2 for queue selection, scheduling and dropping decisions. Each interconnected border router will respect the QoS packet markings. Remapping and packet header remarking will be performed by either AS2BR_2 or AS3BR in order to match the available class sets. Whether or not identical class sets and markings are used in the involved autonomous systems needs to be checked during the separate QoS signalling information exchange. There is no longer a necessity to perform multi-layer ingress classification due to the QoS-based forwarding and consistent marking procedure. The described tunnelling procedure is known as “peer model”, where the tunnelling is confined within the transit AS boundaries. However, if the interconnection links are inter-AS tunnelling enabled, the so called “overlay model” can be used. That is, customer traffic is encapsulated in the sending AS1 and decapsulated in the egress border router of AS2 or ideally the ingress border router of AS3. Such an inter-AS tunnel would use tunnel QoS markings, which need to be agreed on between at least AS1 and AS2. Multiple inter-AS tunnels are also feasible, where AS2 would offer several entry points for differentiated tunnelled transit. The QoS-based tunnelled transport of IP traffic with differentiated traffic classes is optimal for transport of unmodified packets and for consistent QoS marking support. However, it requires cross-domain reachability and QoS marking signalling combined with cross-layer QoS mapping information. Both requirements are addressed improvements of this thesis. 43 17.11.2009 3.3 Architectural scope 3.3.1 Cross-layer QoS Quality of Service support has been targeted in many networking technologies. Circuit switched networks, such as the “Plain old telephone service (POTS)”, reserve for instance separate lines / fibres, separate time slots, separate frequencies, separate wavelengths, etc. for the interconnection of communication endpoints. Such reserved channels inherently provide excellent quality of service being designed and operated with exactly those reservations as required for the targeted service. Packet switched networks such as “Frame Relay (FR)”, “Asynchronous Transfer Mode (ATM)”, Ethernet with “virtual local area network (VLAN)” support, “Multi Protocol Label Switching (MPLS)” and others provide similar channel emulations by means of virtual channels. Packetized information transported in frames is separated by channel identifiers (FR->DLCI, ATM->VPI/VCI, Ethernet->VLAN-ID, MPLS->Label), which are either preconfigured or dynamically setup during connection setup. In either way, such separated channels provide excellent lower layer support for QoSbased tunnelling as described in 3.2.3. If separate QoS-inferred transport “lanes” are used for IP packet transport, such setups will be named “Layer 1 QoS support” in the remaining chapters. All of the named packet switching networks also provide some means for QoS-related frame markings. First of all, the channel identifier can be associated with certain per hop treatments and possibly reservations. Furthermore, explicit “QoS markings” are available such as: • FR Æ “Discard Eligibility (DE)” bit • ATM Æ “Cell Loss Priority (CLP)” bit, • Ethernet-VLAN Æ 3 bit Priority Code Point (PCP) and • MPLS Æ 3 bit Traffic Class" Field [9]. Such quality of service support for tunnelling technologies will be referred to as “Layer 2 QoS support”. If QoS is aimed at in the IP networking layer (layer 3), the underlying QoS support should ideally be incorporated in the QoS-based routing and forwarding process. However, according to the layering concept, IP does not normally be aware of the underlying technologies and their QoS support. The mapping between the upper layer QoS class set and marking and the lower layer QoS class set and marking is neither harmonized nor standardized. The differentiated services approach mentions this drawback in RFC 2475 [32] and recommends in guideline 15, that specifications of per hop behaviour should include such layer two mapping recommendation. However, this is not the case. For more details, see chapter 8.2. Vendors also give configuration guidelines for such mappings to their customers. Some examples of Cisco default mappings are given in chapter 8.2. The Ethernet standardization addresses VLAN-based frame priorities in its standard 802.1p, which is now incorporated in 802.1D [97]. However, IEEE does not provide mapping rules outside the Ethernet VLAN realm. It does standardize so called “User Priority Regeneration” [97] and [98], which defines how priority classes are to be mutually mapped, if different class set granularities are supported. Only the initial setting is stated, which might be changed by port specific configuration. This is looked at more closely in chapter 8.2 as well. In summary, many configuration options are offered to network operators for cross-layer QoS settings and some recommendations and default settings are provided. However, there is no standardized mapping defined and reliable configurations can only be assumed on single administrative domains. All default configurations can be overwritten and 44 17.11.2009 supported markings and mappings need to be dynamically signalled or manually configured based on SLAs. 3.3.2 Cross-domain QoS The “Resource ReSerVation Protocol (RSVP)” [39] is the predominantly used QoS signalling protocol for single packet flow reservations as well as for trunk reservations together with MPLS or other tunnelling mechanisms. The protocol belongs to the integrated services architecture (see 4.1.2) and sends traffic specifications toward the traffic sink and on the way back receives the actually achieved reservation specification. Strict parameter based reservations with quality guarantees can easily be set up by this method. RSVP is an end-to-end applicable reservation protocol, which potentially allows the resource reservations across domain boundaries. This protocol has further gained importance, since MPLS has chosen RSVP augmented with traffic engineering extensions (RSVP-TE) for the predominantly used path setup protocol for traffic engineered MPLS paths [11]. RFC 5151 [73] addresses the inter-AS setup of MPLS paths using three possible methods: contiguous, nested and stitched paths. One crucial signalling requirement for neighbouring domains is the class set and marking information exchange during the setup procedure. This can be achieved within RSVP-TE by means of the so called “DIFFSERV” object. Two types have been defined for 1:1 and n:1 class-to-tunnel mappings and can be found in RFC 3270 [75]. Besides this Inter-AS MPLS approach, there is no generally applicable protocol available, that signals available traffic separation class sets and their encodings across AS boundaries and beyond. The setup of a QoS supporting IP forwarding path across several ASes with potentially different class sets and encodings is currently unsolved and requires explicitly arranged mutual agreements between neighbouring AS peers along the way in order to establish a SLA guided forwarding path. Furthermore, the availability of transit QoS in a certain class set extent as well as the required packet marking for appropriate QoS class selection is not globally known, which complicates the search for QoS capable peering partners to be contacted for SLA negotiations. This major drawback of a currently missing generally available simple QoS signalling mechanism across AS boundaries is resolved by this thesis. Chapter 7 describes the proposed solution. 45 17.11.2009 4 State of the art QoS Concepts As described in chapter 3, over-provisioning is the easiest and currently used quality of service concept in the Internet. As long as the cost for hugely over-dimensioned transfer capacities (factor 5 to 10) is lower than any QoS scheme cost (investment in QoS capable devices, staff training, operation monitoring, debugging, SLA negotiations etc.), there will always be a sensible decision to go for the over-dimensioned network approach. However, QoS concepts have been developed for IP and other networking technologies, which are used within administrative domains. The following sections give a brief overview to them. 4.1 IP QoS The Internet protocol included from its very beginning a marking option in its header Fig. 5 to differentiate the priority and optimisation indication for the packet’s forwarding treatment. Those precedence and type of service marking were, however not used by vendors and network users and used to be ignored widely for many years. The carried IP traffic used to be data traffic only, which is insensitive to packet loss or delay and can cope with bad transfer qualities through retransmissions and packet buffering. Following the classification in Table 1, this is batch transfer, where sending and receiving have bulk characteristic without any strict timing requirements. Throughput is the main transfer parameter. Table 1 Transfer demand matrix – after [79] Source Bulk Bulk Batch transfer e.g. file transfer Stream “Replay” application e.g. video on demand Stream “Recording” application e.g. recording of a measurement signal Signal transfer e.g. voice over IP, video conferencing Sink “Recording” applications are streaming services, where the sender needs to regularly transmit its packetized information. The reception can be delayed since it is not critical for the recording. If buffering is used within the source, packet retransmissions are feasible to cope with packet losses. “Replay” applications are streaming services, which require bounded transfer delay constraints. Some delay variations can be compensated by replay buffers and time shifted replay start points. However, excess delay causes late reception of packets and becomes useless for the sink. Losses can not be repaired by retransmissions. Signal transfer applications are duplex streaming services, which require bounded transfer delay constraints in both directions. Only very limited sender and receiver buffers can be used for transfer delay variation compensation. In both directions, packet retransmissions are not available for loss compensation. Acceptable one way transfer delay times are for instance investigated in ITU-T Recommendation G.114 . According to the analysis of very good user satisfaction for voice over IP, the mount-to-ear-delay ideally stays below 150ms. 46 17.11.2009 The demand for IP QoS has risen with the increased usage of IP networks for the latter three types of time and loss critical services. 4.1.1 DiffServ The Differentiated Services (DiffServ) architecture is a prioritization concept of aggregated traffic providing relative QoS (see 3.1.1) in IP networks. Work on DiffServ started in the late 1990s and resulted in two major standard documents: • RFC 2475 - An Architecture for Differentiated Services [32] and • RFC 2474 - Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers [142]. Three interesting citations out of the work on DiffServ should be mentioned here, which clarify the situation and intention of its developers. “What problem are we solving? Give “better” service to some traffic (at the expense of giving worse service to the rest). ATM marketing fantasies to the contrary, QoS is a zero-sum game: - it does not create bandwidth. - it does not guarantee that you get better service.” Van Jacobson [121] "QoS is managed unfairness." Kathleen Nichols [141] “DiffServ wasn't chartered to solve the end to end QoS problem. It was chartered to define coarse-grained class-of-service differentiation, which is an entirely different (and much easier) goal.” Brian E. Carpenter [44] The formal description of DiffServ’s intention is given in RFC 3086 [143]: “The differentiated services framework enables quality-of-service provisioning within a network domain by applying rules at the edges to create traffic aggregates and coupling each of these with a specific forwarding path treatment in the domain through use of a codepoint in the IP header”. An overview of the Differentiated Services architecture is depicted in Fig. 42. 47 17.11.2009 Fig. 42 Differentiated Services regions, domains and nodes Packets of the different traffic streams are grouped into so called “Behaviour Aggregates (BA)”, which infers the same per hop treatment behaviour (PHB) in relaying nodes along the path. A BA is identified by as differentiated services codepoint (DSCP). All nodes within a DS domain therefore associate a consistently configured set of treatment policies (queuing, scheduling, dropping) to each specific BA. This applies to core as well as to edge nodes. Core nodes classify behaviour aggregates solely by inspecting the packet‘s DSCP information. Edge nodes of a DS domain, however, additionally perform multi-field classification and conditioning functions. That is, the classification / grouping in the domain’s ingress node inspects a combination of possibly several header information fields and the ingress interface of that packet in order to make a policy guided decision about the packet’s aggregation into a certain BA. This in turn results in the appropriate DSCP marking (see Fig. 43). Fig. 43 Behaviour aggregate classification and DSCP marking The edge node’s conditioning comprises several function elements, such as meter, marker, shaper, and dropper. It is important to understand, that DiffServ assumes the correct classification and in particular the conditioning functionality being sufficiently enforced at the edge of a DS domain. This in fact increases the scalability of the concept and takes the burden of computational intensive multi-field classification and conditioning off of the core nodes. Although this approach does not preclude or address the possible burstiness and congestion conditions that will arise in internal traffic aggregation points, it is a good compromise between strict QoS support and required control overhead. Internal over-provisioning can be used as countermeasure. 48 17.11.2009 RFC 2475 [32] depicts the classifier and traffic conditioner structure as show in Fig. 44. Fig. 44 Logical View of a Packet Classifier and Traffic Conditioner The policy-based setup of PHB associated treatment combinations of queuing, scheduling, dropping, metering, marking and re-marking, shaping and dropping is made by each DS domain administrator and can make use of the plethora of mechanisms being described in chapter 3.1.2. Several DS domain may be operated under the same administration, which relieves the edge node operation of neighbouring DS domains. If both domains use the same configuration, the ingress edge can simply operate with core node functionality. Otherwise, full ingress operation needs to be applied. If different administered DS regions are interconnected, there needs to be an agreement on how to setup the ingress classification and conditioning. This is normally done with SLAs containing the so called “Traffic Conditioning Agreement (TCA)”. As described above, the central point of operation of DS domains are behaviour aggregates with associated PHBs. There is no limitation being standardized regarding the number of possibly applicable PHBs. However, the mapping into DSCP is limited to 64 possible values, which results in operator specific PHB-DSCP mappings (see Fig. 45). Fig. 45 PHB ÅÆ DSCP mapping The state limitation of 4096 “global PHB space” is not a strict limit and relates to the definition of so called “Per Hop Behaviour Identification Codes (PHB-ID)” [31] for PHB signalling usage out a consistently managed DS region. A typical example is inter-provider 49 17.11.2009 PHB identification. The PHB ID encoding distinguishes between standard PHBs as described below and non-standardized PHBs. The latter are IANA assignable and currently limited to 12 bit length, hence the 4096 limitation. Fig. 46 PHB encoding [31] PHB ID codes can contain a set bit 14, which indicates PHB sets. This is best understood for the below described 4 classes of AF PHB with 3 dropping precedence encoding each. Besides the free choice for PHB definitions, there have been 22 per hop behaviour defined with recommended DSCP encoding values, which will be briefly explained below. Default PHB - 000000 RFC 2474 [142] defines the default setting of the DSCP header field to “000000”, which is the standard IP encoding for “best effort” service. This is also the last resort choice, if the DS domain ingress node can not associate another DSCP encoding during its classification process. Class-Selector (CS) PHB – xxx000 RFC 2474 [142] has included the 8 DSCP encodings for backward compatibility reasons with the original IP precedence encoding. The three leading bits of the 6 bit DSCP are thereby numbered with the equivalent class selector number CS0 – CS7. CS0 is identical with the default PHB. Some minimal requirements have been stated concerning packet ordering and timely delivery for those classes. It should be noted, that CS-DSCPs all encode the lower 3 bits to 000. However, as DS domain, which only supports class selector PHBs will classify incoming packets on the three leading bits only. This way, different DSCPs might be merged into single CS DSCPs. Expedited Forwarding (EF) PHB - 101110 RFC 3246 [60] defines this expedited forwarding per hop behaviour, that is intended to support low delay, low jitter and low loss services. It is the only PHB, that is normally associated with a rate limitation that is strictly enforced in the DS ingress edge nodes. EF marked traffic is expected to experience short or even empty waiting queues in relaying nodes, which leads to low packet loss as well as low and hardly jittering transport delay times. Separate enqueuing and highly prioritized scheduling are the keys for EF forwarding behaviour. 50 17.11.2009 Assured Forwarding (AF) PHB RFC 2597 [88] standardizes a group of Assured Forwarding PHB. 12 DS codepoints have been allocated for 4 AF classes (AF1, AF2, AF3, AF4) with 3 drop precedence values (1,2,3) each. An important constraint is given about the packet ordering. Within each class, packets marked with the same AF class (and possibly differing drop precedence) must not be reordered during the forwarding process. This is particularly important for multi-path forwarding (e.g. load balancing). Since PHB are aggregates of IP traffic, such AF classes are also referred to as “ordered aggregate”. The recommended encoding is depicted in Fig. 47 and Table 2. Binary Drop always Class Encoding Precedence ‚0‘ AFcd = XYZab0 Fig. 47 Encoding of Assured Forwarding PHBs Table 2 Assured Forwarding DSCP encoding AF class 1 2 3 4 DP low AF11 001010 AF21 010010 AF31 011010 AF41 100010 DP medium = AF12 = 001100 DP high AF13 = 001110 = AF22 = 010100 AF23 = 010110 = AF32 = 011100 AF33 = 011110 = AF42 = 100100 AF43 = 100110 All standardized PHBs are “per hop behaviour” descriptions, which only define the forwarding behaviour of a single node. No statement can be made about the experienced overall forwarding behaviour by a packet crossing a DS domain. Such chained forwarding behaviours, called “Per Domain Behaviour (PDB)”, are addressed in RFC 3086 [143]. Per domain behaviours are requested to state specific metrics that quantify the treatment, which should be measurable, to be used in SLAs. Interestingly there is currently just one PDB standardized, which does not include any quantifiable metrics. It is called “Lower Effort (LE)” PDB and targets traffic of lower importance than the traditional best effort type. There has been another PDB specification approach around called “’Virtual Wire’ PerDomain Behaviour” (draft-ietf-diffserv-pdb-vw-00). However, it has never reached RFC status and expired in 2001. Lower Effort (LE) PDB – 001000 Traffic that is of lower value than normal best effort traffic is currently marked with the default PHB. However, such low value packets are allowed to be starved out in times of congestion and can serve as lowest priority background traffic to effectively use available link capacity without detrimental interaction with other traffic classes. There is no precise DSCP encoding given, but the mentioned CS1 (001000) in the RFC 3662 [34] is expected to be widely used in LE supporting DiffServ setups. 51 17.11.2009 Table 3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Currently specified PHBs DSCP 000 000 001 000 001 010 001 100 001 110 010 000 010 010 010 100 010 110 011 000 011 010 011 100 011 110 100 000 100 010 100 100 100 110 101 000 101 110 110 000 111 000 PHB Default PHB / CS0 LE / CS1 AF11 AF12 AF13 CS2 AF21 AF22 AF23 CS3 AF31 AF32 AF33 CS4 AF41 AF42 AF43 CS5 EF CS6 CS7 It should be clearly stated, that all PHB and PDB encodings are recommendations and network operators may choose alternative DSCP values for the same behaviour. That is why inter-domain PHB signalling should include the global PHB ID signalling together with locally applied encodings. This approach has been proposed in this work and is documented in chapter 7.3.1. A second general statement shall be given on fragmentation. If IP packets are being fragmented along the forwarding path, there is no explicit rule on how DiffServ marking, metering, scheduling and dropping should react to the set “more fragment” bit. As RFC 2474 states: “The policy to apply to packet fragments is outside the scope of this document”. 4.1.2 IntServ The second fundamental QoS architecture for the Internet is the so called “Integrated Services (IntServ)” architecture [38]. In contrast to DiffServ, IntServ is an approach targeting fine-grained end-to-end QoS (see 3.1.1) with guaranteed absolute traffic parameters. Guarantees are enabled through traffic flow specific reservations and connection admission control in each node along the forwarding path. This requires a connection setup procedure with resource request and resource grant messages and leads to flow-specific reservation states in each relaying node. The association of packets to those reserved flow states is not generally based on a single packet header marking, but rather requires multi-field “classification” in each node. In IPv6, the “Flow Label” header field (see Fig. 4) enables efficient IPv6 flow classification [155]. Reservations are application specific end-to-end flow states, which are signalled by means of a specialised signalling protocol called “Resource ReSerVation Protocol (RSVP)” [39]. Reservations exist in hosts and relaying nodes and are unidirectional. RSVP operates on top of IPv4 or IPv6 and follows standard IP routing. Fixed route selection (source routing) can be enforced by so called explicit route objects, which list the pinned down sequence of hops to be used. Soft state reservations are setup, which automatically time out, if not periodically being refreshed again. Refresh cycles of 30 seconds are common. Reservations are receiver initiated. Multicast is supported and upstream reservations are merged in multicast tree joints towards the sender. 52 17.11.2009 Five fundamental questions are addressed in the IntServ architecture’s signalling: • How to identify a flow (associate packets to reserved flows) ? Æ FilterSpec, • What is the sender’s traffic profile ? Æ TSPEC, • What guaranty profile is requested by the receiver? Æ RSPEC, • Which reservation style is used in the FilterSpec? Æ fixed, shared, wildcard and • Which service models are requested for the network element behaviour Æ “Controlled-Load Network Element Service” or “guaranteed service”. Since the resource reservation is flow-oriented, the flow descriptor (see Fig. 48) becomes a central architectural element. Flow Descriptor FilterSpec ‘filter specification’ rules to associate packets to flows Æ classifier setup FlowSpec TSpec RSpec ‘traffic specification’ describe the expected traffic ‘reserve specification’ describe the flow reservations Æ scheduler setup Flow Fig. 48 QoS RSVP flow descriptor structure The FilterSpec is generally not limited to certain header and/or payload fields of the respective IP packets. However, source IP and source port are basically used for flow classification. Three classes of filters (and hence the classifications) are specified: “fixed filter”, which leads to separate reservations for each sender, “shared filter”, which allows for resource sharing between several named senders and “wildcard filter”, which allows for resource sharing between all senders. The working principle of RSVP is as follows. A traffic source offers an application service to potential clients and describes the offered traffic flow characteristics by means of token bucket parameters in the so called TSpec structure. This parameter set is carried unchanged from one node to the next in RSVP PATH messages. The TSpec eventually reaches the potential clients and those receivers decide individually, whether they start a reservation up-stream. This is signalled using RESV messages with concrete receiver TSpec values, which are now referred to as “RSpec values”. Intermediate nodes will reserve the requested resources according to those RSpec values in their scheduler setup. The TSpec and RSpec parameter encoding is defined in RFC2210 [182]. Two QoS service modes of operation have been standardized, which either imply strict bounds on end-to-end datagram queueing delays as “Guaranteed Quality of Service” [166] or approximate QoS by means of capacity based admission control in order to emulate a lightly loaded network to the respective “Controlled-Load” [183] flows. Fig. 49 and Fig. 50 depict the message flow and node internal block diagram structure. Fig. 49 also points out the merging of RSpecs at multicast tree junctions. Single receiver requests are combined and upstream signalling and resource reservation is based on the combined reservations requests. 53 17.11.2009 Fig. 49 RSVP message flow diagram Fig. 50 RSVP support block diagram – after [39] The policy control and the admission control function are normally harmonized across a IntServ domain. Its central management is achieved through policy servers (also referred to as “Policy Decision Point (PDP)”. The information exchange in query response manner makes use of the “Common Open Policy Service (COPS)” protocol [91]. 4.1.3 IntServ / DiffServ combination The fine-granular end-to-end reservations based on IntServ can not be scaled into a globally extended network. However, two solutions exist to tackle this scalability problem: a) create coarse-grained reservations by applying IntServ to tunnels and b) setup the interworking of IntServ with DiffServ networks in order to gain some QoS advantage from non-IntServ enabled networks. The first approach is commonly applied in MPLS-based networks, which is addressed in chapter 4.3. The second approach is standardized in RFC 2998 “A Framework for 54 17.11.2009 Integrated Services Operation over DiffServ Networks“ [29] and will be briefly outlined below. End-to-end QoS in such a IntServ/DiffServ combined network path is being targeted in a way, where whole DiffServ domains are regarded as virtual links between the IntServ capable routers or hosts. That is, fined grained multi-field classification is performed in the IntServ realm and behaviour aggregation (BA) classification (namely DSCP classification) is applied in the virtual links. By the definition of supported PHB and the rate limited admission control into the DiffServ domain, those virtual links obtain predictable forwarding behaviour. As an example, the RSVP support in Cisco routers can be switched between the standard IntServ operation and the RFC2998 mode of operation (see Fig. 51). The difference lies in the way of classification as well as the low latency queuing in the data plane for the IntServ/DiffServ mode. Fig. 51 Cisco’s two RSVP operation models: IntServ and IntServ/DiffServ [53] 4.1.4 ITU-T IP QoS concept Besides the IETF standardization of IP QoS, ITU-T has also targeted IP QoS in its recommendation Y.1221 [107] and Y.1541 [108]. Generally speaking, it distinguishes three transfer capabilities, which are arrangements of traffic control and congestion control functions. A concept of traffic contracts between users and the network is assumed. Y.1541 adds classes of network QoS with objectives for IP network performance parameters in absolute numbers, which will be the base for the respective contracts. Table 4 lists the six specified classes, their parameter limitations as well as the suggested association to DiffServ PHBs. 55 17.11.2009 Table 4 Excerpt of IP QoS class definitions and performance objectives [108] Two further classes (6 and 7) have already been identified, but with provisional status. That is, eight classes can be expected for ITU-T’s IP QoS recommendations in the future. 4.2 Ethernet QoS The predominantly used layer two networking technology in today’s data networks is Ethernet. There are several flavours standardized in respect to the transmission speed and channel allocation behaviour. Commonly used Ethernet variants are Fast Ethernet (100 Mbps) with emulated shared medium operation, Gigabit Ethernet (1 Gbps) with exclusively uses dedicated transmission lines and 10 Gigabit Ethernet (10 Gbps). 40 and 100 Gigabit Ethernet (40 and 100 Gbps) is currently under specification at the IEEE P802.3ba task force. Those Ethernet types differ substantially in the physical and operational specifications, but they all make use of one and the same frame format (see Fig. 52). Fig. 52 Ethernet frame format Almost all local area networks (LAN) and many metro networks rely on Ethernet for device networking. A special development has been made with Wireless LAN (WLAN) [99], which uses an Ethernet-like frame structure as well. Not only, that the Ethernet is the major LAN and metro technology. The Internet in global scale mostly relies on Ethernet based exchange points for the mass data transmission at 56 17.11.2009 the interconnection of ASes in the core. (About 130.000 Terabyte of data is exchanged per month [8].) However, native Ethernet does not provide any QoS functionality and core interconnects are currently based on that standard Ethernet framing. The “virtual LAN (VLAN)” extension to Ethernet added four additional octets in the frame structure for the purpose of grouping end devices into VLAN-ID groups. That is, an Ethernet node can now belong to one or more virtual LANs and communicate only to peers within the same virtual LAN(s). This VLAN-ID filtering in the relay nodes has been standardized in IEEE Std 802.1Q [98]. This tag field not only includes the respective VLAN-ID field, but also provides three priority bits. The actual usage of those user priority bits has been defined by the IEEE P802.1p task group and is now published in IEEE Std 802.1D [97]. Fig. 53 depicts the resulting tagged Ethernet frame structure. Fig. 53 IEEE 802.1p User Priority marking in 802.1q (VLAN) tagged frames VLAN enabled Ethernets are therefore able to classify frames into eight different classes and provide class-based forwarding in Ethernet interconnection devices – called “switches”. IEEE 802.1D gives detailed specifications on how marked frames are to be enqueued (strict priority queuing only) and forwarded depending on the available number of queues in the switching device. Since traffic type “2” is not used, seven so called traffic types are specified as listed in Table 5. Interestingly, user priority “1” and “2” are handled to be of less priority than best effort (“0”) traffic type. Table 5 Ethernet traffic types [97] Table 6 shows the mapping and merging of traffic types according to the number of traffic separating queues. 57 17.11.2009 Table 6 Mapping of traffic types to available queues [97] Because of the widespread usage of Ethernet in modern businesses and its available traffic separation with strict priority queuing, IP QoS and Ethernet QoS are often combined to achieve maximum QoS support in company internal networks – so called “Intranets”. Chemnitz University for example applies the mapping between Ethernet priorities and IP DSCP markings as given in Table 17. Table 7 Chemnitz University applied Ethernet-priority-to-DSCP mapping A further development in QoS-based Ethernet deployment is currently worked on under the title “Carrier-grade Ethernet”. Several companies are pushing the standardization process with different proposals for frame encapsulation and operational twists. Without going into detail, Fig. 54, Fig. 55 and Fig. 56 show the resulting frame structures. Fig. 54 VLAN Cross Connect / VLAN XC [123] 58 17.11.2009 Fig. 55 Q-in-Q / stacked VLAN / Provider Bridges - IEEE 802.1ad [100] Fig. 56 MAC-in-MAC / Provider Backbone Bridges (PBB) – IEEE 802.1ah [101] The most prominent approach is called “PBB Traffic Engineering (PBB-TE) or Provider Backbone Transport (PBT)”, which however reuses the 802.1ah frame structure. No matter, which approach will prevail, they all provide one or even several times the user priority encoding in their respective frame structure. Furthermore, the encapsulation schemes of Ethernet frames being nested in a second Ethernet frame (MAC-in-MAC approach) allows for tunnelled transport of customer frames. That is, not only standardized Ethernet-QoS forwarding is available, but rather QoS-based tunnelling and QoS-based layer two “routing”. This plethora of choice is of particular interest for this thesis due to the importance of Ethernet for future Internet setups as well as the good match to the crosslayer mappings enabled by the new Inter-AS QoS concept (see chapter 7). QoS-based Ethernet congestion control The last paragraph on Ethernet-QoS will touch on a further QoS-related approach currently under development. Ethernet defines a congestion control mechanism, which allows switching devices to stop the upstream neighbour’s sending by means of so called PAUSE control frames. Such a locally generated frame signals a pause request in milliseconds in times of congestion. Upstream neighbours can in turn be forced to pause 59 17.11.2009 their traffic sources creating a chain of backpressure under high network load situations. However, sending selective PAUSE frames to upstream neighbours with differentiated pause times for the 7 traffic classes appears to be a simple and promising approach in order to confine the congestion effects in lower layers to low priority traffic types. Such “Priority-based Flow Control (PFC)” has already been addressed e.g. in [171] and [126]. IEEE has started the work on PFC within a task group 802.1Qbb by the end of 2007. It extends the original PAUSE based flow control as of IEEE 802.3x [102]. In principle, the send stop request will no longer lead to a full stop, but only to a sending stop for a signalled priority. Given that Ethernet traffic is sorted into eight lanes (priorities), Fig. 57 depicts the concept’s idea. Fig. 57 Priority Flow Control [56] The simulator OMNET has been extended to include priority-based PAUSE functionality. However, the research is not yet completed and will not be published in this thesis. Generally, backpressure flow control schemes do raise the hazard of congestion spreading, if several flows to different output directions are sent by a (intermediate) source, which only partially add to a single output congestion. The resulting PAUSE backpressure, however, does stop all flows undifferentiated, which artificially spreads the congestion to practically un-congested forwarding paths as well. Fig. 58 Congestion spreading [103] 60 17.11.2009 The current 802.1Qbb approach therefore includes a congestion notification mechanism as described in 802.1Qau [104]. Due to this combined approach, the right source will eventually receive the congestion notice and slow down selectively. A more simple mechanism is being investigated, which sends the priority-based pause together with a destination MAC address. That is, the sending intermediate node stops only those prioritized frames, which are destined to that MAC address, which has been identified as the highest load on the congested output queue. Research results will be published outside of this thesis. Generally speaking, the VLAN-based QoS support within Ethernet and the fast growing deployment of Ethernet technology in all networking areas underlines the importance of a close cross-layer coordination of traffic separation and its marking with the ubiquitous IP networking and the proposed CoS concept. 4.3 MPLS QoS “Multi-Protocol Label Switching (MPLS)” is the widely accepted tunnelling mechanism in IP networks, which is used to introduce some connection-oriented forwarding behaviour in the datagram network. Although MPLS is capable to encapsulate any networking protocol data units, it is currently exclusively used for IP packet forwarding. MPLS introduces in most cases an additional header structure, called “shim header”, which is effectively used to encapsulate the packet data behind it. The second important characteristic is a new 20 bit addressing scheme, called “label”, which has interconnection local significance. Labels are swapped between input and output labels during the MPLS forwarding procedure using a “label information base (LIB)”. The strict swapping operation together with a LIB establishment procedure creates a forwarding chain with fixed relaying nodes – the so called “Label Switched Path (LSP)”. This concept of fixed length local addresses with chained relay is not new, but can be found e.g. in ATM and Frame Relay networks as well. Although MPLS can be used on top of all underlying packet transport capable networking technologies, separate mapping procedures exist for ATM and FR due to this addressing similarity. The pre-established LSPs are excellent tunnels for traffic engineering approaches. MPLS is therefore widely used to perform traffic engineering (flow steering and fast restoration) besides the mandatory available shortest path first IP routing. Fig. 59 depicts the shim header (label stack entry format [160]) and its usage between e.g. an Ethernet transport in layer two and IP payload in layer three. The figure also implies the label stacking option, which is explicitly shown in Fig. 60. Such hierarchy of tunnels is a new and powerful option for scalable, fast restorable and transparent transport even of already tunnelled customer traffic. This capability is of particular interest, since customer traffic is most elegantly transported in tunnels and might be forwarded in nested tunnels in carrier’s carrier scenarios. The latter implies inter-carrier (that normally means inter-AS) MPLS LSP scenarios, which is not commonly used nowadays, but are envisioned for the near future. In terms of QoS support, the shim header “Traffic Class (TC)” field is the obvious class encoding. It is a three bit field, which allows for eight classes of tunnelled traffic. Historically, this field used to be named “EXP”, since the bits’ usage was for experimental purposes only. However, RFC 3270 [75] explicitly targets the MPLS support for Differentiated Services and introduces the so called “EXP-Inferred-PSC LSPs (E-LSP)”. The eight available behaviour aggregates (BAs) recall DiffServ behaviour aggregates with associated per hop treatments. However, the RFC deliberately delegates the mapping between DSCP encoding and E-LSP encoding to some unspecified signalling or preconfigured setup. 61 17.11.2009 Fig. 59 MPLS shim header structure and hierarchy usage Fig. 60 MPLS Label stack structure Because of the QoS dedication of the EXP bits, a renaming of the field towards “TC” has occurred just recently. RFC 5462 [9] updates the respective DiffServ related MPLS RFCs, but keeps the “E-LSP” abbreviation. The second mapping option for IP traffic classes onto MPLS tunnels is given in so called “Label-Only-Inferred-PSC LSPs (L-LSP)”. Here, each traffic class is associated to its separate tunnel, represented by a different entry label. This extents the limitation of eight classes up to – theoretically – a million supportable classes. This L-LSP QoS support picks up the generally available class separation available in all “tunnel or virtual channel based” transport technologies where each class is mapped into a separate encapsulation, which might also involve separate forwarding paths per class. The EXP/TC-inferred QoS forwarding behaviour combined with Label-inferred QoS tunnelling combined with LSP hierarchies enables numerous quality of service support options. This plethora of choice is of particular interest for this thesis due to the importance of MPLS for future Internet setups as well as the good match to the cross-layer mappings enabled by the new Inter-AS QoS concept (see chapter 7). A crucial component of the MPLS QoS support is the control part for path setup (label distribution) and TC marking distribution. The MPLS working group developed two major signalling branches for QoS-related traffic engineering. One branch extended a specifically developed “Label Distribution Protocol (LDP)” [10] into RFC 3212 “Constraint-based 62 17.11.2009 Routed LDP (CR-LDP)” [122]. The second branch reused the existing IntServ signalling protocol, RSVP and specified the traffic engineering extensions to it. The resulting “Resource Reservation Protocol-Traffic Engineering (RSVP-TE)” [13] defines several protocol objects, which are most importantly used to convey the label information. For this thesis, it is important to mention, that RSVP-TE does not directly signal EXP/TC markings during the LSP setup procedure. A more generalized “colour” concept is used, which is implemented as resource class attribute [14]. This new attribute becomes part of a path description and can for instance be used for constraint-based routing. In 2003, the MPLS working group has come to the decision to favour RSVP-TE for MPLS LSP path setup and to “refrain from entertaining work that intends to progress RFC 3212 or related RFCs beyond proposed standard” [11]. Two major improvements to MPLS have been tackled during the past few years. InterDomain MPLS and GMPLS Traffic Engineering are of importance to the thesis’ class of service concept. Inter-Domain MPLS RFC 5151 [73] is the RSVP-TE extension to a defined Inter-domain MPLS TE framework [74]. Taking the operators’ independence in terms of MPLS support and configuration into account, three methods of LSP establishment have been identified as depicted in Fig. 61. Contiguous LSPs are setup following the procedures from RFC 3209 [13] and 3473 [26]. Nested LSPs follow RFC 4206 [128] and stitched LSPs are described in RFC 5150 [15]. Interestingly, those setup procedures do not specify, how inter-domain EXP/TC encoding information is to be exchanged. Coloured routing is again proclaimed. Fig. 61 MPLS LSP signalling: contiguous, nested, stitched 63 17.11.2009 Generalized Multi-Protocol Label Switching (GMPLS) GMPLS is the logical abstraction of the MPLS approach towards physical representations of “labels”. RFC 3471 [25] describes the functional signalling part of the extended control plane. Targeted physical label switching representations are: time-division (e.g., Synchronous Optical Network and Synchronous Digital Hierarchy, SONET/SDH), wavelength (optical lambdas) and spatial switching (e.g., incoming port or fibre to outgoing port or fibre) – see Fig. 62. Fig. 62 GMPLS label representations Although such generalized labels are not markable with traffic class bits, the virtual channel concept, which those time division, lambda or spatial encapsulations represent, is well capable of QoS based tunnelling support – even more so with the decoupled and hugely hierarchical generalized encapsulation concept (Fig. 63). Fig. 63 GMPLS LSP hierarchy 64 17.11.2009 The class of service concept of this thesis therefore encompasses virtual channel based traffic class separations as layer one QoS approach and includes GMPLS based tunnelling on purpose. 4.4 QoS in access networks Speaking about wired and wireless access networks in a thesis about BGP signalled class of service support is rectified by the increasingly used intra-AS QoS support up to the provider-customer edge of a network. Fast increasing access rates in multi-megabit scale and fast growing acceptance of heavy data load application services by the Internet community causes not only large volumes of traffic, but also a sensitive mixture of loss and time critical data streams in that volume. Standardized access technologies all provide means for QoS-based traffic separation and Europe seems to take a leading role in actually applying them on customer lines and channels. The following section briefly lists some typical QoS class sets, which are often even standardized with fixed parameter limits for loss, delay and delay variation. Wireless access technologies – UMTS The currently widely available wireless access technology, “Universal Mobile Telecommunications System (UMTS)” is a third generation (3G) mobile telecommunications technology. It offers mobile data transfer services, which enable mobile devices to participate in IP communication networks. The UMTS standard documents (e.g. 3GPP TS 23.107 [1]) a detailed quality of service concept and architecture, which is based on four so called UMTS bearer services. Table 8 lists the resulting QoS classes and their respective characteristics and applications. The conversational, streaming, interactive and background class have detailed specifications for requirement and service parameters (see Table 9). Table 8 UMTS QoS classes [1] 65 17.11.2009 Table 9 UMTS Bearer Service Attributes [1] Customer traffic entering through prioritised UMTS bearers will certainly be prioritized in DiffServ capable provider’s backhaul network and should be passed on in e.g. a class set of four across AS boundaries towards the destination network. Wireless access technologies – LTE The highest demand for QoS concepts in the core of the Internet builds up, if the pace of speed increase in the core is slower than the increase pace in the access. This is particularly true for the 3GPP “Long Term Evolution (LTE)” standard, which is currently developed as a set of enhancements to UMTS. The official target of achievable customer data rates is 100 Mbps downlink and 50 Mbps uplink speed [3]. This rapid speed increase becomes even worse for the operator’s core network, if LTE-Advanced gets specified, which promises up to 1Gbps access rate for a slow mobility usage case [4]. LTE addresses the different needs of such potentially high speed interactive services with the definition of nine QoS classes as shown in Table 10. The repetition of classes is due to the distinction between traffic of premium versus non privileged subscribers. 66 17.11.2009 Table 10 LTE QoS class attributes [2] Wireless access technologies – WiMAX The “Worldwide Interoperability for Microwave Access (WiMAX)” wireless telecommunication technology has been standardized under IEEE 802.16 [105] and is regarded as alternative to wired broadband access solutions. WiMAX defines several QoS setups (provisioned, admitted, active), which are unidirectional, flow-based and either dynamically signalled or statically configured. Generally, the supported so called “service flows” are parameter signalled with a detailed set of QoS parameters associated to each active service flow. Large scale WiMAX deployments can, however, not guarantee the same service flow setups being consistently available throughout the network at all time. The solution are so called “global service classes”. A vital part of the WiMAX QoS concept is the naming of service flows, which are then associated with parameters sets. That is, global service classes should carry standardized names, which in this case are standardized by naming rules. The resulting names can therefore be parsed and reveal pointers into standardized tables of fixed options or parameters settings. In a way, those parameter tables and their combined referencing constitute the provided class sets. This spans a large class set range, which is in no way comparable to the aforementioned simple IP or Ethernet class sets. Simple flow prioritization is not provided in WiMAX except for the case, when two flows have identical parameter sets but different priorities, a so called “traffic priority parameter” decides upon the precedence. The interworking of WiMAX with its detailed QoS approach with simple IP or Ethernet classes of service is not trivial. WiMAX, however, provides a means to associate e.g. Ethernet priorities with WiMAX service flows by means of classifier rules. Among others, the Ethernet user priority or the IP DSCP value can be added as classification criteria to a specific flow. Wired access technologies – DSL / ATM “Digital Subscriber Line (DSL)” is a family of wired access technologies, which all use the traditional two-wire access lines for higher speed packetized data access services. The most prominent family members are: “Asymmetric Digital Subscriber Line (ADSL)” [112][113], “ADSL2” [114][115], “Very-high-bitrate DSL (VDSL)” [116] and “VDSL2” [117]. The digital subscriber line standards do not address traffic separation and prioritization, but provide bit pipes for higher layer protocols. DSL uses Asynchronous Transfer Mode (ATM) 67 17.11.2009 as framing and networking technology for packetized data transport services. The only natively available prioritization is found within the “High-Level Data Link Control (HDLC)” encapsulated transport of overhead messages. In order to understand the traffic separation in ATM-based DSL access, Fig. 64 depicts the structure of the fixed length ATM packets, called “cells”. For cell prioritization, there is only a single header bit available, which indicates a higher cell less priority if set. It is a sort of punishment marking for cells and can not be regarded as traffic class marking as seen with MPLS TC, IP DSCP or Ethernet priority. However, ATM cells carry local scope addresses (VPI/VCI), which are swapped at the forwarding switches and need to be setup during a connection setup phase. That is, the set up chain of VPI/VCI forwarding tables again creates a tunnelling scheme for the ATM cell payload by means of the resulting virtual channels. Several virtual channels can be established across a DSL access line and might be used for traffic class separations. Especially in Europe, it is common for DSL providers to configure e.g. 4 or 6 ATM VCs onto DSL lines for QoS purposes. Fig. 64 ATM cell structure Traditionally, the ATM transport not only included the actual access line transport, but also the aggregation network at the providers edge. This trend has changed towards an Ethernet-based aggregation network as specified in TR-101 [68]. Furthermore, TR-101 also includes interface options, where direct usage of Ethernet instead of ATM on the DSL line is recommended. Together with this Ethernet transition, the Ethernet priority support for QoS has explicitly been standardized. It is included as a “MAY” option. Given the above configurations and trends, it is expected to see separated traffics of four to eight classes within DSL access networks. Wired access technologies – GPON Due to the continuing demand for higher speed access rates on one hand and the high transmission capacity of fibre based technologies on the other hand, network operators push fibre as close to the customer as possible. This will include aggregation points in street cabinets or even fibre in customers’ houses. The predominantly used technology today for this purpose is “Gigabit-capable passive optical networks (GPON)” [111]. Within GPON, four service-bearing transmission containers are provided, which represent four differentiated classes of service. GPON specifically offers an Ethernet data service (see Fig. 65), which includes TR-101 [68] functionality. Furthermore, GPON is often combined with VDSL2, which again can lead to Ethernet transport with optional user priority 68 17.11.2009 encoding. Due to the underlying four bearers, this Ethernet transport can additionally be stretched into four parallel Ethernet transport channels. It is expected to see separated traffics of four to eight classes within GPON based access networks. Fig. 65 Functional layering structure for the Ethernet data service [111] 4.5 Summary of expected Class of Service support Given the above listed variants of QoS support in different networking technologies, the following table summarizes the expected number of separated traffic classes. 69 17.11.2009 Table 11 Overview of available layer 2 and 3 quality of service classes ¹ ATM has not been extensively described in this thesis, due to its declining usage. However it needs to be mentioned that ATM had long before one of the most detailed and researched into QoS concept with parameter negotiation, detailed measurement, traffic conditioning, admission control and management functions. Many recent developments – especially in the field of MPLS – seem to have adopted and learned from ATM. The mentioned QoS categories relate to classes of service (CBR, VBR, ABR and UBR as defined by the ATM-Forum). 70 17.11.2009 5 State of the art AS interconnection The Internet is a patchwork of interconnected autonomous systems, which exchange IP traffic. By means of an Exterior Gateway Routing protocol, BGP version 4, each AS announces the IP networks, represented by IP prefixes, which are reachable through that AS. Interconnected autonomous systems establish so called BGP peering sessions for this reachability information exchange. The “Network Layer Reachability Information (NLRI)” is advertised in BGP UPDATE messages together with associated path attributes of the announced routes. Each AS in turn processes the route advertisements and determines, which routes it takes in its own routing table as well as which routes it relays to other external peers. Those policy decisions are taken based on the interconnection topology of that AS, as well as the operator’s policy rules, which process the BGP path attributes. Fig. 66 depicts the basic options for physical AS interconnection, being direct point-to-point links and Ethernet based traffic hubs, the Internet Exchange Points. Fig. 66 AS interconnection options Whether or not an AS decides to establish a direct interconnection or an IXP interconnection depends mainly on the link cost as well as on the business gain through interconnections at central exchange points. The interconnection link technology is increasingly based on Ethernet. It is mandatory for IXP access due to their Ethernet switch based platform, but also has become popular for direct lines. An AS is usually solely responsible for the link towards its IXP switch port or towards the interconnected AS. In the latter case, either the two parties share the cost or the smaller one takes the lead in order to get connected to the bigger one. This can either be fixed line cost or part of the traffic cost in non-zero settlements. The link can be owned or rented. The link speed is normally in the range of 100 Mbps up to 10 Gbps (sometimes bundles of n x 10Gbps) and therefore mostly fibre based. However, dark fibres used with own transmission equipment are more common in the 71 17.11.2009 direct interconnection case. Otherwise, rented virtual channels are used by the vast majority. If the interconnected parties are not co-located in the same building, where direct Ethernet interconnect is possible, Ethernet over carrier (wavelength, MPLS, SDH) seems to be the most often used linking technology. Whether or not a point-to-point interconnection is chosen depends on several factors. Driving forces are geographic location, customer base and customer traffic destinations as well as zero or non-zero settlement based interconnection. Advantages of point-to-point interconnections (private interconnection): - One or only a few peering ASes are geographically close and terminate the majority of traffic loads of the own customer base. So it is worthwhile to rent or even build the direct link, - Link speed requirements to only a few peering ASes exceed the normal IXP interconnection speed, - Private mutual agreements (e.g. QoS requirements, MTU sizes, and other technical twists) should neither be made public nor can be provided by public platforms, - High security and confidentiality requirements request for a private interconnection, - Point-to-Point interconnection is setup to important partner ASes for backup purposes and - Accounting for paid traffic exchange is easy with the link interface counters. Advantages of IXP based interconnections (public interconnection): - The high link (rent) cost are prohibitive for the setup of many single link interconnections to other ASes for large scale connectivity, if only a single link to a major exchange point serves the job (e.g. single line from Africa to a large European exchange point), - The customers’ traffic demand is distributed and most of the high load terminating ASes are present a the chosen IXP, - The reception of several paths towards the same network increases routing robustness, if one interconnected AS fails to reach that destination, - Interconnection at IXPs is normally a zero settlement, - Direct interconnection across a switch towards many other networks reduces latency, which would otherwise build up with up and down transitions through higher tier ASes, - accounting based on source MAC address filtering is possible and - Route servers at IXPs reduce the number of BGP sessions to a single one. The “Router Server (RS)” argument needs to be briefly explained. Since IXPs allow for the interconnection of several hundred ASes at one place, the BGP session load of the single interconnection AS border router is enormous. However, it can be cut down to just one BGP session, if the IXP offers a central route server for its customers. Such a route server acts similar to a route reflector, but is peered with using eBGP with all resulting in and out filter policies in the peering ASes. Route announcements and receptions are therefore exchanged with the central route server only. The actual traffic exchange, however, takes place directly between the interconnected ASes. In numbers, many ASes are interconnected via public exchanges, but most of the Internet traffic load is carried over point-to-point interconnections. Large companies such as Google and Amazon are present at many Internet exchanges for close user proximity. Transit providers with global networks are also commonly present at IXPs, which use the platform as customer aggregation points. 72 17.11.2009 Quality of service support is currently almost exclusively provided through private interconnections. Public exchanges, however, start to support Ethernet based traffic separation [127]. Fig. 67 exemplarily depicts the internal topology of the German Internet exchange, DE-CIX, in Frankfurt, which is distributed across four locations in Frankfurt and has proven to support VLAN user priority marking upon request. Fig. 67 DE-CIX topology 2009 [61] The native interconnection between two ASes does not reveal the associated payment structure – settlement – involved in the interconnection. Payment based wordings are: • “transit” / “non-zero settlement” for paid traffic exchange (see 5.1) and • “peering” / “zero settlement” for free of charge traffic exchange (see 5.2). Both words “transit” and “peering” are, however, also used in purely technical terms. That is, “transit traffic” / “transit network” could simply describe the forwarding architecture across a different network towards the destination, regardless of any payments. The same applies to the word “peering”, which can be used to describe any interconnection between adjacent networks (peers). Moreover, the BGP protocol always speaks about peers and peering sessions no matter what settlement is associated with that interconnection. The level of interconnection of a single AS determines its rank in a global AS hierarchy, the so called “tier structure”. Lower level tiers pay for their traffic exchange with higher level tiers (“they buy transit”), where as same level tiers might “peer” with each other for mutual benefit for free. 73 17.11.2009 In theory, the highest tier group, “tier 1”, applies to ASes, which only interconnect via free of charge peerings with other tier 1 ASes and thereby gain full connectivity to all globally available Internet destinations. Due to their fully meshed interconnection, the routing table of those AS border routers does not contain a default route entry, but holds specific routing entries to all routable prefixes. Routers with such a full routing table make up the so called “Default Free Zone (DFZ)”. At the lower end, “tier 3” ASes exchange all their external traffic through transit interconnections with higher tiers. The vast majority of ASes are “tier 2” ASes, which hold some free of charge peerings to other tier 2 ASes and buy transit for the remaining global connectivity. Deutsche Telekom is such a tier 2 example, which runs a rather widely interconnected network with just one upstream transit (AS1239). Fig. 68 depicts the described Internet hierarchy. Fig. 68 Internet hierarchy 5.1 IP transit The easiest way to achieve global connectivity for its customer base is through buying of transit from the global players, either largely interconnected tier 2 or from one of the roughly 10 core ASes of type tier 1. The selling party is the provider and the buying party the customer. There are many factors, which influence this interconnection decision – cost and reliability being the major drivers. Non-zero settlements are not published and the two parties normally sign non-disclosure agreements about the interconnection specifics and rates. Two major settlement philosophies are common, being classical IP interconnection with volume based charges and upcoming Voice interconnections with time based charges. The latter is a new development, which is coming up through the current transition trend from time based circuit-switched voice interconnections towards “Voice over IP (VoIP)” type voice interconnections. It is assumed to be currently common to have two separate 74 17.11.2009 interconnections for the Internet and the voice services, but the drive towards a purely IP interconnected service platform might lead to a volume based voice interconnect as well. The easiest volume interconnection charging model is the “difference in volume” charge. Incoming traffic volume is subtracted from the outgoing traffic volume at the interface of the upper tier. Since large traffic volumes will stream down from the upper peer to the lower one, the lower gets the positive sign and pays for the difference. This in turn reveals that traffic from the lower tier towards the higher tier limits its revenue. A second charging model is a base and offset model, which includes a monthly fixed transit cost amount for a ”Committed Information Rate (CIR)” and a volume based excess charge for unexpected excess traffic. In densely interconnected regions, such as the US, Europe, Japan etc., a plethora of interconnection choices exist, which enable transit competition due to the cheaply available interconnection links. The trade-off between higher transit cost and alternative interconnection link expenses make the Internet more expensive in remote or underdeveloped regions of the world. Operators tend to establish several transit interconnections for backup and strategic reasons. Contracts are constantly renegotiated, which leads to a ever changing AS interconnection topology. Separate companies have specialized in data mining the AS tree in order to provide consultancy to transit selling parties, monitor the interconnection changes for debugging or trend analysis, document the topology for fault tracking and usage statistics etc. Even some vague conclusions can be drawn from the observed connectivity changes and interconnection path announcements, which estimate customer churn and undisclosed price movements. The clue to the selection of possibly multiple transit paths as well as the base for the mentioned analysis work is the policy based routing protocol BGP. Route attributes are used for mutual or single sided path selection by filtering, e.g. a cheap unreliable transit for the normal traffic load and a high priced transit for some important customers or services. This is often prefix oriented and used in a multi-homing scenario for transit cost optimization. BGP as a path vector protocol creates a record of crossed ASes in the route advertisement process, which provides partial knowledge about the AS graph structure. The introduction of the currently missing quality of service support on transit interconnection raises several issues, which need to be observed and addressed in future analysis. Expected issues are: - transit partner selection based on available QoS support, - transit partner selection and or negotiation about QoS class granularity, marking/remarking and traffic handling (shaping, scheduling, dropping etc.), - possible QoS-related charging models instead of the single class model today and - extended reliability discussion on whether the QoS support is only interconnection local or extends across further interconnections. The thesis’ class of service concept provides a simple marking and rate limitation mechanism, which addresses some basic QoS improvements in a transitive inter-domain manner. 5.2 IP peering The interconnection of ASes for mutual benefit for no cost (zero settlement) is called “peering” and happens largely at internet exchange points. Partnering ASes agree to setup interconnection links with BGP peering sessions to exchange reachability information. If the customer base of each party often exchanges traffic with the opponent customer base, this short-cut interconnection saves both sides transit costs via higher level tiers as well as transfer delay time due to the shortened forwarding path. 75 17.11.2009 Peering requests can be raised informally and the resulting agreements can be rather loose. That is, no complex service level agreement is mandatory and as long as both partners are content, even a handshake can be sufficient. Emails with contact details are exchanged and the IP addresses and AS numbers of the peering equipment are sufficient knowledge. Such a simple setup leaves the questions about interconnection quality, fault handling, service reliability etc. open. If either party becomes unhappy with the way of operation, “depeering” can occur. Generally, peering requests will be refused in the following cases: • requests from own customers, • requests from potential customers, • requests from other peer’s customers, • requests from providers with bad track records, • requests from providers with low infrastructure investment policy and • requests from providers, where the mutual benefit is questioned. The peering / depeering business is a playground for multi-dimensional optimization strategies and resulting business models. Social networking and market influence by route manipulations have strong impact in the request and grant procedure. In the transit case, attracting customers of competitive transit providers by financial or technical incentives to use the own transit path is common practice. However, attracting customers of competitive transit providers to use the own peering agreement is considered bad practice. The peering policies of network providers are often very short and publicly available. They are mainly published to attract potential partners. Existing providers and new players use the information to optimize their peering relationships and to find the right geographical and technical base for the interconnection. Examples of publicly available peering policies are e.g. [162] and [174]. The most comprehensive resource for publicly available provider information, their geographical presence at exchange points, their public policy, their interconnection types and speeds etc. can be found at the so called “PeeringDB” [151]. An interesting case study is the “aggressive, user-driven rollout” peering strategy of the company Google [58]. In order to cut down delay times for good user experience of Google services, the company is welcoming direct peering interconnections with customer networks at many places around the world. Furthermore, the company is a leading force for the ongoing IPv6 transition and mitigates the currently missing global IPv6 transit availability by means of direct IPv6 peering. Google has also made a scaleable “signup” procedure proposal, which is of particular interest for this thesis class of service support signalling approach. Google is running a large IPv6 connectivity test with so called “trusted tester” peering partners. However, the company can not possibly sign up testing agreements with each interested peering AS, considering the number of possibly 55000 assigned and about 30000 actively participating ASes in the Internet [93]. Therefore, a proposal was made for sending a BGP community value of “15169:6666” in the advertisement of a specific IPv4 prefix to “sign up” for the trusted tester program [58]. 76 17.11.2009 5.3 Internet Routing Registry - IRR The “Internet Routing Registry (IRR)” is a database system to store and globally access routing policy information in a structured, humanly readable and automatically usable way. 32 IRRs are currently listed [137], which differ in administration and database quality (performance, consistency and availability of database tools). In theory, the distributed registry information of the different regions in the world would comprise of one consistent information base about all AS routing policies. In practise, operators mainly rely on the information stored in the RIPE database, which is known to be the most advanced and consistently administered database. The representation of stored IP routing policies has been standardized in RIPE document “ripe-81” [159], which has been republished as RFC 1786 [21]. Many registries make use of the “Routing Policy Specification Language (RPSL)” [6] or “Routing Policy Specification Language next generation (RPSLng)” [35]. The biggest advantage of this specification language is its usage for automated router configuration generation. Network operators specify their AS number and assigned prefixes as well as their import and export policies in the registry. Direct interconnection partners can draw their generated filter rules based on that structured information and ASes further away of the originating AS can verify the incoming route information for plausibility and sanity. If all operators would rely on this registry mechanism, malicious route advertisements and the spreading of such fake information could largely filtered out. As the “Youtube outage” [41] in February 2008 showed, however, many operators are still not making use of this valuable Internet routing registry in their update sanity checks. 77 17.11.2009 6 Related work A number of QoS improvement approaches have been proposed before, but none has been standardized and actually used for QoS support in the public inter-domain case. Private inter-domain QoS setups do exist, but are not made public. In such cases, the QoS configurations and parameter settings are agreed on offline and documented in service level agreements. Three major characteristics have been identified about the past QoS improvement approaches: 1. Quality of Service is targeted end-to-end and includes the inter-domain interconnections for the case of sending and receiving parties being in separate ASes, 2. Quality of Service is targeted in a guaranteed quality fashion, which requires detailed parameter signalling, QoS enforcement functions, QoS parameter measurements, violation detection and fining and 3. Quality of Service is targeted in a homogeneous fashion, that is, all participating ASes need to support the same QoS setup. This includes the common signalling protocol, common setup of class sets with the respective classification, scheduling and dropping functionality. Some important examples of those past QoS improvement approaches are addressed below. France Telecom and Alcatel submitted an Internet draft, draft-jacquenet-bgp-qos-00 [59], in 2004, which introduced the so called “QOS_NLRI” attribute in BGP. It is used for propagating QoS-related information associated to the NLRI (Network Layer Reachability Information) information conveyed in a BGP UPDATE message. Single so called "QoS routes" are signalled, which fulfil certain QoS requirements. Several information types are defined for the attribute, which concentrate on rate and delay type parameters. This approach therefore addresses QoS guarantees for selected end-to-end routes. QoS parameters, such as packet rates, loss rates, one-way delays and inter-packet delay variation are signalled in absolute numbers and might need to be re-signalled, if end-toend requirements or network load conditions require adoption. Parameter signalling, however, introduces two major drawbacks in global scale operation. The first is the resource accounting, which actually registers used and available capacity shares together with triggered signalling and admission control. This effort is justifiable for just some QoS routes, but is unmanageable if most of the routes have associated parameter sets. The second drawback is protocol stability. BGP has been designed to trade-off routing dynamic for routing stability. Frequent parameter signalling is therefore counterproductive. In support of this argument, the BGP route flap dampening (RFC 2439 [175]) behaviour should be briefly described. If enabled, BGP will suppress frequent route advertisements based on a penalty scheme with hysteresis thresholds. In this dampening concept, each route flap (withdraw/announcement pair) accounts for a penalty of 1000 and each attribute change for a penalty of 500. Penalties accumulate for each advertisement event and decays by 50% in a configurable time period. Penalty counts above a suppress limit prevents the advertisement relay of a given prefix and its attributes. Penalty counts below a reuse limit switch to normal router advertisement operation. Fig. 69 depicts the dampening characteristic. 78 17.11.2009 Fig. 69 Route Flap Dampening [167] The work on this draft is embedded within a European Union funded project called “Management of End-to-end Quality of Service Across the Internet at Large (MESCAL)” [139]. This extensive project work on inter-domain QoS support goes far beyond the limited class of service approach of this thesis. It is based on a so called “cascaded signalling approach”, which assumes AS-internal DiffServ QoS class support – referred to as “local-QoS-class (l-QC)” – and inter-AS QoS class support with resulting “extendedQoS-classes (e-QC)”. Fig. 70 depicts the cascaded approach with the l-QC and e-QC after interconnection. The MESCAL project targets end-to-end QoS guarantees and therefore signals parameters between ASes. Such parameters are part of the l-QC QoS specification and the e-QCs are constructed recursively from them following parameter specific combination rules. Fig. 70 MESCAL - Cascaded Approach [139] As mentioned before, such QoS guarantees with parameters signalling are out of focus for this thesis’ class of service concept. However, the MESACAL project also includes an option - called “loose guarantees solution” - that renounces end-to-end QoS guarantees. 79 17.11.2009 However, it still performs mutual negotiations on performance parameters and bandwidth requirements and requires either globally understood class indicators (e.g. DSCPs) or SLA-based local indication agreements. Dynamic signalling of (re-)marking information and marking preservation is not provided. France Telecom started a second Internet draft, draft-boucadair-qos-bgp-spec-01 [37], in 2005. It is based on the specified QOS_NLRI attribute and introduces some modifications to it. The notion of AS-local and extended QoS classes is used, which effectively describes the local set of QoS performance parameters or their cross-domain combined result. Two groups of QoS delivery services are distinguished, where the second group concentrates on ID associated QoS parameter propagation between adjacent peers. The first group is of more interest for this thesis’ work, since it concentrates on the "identifier propagation" such as the DSCP value for example. However, this signalling is specified for the information exchange between adjacent peers only and assumes the existence of extended QoS classes and offline traffic engineering functions. The limitations of the inherited QOS_NLRI attribute remain. The co-workers at France Telecom, Christian Jacquenet and Mohamed Boucadair, hold large contributions to the mentioned work and explain provisioning techniques for the currently topical IP/MPLS networking case (see [120]). However, the exchange of class identification (marking) is also not addressed. Another approach has been raised by a group of researchers at Johns Hopkins University. It is described in [24]. The Internet draft associates a list of QoS metrics with each prefix by extending the existing AS_PATH attribute format. Hop-by-hop metric accumulation is performed as the AS_PATH gets extended in relaying ASes. Metrics are generically specified as a list of TLV-style attribute elements. The metrics such as bandwidth and delay are exemplarily mentioned in the draft. One contribution specialized in the signalling of Type Of Service (TOS) values which are in turn directly mapped to DSCP values in section 3.2 of the draft [185]. The TOS value is signalled within an Extended Community Attribute and, if it is understood correctly, will be applied to a certain route. An additional value field is used to identify, which routes belong to which signalled TOS community. Who advertises such attributes and whether they are of transitive or non-transitive type remains unspecified. Advertising multiple paths (and associated metrics) for one prefix is addressed and a new path selection algorithm had been proposed. The concept would therefore support the packet classification and classbased route selection. The draft expired in December 2006. The most comprehensive analysis (although not an IETF draft) is given in [7]. This "Interprovider Quality of Service" white paper examines the inter-domain QoS requirements and derives a comprehensive approach for the introduction of at least one QoS class with guaranteed delay parameters. The implementation aspects of metering, monitoring, parameter feedback and impairment allocations are all considered in the white paper. However, QoS guarantees and frequent parameter signalling have been identified as critical characteristics for the inter-domain global scale routing system and the BGP protocol stability. It is valuable work for the fine-grained QoS setup for an arguably large number of selected end-to-end routes. A general applicability of the concept for nearly all Internet routes is not feasible. A more economically inclined approach has been published during the IEEE ICC2002 conference under the title “Enabling dynamic market-managed QoS interconnection in the next generation internet by a modified BGP mechanism” [94]. It relates to this thesis for two reasons. Firstly, the intention of QoS support at inter-domain interconnections and 80 17.11.2009 secondly the usage of BGP for signalling. It is again based on the QOS_NLRI as described above, but includes price information as well. Although the economical characteristics might become the clinching argument for or against any QoS-based interconnection, it is not expected to undisclose such information in publicly seen routing protocol messages. Furthermore, the limitations of QoS guarantees and the associated parameter signalling have already been described above. A further concept proposes BGP-based QoS service capability signalling for groups of NLRI. This Internet draft was launched in October 2006 under the name “draft-djernaessimple-context-update-00” [68]. The draft does not specify the precise signalling encoding of QoS class markings and parameter signalling, but rather retreats to a more general QOS Service signalling, which might optionally involve interconnection local marking signalling. The fundamental idea in this signalling concept is the grouping of reachability information (prefixes) in QoS Service address families. BGP always signals in the UPDATE messages, which address family the contained network layer reachability information belongs to. The respective concept of “address family identifier (AFI)” and “subsequence address family identifier (SAFI)” has been defined in RFC 4760 [19]. The draft proposes to define a new AFI/SAFI for QoS Service signalling and all NLRI contained in such an UPDATE message belong to that QoS Service context. This approach is attractive for two reasons. Firstly, the signalling overhead scales well due to the grouping effect of possibly numerous prefixes under a common AFI/SAFI based signalling. Secondly, QoS related UPDATE information can selectively be signalled for the separate AFI/SAFI. The same is true for selective route refreshes and soft-reconfigurations. However, this thesis aims for a global scale class of service interconnection support for possibly all Internet routes. Using the capability signalling concept, this would result in double signalling of all prefixes, one time for traditional reachability and a second time for the QoS service context association. Further observations on existing QoS signalling approaches are summarized in RFC 4094 [131] – a review analysis produced by the “Next Steps in Signaling (nsis)” working group. Half of the document is dedicated to RSVP analysis (see 4.1.2), being the most important QoS reservation protocol in today’s networks. Since RSVP is an end-to-end QoS signalling protocol, which is also augmented to establish MPLS traffic engineering tunnels, it has high potential to setup (possibly tunnelled) inter-domain QoS paths. RFC 2814 [184] and RFC 2815 [163] also address the mapping issue of Integrated Services QoS into Ethernet User Priorities. However, no fixed mapping is – and can be – defined, but rather a request and response negotiation between neighbouring nodes about locally available Ethernet resources is suggested. The dissemination of DSCP values has also been standardized for RSVP in RFC 2996 [27]. The usage of RSVP, however, raises concerns about scalability (due to the flow-based end-to-end nature and soft-state signalling behaviour) and lately about direct user <-> provider equipment interaction (see Fig. 73). RFC 4094 analyses several intra-domain signalling protocols, which similarly to RSVP allow for resource reservations for traffic flows. The protocols are Tenet [18], ST-II [65], YESSIR (YEt another Sender Session Internet Reservations) [148], Boomerang [76] and INSIGNIA [129]. Differing aspects are signalling complexity, sender or receiver initiated reservations, and multicast reservation support. Three inter-domain reservation protocols are also analysed, which is closely related to this thesis. The first is the “Border Gateway Reservation Protocol (BGRP)” [147]. BGRP creates a sink-tree reservation structure limiting the reservation states in border nodes. DiffServ forwarding is expected and sender-initiated PROBE/GRAFT reservation messages aggregate resource requests along the way by reusing and re-allocating existing reservations. 81 17.11.2009 The reservation tree structure can not fully aggregate reservations, due to the possibly differing roots of multiple trees. Therefore, a second inter-domain protocol, called “Sharedsegment Inter-domain Control Aggregation protocol (SICAP)” [168] has been defined, which optimizes the aggregation using shared-segment aggregations instead of a tree structure. Due to this change in reservation structure, the state information in border routers can be significantly reduced with SICAP. “Dynamic Aggregation of Reservations for Internet Services (DARIS)” [33], the last analyzed protocol in RFC 4094, provides a threshold based dynamic inter-domain aggregation scheme. Individual reservations are monitored and trigger the setup of an aggregation reservation by crossing a configured threshold. This approach also establishes shared segment reservations along AS path routes. Intermediate ASes can in turn remove individual reservation states and rely on the aggregate instead. All of the analysed protocols have in common, that they setup flow reservations with fixed parameters. This is far too complex for an approach that targets general traffic separation for potentially all flows in the Internet without explicit resource reservations and QoS guarantees. The same argument holds true for the Next Steps in Signalling (nsis) concept, which is standardized within an official IETF working group, which has already produced five RFCs. They focus on signalling framework, protocol design and signalling security. Fig. 71 depicts the layered structure of interconnected NSIS components in a node, which shows the general structure of the concept. The “NSIS Signaling Layer protocols (NSLP)” and the “NSIS Transport Layer protocols (NTLP)” represent the two-layer framework structure. In the upper half, the NSLP for QoS signalling [132] is of most interest here. It is still in proposed standard draft status. The last draft version (-16) expired in August last year. The lower half is dominated by the universal transport layer protocol “General Internet Signaling Transport (GIST)” [164]. It has recently been changed into an “experimental” draft status. Both together provide similar ways of operation and achievable functionality as RSVP. However, the NSLP QoS does not depend on a specific underlying QoS model and supports different reservation types (such as edge-to-edge, access-toedge, edge-to-end). NSIS is a universal signalling concept that appears to be well applicable for a wide range of resource reservation for flows of different granularities. Three independent implementations exist, which are all based on Linux platforms. None of the commercial router producers has NSIS implementations in their products. It is expected, that NSIS is well suited to achieve similar traffic separations as targeted in this thesis at least within the networking layer. However, it has not been chosen as base for the new cross-domain and cross-layer coarse grained Quality of Service support concept for several reasons. First of all, the flow-based reservation signalling is considered counterproductive as mentioned before. Secondly, the concept is too complex for the aspired simple traffic separation. Thirdly, the lack of support in commercial routers would delay the adoption of the proposed concept of this thesis in provider networks at large scale. Lastly, the recent shift towards experimental status is a major drawback on the road to commercial deployment. Especially the reasoning of the “Internet Engineering Steering Group (IESG)”, for the downgrade from proposed standard to experimental standard is remarkable. Fig. 73 and Fig. 72 document the current situation of the GIST standardization process. 82 17.11.2009 Fig. 71 Components of a NSIS node - [80] Next Steps in Signaling Internet-Draft Intended status: Experimental Expires: December 5, 2009 H. Schulzrinne Columbia U. R. Hancock RMR June 3, 2009 GIST: General Internet Signalling Transport draft-ietf-nsis-ntlp-20 ... Fig. 72 GIST protocol change to “Experimental“ status [164] 83 17.11.2009 To: Gerald Ash <gash5107 at yahoo.com>, "iesg at ietf.org" <iesg at ietf.org> Subject: Re: [NSIS] FW: I-D Action:draft-ietf-nsis-ntlp-20.txt From: Ross Callon <rcallon at juniper.net> Date: Thu, 11 Jun 2009 23:17:47 -0400 The fundamental problem with GIST is that is allows normal hosts (laptops, desktops, …) to send traffic to the control plane of routers. This opens up a new vector for hosts to be the source of DDOS attacks against the control plane of the routers. Note that such DDOS attacks are not just theory -- in fact multi-gigabit DDOS attacks against routers have occurred and do occur, and thus protecting against these is critical. It is therefore normal for service providers to prohibit "host to router" signaling packets (such as RSVP packets) from entering their network from the customer networks, for example by discarding these at the CE/PE boundary. Unfortunately the fact that such DDOS attacks are facilitated is not dependent upon the method that the router uses to recognize the packets as signaling packets. So long as a host has the ability to send traffic to the control plane of routers, then attackers will be able to harness the power of thousands of compromised hosts to attack routers. Of course the same issue could come up with RSVP. It became a standards track protocol a very long time ago, and would probably face the same scrutiny if it were a new protocol being proposed today. The current widespread use of RSVP is generally in ISPs limited to support of MPLS within a service provider. The same issue comes up in terms of DDOS attacks against application servers. Here one issue is that we don't have an alternative: hosts have to be able to send traffic to servers. Also, in general at least the largest DDOS attacks against servers need to be dealt with by putting appropriate packet filters / rate limits in place in routers (assuming that the router network is operating, and wasn't taken down by a different DDOS attack). In terms of the right want to deal with such DOS attacks: The reality is that it would be quite a major undertaking to deploy sufficient protection to allow hosts in general to signal to the router's control plane while still protecting against such attacks. For example control traffic would need to be rate limited at pretty much every entry to every major service provider network, and the effect that any DDOS attack would have on legitimate control traffic would need to be understood. If the attack came from a very large number of sources, then the rate at each entry point might be quite low, implying that either the widely deployed rate limits would need to also be very low, or they would need to be adjusted in response to an attack. All of this would need to be documented. However, the amount of difficulty that would be encountered in deploying such a system suggests that this is not an appropriate thing to put into the IETF standards track unless and until there is clear and well documented motivation for whatever new signaling protocol is being proposed. It is also possible that a signaling protocol could be used in a sort of "walled garden" scenario, where the hosts that are permitted to initiate control traffic are known and are protected from compromise. The current use of RSVP within some enterprise networks could be thought of as one example of such a "walled garden". If deployment experience of NSIS is collected from the experiment and presented with a clear definition of the walled garden within which the protocol can be safely operated, then this work might be more likely to be progressed to standards track (with the description of how and why the deployment is limited to that garden). Ross (speaking for myself, but having discussed the issue with other IESG members) Fig. 73 GIST protocol objections explained by Ross Callon [43] Following the reasoning of routing area director Ross Callon, any general signalling protocol that aims for end-to-end resource reservations will no longer pass the IESG for potential denial of service reasons. Either the protocol scope retreats from the end hosts (and applications) or a “walled garden” scenario is implemented, which strictly limits the user to network interaction to non-harmful functionality. Under those circumstances, RSVP appears to be the historical standardization flaw that will prevail. Further work has been completed in the field of guaranteed inter-domain QoS reservations [165]. It introduces a refined version of BGP, called “BGP+”, which is optimized for fast convergence. BGP+ also includes the ability to judge about a route’s QoS capabilities and to exchange this information with QOS NSLP and NTLP (GIST). This is required for the reservation adoption in the case of route changes. 84 17.11.2009 However, this work is again too complex for the aspired generally applicable simple class of service concept of this thesis. Its strong ties with GIST, the guaranteed resource reservation and the requirement of a homogenous set of supported classes of service in all participating provider networks prohibit its usage for the new concept. Related work on QoS provisioning in a wider sense can include the concept of “PreCongestion Notification (PCN)”. Since resource reservations and traffic prioritization provide significant QoS enhancement in highly loaded (congested) networks, the avoidance of congestion by sending rate reductions is an effective means of QoS provisioning. This goal is targeted by PCN using token bucket metering and packet marking for early congestion warning. This marking can either guide the intermediate nodes to select the “right” packet for dropping in the case of congestion or trigger the egress edge of a PCN domain to inform the ingress edge about the congestion. This ingress in turn is responsible for admission control to the PCN domain and possibly flow termination, if the already admitted flows’ QoS degradation does not allow extra flows to enter the domain. Fig. 74 depicts the major PCN components and its working principle. Fig. 74 PCN working principle - [136] From an inter-domain perspective with a general traffic separation scheme in mind, the PCN concept reveals two major drawbacks. The first is its limitation to a single domain and the more subtle second limitation comes out from the missing PCN marking encoding in packet headers. The IPv4 header (see Fig. 1) has no PCN marking bits, but rather 6 DSCP bits and two bits for the explicit congestion notification [156]. However, PCN defines three level of congestion marking (no congestion, admission stop marking and excess traffic marking), which need to be encoded in the packet header. The found compromise is PCN’s limitation on just one DSCP value marking for PCN signalling. That is the so called “DSCP for Capacity-Admitted Traffic” [17] will be used for PCN and redefines the ECN bits accordingly. All other DSCP marked traffic can either not be admitted into a PCN domain, or will be remarked to PCN DSCP or can not be used for PCN marking. Both strong limitations clearly show, that PCN will not be an equivalent replacement for the proposed general concept of cross-domain and cross-layer coarse grained Quality of Service support in IP-based networks as described in chapter 7. 85 17.11.2009 7 New (coarse grained) CoS concept 7.1 Motivation and target The current situation of Quality of Service support within Autonomous Systems and the support between ASes at interconnection points differs dramatically. The fast growing number of Internet users and the rapid increase in access line transmission capacity lead to a steady growth of Internet traffic. Fig. 75 exemplarily shows the overall exchanged traffic statistics of the German Internet exchange in Frankfurt. Fig. 75 DE-CIX yearly traffic graph - [62] Furthermore, service providers are increasingly offering voice over IP or IP-TV services to their customers. In order to ensure the right transfer quality of those delay-sensitive services, most providers still choose a high degree of over-provisioning (less than 20% network load) as the easiest and still cheapest solution. However, European providers seem to take a leading role in consistently setting up Differentiated Services forwarding within their network in order to ensure separated traffic handling and to cut down on the over-provisioning cost. Four to six traffic classes of service are common. However, this approach can only be consistently and homogenously applied intra-domain. Public inter-domain interconnections are still run without any QoS support (“Best-Effort (BE)” interconnection) and solely rely on the over-provisioning solution. Such interconnected “quality islands” exist independently, peer with BE traffic, perform costly multi-parameter ingress classification to locally guess and match on the incoming traffic class, run uncoordinated QoS concepts and might not even be known globally. 86 17.11.2009 Due to the fast access speed increase and the high quality expectations of their customers, service providers are increasingly forced to frequent and costly interface speed upgrades. This new coarse grained CoS concept therefore targets the inter-domain BE interconnection and aims for a traffic separating interconnection style without QoS guarantees. Neighbouring providers are already able to setup such CoS enabled interconnections by means of mutual SLA-based agreements about the supported classes of service, their encoding and mapping. However, the new concept extents this manual local interconnection CoS support by means of transitive signalling of available classes of service and their respective class markings. This enables multi-domain CoS transit paths with automated class of service transfer adoption. The locally available CoS support is disclosed to all providers, which will adopt the forwarded traffic to the neighbour’s class set and encoding. Based on this basic functionality, CoS routing and tunnelling are expected to evolve on top. Furthermore, the new coarse grained CoS concept extends the traffic separation in locally available classes of service across networking layers. IP CoS support is the anchor point of the class set signalling using the inter-domain PHB ID encoding (see Fig. 46). However, depending of the availability of MPLS tunnels, Ethernet QoS support or virtual channels for traffic separation, the concept’s signalling associates the classes and their markings of different layer technologies. This is required, since no definitive standards exist, which defined this cross-layer mapping of available class sets. However, service providers do individually define mapping rules within their domain and now have the means to signal this cross-layer mapping to interconnection partners. This again enables class set approximation, but now augmented to a consistent traffic separation in all CoS forwarding enabled layers. Lastly, the new concept introduces an optional second mechanism, which prevents the possible excessive misuse of higher priority traffic classes. Class-based ingress limitation using token bucket metering with associated dropping or remarking rules for excess traffic protects the CoS enabled AS from overload in high priority traffic classes. Those ingress filter parameters are signalled to adjacent interconnection partners. This results in a predictable forwarding behaviour and allows for informed traffic planning and possibly shaping at the sending AS egress edge. The new coarse grained CoS concept therefore: • provides knowledge about the available traffic separations and markings by means of transitive Cross-domain marking signalling with associated Cross-layer mapping, • enables marking adoption (and possibly route selection) without guarantees and • performs fair signalling of class overload limitations and excess traffic handling. This twofold “free to join” concept about global class set marking signalling with cross-layer mapping and rate limitation signalling is optimized for simplicity. Quality of Service guarantees are waived in favour of signalling, metering, debugging and operating simplicity. QoS in this approach therefore refers to primitive traffic separation into several classes, which will experience differently prioritized forwarding behaviour in relaying nodes. Enqueueing in separate queues is aimed for. Inter-AS Class of Service is targeted by the concept, since simple traffic separation is identified as key characteristic. If widely applied, the public Internet will evolve into a public “Betternet” in the future. 87 17.11.2009 The concept has been formulated in two Internet drafts [124] and [125] and was widely discussed and published in the networking community. The resulting feedback acknowledges the reduced complexity and expressed the preference before the aforementioned more QoS guaranteeing approaches. The concept is expected to be applied in global scale, possibly combined with SLA-based QoS guaranteeing solutions at individual interconnections. Due to the targeted global deployment, scalability, router resource consumption and operational stability has been analyzed. 7.2 Usage of BGP for QoS signalling Signalling Class of Service sets and markings between interconnection partners can either be performed as piggyback on already deployed protocols or by means of a separate signalling protocol. Static CoS sets and markings would be a third and in theory the best solution, when all providers would agree on a single globally available inter-domain CoS set. However, the latter is neither existent nor likely to be standardized any time soon. The definition and usage of specialized signalling protocols for the possibly frequent exchange of load statistics and flow-based quality requests and grants is an appropriate solution and likely to happen for QoS guaranteeing approaches. The concept of this thesis, however, does not require such separate handling due to its simplicity and coarse-grained global signalling scope. Reusing existing protocols with simple extensions is therefore envisaged. An attractive reuse candidate out of the existing signalling protocols is the NSIS protocol family. It has not been chosen for two reasons: 1. None of the existing AS border routers currently runs NSIS protocol entities and 2. The recently observed IETF objections against NSIS seem to dramatically delay its appearance as a proposed standard of the Internet (see Fig. 73). The only IETF standard protocol that is readily available at interconnection points for the reuse purpose is the Border Gateway Protocol. The following two sections explain the pros and cons of this choice. Why to use BGP for signalling BGP is the de-facto interconnection protocol and therefore globally accepted and globally available. It is a well designed flexible protocol that allows for simple signalling extensions. BGP exchanges reachability information and can tag this information with route related attributes. Such attributes have IANA assigned type values listed in respective IANA registries and associated attribute structures. This attribute and IANA registry approach allows for the flexible extension of BGP. All attributes (existing or newly defined attributes) are automatically associated with the network layer reachability information advertised in the respective BGP UPDATE messages. Attributes of transitive type are even relayed globally together with the NRLI. Why not to use BGP for signalling BGP’s stability is achieved through dampened UPDATE message rates and the concept of failure confinement within routing areas or confederations (see chapter 2.2.2). Any fast changing signalling information is therefore not suited for BGP. BGP might also be avoided, if long lived signalling information can be placed in Internet Routing Registries (see 5.3) instead of the UPDATE message transport. 88 17.11.2009 Lastly, since all AS border routers of the Internet need to store and process the large and ever growing BGP communicated reachability information, any extension should barely put any extra burden on the routers’ resource consumption. The new coarse grained CoS concept does make use of BGP and its already defined extended community attribute structure for the following reasons: 1. BGP is readily available for the concept’s deployment, in particular, if widely implemented attribute structures can be used for the CoS signalling, 2. The concept’s CoS signalling is small in size and not rapidly changing, 3. Service providers are familiar with BGP’s community philosophy and can easily adopt to the proposed CoS extensions and 4. The Internet Routing Registry signalling approach is included in the concept’s specification for backup and security purposes, but not exclusively relied on. This is due to the fact, the many service providers are still not making use of IRR information in their border router configurations. Within BGP, the choice of Extended Community attributes for the CoS signalling has been made, since the container size of 8 byte is sufficient and the automatism of associating attributes with all NLRI of an UPDATE message matches the concept’s target of signalling simplicity and efficiency. Specification details are outlined in chapter 7.3. The different approach in related work (see chapter 6) of using a separate address family for “QoS route” signalling has been deselected. The new coarse grained CoS concept’s signalling information is expected to be associated with the majority of Internet routes. The use of a separate address family would require doubled signalling for reachability and CoS support purposes, which is not an efficient signalling solution. 7.3 Definitions and information processing The following two sections outline the design principle, attribute definitions and processing as specified in the respective two IETF draft documents [124] and [125]. They are an integral part of this thesis work. 7.3.1 BGP extended community attribute for CoS marking Cross-domain CoS marking and cross-layer mapping signalling is specified in “draft-knollidr-qos-attribute-04” [124] as follows. Reachability information of IP prefixes is augmented by possibly several instances of a new BGP Extended Community. Each instance signals the availability of a certain class of service together with its technology dependent marking encoding. Several such Extended Communities are needed in order to signal more available classes as well as more associated cross-layer representations in other networking technologies. As a design principle, only the IP prefix originating AS is allowed to initially associate such a set of Extended Communities of supported classes with the advertisement of their own prefixes. Neighbouring and more distant ASes will then: - learn about the available classes and marking encodings, - possibly use the information for best path or multi-path decision making, - relay the respective best path and associated transitive attribute information to their neighbours – possibly adopting the signalled locally applied marking and 89 17.11.2009 use the learned class marking for downstream packet forwarding (including possible remarking at the outgoing edge interface). Transit ASes perform class marking approximation for an as close as possible class set mapping and forwarding adoption. ASes are free to ignore single classes or cross-layer mappings of the classes, but need to indicate this by means of a provided “ignore” flag. Fig. 76 depicts the resulting signalling and traffic forwarding procedure. - Fig. 76 Cross-Domain CoS marking concept Several QoS Marking communities may be included in a single BGP UPDATE message. They are virtually linked together by means of an identical "QoS Set Number" field. Each QoS Marking community is encoded as 8-octet tuple, as defined in [124]. Signalled QoS Class Sets are assumed to be valid for traffic crossing this AS. If different QoS strategies are used with an AS, its provider is responsible for consistent transport of transit traffic across this inhomogeneous domain. In all transit forwarding cases, QoS based tunnelling mechanisms are the means of choice for transparent traffic transport. The availability of the "Best Effort" forwarding class is implied and defaults to a zero encoding on all signalled layers. It is therefore not necessary to include QoS Marking communities for the Best Effort Class as long as the default encoding is in place. 7.3.1.1 Extended Community Type The new QoS Marking community is encoded in a BGP Extended Community Attribute [161]. It is therefore a transitive optional BGP attribute with Type Code 16. The actual encoding within the BGP Extended Community Attribute is as follows. The QoS Marking community is of regular type which results in a 1 octet Type field followed by 7 octets for the QoS marking structure. The Type is IANA-assignable and marks the community as transitive across ASes. The type number has been assigned by IANA to 0x04 (see Fig. 77). Optionally, a non-transitive Type value assignment of 0x44 is provided, which allows for the AS internal marking information exchange. The community format remains untouched for this non-transitive version. Fig. 78 depicts the BGP Extended Community Attribute structure. 90 17.11.2009 http://www.iana.org/assignments/bgp-extended-communities Border Gateway Protocol (BGP) Data Collection Standard Communities (last updated 2009-06-02) … Registry Name: BGP Extended Communities Type - regular, transitive Reference: [RFC4360] Range Registration Procedures ----------- --------------------------------------0x90-0xbf Standards Action/Early IANA Allocation 0x00-0x3f First Come First Served Registry: Type Value Name Reference Registration Date ----------- ------------------------------------ --------- ----------------0x04 QoS Marking [Knoll] 2008-12-30 0x05 CoS Capability [Knoll] 2009-05-18 Registry Name: BGP Extended Communities Type - regular, non-transitive Reference: [RFC4360] Range Registration Procedures ----------- --------------------------------------0xd0-0xff Standards Action/Early IANA Allocation 0x40-0x7f First Come First Served Registry: Type Value Name Reference Registration Date ---------- ------------------------------------- --------- ----------------0x40 Link Bandwidth Extended Community [draft-ietf-idr-link-bandwidth-00] 2009-05-18 0x44 QoS Marking [Knoll] 2008-12-30 … Fig. 77 IANA registry for BGP Extended Community type numbers As already made clear in chapter 2.2.2, it is important to distinguish between the “transitive attribute” and a “transitive community”. This depicted attribute structure is by default of transitive type and will therefore be always relayed across ASes – regardless of the actual processing of it. A special marking flag, so called “partial flag”, is defined for BGP path attributes, which will be set by ASes, which do not interpret such an Extended Community Attribute. With the decision to use Extended Community Attributes as “container structure” for the Extended Communities for CoS marking, it is ensured, that the signalling relay will actually reach all ASes of the Internet. However, one limitation still exists in practise and that is, that providers might decide to generally suppress the relay of Extended Community Attributes no matter what communities are enclosed. In such a case, all ASes up to the blocking one will receive the class set information and might make use of it. Further upstream ASes, however, will not receive the CoS signalling via this relay path. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0 1 0 0 0 x 0 0| | +-+-+-+-+-+-+-+-+ 7 octet QoS Marking community structure | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Fig. 78 BGP Extended Community Attribute structure with type 0x40 or 0x44 BGP UPDATE messages can by definition only include a single BGP Extended Community Attribute. However, each attribute can enclose several Extended Communities. Such Extended Communities are in turn again classified as “transitive” or “non-transitive” community type. Here, “transitive” stands for the distinction of whether a community can be signalled across an eBGP session or whether the community is confined to communi- 91 17.11.2009 cation sessions with iBGP peers only. The CoS marking Extended Community has been assigned a transitive as well as a non-transitive type number to give providers the choice for AS external or AS internal only usage of the signalling structure. The remaining explanation will assume the usage as transitive type, since cross-domain signalling is an important target of the concept. 7.3.1.2 QoS Marking Extended Community Structure 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0 0 P R I A 0 0| QoS Set Number|Technology Type| QoS Marking Oh| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | QoS Marking Ol| QoS Marking A |0 0 0 0 0 0 0 0| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Fig. 79 Structure of the QoS Marking Community As shown in Fig. 79, each signalled Extended Community contains a “Flags” field, a “QoS Set Number”, a “Technology Type” and two “QoS Marking” fields. The first octet contains four flags, ‘P, R, I and A’, which are used to indicate processing status and results. The 'P' flag indicates the preservation of incoming markings during the transit forwarding process. The IP prefix originating AS should set the flag to '1', which is otherwise implied by an AS_PATH length of 1 AS. Transit ASes must set the flag to '1', if the advertised Marking A is accepted at the ingress and is sent out unchanged at the egress. That is, no remarking occurs - neither for marking adoption with the neighbouring downstream AS nor by resetting the markings. This flag field is set and cleared by each relaying AS according to its handling of markings - irrespective of the possible ignorance of the particular Marking A in the internal per hop forwarding behaviour. The "R, I and A" flags are set to '0' in the advertisement by the IP prefix originating AS. Transit ASes must change the flag value to '1' once the respective event occurred. If the QoS marking actively used in the transit AS internal forwarding is different from the advertised original one, the 'Remarking (R)' flag is set to '1'. This must be signalled separately for each technology type community within the set of Extended Communities. The same applies to the 'Ignore (I)' flag, if the respective advertised QoS marking is ignored in the transit AS internal forwarding. The 'Aggregation (A)' flag must be set to '1' by the UPDATE message relaying transit AS, if the respective IP prefixes will be advertised inside an IP prefix aggregate constituted from differing Class Sets. The handling of prefix aggregation is vital for routing table size reduction and routing stability. However, this aggregation can easily result in the merging of routes to the more specific prefixes with differing class of service sets. In this case, the aggregator becomes the IP prefix originating AS for the prefix aggregate and is responsible for the mapping between the upstream signalled merged class set and the downstream available differing class sets. It is the provider’s responsibility to care for close class set approximation in terms of forwarding and marking behaviour. If the defined "R, I and A" flags are cleared - and by means of the cleared 'Partial' flag of the BGP attribute it is shown, that no "QoS Class ignorant" AS is involved in the forwarding path – a consistent class based overall traffic separated forwarding is available along this path. Several single QoS Marking communities can be logically grouped into a QoS Marking community Set characterized by a identical QoS Set Number. This grouping of the single QoS Marking communities into a set provides cross-layer linking between the QoS class encodings. The number of signalled QoS Marking communities as well as QoS Marking community Sets is at the operator's choice of the originating AS. The enumerated QoS set numbers have BGP UPDATE message local significance starting with set number 0x00. 92 17.11.2009 Since all signalled marking are networking technology specific, the Technology Type field indicates, which technology the marking refers to. Extensive searching has been performed in the course of defining this signalling for existent technology type enumerations. The closest result was the “IANAifType-MIB” enumeration [96]. However, this enumeration is far too detailed and the registry maintainers have discouraged its usage for existing consistency weaknesses. Therefore, a short and simple enumeration has been defined as shown in Table 12. Table 12 Value 0x00 0x01 0x02 0x03 0x04 0x05 0x06 Technology Type Enumeration Technology Type DiffServ enabled IP (DSCP encoding) Ethernet using 802.1q priority tag MPLS using E-LSP Virtual Channel (VC) encoding using separate channels for QoS forwarding / one channel per class (e.g. ATM VCs, FR VCs, MPLS L-LSPs) GMPLS - time slot encoding GMPLS - lambda encoding GMPLS - fibre encoding The two most important fields of the new QoS Marking Extended Community structure are the QoS Marking O and A field. The interpretation of these fields depends on the selected layer and technology. ASes, which process the community and support the given QoS Class by means of a QoS mechanism using bit encodings for the targeted behaviour (e.g. IP DSCP, Ethernet User Priority, MPLS TC etc.) must use a copy of the encoding in the "QoS Marking A" community field. Unused higher order bits default to '0'. Other technologies, which use separate forwarding channels for different classes (such as L-LSPs, VPI/VCI inferred ATM classes, lambda inferred priority, etc.) shall use class enumerations as encoding in this community field. The enumeration count starts with zero for the best effort traffic class and rises by one with each available higher priority class. There are two QoS Marking fields within the QoS Marking community for the "original (O)" and the "active (A)" QoS marking. Higher order bits of those fields, which are not used for the respective behaviour encoding, default to zero. The QoS Marking O (Original QoS Marking) field is a 16 bit QoS Marking field, which consists of a high ("Oh") and a low ("Ol") octet. The IP prefix originating AS copies the internally associated QoS encoding of the given Technology Type into this one octet field. The field value is right-aligned depending on the number of encoded bits. For the IP technology, the encoding of Per Hop Behaviour Codes has to follow the definitions stated in [31]. The field must remain unchanged in BGP UPDATE messages of relaying nodes. QoS Marking A (Active QoS Marking) and QoS Marking O must be identically encoded by the IP prefix originating AS, except for the case, where IP technology Per Hop Behaviours are addressed. "QoS Marking A" will always contain the locally applied encoding for the targeted PHB. All other ASes use this Active QoS Marking field to advertise their locally applied internal QoS encoding of the given class and technology at the interconnection point. The field value is right-aligned depending on the number of encoded bits. A cleared Marking field (all zero) signals that this traffic class experiences default traffic treatment within the transit AS forwarding technology. 7.3.1.3 QoS Marking Extended Community Usage Providers may choose to process the QoS Marking communities and adopt the behaviour encoding and tunnel selection according to their local policy. This may also lead to different IGP routing decisions or even effect BGP update filters. 93 17.11.2009 Only the IP prefix originating AS is allowed to signal the QoS Marking communities and Sets. All advertised prefixes, which originate from that AS will be sent with the same QoS Marking community Set in the respective UPDATE message. Transit ASes must not modify or extend the QoS Marking community Set except for the update of each 'QoS Marking A' field contained in the community Set and the respective "P, R, I, A" flags. Prefixes with associated identical QoS Marking community Sets are to be advertised together in common UPDATE messages in relaying nodes. Fig. 80 shows an AS interconnection example with different Class Sets. It shows the case in AS 5 where different Class Sets are used internally and externally. The proposed QoS Class Set signalling will always use the external definitions within the UPDATE message QoS Marking communities. The example also shows, that IP prefixes, which originate in AS 5 and AS 3 can be advertised together with the same QoS Marking community Set as long as their Layer 2 encoding is identical. Fig. 80 CoS enabled AS interconnection example topology IP packet forwarding based on packet header QoS encoding might require remarking of packets in order to match AS internal policies and encodings of neighbouring ASes. Identical QoS class sets and encodings between neighbouring ASes do not require any remarking. Different encodings will be matched on the outgoing traffic. Outgoing traffic for a given IP prefix uses the 'QoS Marking A' information of the respective BGP UPDATE message QoS Marking community for adopted remarking of the forwarded packet. If the 'I' flag is set for a given encoding, the outgoing traffic remarking should still be applied despite the signalled lack of QoS Class forwarding support. This is particularly important, if the preserve flag 'P' is signalled together with the 'I' flag. Several IP prefixes of different IP prefix originating ASes may be aggregated to a shorter IP prefix in transit ASes. If the original Class Sets of the aggregated prefixes are identical, the aggregate will use the same Set. In all other cases, the resulting IP prefix aggregate is handled the same as if the transit AS were the originating AS for this aggregated prefix. The transit provider may care for AS internal mechanisms, which map the signalled aggregate QoS Class Set to the different original Class Sets in the internal forwarding 94 17.11.2009 path. In case of IP prefix aggregation with different QoS Class Sets, the 'Aggregation (A)' flag of each QoS Marking community within the Set must be set to '1'. 7.3.1.4 Confidentiality Considerations The disclosure of confidential AS intrinsic information is of no concern since the signalled marking for QoS class encodings can be adopted prior to the UPDATE advertisement of the IP prefix originating AS. This way, a distinction between internal and external QoS Class Sets can be achieved. AS internal cross-layer marking adaptation and policy based update filtering allows for consistent QoS class support despite made up QoS Class Set and encoding information within UPDATE advertisements. In case of such policy hiding strategy, the required AS internal ingress and egress adaptation shall be done transparently without explicit "Active Marking" and 'R' flag signalling. 7.3.1.5 QoS Marking Example The example AS is advertising several IP prefixes, which experience equal QoS treatment from AS internal networks. The IP packet forwarding policy within this originating AS defines e.g. 3 traffic classes for IP traffic (DSCP1, DSCP2 and DSCP3). These three classes are also consistently taken care of within a TC bit supporting MPLS tunnel forwarding. The BGP UPDATE message for the announced IP prefixes will contain the following QoS Marking community Set together with the IP prefix NLRI. 95 17.11.2009 Fig. 81 QoS Marking Extended Community signalling example 7.3.2 BGP class of service interconnection Class-overload prevention is specified in “draft-knoll-idr-cos-interconnect-03” [125] as follows. The new coarse grained CoS concept is a twofold concept and provides in its second half an optional mechanism, that prevents the possible excessive misuse of higher priority traffic classes. Class-based ingress limitation using token bucket metering with associated dropping or remarking rules for excess traffic protects the CoS enabled AS from overload in high priority traffic classes. Those ingress filter parameters are signalled to adjacent interconnection partners. This results in a predictable forwarding behaviour and allows for informed traffic planning and possibly shaping at the sending AS egress edge. This fair and square interconnection limitation signalling is specified using two BGP attributes. Two new transitive attributes are specified, which enable adjacent peers to signal Class of Service Capabilities and token bucket Class of Service admission control Parameters. The new "CoS Capability" is deliberately kept simple and denotes the general EF, AF Group, BE and LE forwarding support across the advertising AS. The second "CoS Parameter Attribute" is of variable length and contains a more detailed description of available forwarding behaviours using the PHB ID Code encoding. Each PHB ID Code is associated with rate and size based traffic parameters, which will be applied in the ingress AS Border Router for admission control purposes to a given forwarding behaviour. 96 17.11.2009 A Basic Set of supported Classes, called "Basic CoS" is defined here, which consists of the primitive "Best Effort (BE)" PHB, the "Expedited Forwarding (EF)" PHB [60], the "Assured Forwarding (AF)" PHB Group [88] and the "Lower Effort" Per-Domain Behaviour (PDB) [34]. Providers, which can support this Basic CoS, signal this capability to their interconnection partners by means of the new CoS Capability Extended Community defined below. 4 AF PHB classes have been defined so far, which will be grouped into the generally signalled "AF Group". That is, as long as the AS provider can support at least one out of the 4 AF classes in his externally supported CoS Set, this AS is regarded as AF capable. A second transitive attribute is defined for parameter signalling about the applied access control within the ingress AS border router. The reason for this traffic limitation is the fact, that certain high quality forwarding behaviours can only be achieved, if the percentage of high priority traffic within the traffic mix lies below a certain threshold. This attribute informs the interconnection partner about the applied limitation, which can in turn be used to perform traffic shaping at the neighbouring AS egress. The attribute allows this limitation signalling either associated to the NLRI within the same UPDATE message or with "global" scope to describe the generally applied ingress limitation. Both attributes are likely to be used together, if ingress class limitation is used for the respective AS. Fig. 82 depicts the resulting class overload limitation concept and outlines, how excess traffic can either experience dropping or remarking punishment actions. Fig. 82 Class overload limitation concept 7.3.2.1 CoS Capability Extended Community Structure The CoS Capability Extended Community is encoded as BGP Extended Community path attribute as described in section 7.3.1.1. It is deliberately kept very simple and is defined as outlined in Fig. 83. It is a transitive Extended Community of regular type with the IANA assigned type value of 0x05 (see Fig. 77). The binary encoded support of per hop behaviour classes is detailed in Table 13. 97 17.11.2009 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |B E A L| Currently Unused - default to '0' | |E F F E| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Currently Unused - default to '0' | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Fig. 83 CoS Capability Extended Community Structure Table 13 CoS Capability Attribute – binary class encoding Bit 0 1 2 3 4 .. 7 Flag BE EF AF LE unused Encoding Default to ‘1’ to signal general “Best Effort” PHB support ‘1’ … “Expedited Forwarding” PHB support [60] ‘1’ … “Assured Forwarding” PHB group support [88] ‘1’ … “Lower Effort” PHB support [34] Default to ‘0’ The implied Per-Hop-Behaviour Identification Codes follow the definition as standardized in RFC 3140 [31]. The AF Group needs to consist of at least one of the currently available AF1x, AF2x, AF3x and AF4x. Fig. 84 Per-Hop-Behaviour Identification Codes implied by CoS Capability 98 17.11.2009 7.3.2.2 CoS Parameter Attribute structure The second attribute is a new optional transitive BGP path attribute of variable length. The attribute type number of 0xFF is currently used as specified in RFC 2042 [133]. The attribute contains one or more of the following token bucket parameter sets as shown in Fig. 85. Fig. 85 CoS Parameter Attribute structure The PHB ID Code field associates the respective signalled PHB support with the consecutively followed token bucket parameter set. This parameter set follows the specifications as given in RFC 2210 [182], which is also used within the IETF Integrated Services architecture. Only two flags (‘G’ and ‘DR’) are defined within the one octet Flags field. The 'G' flag signals, whether the limitations have global scope on all incoming traffic ('1') or are associated to traffic that is destined to destinations within the NLRI of the UPDATE message ('0'). NLRI specific limitation will supersede globally signalled ones for traffic destined to those NLRI destinations. The 'DR' flag signals the applied handling of non-confirming traffic. DR='0' signals strict dropping of excess traffic. DR='1' signals the performed remarking of excess traffic packets to Best Effort traffic marking. In order to correctly identify the originator of the signalled limitations, the “ASN of sending AS” holds the corresponding AS number. Depending on the 2-octet or 4-octet AS peering type, the sending AS of the attribute must encode its AS number as right-aligned 32 bit number. 7.3.2.3 CoS Parameter Attribute Usage The signalled parameters are used for PHB ID Code based ingress limitation. Depending on which PHB ID Codes a BGP peer signals in this attribute to its neighbour, it is said, that the respective PHB ID Code is supported and will experience the defined limitations. 99 17.11.2009 Those limitations can be applied to all incoming traffic of a specific PHB ID Code (marked as 'G') or only for incoming traffic, that is destined for the NLRI of the given UPDATE message. The resulting treatment for non-confirming traffic is signalled through the 'DR' flag. All limitations have AS local scope for the advertising AS and the neighbouring AS might or might not adopt its sending behaviour to those advertised limitations. Despite of the transitive nature of the new attribute, its usage for ingress limitation is confined to neighbouring ASes. Processing of the conveyed parameters is only valid for peers, who are peering with the AS specified in the ASN field of the attribute. The attribute should not be transitively relayed to non-adjacent interconnection partners. Since non-transitive BGP path attributes can be sent out into eBGP peering sessions, the CoS Parameter attribute would have been sufficiently defined as of non-transitive type. However, current commercial routers are not aware of this new attribute and will silently discard it. Therefore, the attribute has been defined as transitive type in order to allow for remote router configuration control as outlined in chapter 12. 100 17.11.2009 8 Mapping strategies 8.1 Problem statement The new coarse grained CoS concept of this thesis allows for the cross-domain and crosslayer signalling of supported classes of service, their marking encoding and mapping. The number of supported classes, their networking technology dependent marking and in particular their mapping between networking technologies is chosen independently by different network providers. The intra-domain mapping between technologies and the interdomain mapping between class sets of each layer constitutes the heterogeneity of the internetworking situation. The targeted aim in all such mappings is, that the QoS efforts taken up in one layer or domain should not be destroyed in the other. Fig. 86 Classification of the Mapping scope Three general criteria need to be distinguished: 1. QoS enabled networking technology (e.g. DiffServ, Ethernet QoS, MPLS E-LSPs), 2. Class encoding (marking) and 3. Associated parameter and treatment characteristics. 8.1.1 mapping between different class sets of the same layer This section outlines the difficulties in mapping operations within the same layer and specifically within the same networking technology. The simplest case is the 1:1 mapping between class sets with the same number of classes and associated treatment characteristics, but differing markings. Simple mapping can be performed as long as the resulting remarking is set up consistently. Such a mapping would 101 17.11.2009 not change the experienced forwarding behaviour, but rather enable providers to freely choose the marking value space. In terms of this new CoS concept, the signalling would include the same technology encoding, the same Marking O, but differing Marking A. The more likely and complex mapping occurs, if n original classes would be mapped into m resulting classes. This results in a mapping table with the following categorisation: • n > m Æ Class aggregation, • n = m Æ 1:1 mapping as described above and • n < m Æ Class splitting or Class wastage. The latter case appears to be the easier one, since no traffic merging is possibly required and class marking can be preserved. The degree of traffic separation, as targeted in this coarse-grained concept, is preserved and no detrimental effects are expected. This holds true, if all incoming traffic flows into the finer-grained QoS enabled AS arrive with the same limited class set n as described above. However, the respective AS merges those incoming traffic flows onto its internal forwarding paths and the merging might lead to some finer-grained traffic taking up higher priority classes as compared to the coarsegrained ones. For example, a 5:7 mapping as opposed to a 3:7 mapping might end up with a higher link share on the combined forwarding paths. This can be addressed by a priority scaled mapping, where the scaling in the 5:7 case, leads to 40% priority increase and in the 3:7 case to a 233% priority increase. The uniform distribution of class priorities in this simple case would assume a linear priority scheme in the original and resulting class sets. Un-scaled and scaled mapping would leave the additional classes unused. The difference only lies on the decision, which classes remain unused. Class splitting, however, would make use of all additional classes and differentiate the incoming traffic further. Such an approach could not only rely on the class markings, but rather perform further classification on additional header fields. Depending on the networking technology header, this could be the protocol or type field, the destination MAC or IP address etc. In general, this could also result in waiving the original classification and performing multi-layer packet inspection and re-classification. Class aggregation is the opposite case, where existing traffic classes are merged onto a common forwarding treatment. This can either be realised through a remarking function, where several incoming class markings are merged into one resultant marking or through the unchanged marking, but merged forwarding treatment. The latter will be referred to as “funnel treatment”. For the considered AS, the resulting forwarding treatment will be the same for each traffic class. However, the original traffic separation at the outgoing edge of the AS domain can not be restored in the remarking case and the smallest degree of traffic separation will prevail along the remaining forwarding path. Marking preservation is the preferred solution. The required funnel treatment can either be achieved through classification merging at each relaying AS internal node or through tunnelling. Either, the markings remain untouched within the tunnel and the tunnel marking determines the forwarding behaviour, or separate tunnels can be used to de-aggregate the traffic flows and markings. One specific tunnelling solution for all IP networks is specified in RFC 2983 [30] and formerly in RFC 2003 [152]. This IP in IP encapsulation is attractive for three reasons: 1. The encapsulated IP header remains entirely unchanged, 2. The mapping result and forwarding header processing is solely based on the outer IP header and 3. All internetworked ASes are capable of providing IP tunnelling as long as their edge devices can handle the encapsulation and de-capsulation procedures. 102 17.11.2009 Fundamental drawbacks of this IP tunnelling are the increased processing load on edge devices and the reduced maximum transmission unit size for transit payloads. At least 20 byte MTU reduction will occur due to the outer IP header. The same behaviour can be achieved with the more generic IP-based encapsulation protocol, called “Generic Routing Encapsulation (GRE)”, defined in RFC 2784 [72]. IP encapsulation within GRE would add at least another 4 byte MTU reduction for the minimal GRE header structure. Existing class aggregation recommendations for DiffServ classes and Ethernet priorities are listed in chapter 8.2. Encapsulation (tunnelling) of transit (customer) traffic is highly recommended by this new coarse grained CoS concept. IP-in-IP tunnelling, MPLS label switched paths or some kind of Ethernet encapsulation (refer to chapter 4.2) are strongly encouraged for traffic separation preservation and ease of CoS deployment. 8.1.2 mapping between different class sets of different layers Service providers, who offer Differentiated Service in the IP layer, are likely to undergird this approach with QoS mechanisms in lower layers. Typical QoS support of such link layer technologies are listed in chapter 4. Ideally, the underlying QoS support exactly matches the granularity and forwarding treatment requirements of the DiffServ PHBs. However, in practise this can hardly be achieved due to the differing number of available classes and the lack of standardized mappings between the CoS sets of each technology. Service providers will therefore independently decide on the cross-layer mappings and classification policies applied within their networks. Those configuration decisions are under sole control of the respective providers and therefore expected to be setup consistently and appropriately. The resulting ⎛n⎞ mapping ⎜ ⎟ is a vertical association of n classes of service and their markings to x ⎝ x⎠ underlying classes of service and their markings in the lower layer technology. This can possibly be setup in multiple rows, if several encapsulating technologies are in place. For the inter-domain case, the explained n:m mapping within the same layer is therefore multiplied by the number of technology combinations (such as e.g. IP in MPLS-LSP in Ethernet in FR-VC) in lower layers. ⎛n : m⎞ ⎜ ⎟ The matrix ⎜ ... ⎟ is the resulting layered mapping table in a fully meshed cross-layer ⎜x: y ⎟ ⎝ ⎠ mapping case. Here, each original class within each encapsulation is mapped into a resulting class in each new encapsulation. Such cases need to be addressed in interdomain cases, where the traffic exchange at an interconnection point is not just IP based, but encompasses inter-domain tunnels. 103 17.11.2009 ⎛n:n⎞ ⎜ ⎟ Here, either, the encapsulated class set and marking will prevail ⎜ ... ⎟ or the tunnel CoS ⎜ x: y⎟ ⎝ ⎠ ⎛n : m⎞ ⎛n : n⎞ ⎜ ⎟ ⎜ ⎟ will be harmonized ⎜ ... ⎟ or the encapsulated and the tunnelled CoS ⎜ ... ⎟ are forced ⎜x:x ⎟ ⎜ x : x⎟ ⎝ ⎠ ⎝ ⎠ to be identical between the interconnection partners. As mentioned earlier, this intra-domain and possibly inter-domain tunnelled forwarding is particularly important at places, where class aggregation needs to be performed. This aggregation would be applied by means of a reduced number of tunnel-based forwarding treatments, but will preserve the traffic separation granularity for the encapsulated traffic. The respective signalling of marking preservation is available through the “P” flag in the QoS Marking Extended Community. 8.2 Existing recommendations In the course of work on this thesis, a number of readings and talks revealed, that service providers are occasionally involved in QoS related research and discussions, but are reluctant to turn QoS on for inter-domain interconnections. This explains the current gap between the many QoS proposals and the still missing deployment. However, deployment recommendations are incentives and guidelines to make configuration easier and concepts more acceptable. As outlined in chapter 4 and 6, numerous QoS specifications and approaches exist, which have not yet led to QoS enabled interconnections in large scale. Differentiated Services, however, is widely accepted and increasingly applied within ASes. The same applies to MPLS and its TC bits based DiffServ support. Since the protocols of IETF dominate the AS interconnection and the capabilities of devices within the ASes at each side, the following section will concentrate on recommendations given in RFC 4594 [16] and RFC 5127 [45]. Ethernet specifications for user priority support and the respective priority setup specifications will follow, based on the definitions of IEEE 802.1D [97]. It should be noted, that the large variety of configurable class sets, their encoding and intra-layer and cross-layer mapping contributes to the complexity and uncertainty of an overall class of service based packet forwarding transport. The lack of a single standardized and globally supported class set and its encoding becomes obvious. However, service providers are reluctant to apply such a fixed class of service support and might not be able to define the common CoS base for such a standard activity. This is the sacrifice for standing out from each other in competition and for the valuable freedom of administration of the respective AS. On the contrary, the introduction of PHB IDs revealed, that service providers even requested a 16 bit encoding instead of the 6 bit DSCP encoding. Since the currently applied fallback solution of mere over-provisioning is expected to exceed the capital and operational expenditure budget of declining interface cost limitations, coarse-grained CoS interconnection with class-based over-provisioning will arise. Existing recommendations will be taken up for granted by larger Internet Service Providers and their smaller interconnection partners will adopt it. This is claimed to happen for simple DiffServ-based class setups and a respective recommendation is given in chapter 8.3. Cross-layer mappings will be applied AS-internally, but might evolve inter-domain as well. 104 17.11.2009 Especially the IXP-based CoS interconnection is expected to be used and the new concept provides the means for a consistent automatic setup. Configuration Guidelines for DiffServ Service Classes - RFC 4594 The fundamental guideline document for the deployment of DiffServ is RFC 4594 [16], which gives configuration guidelines to network operators for sensible service class selection, their usage and construction out of queueing, traffic management, PHB selection and DSCP marking elements. All definitions therein are given as recommendations and the authors point out several times, that all can be changed and applied differently, at the provider’s choice. However, they do point out consistency as being of importance for interoperability. The universal deployment of this DiffServ Service Class recommendation is, as the name suggests, based on the “Service Class” approach. That is, all applications that generate similar traffic characteristics and require similar traffic forwarding behaviour can be grouped into classes, so called “Service Classes”. The required characteristics of the traffic aggregate are represented by a PHB, which will in turn be encoded as DSCP value. Network traffic has been classified into two groups: “network control traffic” and “user/subscriber traffic”. The first group is divided into two service classes, being "Network Control" and “OAM”. As shown in Fig. 87, in the “user/subscriber traffic” group, ten Service Classes have been identified. They are grouped into four application categories and reflected against the so called “End-user multimedia QoS categories” as defined by ITU-T in G.1010 [118]. Fig. 87 User/Subscriber Service Classes Grouping - [16] The service classes in turn are associated with descriptive characteristics as well as with coarse statements about the respective loss, delay and jitter tolerance. No absolute values are given, but treatment orientation can easily be taken out from this definition. Fig. 88 depicts the table of associated characteristics. 105 17.11.2009 Fig. 88 Service Class Characteristics - [16] All service classes are given DSCP encoding names and values together with application examples. This sums up to 20 DSCP encodings for all 12 service classes as shown in Fig. 89. Lastly, each service class is guided by recommendations for applicable conditioning at the ingress of the network, the queuing and the queue management. The resulting table is depicted in Fig. 90. It is important to note, that a “Low-Priority Data” Service Class has been defined here with CS1 (‘001000’) encoding. This is of particular interest, since the support of Lower Effort PDB (RFC 3662 [34]) is expected to become important for inter-domain deployment and matches the recommended set of service classes. In practise, DiffServ support will not start off with all 12 service classes being configured at once. A start with fewer classes and a gradual increase as differing demand arises is likely and at the same time supported by the recommendations. 106 17.11.2009 Fig. 89 DSCP to Service Class Mapping - [16] Fig. 90 QoS Mechanisms Used for Each Service Class - [16] 107 17.11.2009 Given the flexible and coarse design of the service class recommendations, its readiness for application as well as the authors’ reputation and company support, it is likely to find such class sets being configured and deployed within provider networks. Since standard DSCP values are specified in the configuration guidelines, a smooth interworking between providers’ networks is feasible, possibly with a reduced number of service classes. Aggregation of DiffServ Service Classes – RFC 5127 [45] As described above, service providers are likely to adopt the DiffServ recommendations given in RFC 4594. However, the possible granularity of 12 service classes with up to 20 DSCP encoding values is far too detailed in core network areas. RFC 5127 [45] therefore gives guidelines on how multiple service classes may be aggregated into a fewer set of so called “forwarding treatment aggregates”. The usage and DSCP encoding of RFC 4594 based Service Classes is assumed and funnel mapping of several different DSCP encoded traffic flows with about the same forwarding treatment requirements into one treatment aggregate is specified. Marking preservation is expected. Treatment aggregates are created by funnel mapping of Service Classes grouped by loss, delay, and jitter requirements. Fig. 91 exemplarily shows the funnel mapping of the 12 Service Classes into e.g. 4 treatment aggregates. Fig. 91 Treatment Aggregate / Service Class Performance Requirements - [45] Fig. 92 in turn adds the funnel mapping of inter-DSCP association for the Treatment Aggregate Behaviour. 108 17.11.2009 Fig. 92 Treatment Aggregate Behaviour - [45] As mentioned above, such bundling of Service Classes into coarser forwarding treatments is typically deployed in high capacity core networks, where statistical multiplexing of similar traffic streams into a few forwarding classes combined with moderate over-provisioning is manageable and sufficient. Fine-grained QoS schemes in the core do not scale, increase processing load, hamper debugging etc. and are thus not acceptable by network providers. The four treatment aggregate example, however, seems to be a realistic, feasible and acceptable compromise. Fig. 93 MPLS E-LSP mapping of Treatment Aggregates - [45] The inter-domain case is also addressed briefly in the recommendation stating, that peering parties need to agree on the exact treatment aggregate content and representation. This results in mutual agreements and limits the extent of the QoS enable intercon- 109 17.11.2009 nection path. Furthermore, the RFC targets not just traffic separation, but rather Services Class based interconnection possibly including the MIT QoS scheme (see chapter 6) combined with precise parameter bounds as given in Y.1541 [108]. Both drawbacks are comprehensible for this higher class QoS interconnection, but distract from the aim of this thesis. RFC 5127 stresses the recommendation of marking preservation despite of treatment aggregate based forwarding. This is completely in line with this thesis’ strong recommendation of tunnelled customer transport. As a common example, the RFC documents the mapping suggestion of treatment aggregates into E-LSP of a MPLS domain. This choice is realistic, since nowadays most providers use MPLS in their network. Fig. 93 depicts the proposed mapping table of the 20 possible DSCP values as of RFC 4594 into 6 TC bit (former EXP bit [9]) encodings of the E-LSP. Four traffic aggregates are claimed to be supported, but sub-differentiated into two dropping levels for the default and the assured elastic aggregate. Again, the lower effort CS1 encoding is consistently used as forwarding treatment with lowest priority. All stated aggregate selections, DSCP encodings and TC bit associations are explicitly defined as recommendations. However, no strong commitment is requested from service providers. IEEE 802.1D user priority mapping definitions Ethernet QoS, in contrast to the above mentioned loose mapping commitment, precisely specifies the available user priorities, the mapping to strict priority forwarding queues and priority regeneration rules. Those specifications are part of IEEE 802.1D [97] and have been already addressed in chapter 4.2. Ethernet priority tagged frames offer 3 bits for 8 available forwarding priorities (see Table 5). Strict mapping is given for switch devices, which support 8 or less internal queues (see Table 6). Furthermore, the standard allows for priority reassignment by means of a configurable regeneration table. The default configuration would simply map the 8 classes to each other without changes. However, the combination with IP and/or MPLS based treatment aggregates could well be consistently reflected by mapped user priority settings. Hence, the queue mapping table can in turn be applied for treatment aggregation in single step granularities between one and eight resulting priorities (see Table 14). Table 14 Queue mapping reuse for priority mapping 110 17.11.2009 Systematic QoS Class Mapping Framework Although not widely deployed, the “Systematic QoS Class Mapping Framework” [150] of the Information and Communications University in Korea is worth mentioning. It targets the QoS class mapping in two steps: “Parameter-To- Class mapping” at the forwarding path ingress and “Class-to-Class Mapping” at the network borders along the way. Fig. 94 QoS Class Mapping framework - [150] For this thesis’ mapping analysis, the framework’s class-to-class mapping is of importance. Since not signalling of supported classes and their encoding is included, the framework is based on the six ITU-T defined IP QoS classes [108]. In order to extend this limited class set, the framework introduces a “Location Information (LI)” and performs a parameter-to-LI mapping at the entry and LI-to-class mapping at each relay boundary. LI is encoded as an integer in a redefined IP TOS field and addresses a 4x16 (loss x delay bounds) matrix. However, this interesting approach is unlikely to gain community acceptance because of the TOS field redefinition and the required LI interpretation and mapping function within all participating interconnected service provider networks. ITU-T NGN Focus Group - Proceedings Part II The NGN Focus Group within ITU-T has also developed a mapping for their defined six IP QoS classes, which is documented in paper [109]. However, the mapping concept is limited to the respective classes 0..5 and considers the mapping into four ATM, Frame Relay and UMTS QoS classes. It is not generally applicable to DiffServ based networks and will therefore not be considered in detail. QoS class mapping in Cisco devices The vendor Cisco gives detailed instructions on how to enable and to configure QoS support in Ethernet switches (see [55]). This includes configurable mapping tables for Ethernet priority (called “CoS”) to DSCP mappings with default settings. As Table 15 and Table 16 show, Cisco relies on the upper 3 bit DSCP value, which relates to the class selector (former IP precedence) values. The eight priority values are exactly matched to the respective class selector and in the opposite mapping direction, funnel mapping based on the upper 3 bit value is performed. Interestingly, one of the most important DSCP encoding, the one for EF traffic (= 46), is not directly covered. However, mapping tables are configurable and for example the fully Cisco equipped Chemnitz University applied a slightly modified mapping as shown in Table 17. 111 17.11.2009 Table 15 Cisco’s default CoS-to-DSCP mapping [55] Table 16 Cisco’s default DSCP-to-CoS mapping [55] Table 17 Chemnitz University applied CoS-to-DSCP mapping MPLS capable devices are as well challenged to define mappings between DSCP encodings and the 3 bit E-LSP encoding. For consistency and simplicity reasons, service providers will configure the DSCP <-> CoS and DSCP <-> TC identically. It should be noted, that this is a best common practice approach and no standard. RFC 3270 [75] describes the Support of Differentiated Services within MPLS domains. This includes E-LSP and L-LSP mappings. 112 17.11.2009 E-LSP mapping as mentioned above, is freely configurable, but requires having a consistent mapping strategy throughout a MPLS domain. If this is not the case for interconnected domains, consistent remapping at ingress and/or egress needs to be cared for. By default, if no mappings are configured, all encodings will map to Default PHB treatment. Special care needs to be taken for L-LSP encodings, since 20 bit label information and 3 bit TC information is combined for QoS treatment selection. L-LSP paths are associated with PHB types, which influences the embedded TC bit encoding. Mandatory mapping exists for TC bit encodings, which must be followed in outgoing marking and incoming classification. Fig. 95 depicts the bidirectional mapping requirement. Fig. 95 Mandatory L-LSP encoding rules - [75] 8.3 Coarse grained CoS mapping recommendations The new coarse grained CoS concept emphasized traffic separation, simplicity and implicitly customer traffic tunnelling. Service Class specifications combined with treatment aggregation creates the base for the selection of proposed simple class sets within the concept. The wide spread deployment and precise mapping of Ethernet priorities (see Table 14) adds a possible mapping strategy for dynamic class set granularity. A subset of eight class encodings is widely available in QoS enabled networking technologies and might be considered as realistic upper bound. However, much simpler setups are envisioned for the upcoming inter-domain class of service support. The default behaviour (“best effort (BE)” PHB) will and must always be available and defaults to an encoding of zero in all technologies. Direct mappings or funnel mappings with unspecified encoding ranges must always map the unknown codepoints into the default behaviour. Secondly, the lower effort (LE) behaviour is strongly encouraged for the inter-domain use case, since the existing BE interconnection can easily be augmented with an exchange of lower value traffic. Hence, the first and simplest recommended class set would consist of BE & LE classes encoded as ‘000000’ / ‘000’ and ‘001000’ / ‘001’ for the six or three bit representations. As for the mapping between BE & LE and other class sets, the LE will either be mapped into the LE class and all other classes into BE – resulting in separate enqueuing for both supported classes with lower scheduling and higher dropping priority for LE. However, in purely Ethernet related setups, BE and LE would both be enqueued in 113 17.11.2009 the same queue, but with possibly differing dropping probabilities. This single queue mapping conforms to the Service Class concept, but is not as well recommended for the simple BE&LE class set case. A second class setup is envisioned, which does not make use of LE, but rather introduces EF for high priority, delay sensitive and AF as medium priority forwarding class. The EF encoding of ‘101110’ would normally map to a priority of 5 (‘101’). However, for Ethernet related enqueuing and mapping, a priority of 6 (‘110’) is recommended. The AF support, is not associated to a single precise DSCP encoding. The four AF classes with 3 dropping priorities each result in 12 possible DSCP values. However, for reasons of simplicity, all AF encodings will possibly be funnel mapped into a single 3 bit encoding. ‘100’ is recommended for Ethernet related enqueuing and mapping. Lastly, a combination of all four basic behaviours (LE, BE, AF and EF) is recommended as inter-domain class of service support. The mapping of the EF&AF&BE case and separate low priority enqueuing for LE (‘001000’ / ‘001’) is aimed for. The described class sets and mapping are recommendations only and can be freely adapted to specific needs. However, in such cases, consistent signalling of supported encodings and cross-layer mappings is required by means of the QoS Marking Extended Community. If virtual channel encodings are used in the interconnection case, providers at both ends need to abide by the mandatory marking requirements for ATM, FR and MPLS L-LSPs as defined in RFC 3290. 114 17.11.2009 9 Simulation results The new cross-domain and cross-layer coarse grained Quality of Service support concept aims for both a general and global deployment. Therefore potentially all kinds of traffic will be carried along short as well as long multi-AS forwarding paths. This universal usage can not be covered by simulation. However, the comparison of the new concept’s improved class of service support contrasted with the current best effort only forwarding will be documented in the following section on simple topology examples with extensively simulated configuration combinations. The actual signalling of marking information does not need to be simulated, because proven BGP UPDATE message based signalling is not fundamentally influenced by the addition of a few Extended Communities. Practical feasibility and scalability are of more concern here, which is addressed in real world tests in chapter 11. Two simulation sections follow below, which address the resulting QoS marking and forwarding behaviour as well as the functionality of token bucket ingress limitation filters. 9.1 Setup selection for QoS marking and forwarding The packet transmission of different traffic type flows across differently configured single nodes as well as ASes has been simulated extensively. Besides the network topology, parameters have been varied, such as the number of traffic sources with appropriate traffic markings, the number of supported classes of service along the forwarding path as well as the scheduling and queuing configurations applied within the relaying nodes. The recommended simple CoS setup consisting of at most LE & BE & AF and EF coarse grained classes has been used. Three queuing disciplines have been varied, which are “no priority = round robin queuing”, “strict priority queuing” and “class based weighted fair queuing”. Varying those parameters resulted in more than 3000 simulations, which will be documented using a few selected examples. The complete result set is available upon request. The simulation of QoS marking and forwarding influence follows in scenario 1 to 7 in different topological granularities. Starting from a single node interconnection, the topology grows into interconnected ASes, into multiple interconnected ASes up to a chain of four interconnected transit-AS with multiple stub-ASes at either side. Scenario 7 concludes the marking and forwarding simulation section with the comparative simulation of cross-layer interworking between IP and Ethernet QoS enabled forwarding. The simulator OMNeT++ [12] has been used for the majority of the simulations. It is a modular, C++ programmed, discrete event simulation framework, which is freely available for academic research as open-source software. The so called “INET Framework for OMNeT++” creates the base for Internet protocol modelling and simulation. Additionally, two further modules have been used for realistic VoIP modelling and topology generation for Autonomous System based simulations. The Voice over IP traffic generation makes use of the “voiptool”, which is documented and available at [36]. The AS based topology generation is done by the “Realistic Simulation Environment for IPbased Networks (ReaSE)” [83]. 115 17.11.2009 Not readily available was the DSCP marking option for simulated sources, DSCP based enqueuing in multi-queue setups, class based weighted fair queuing, class based throughput metering and class based enqueuing and scheduling within the Ethernet switch simulation module. The work is documented in [134]. As mentioned above, the simulation abstracts the numerous traffic sources in real world scenarios into the recommended four basic traffic types being: • EF - Expedited Forwarding, • AF - Assured Forwarding, • BE - Best Effort and • LE - Lower Effort. The EF class is normally used for delay and delay variation critical applications such as Voice over IP, video conferencing etc. To get as close as possible to realistic VoIP packet streams, the voiptool [36] has been used to feed the EF sources and to record the received packets at the respective sink. This tool actually sends out audio samples read in from a wave file and records the received wave packets at the EF sink. Afterwards, the received wave file can be compared to the original one in order to calculate the perceived audio quality expressed in “Perceptual evaluation of speech quality (PESQ)” values [119]. If not stated differently, the simulation parameters were as given in Table 18. Table 18 Traffic source configuration parameters Traffic Type VoIP (EF source with differing DSCP and PESQ meter) Parameter settings Coding rate: 40000 bps DSCP: 101 000 // 0x28 mapped to ‘110’ Sending intervall:8 ms Packet size: 79 Byte Sending rate: 79 kbps Traffic Type CBR0 (BE source) Parameter settings DSCP: 0 Sending interval: 5ms Packet size: 827 Byte Sending rate: 1.323200 Mbps FTP (TCP based AF source with differing DSCP) Port: 21 DSCP: 010 000 // 0x10 Packet size: 1044 Byte Sending rate: variable TCP type: Reno Maximum Segment Size: 1024 Advertised Window: 14336 CBR1 (LE source with drop rate meter) DSCP: 001 000 // 0x08 Sending interval: 1ms Packet size: 83 Byte Sending rate: 664 kbps The differing DSCP encodings for the four recommended forwarding behaviours is due to the fact, that the simulator classification for enqueuing is based on the upper 3 DSCP bits only - hence the truncated EF and AF encodings. Waiting queues are drop tail queues by default and the metering resolution of the throughput meter was set to 250ms. Depending on the number of configured waiting queues for the different simulation runs, the queue mapping strategy follows the one given in Table 14. Thus, the DSCP marking for EF traffic will be mapped to priority 6 (‘110’) for smooth mapping into the IEEE VO priority class. The parameter combinations used within each topology, described later, are arranged in Table 19. The sources are gradually started to create different traffic mixes and the columns (a) to (f) show the supported classes (including separate enqueuing) along the 116 17.11.2009 forwarding path. All combinations were simulated with three scheduling schemes: no priority (round robin), strict priority and class based weighted fair queuing. The latter requires the setup of queue weights, which are given in the last row for the respective supported class set in each column. Table 19 Class and traffic type variations in simulations 9.2 Simulation results for QoS marking and forwarding The following sections will document some simulation results for the setups described above. Six scenarios differ in topological complexity with resulting traffic load and simulation results. Congestion is artificially caused by differing link capacities in order to demonstrate the influence even of this simple class of service on the resulting forwarding quality. Scenario 6 also addresses the mapping between differently setup class support along the AS forwarding chain under the light of marking preservation as opposed to packet remarking. Scenario 7 documents the influence of CoS setup harmonization between different networking technologies. The IP QoS and Ethernet QoS are combined in this cross-layer simulation, which required modified link capacities and adopted source sending rates. 9.2.1 Scenario 1: single node interconnection The topology of the first scenario is deliberately kept simple for demonstration purposes of the used sources and meters. Simulation results are deduced, which allow the direct mapping between source characteristic, relaying behaviour and reception result. Fig. 96 depicts the single node interconnection of four sources 1:1 mapped to their respective sinks and a bottleneck link between two routers. Router 0 is the contention point, where packet losses occur. 117 17.11.2009 Fig. 96 Scenario 1: single node interconnection Fig. 97 exemplarily shows the sending and receiving characteristic of the VoIP source in the four class CoS support case. Slight variations in the sending rate are due to the audio encoding characteristics in wave files, where a larger packet with some replay overhead is consecutively followed by three smaller audio sample packets. A PESQ value of 4,334 and a LE drop rate of 83.8 % has been achieved. Fig. 98 depicts the same situation but with no CoS support. All traffic types are roughly equally affected by the packet drops, which balances the loss rate according to the sourced traffic load. A PESQ value of 4,329 and a LE drop rate of 30 % has been worked out. Fig. 97 S1: 9-f-cbwfq Fig. 98 118 S1: 9-a-no-priority 17.11.2009 Fig. 99 S1: 9-f-cbwfq Fig. 100 S1: 9-a-no-priority Fig. 99 depicts the resulting traffic mix after the contention point. It can be seen, that all traffic classes are transmitted. The rather slow EF traffic passes through without discrimination. The TCP type AF traffic gets a fair capacity share according to its queue weight. Best effort traffic is limited below 500 kbps and LE traffic gets a minimal share. As stated above, about 80% of LE traffic gets discarded. Fig. 100 gives a different picture. EF traffic is varying as already shown in Fig. 98. However, BE traffic uses up most of the capacity followed by LE traffic. TCP based AF traffic is completely starved out. This is due to the fact, that BE and LE combined have been configured to exceed the bottleneck link capacity. However, starvation effects can also result from CoS deployment with strict priority queueing. This is documented in Fig. 101. EF traffic gets excellent forwarding and the TCP based AF traffic uses all remaining capacity. BE and LE traffic dies off. Fig. 101 S1: 9-f-strict-priority 119 17.11.2009 9.2.2 Scenario 2: AS interconnection – Single AS Scenario 2 uses a topology of two stub ASes (AS00 and AS20) and one transit AS (AS10). Each AS is simulated with double sources and sinks. No crossing traffic is modelled in AS10, so that a simple router model can be applied. Fig. 102 Scenario 2: AS Interconnection – Single AS Fig. 103 S2: 9-f-cbwfq Fig. 104 S2: 9-a-no-priority Fig. 103 and Fig. 104 show the resulting throughput graphs sorted by traffic classes. The qualitative result is comparable to the on in Scenario 1. However the usage rate (ur) increase in the CBWFQ case needs to be explained. The sending rate of all fixed rate sources sums up to about 4,1 Mbps. This rate is roughly achieved in the non priority case on the 10 Mbps link between AS00 and AS10. The reading of the diagram distinguishes the rates before the bottleneck with index ‘1’ and the ones after AS10 with index ‘2’. As Fig. 104 shows, TCP traffic is almost completely starved out. Packet drops trigger TCP’s congestion avoidance mechanism, which leads to an ever decreasing sending rate. In the CBWFQ case, the configured fair capacity share is given to the AF class, which leads to a 120 17.11.2009 sustainable TCP throughput rate. Before the bottleneck, the non-responsive CBR sources fill the 10 Mbps link to their nominal sending rate – hence the higher link usage. Fig. 105 S2: 9-b-cbwfq Fig. 106 S2: 9-e-cbwfq Fig. 105 depicts a similar result compared to the four class CBWFQ setup. This time, only two traffic classes (BE & EF) were supported and the classifier would map AF into the EF queue. IEEE based classifiers would therefore require an AF group encoding of ‘100xxx’. However, EF&AF&BE would be a recommended class set, which results in the very well acceptable forwarding behaviour as depicted in Fig. 106. 9.2.3 Scenario 3: AS interconnection – Multi-AS Scenario 3 uses a topology of six stub ASes (AS00, AS01, AS02 and AS20, AS21, AS22) and two transit AS (AS10 and AS11). Fig. 107 Scenario 3: AS interconnection – Multi-AS 121 17.11.2009 Fig. 108 S3: 9-f-cbwfq Fig. 109 S3: 9-a-no-priority Fig. 108 and Fig. 109 clearly demonstrate the superior forwarding behaviour of a simple CoS enabled forwarding behaviour. The increased number of sources, which are mixed with the transit links of AS10 smooth out the multiplexed traffic class throughput graphs. 9.2.4 Scenario 4: AS interconnection – Multi-AS 2 Scenario 4 is a slight modification of the topology of scenario 3. The transit path includes one more AS and the transit links decrease in capacity. This results in two contention places and multiplies the class separation effect. The results for BE only support, in AS10 and AS11 compared to identical four class support in both ASes, is not printed out, since the characteristic throughput graph in each case is found. It is however of more interest to vary the supported class sets along the AS chain. Fig. 110 Scenario 4: AS interconnection – Multi-AS 2 Fig. 111 and Fig. 112 reveal, that the order of the limited class sets to pass through does matter, even in a symmetrical topology. In Fig. 111, all four traffic classes will be taken care of in AS10 before the non-prioritized BE only forwarding in AS11 occurs. Fig. 112 takes the opposite configuration and performs worse, particularly for the TCP based traffic. 122 17.11.2009 Fig. 111 S4: CBWFQ / no priority Fig. 112 S4: no priority / CBWFQ The reason for this behaviour is the resulting traffic mix after AS10. In the first case, a high percentage of AF traffic will remain in the mix, which is then equally discriminated in AS11. In the latter case, hardly any AF traffic is contained in the mix after AS10 and gets starved out in AS11. 9.2.5 Scenario 5: AS interconnection – Multi-AS 3 Scenario 5 is again a slight modification of the topology of scenario 4. A further transit AS has been introduced accompanied with a further link capacity reduction. Fig. 113 Scenario 5: AS interconnection – Multi-AS 3 As Fig. 114 and Fig. 115 show, the advantage of late CoS bottlenecks along a forwarding path remains. However, each introduced transit AS has detrimental effects on the variation and the throughput level of rate adopting sources. However, Fig. 116 and Fig. 117 document the case, where the CoS bottlenecks would at least separate two traffic classes. The resulting traffic characteristics and throughput numbers underline the thesis’ strong request for coarse-grained inter-domain CoS support. 123 17.11.2009 Fig. 114 S5: 2x CBWFQ / no priority Fig. 115 S5: no priority / 2x CBWFQ Fig. 116 S5: 2x CBWFQ / EF&BE Fig. 117 S5: EF&BE / 2x CBWFQ 9.2.6 Scenario 6: AS interconnection – Multi-AS 4 Scenario 6 is again a slight modification of the topology of scenario 5. A further transit AS has been introduced accompanied with a further link capacity reduction. No further fundamental knowledge might be gained from this extended topology, except for the comparison of remarking and non-remarking simulation results. All scenario simulations and results so far assumed, that enqueued traffic would keep its class marking regardless of the respective class support at each relay node. Although this 124 17.11.2009 concept strongly recommends such funnel mapping and marking preservation, network operators are free to remark packets as they traverse their AS. Fig. 118 Scenario 6: AS interconnection – Multi-AS 4 The remarking behaviour will therefore be documented on 2 selected examples. It is assumed, that class mappings for enqueuing and class remarking will be applied identically by the respective CoS limited AS. Fig. 119 S6: 2 classes w/o remark. Fig. 120 S6: 2 classes with remark. Fig. 119 and Fig. 120 depict the result when either the CoS bottleneck is performing marking preservation or remarking. In the latter case, traffic classes are either upgraded (as shown with the AF traffic being remarked into EF type) or downgraded. Consecutive forwarding behaviour can therefore no longer be class type specifically applied. The situation gets worse in the best effort only support case, where all traffic passes through a single (BE) class. As clearly shown in Fig. 121 and Fig. 122, the forwarding within AS 12 applied single class funnel classification without remarking on the left and with remarking on the right. In the remarking case, consecutive transit ASes will no longer be able to distinguish the all BE marked packets. This effectively results in traditional best effort forwarding along the remaining transit path segments. 125 17.11.2009 Fig. 121 S6: 1 class w/o remarking Fig. 122 S6: 1 class with remarking 9.2.7 Scenario 7: AS interconnection – Cross-Layer Scenario 7 addresses the cross-layer marking and mapping challenge, which arises with any underlying transport networking technology. Since AS interconnection is increasingly based on Ethernet links (being IXP platforms or point-to-point links), the example will focus on the interworking of IP QoS and Ethernet QoS. Fig. 123 depicts the selected topology, where AS00 and AS20 are interconnected across an Ethernet switch. Fig. 123 Scenario 7: AS interconnection – Cross-Layer The introduction of an Ethernet model requires the selection of 10 or 100 Mbps link capacities. However, this requires an increase of EF load on the network as well. An aggregated traffic load of 2.64 Mbps VoIP, 13.2 Mbps CBR0 and 8.43 Mbps CBR1 has been chosen. Furthermore, a ‘110’ mapping has been applied for the VoIP traffic in order to conform to the IEEE voice encoding. 126 17.11.2009 Fig. 124 S7: with Ethernet QoS Fig. 125 S7: without Ethernet QoS Fig. 124 and Fig. 125 depict the resulting CoS based throughput in the traffic mix after the 4 queue CoS switch or after the BE only switch, respectively. The best effort only switch virtually destroys the prioritization for EF and AF traffic and prefers the high volume CBR traffic type. Therefore, the underlying CoS support with consistent cross-layer mapping is important for the successful overall performance. It is a constituent part of the proposed CoS concept. 9.3 Setup selection for token bucket ingress filtering Class of Service support will require class overload protection using token bucket based ingress rate limitation. Since the OMNET++ does not support token bucket filtering, the “Network Simulator 2 (ns2)” [145] has been used for these simulations. The simulator’s classification scheme differs from the one used in OMNET++ and generally refers to so called “flow IDs (fid)”. Those source-to-destination flow identifiers allow for the mapping of packets into DiffServ DSCP encodings in DiffServ enabled nodes. Such DiffServ domains are modelled by means of three nodes, the edge node performing ingress classification, the core performing CoS based enqueuing and dropping and the outgoing edge node performing CoS based dequeuing and rate limited priority scheduling. Those nodes can be abstracted into a single ingress rate limited node, when they are arranged in a single forwarding line. Fig. 126 depicts the resulting simulation topology. 127 17.11.2009 Fig. 126 Single node structure with token bucket filtering The rate limitation by means of the ns2 token bucket implementation is a combination of token bucket based metering and the resulting increased dropping probability for excess traffic and a strict rate limitation on CoS enabled enqueuing. Table 1 lists the applied parameters, whereby the queue rate is the limitation setting for each of the four traffic types. 3 Mbps link it Table 20 Simulation parameter settings All simulated packets were of 1000 byte size. Strict priority queuing with the highest queue being queue 1 was applied. 9.4 Simulation results for token bucket ingress filtering The simulation of this single node example did yield the expected rate limitation result and is documented in Fig. 127, Fig. 128, Fig. 129 and Fig. 130. The sending behaviour of the 128 17.11.2009 four sources remains the same, but the mapping groups LE into BE (4:3), then additionally AF with LE into BE (4:2) and lastly all traffic classes into BE. The legend of the graphs reads with the curves “Before TB” as the throughput observed before the token bucket limitation and “At Dest” as the throughput received at the destination. All but Class 2 have constant sending rates. The AF source is a TCP source, which always sends as fast as possible. The sending and queueing rates are deliberately chosen above the physical bottleneck link capacity in order to demonstrate the hard rate limitations (e.g. of 0.5 Mbps for EF) combined with priority based queuing and dropping. Fig. 127 Single node TB 4->4 Fig. 128 Single node TB 4->3 The start time of each source is varied to demonstrate the source’s influence. EF starts first and is limited to a “committed information rate (CIR)” of 500kbps. This slightly reduces the constant sending rate of 600kbps. It has the highest priority and as long as there are two queues available, hardly notices any surrounding traffic changes. It is a typical rate limited EF forwarding service with best forwarding quality. Secondly, a constant bit rate class of type LE is turned on with a nominal sending rate of 2 Mbps. However, token bucket limitation is applied for the selective LE queue in Fig. 127. This limits the LE traffic to just 400kbps. Thirdly, a third constant bit rate class of type BE is turned on with a nominal sending rate of 2.5 Mbps. Its token bucket limitation is set to 2.5 Mbps in the 4 class case. However, token bucket limitation is applied for the selective BE queue in Fig. 127. This limits the BE traffic to 2.2 Mbps. The bottleneck link with 3 Mbps capacity is completely used up. Only 300kbps remain for the Lower Effort class. Lastly, the TCP source starts sending with the highest achievable rate and a priority of 4. The rate limitation of 2.2 Mbps is not reached because of the congestion avoidance mechanism in TCP. LE type traffic gets starved out and BE traffic is reduced to about 800kbps. The TCP stream uses about 1.6 Mbps on average. A similar behaviour is observed in the 3 class setup, where BE and LE share the same unlimited low priority queue. Initially, LE can transmit its full 2 Mbps link load in the uncongested phase. With the BE source in place, both streams equally share the remaining 2.5 Mbps link capacity. Together with the TCP load, BE and LE share the formerly available 800kbps BE link share and the TCP traffic yields the same throughput as in the 4 class support case. 129 17.11.2009 Fig. 129 Single node TB 4->2 Fig. 130 Single node TB 4->1 Fig. 129 depicts the minimal CoS support setup, where a 500kbps rate limited high priority EF class is contrasted with the common link share of the LE, BE and AF sources. It can be seen, that the remaining 2.5 Mbps link capacity is taken up by the constant bit rate BE and LE sources in the relation of their sending rates. The congestion controlled TCP stream dies off, due to the lost prioritization. This situation is stepped up further in the single class support case depicted in Fig. 130. The EF class gains its full transmission rate in the uncongested phase, but is drastically reduced under the view of all sources being active. EF, BE and LE divide the 3 Mbps capacity in equal shares according to their sending rates. No TCP type traffic can sustain this link’s 100% congestion phase. 9.5 Summary of simulation results The simulation results of the QoS marking and forwarding behaviour as well as the functionality of token bucket ingress limitation filters clearly demonstrate the superior class of service forwarding operational quality as contrasted with the currently deployed best effort only transmission capabilities. Due to the impossible handling of arbitrarily complex Internet traffic models for this simulation effort, the concept’s coarse-grained class of service support has been applied in the modelling as well. Up to four most commonly found traffic classes have been distinguished in the setups combined with extensive parameter simulations and scheduling strategy variations. Single node as well as AS interconnection setups have been modelled, which allowed for simulations of varying class support situations in interconnected ASes. Only some few examples of the gained results have been documented in here. The complete result set of all combinations is available upon request. The concept’s expectation of sensible usage shares with matching class set support has been proven. The applicability of strict priority queuing and class based fair queuing is both valid and leads to satisfactory results. However class based fair queuing is preferred due to its configurable prevention of traffic starvation in all classes. The interconnection scenario of multiple-ASes revealed the general advantage even of consistent two class support in transit networks. CoS bottleneck simulations revealed, that the ordering of class support granularities along a forwarding path does matter. The later a merging of traffic occurs, the better. Furthermore, the advantage of cases where marking preservation is performed in CoS bottlenecks as compared with remarking cases has been demonstrated. This backs up the concept’s strong recommendation of tunnelled customer traffic transport with matched tunnel CoS support. 130 17.11.2009 The consequences of missing tunnel CoS support have been simulated and exposed. Class overload prevention will be performed by token bucket ingress filtering as specified in the second IETF draft document. Therefore, the precise limitation characteristics and some typical application scenarios have been simulated. Token bucket metering combined with prioritised queuing is a simple but powerful means for network protection and sensible rate limited class of service based forwarding of traffic. The fundamental building blocks of the new cross-domain and cross-layer coarse grained Quality of Service support concept have been successfully simulated. Given the high level of traffic aggregation across interconnection links and the current poor best effort forwarding situation, even a simple two class of service interconnection is shown to be highly beneficial for the separated transport of prioritized traffic. 131 17.11.2009 10 Concept implementation New concepts and ideas can only become Internet standards, if they are contributed to the respective working group within the “Internet Engineering Task Force (IETF)” and gain community support there. RFC 2418 [40] defines the guidelines and procedures for the working group operation. This formally defines, what is cited as an early quote by David Clark: “We reject kings, presidents and voting. We believe in rough consensus and running code”. Especially the philosophy of running code as a way to find out possibly missing subtleties in specifications and at the same time ensuring that the new specification can actually be used straight away is a fundamental building block in the IETF standardization work. Consequently, the new cross-domain and cross-layer coarse grained Quality of Service support was not only submitted to the “Inter-Domain Routing (idr)” working group of the IETF, but also implemented. Basic functionality has already been achieved and used for test runs in the University laboratories as well as with service providers. 10.1 Linux implementation The open-source operating system Linux includes the routing suite software “Quagga” [154], which includes the implementation of several routing protocols as well as local routing table management. The Border Gateway Protocol is also supported, which has been used and extended for the implementation of the new signalling concept. Fig. 131 Quagga Routing Suite structure Fig. 131 depicts the structure of the routing suite software. It consist of the central software process (daemon) “Zebra” and several processes (daemons) for the depicted routing protocols. Each process has an associated command line interface called “Virtual 132 17.11.2009 TeletYpe shell (vtysh)”. Router administrators can therefore connect to each process and issue control commands to them. All processes exchange their routing information with external peers as well as with the central Zebra daemon for node local routing information and table updates. The BGP daemon and its associated vtysh has been modified for the implementation of the concept’s new routing information exchange as well as for the required new vytsh commands for its configuration. Fig. 132, Fig. 133 and Fig. 134 give example setups for both new Extended Communities as well as for the token bucket rate limitation signalling. All configuration command create the required internal data structures, initiate the respective sending and show up in the “show running-config“ command, which displays the current router configuration. The configuration and activation of the new communities and attributes targets the so called “route-map” mechanism, which is used in such configurations for triggered actions on matched criteria. This powerful mechanism is now extended to selectively output the CoS signalling data to neighbours by attaching the respective route-map to this peering session. Fig. 132 Example setup for 4 QoS Marking Ex. Communities for IP-DiffServ Fig. 133 Example setup for a CoS Capability Ex. Community 133 17.11.2009 Fig. 134 Example setup for a CoS Parameter Attribute The modifications of the BGP daemon within the Quagga routing suite have not yet been submitted to the Quagga team for inclusion. They are still under development and testing, but have proven operational stability and functionality. Table 21 lists all newly added commands and their available parameters. A detailed description of the commands and their parameter handling will be published after the official code adoption within the Quagga software project. Chapter 11 documents the test results, which were achieved by means of this modified Linux routing software. 134 17.11.2009 Table 21 Extended command line syntax for CoS configurations 135 17.11.2009 10.2 Wireshark implementation Measurement tools are required to analyze test results and to aid the debugging process. The most widely used network analyzer tool for data communication networks is the freely available software “Wireshark” [181]. This successor of the former “ethereal” software holds a comprehensive set of protocol dissectors, which allows the user to analyze almost all types of captured data packets in a cleanly structured way and detailed to every bit of the packet’s control information. Since all programming sources of the Wireshark package are available, the new Extended Communities have been added to the dissection repository. The modifications have been submitted to the Wireshark team and were accepted for inclusion. The newest official release therefore includes the decoding functionality for the data structures. Fig. 135 depicts a screenshot of the software. The enlargement shows some examples of transmitted Extended Communities within a BGP UPDATE message. Fig. 135 Wireshark screenshot with captured Extended Communities 136 17.11.2009 10.3 Online debug form The new CoS signalling capabilities in Linux based BGP speakers is not yet available in commercial routers. Therefore, many network operators might well be able to receive those Extended Communities and attributes, but will be confronted with decimal or hexadecimal encodings of the information as shown in Fig. 136. Fig. 136 Reception example of Extended Communities in commercial routers In order to decode the received information, such operators would be forced to use the augmented Wireshark functionality. However, this is will not happen in production style setups. Therefore, a second means of decoding is provided, which accepts the un-decoded command line output and displays the decoding result. This service has been set up as online debug form and can be accessed at the following URL: http://www.bgp-qos.org/draft-knoll/decode_attributes.php . Either single encodings (e.g. “0x420:11778:3422565120”) or complete command line log files can be submitted in the online form for decoding. The result will be returned in structured table output style as shown in Fig. 137. Fig. 137 Decoding result of the online form 137 17.11.2009 11 Implementation test Service providers run large network setups and almost exclusively use commercial router equipment. The interoperability of the Linux-based concept’s implementation with such routers is therefore vital and needs to be tested. Due to the selection of BGP as signalling protocol and the reuse of Extended Communities for most of the signalling information transport allows for the interconnection of modified Linux and commercial router systems. More over, the Extended Community attribute is by definition a transitive attribute and the CoS Parameter attribute has been designed as such as well. Thus, all commercial routers of all vendors will receive and store the attributes and eventually relay them unprocessed. In practise, the latter holds only true, if the network operator has not configured the discard filter for Extended Communities and for unknown attributes. The simple interconnection of a Linux-PC with several types of Cisco routers has been tested. The establishment of a peering session, the exchange of routing information and the attachment of some of the new path attributes was successfully realized. The more challenging testing of the attribute relay between commercial routers as well as the extensive signalling of the new path attributes under DFZ routing conditions has been successfully completed and is documented below. 11.1 Test setup The intention of the test setup as shown in Fig. 138 is to learn a so called “full feed” global routing table view from a public peering, to augment this information with some CoS tagged self-originated prefixes and to relay this advertisement to a commercial Router 1. This in turn relays the full routing information, including the injected CoS signalling information, towards a second commercial Router 2. Fig. 138 Implementation test setup 138 17.11.2009 Three Linux PCs and two Cisco 2811 routers have been used in the testing arrangement. The test has been performed in test labs of independent service providers for the global Internet peering connectivity. All information exchanges were to be captured for documentation and offline analysis purposes. Wireshark was used for this task and the modified Linux PC (“themis” in the figure) was able to do the packet capture for the public interconnection link. The direct links between the Linux-PC and Router 1 as well as the one between Router 1 and Router 2 needed to be eavesdropped by means of two simple Ethernet hubs and two Wireshark equipped Linux-PCs (“leda” and “maia” in the figure). 11.2 Test result and observations The session establishment between the BGP speakers of the described test setup could be successfully realized and routing advertisements within BGP UPDATE messages were observed. The public Internet peering for global connectivity was configured to relay Internet to test setup advertisements and to filter out all locally generated or repeated advertisements. This way, all unacknowledged announcements could be suppressed. This is particularly important, since service level agreements and mutual information exchange within BGP peering sessions are of legal relevance and must be kept clear of any unwanted or uncontrolled leakage of information. Fig. 139 documents the successful reception of a full routing table feed from the public Internet along the modified Linux PC into the Router 1 BGP routing process. The routing table contained 273109 IP prefixes and consumed about 56 MB of RIB memory space. The time between session establishment and table convergence at the Linux PC came to about 6 minutes. The same time was needed for the unchanged relay operation toward Router 1 and again to Router 2. Fig. 139 Router1: show ip bgp sum – full feed In a second test series, the Linux PC started to send out route-map matched reachability UPDATE of its own networks. Some made up networks were configured and associated with a four CoS Extended communities. The signalling of the respective CoS Extended Communities was successful and could be observed in Wireshark and in the routers’ debug and statistic outputs. Fig. 140 and Fig. 141 exemplarily show the resulting memory consumption of 40 bytes for the four attached Extended Communities, which were received in one Extended Community attribute. This consumption is interesting in two ways. Firstly, the Extended Community itself is an 8 byte structure and four such structures should result in 32 bytes memory usage and secondly, no difference in consumption 139 17.11.2009 was found between 1, 10, 100 or several hundred announced prefixes with the same associated CoS Extended Communities. Fig. 140 Router 1: show ip bgp sum – single prefix with 4 communities Fig. 141 Router 1: show ip bgp sum – 10 prefixes with 4 communities Fig. 142 in turn documents the maximum tested simulation run for a single CoS Extended Community being deliberately attached to all incoming full feed announced prefixes. This forced behaviour is not conformant to the specified concept, but clearly proves the sending capability of the Linux-PC as well as the stable handling and efficient storing of this massive load of incoming CoS attributes by commercial routers. Fig. 142 all Router1: show ip bgp sum – full feed with single community attached to Further test runs have been performed, where hundreds of prefixes were associated with thousands of differing CoS Extended Communities in the connectivity advertisements. It can be stated, that all tests have successfully been passed and the resource usage analysis of this testing is discussed in chapter 11.4. 140 17.11.2009 One further observation has been made, which revealed a still unresolved BGP signalling flaw. Fig. 143 Completely processed Extended Community attribute example Fig. 143 depicts the reception of four CoS Extended Communities contained in one Extended Community attribute. However, this UPDATE message passed through a Cisco router which marked the attribute as “completely” processed. This marking flag (complete vs. partial processing) is a mandatory field associated with all BGP path attributes. The standard requests, that all BGP speakers, that receive an unknown to them transitive attribute, must relay the attribute with raised “partially processed” marking. In the Extended Community attribute case, the Cisco router obviously did not raise the partial flag because of its familiarity with Extended Community attributes as such. The unknown content, namely the new QoS Marking Extended Communities, is therefore silently relayed as completely processed. This signalling inconsistency has been acknowledged by the two major router vendors. 141 17.11.2009 11.3 Ethernet QoS support test at IXPs The new cross-domain and cross-layer coarse grained Quality of Service support concept places emphasis on the CoS interworking not only between networking domains, but also between networking layers. Since many potentially CoS capable interconnected Service Providers peer across public Internet Exchange Points, the underlying Ethernet QoS support needed to be tested as well. All major Internet Exchange Points in the world are currently not QoS enabled and switch untagged Ethernet frames. However, talks to IXP operators revealed, that they are willing to support their customers in high class peering setups and want to be prepared for it. Allowing customers to configure VLAN tagged peerings across an IXP platform is a prerequisite for QoS support. The support itself can be divided into QoS marking and marking preservation only support or QoS marking and QoS forwarding support. The latter is unlikely to be enabled soon. Fig. 144 VLAN User Priority test at DE-CIX [144] 142 17.11.2009 The IXP administration at the German Internet Exchange Point in Frankfurt was kind enough to perform the raised testing request on their cascaded platform. Two PCs, languard1 and languard4 were exchanging IEEE 802.1Q tagged Ethernet frames with configured user priority markings. It could be shown, that all marks traversed the Force10 and Foundry based switching platform unchanged. This is a fundamental building block for VLAN tunnelled and priority marked AS interconnections. IXP customers are supplied with VLAN based platform access upon request. The switch hardware of the platform is even capable of performing multi-class prioritised forwarding. Other IXPs are expected to offer high class peerings as well. However, no central database exists currently, which could guide potential peering partners to the QoS enabled peering platforms. Therefore, a new database about QoS enabled IXPs has been initiated. This registry of QoS-enabled IXPs can be found at: http://www.bgp-qos.org/qos-ixp/ . As Fig. 145 already shows, the second European IXP has also acknowledged support for VLAN tagged peering with user priority preservation. Differences are in the number of priority queues supported on the Ethernet hardware platform. Fig. 145 Major European IXPs with VLAN User Priority support 11.4 Resource usage estimates The applicability of an inter-domain concept depends on three major criteria: • simplicity to gain common understanding and usage, • scalability to large number of interconnections and routing table entries and • modesty in resource usage. The latter two are due to the facts of ever increasing Internet routes and autonomous systems. Fig. 146 and Fig. 147 document this continuing growth trend. 143 17.11.2009 Fig. 146 Active BGP entries over time [year] - [93] Fig. 147 Unique ASes over time [year] - [93] Network operators are strongly concerned about the growth rate. The situation is further intensified by AS multi-homing. Here, stub ASes connect via two or more Internet service providers to the Internet. Due to the BGP best path selection, only one homing path would be used at a time and frequent swapping between them is not aimed for and on the contrary is harmful for inter-domain routing stability. Therefore, stub-ASes tend to split their currently aggregated IP address spaces into de-aggregated ranges. This leads to larger prefix lengths and two routing table entries instead of one. This in turn results in an increase in routing table size and has led to the commonly accepted prefix length limitation policy. Prefixes of more than 24 bit length are filtered out in the BGP processing, which virtually disconnects any finer-grained IP address range. Fig. 148 shows the average rate of updated and withdrawn prefixes. Less than ten such prefix manipulations are on average to be processed by any BGP speaker in the Internet. This reveals, that UPDATE processing and UPDATE message size is of minor importance 144 17.11.2009 during time of normal operation. However, it becomes critical for the initialisation phase of BGP sessions. Here, the full routing table is exchanged, which results in considerable UPDATE message amounts and high processing loads. Fig. 148 Hourly Average of Updated and Withdrawn Prefix Rate - [93] Therefore BGP UPDATE messages were generated, which included just one prefix associated with 173 Extended Communities. This extreme message design yielded no operational flaws and no measurable processing increase. The new coarse grained CoS concept has proven to scale well, under the light of thousands of prefixes being associated with CoS signalling in the UPDATE procedure, as well as in the number of differing CoS Extended Communities being received, stored and relayed by BGP speakers. The resource estimation depends on a number of factors, which can vary widely depending on the number of providers adopting this CoS concept. However, due to the limited number of different classes being sensibly used and the fact that identical attributes are going to be stored as a single instance, the overall resource usage is expected to be rather small – especially in terms of additional memory consumption. 11.4.1 Increase in routing update information size The resource usage for the sole transmission of BGP UPDATE messages is best analyzed under worst case conditions. An upcoming BGP peer in a newly established BGP session will require the transmission of the full routing table with all prefixes and associated attributes. This situation will be taken as the starting point for the calculations, which follow below. However, it is to be noted, that such a dense full table exchange does provide the highest information packing density. As far as possible, the sender will group all prefixes with identical attribute sets into one UPDATE message for transmission. The addition of a new CoS attribute will therefore be applied to all advertised prefixes within the message as well, which results in 8 byte signalling overhead for several hundred CoS enabled prefixes. Under normal network operation, single prefixes with associated attributes will be exchanged. Such an UPDATE message of one IPv4 prefix and only the limited set of mandatory attributes (ORIGIN – 4 bytes, AS_PATH with one AS and NEXT_HOP) accounts for a message size of 32 byte. The addition of one CoS Extended Community would need the Extended Community attribute control information (3 byte) and the actual Extended Community (8 byte). 43 bytes would therefore be needed. This yields an overhead of 11 bytes to the original size, which is 34.375 %. 145 17.11.2009 Associating two CoS Extended Communities in the single prefix case requires 3+16=19 bytes overhead, which yields 59.375 % total overhead and goes down to 29.69 % overhead per CoS Extended Community. Fig. 149 depicts the resulting overhead graph. Fig. 149 CoS signalling UPDATE message overhead – single prefix case As mentioned earlier a limitation of 173 Extended Communities within an UPDATE message exists. This limitation is caused by the Ethernet MTU size of 1500 byte. Furthermore, if the same prefix is associated with 173 Extended Communities in a first UPDATE message and the same prefix is announced a second time with 173 Extended Communities in a second UPDATE message, those communities do not accumulate at the receiver, but rather are regarded as new information replacing the old one. As Fig. 150 shows, the UPDATE message with 173 Extended Communities contained 173 * 8 byte = 1384 bytes of Extended Communities, led to 1444 bytes of BGP UPDATE message size and grew to 1498 bytes Ethernet frame size due to the IP, TCP and Ethernet header information. 146 17.11.2009 Fig. 150 Wireshark screenshot with 173 Extended Communities UPDATE The signalling of CoS marking information is done transitively and globally in QoS Marking Extended Communities. The rate-limitation signalling in CoS Parameter Extended Communities, however, is of interconnection local significance (see chapter 7.3.2.3) and therefore of limited concern for resource usage estimation. The signalling of class markings in the newly defined Extended Communities is realized by the construction of single Extended Communities for each class of service and each of its technology representations. The additional signalling bytes transferred in a single update message are therefore calculated as follows. Supdate = classes * technologies * 8 byte + 3 byte To limit the expected attribute’s usage to practical scenarios, one needs to consider the following facts: • most vendors support a maximum of 8 classes (e.g. Ethernet priority, MPLS E-LSP) in their network devices, • most router interfaces support only four hardware queues, • CoS marking signalling for the IP technology does not distinguish between IP version 4 and 6, because of the common DSCP marking design and • common transport technologies for IP packets are Ethernet with or without intermediate MPLS tunnelling support. The maximum UPDATE size increase for the CoS support signalling given the above mentioned assumption for 8 traffic classes encoded within 3 networking technologies yields Supdate = 195 byte / update message. 147 17.11.2009 The exchange of the full feed routing table as shown in Fig. 142 required the sending of 95160 update messages. Under the assumption that all globally reachable prefixes would need to be associated with the 195 byte CoS signalling in their UPDATE advertisements, this would yield an increased UPDATE transfer size of additional 17.7 MB. Given the commonly used interconnection speed of 1Gbps, this results in additional 149 μs transfer delay. The coarse-grained CoS support, which is aimed for in this concept targets only four classes. Because of the intensive usage of IXPs for AS interconnection, It is expected, that CoS support signalling will potentially be limited to IP and Ethernet CoS markings. This two class / two technologies setup sums up to Supdate = 67 byte / update message. Given the current 95160 update messages, this yields a maximum transfer size of 6.1 MB. Given the current interconnection speed of 1Gbit/s, this results in 51 μs transfer delay. An UPDATE size increase for a full BGP table feed will even reduce to a size of 3.2 MB (and a respective 26,6 μs transfer delay on a gigabit interconnection link), if only IP CoS support signalling is deployed. Because of this small additional UPDATE message size increase, the more important factor in terms of resource usage and possible machine update requirement results from the memory usage estimate. 11.4.2 Increase in memory consumption with routers The storage of globally flooded QoS Marking Extended Communities is of highest concern. Each BGP speaker in the world will receive the announced CoS support by the IP prefix originating AS and store this information in its local BGP memory space. Estimating the actual memory usage is non-trivial due to the observed storage concept. Given that the memory consumption for extended community attribute storage is independent from the number of prefixes being advertised, the estimate needs to focus on the number of different attribute sets being advertised. This, however, is not only dependent on the number of classes and technologies being addressed but also on the markings and flags being conveyed in those attributes. Only those attributes, that are identical in every single bit, can be stored as a single attribute instance. Otherwise, they need to be stored separately. This leads to different estimation approaches. 1. Estimate the number of class sets and markings (possibly belonging to different ASes) and calculate the resulting memory space, taking into consideration that flags might change as well. 2. Analyze the attribute structure and multiply the combinations of each field that might independently vary its value. Taking both approaches into account, the actual estimate is the minimum value of both. For convenience, the structure of the QoS Marking Community is again depicted in Fig. 151 below. 148 17.11.2009 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0 0 P R I A 0 0| QoS Set Number|Technology Type| QoS Marking Oh| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | QoS Marking Ol| QoS Marking A |0 0 0 0 0 0 0 0| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Fig. 151 Structure of the QoS Marking Community Based on possibly varying attributes fields, the maximum (and unrealistic) number of storable binary combinations sums up as follows: • 4 varying flags bits contribute 16 combinations, • 8 bit QoS Set number contribute 256 combinations, • 6 defined technologies contribute 6 combinations, • 6 DSCP bits and 1 group bit within Marking O contribute 128 combinations and • 6 bit DSCP bits within Marking A contribute 64 combinations. Assuming that no dependencies exist between those fields, a total number of 201,326,592 combinations can theoretically exist. Since each Extended Community is of 8 byte size, up to 1536 MB of additional memory space were required. This theoretical resource estimate is prohibitive for the deployment of the concept. However, the made worst case assumptions are partially dependent and highly unlikely. The respective limitations will therefore be discussed and considered for a more realistic estimate calculation. The four flags from the Flags field indicate processing states of the respective community and might well be observed in all 16 constellations. They will not be observed for a given CoS signalling in all 16 combinations at once. Combined with the combinations of the remaining structure fields, only one of those 16 combinations can be found for a given QoS Marking Community at a given router device. That is only one instance of this Community must be stored in this device. Secondly, the QoS Set Number is the linking field between multiple QoS Marking Communities of different networking technologies. One such number is needed for the cross-layer technology linking per class. It has UPDATE message local significance and recounts with a value of zero. Given the realistic assumption of a maximum of 8 classes for the AS interconnection, this QoS Set Number will also vary between 0 and 7. Thirdly, because of the current best practice for interconnections, the Technology Type field will commonly contain the IP, Ethernet and MPLS E-LSP enumeration values. This yields a value of 3 combinations. Fourthly, Table 3 lists 21 DSCP values, which might be combined with the grouping bit 14 for AF DSCP in the QoS Marking O field. Furthermore, 3 bit priority based technologies will potentially signal up to 8 combinations in this field. The resulting number of combinations is 41. Lastly, the mentioned 21 DSCP values as well as the 8 priority encodings are most likely to be found in the Marking A field. 29 binary combinations are therefore estimated for this field. 149 17.11.2009 Taken the above assumptions into account, the following number of storable binary combinations needs to be considered: • 4 varying flags bits contribute 1 combination, • the 8 bit QoS Set number contributes 8 combinations, • the Technology Type field contributes 3 combinations, • the Marking O field contributes 41 combinations and • the Marking A contributes 29 combinations. This yields a total of 28536 combinations. Since each combination uses 8 byte memory storage, this sums up to 223 KB of additional memory. Taking one more technology (e.g. separate virtual channels per class) into account, this result would change to 57,072 combinations or 446 KB, respectively. Fig. 152 depicts the resulting memory usage estimate for up to 8 classes and four technologies. Fig. 152 Memory usage estimates for up to 8 classes and four technologies To conclude the estimate on memory usage analysis, the following two figures document real world measurement examples of BGP memory consumptions. Fig. 153 correlates the number of Extended Communities associated to a prefix and the displayed statistics about BGP memory consumption for those communities. A single step measurement has been made, which documents non-linear memory consumption for Cisco implementation specific memory allocation. The most obvious non-linearity occurs with the storage of 7 Extended Communities, which raises the memory allocation to 250 byte. The next increase happens with Extended Community 31. 150 17.11.2009 Fig. 153 Memory usage for ext. communities sent within one UPDATE message Fig. 154 Memory usage for large quantities of sent extended communities Fig. 154 documents the linear increase in memory usage under the light of thousands of stored Extended Communities. The storage of e.g. 65567 Communities consumes 533KB of memory. The test clearly demonstrated the technical feasibility and scalability of the proposed signalling solution to global scale deployment. 151 17.11.2009 12 Summary and outlook This thesis focuses on interconnections of autonomous systems with emphasis on the introduction of a currently missing class of service support. In general, this work has three main contributions. In the first part, a comprehensive compilation of quality of service support concepts with detailed network and node internal building block descriptions has been arranged, which proves the technical readiness of currently deployed devices for an inter-domain class of service based interconnection. Combined with an oral survey among major European, American and Middle East networks operators, this contribution led to the strong request for a simple, understandable and manageable concept design. In the second part, the specification of the new interdomain CoS concept has been drafted and submitted to the IETF for standardization. In the third part, simulations and implementations of vital building blocks of the concept have been made to underline its functionality and technical feasibility. Resource estimates and successful field trials provide evidence for its scalable and functioning design. 12.1 Contributions and results In particular, the following contributions have been made: • The interconnection of autonomous systems for global Internet connectivity is a critical point between network providers in technical and economical terms. Current deployments are solely based on basic public Internet Protocol interconnection only without any quality of service support. Capacity over-provisioning and network internal QoS control have been found as state of the art operation strategies. Due to the continuing fast growth of Internet traffic, the thesis forecasts rising capacity provisioning costs combined with a raised level of congestion on the interconnection links. To address this foreseeable trend, a new class of service interconnection concept of global scale has been designed. • Simplicity has been identified as most important design factor for the concept’s acceptance in the Internet community. This simplicity design goal stretches into the designed signalling structures and handling procedures as well as in the actual extent of supported traffic classes. • The importance to introduce at least two and recommendable four classes of service at AS interconnections has been stated and underlined with simulations. • Simple traffic separation as opposed to existing complex quality of service support concepts with delay, loss and jitter guarantees is strongly aspired to in order to avoid complex, costly and prohibitive deployment restrictions. • Simplicity in terms of waived quality guarantees is a prerequisite of the concept and contributes to global deployment. • The analysis of existing and possibly newly defined signalling protocols for the concept’s dissemination of CoS support information led to the selection and reuse of BGP as commonly available signalling protocol at interconnection points. • New Extended Communities and a new BGP path attribute have been designed for the required signalling of cross-domain and cross-layer CoS support information. 152 17.11.2009 • • • • • • The design of transitive relay functionality of CoS signalling via Extended Communities, as well as the provider controllable mapping of CoS support information between different networking technology CoS support concepts, is a novel principle and fundamental contribution. Elaborate simulation results on single node and AS level class of service support have exemplarily been documented within this thesis and are freely available upon request. Implementation test results have been contributed, which prove the concept’s applicability and interoperability with existing networking equipment. Resource estimates have been worked out, which revealed a negligible influence of the new CoS signalling on routing UPDATE message exchanges and moderate memory consumption within routing devices. The analysis of realistic CoS support scenarios documents the concept’s applicability in large scale. The design of the simple CoS concept does not prohibit the selective application of more complex QoS guaranteeing concepts. In fact, the concurrent deployment of the generally available CoS support combined with QoS guaranteeing setups for a limited set of interconnections or transit paths is supported. A global class-based Internet with at least 2 and recommendable 4 generally available classes of service is recommended by this new CoS concept. 12.2 Practical usage Emphasis has been placed on the practical usage of the concept. The following achievements address some important milestones for the deployment. • • • • The intellectual property right free submission of the concept’s design specification to the IETF standardization body prohibits possible patent applications. Free global deployment is aimed for and provider internal cost savings contribute to the benefit of the concept’s deployment. The implementation results within the Linux routing suite Quagga and the network protocol analyzer software Wireshark are freely available. The Wireshark extension has already been contributed to the official software release and the Quagga implementation will be submitted for inclusion in the official source tree. An online service for decoding of raw CoS signalling data has been setup and can be used at the following location: http://www.bgp-qos.org/draftknoll/decode_attributes.php Type number assignments have been granted by IANA, which already enables the public signalling of QoS Markings and CoS Capabilities in production style network operation. The concept has thereby crossed the border from laboratory confined setups into public applicability. 12.3 Outlook The current status of the new cross-domain and cross-layer coarse grained Quality of Service support concept, limits its deployment to Linux based internetworking devices. Ongoing discussions with network operators and router vendors aim for a general concept support in commercial routers. Technical feasibility has been attested by the discussion partners and deployment interest has been raised by European providers. 153 17.11.2009 Future deployment experiences and adoption requests will lead to concept and implementation refinements. To foster the concept’s deployment in production style networks, the augmentation of legacy commercial router equipment by means of an interactive Linux-based remote management mechanism is currently under development. Fig. 155 depicts the concealed CoS control of commercial border routers by an AS internal Linux-PC. The transitive design of all signalling elements ensures that the passive bidirectional signalling relay within the commercial border actually forward the signalling information to and from the Linux-PC. This PC is in charge of the CoS signalling processing and generation and simply uses the router as signalling relay. A second connection of the Linux-PC to the command line interface of the router will be used to issue the respective control commands for the configuration and activation of the router’s existing class of service support functionality. This intermediate solution will allow operators to enable inter-domain CoS support without costly software or hardware upgrades. Fig. 155 Linux remote control of existing commercial AS border router An ongoing discussion on “Network neutrality” is influencing the vendors’ support and operators’ deployment of any inter-domain quality of service enhancements. A neutral Internet operation without any service blockings, content filtering or any favouring of Internet users over other Internet users is requested. Discussions with service providers and federal network agencies revealed, that the designed CoS concept with its simple and generally applicable structure is likely to be regarded as non-discriminating and possibly omnipresent Internet enhancement. Further techno-economic studies on the cost reduction potential of the concept will need to be carried out to guide the device upgrade and CoS deployment decision process. The BGP Community based signup procedure for new services and concepts, proposed by the company Google, is briefly described in chapter 5.2. Depending on the outcome, this CoS support concept can even be used as the contractual base for inter-provider class of service support agreements. 154 17.11.2009 Bibliography [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] 3GPP, “Quality of Service (QoS) concept and architecture (Release 5)”,3GPP TS 23.107 V5.8.0, 2003. 3GPP,” Policy and charging control architecture (Release 8)”, 3GPP TS 23.203 V8.6.0, 2009. 3GPP, ”UTRA-UTRAN Long Term Evolution (LTE) and 3GPP System Architecture Evolution (SAE)”, 3GPP, 2006, [Online]. Available: ftp://ftp.3gpp.org/Inbox/2008_web_files/LTA_Paper.pdf 3GPP, “Requirements for further advancements for Evolved Universal Terrestrial Radio Access (E-UTRA) (LTE-Advanced) (Release 8)”,3GPP TR 36.913, 2009, [Online]. Available: http://www.3gpp.org/ftp/Specs/archive/36_series/36.913/ Abley, J.; Savola, P. and Neville-Neil, G., "Deprecation of Type 0 Routing Headers in IPv6", RFC 5095, IETF, 2007. Alaettinoglu, C.; Villamizar, C.; Gerich, E.; Kessens, D.; Meyer, D.; Bates, T.; Karrenberg, D., Terpstra, M., "Routing Policy Specification Language (RPSL)", RFC 2622, IETF, 1999. Amante, S., Bitar, N., Bjorkman, N., et. al., "Inter-provider Quality of Service - White paper draft 1.1", 2006, [Online]. Available: http://cfp.mit.edu/docs/interprovider-qosnov2006.pdf AMS-IX, “AMS-IX Monthly Reporting”, 2009, [Online]. Available: http://www.amsix.net/technical/stats/CUMU/ Andersson, L., Asati, R., "Multiprotocol Label Switching (MPLS) Label Stack Entry: EXP Field Renamed to Traffic Class Field", RFC 5462, IETF, 2009. Andersson, L.; Minei, I. & Thomas, B., "LDP Specification", RFC 5036, IETF, 2007. Andersson, L., Swallow, G., "The Multiprotocol Label Switching (MPLS) Working Group decision on MPLS signaling protocols", RFC 3468, IETF, 2003. Andras, V., “OMNeT++”, OMNeT Development Team, 2009, [Online]. Available: http://www.omnetpp.org Awduche, D.; Berger, L.; Gan, D.; Li, T.; Srinivasan, V. and Swallow, G., "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, IETF, 2001. Awduche, D.; Malcolm, J.; Agogbua, J.; O'Dell, M. and McManus, J., "Requirements for Traffic Engineering Over MPLS", RFC 2702, IETF, 1999. Ayyangar, A.; Kompella, K.; Vasseur, J. and Farrel, A., "Label Switched Path Stitching with Generalized Multiprotocol Label Switching Traffic Engineering (GMPLS TE)", RFC 5150, IETF, 2008. Babiarz, J.; Chan, K. and Baker, F., "Configuration Guidelines for DiffServ Service Classes", RFC 4594, IETF, 2006. Baker, F.; Polk, J. and Dolly, M., "DSCP for Capacity-Admitted Traffic", InternetDraft draft-ietf-tsvwg-admitted-realtime-dscp-05, IETF, Work in progress, 2008. Banerjea, A., Ferrari, D., et. al., "The Tenet Real-Time Protocol Suite: Design, Implementation, and Experiences", IEEE/ACM Transactions on Networking, Volume 4, Issue 1, pp. 1-10, 1996. Bates, T.; Chandra, R.; Katz, D. & Rekhter, Y., "Multiprotocol Extensions for BGP4", RFC 4760, IETF, 2007. Bates, T.; Chen, E. & Chandra, R., "BGP Route Reflection: An Alternative to Full Mesh Internal BGP (IBGP)", RFC 4456, IETF, 2006. Bates, T.; Gerich, E.; Joncheray, L.; Jouanigot, J.-M.; Karrenberg, D.; Terpstra, M. and Yu, J., "Representation of IP Routing Policies in a Routing Registry (ripe81++)", RFC 1786, IETF, 1995. 155 17.11.2009 [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] Bauschert, T., “Lecture: Data Communications”, Chemnitz University of Technology, 2008. Bellman, R. E., "Dynamic Programming", Princeton University Press, Princeton, N.J., 1957. Benmohamed, L.; Liang, C.; Naber, E.; Terzis, A., "QoS Enhancements to BGP in Support of Multiple Classes of Service", draft-liang-bgp-qos-00 (work in progress), IETF, June 2006. Berger, L., "Generalized Multi-Protocol Label Switching (GMPLS) Signaling Functional Description", RFC 3471, IETF, 2003. Berger, L., "Generalized Multi-Protocol Label Switching (GMPLS) Signaling Resource ReserVation Protocol-Traffic Engineering (RSVP-TE) Extensions", RFC 3473, IETF, 2003. Bernet, Y., "Format of the RSVP DCLASS Object", RFC 2996, IETF, 2000. Bernet, Y.; Blake, S.; Grossman, D. and Smith, A., "An Informal Management Model for Diffserv Routers", RFC 3290, IETF, 2002. Bernet, Y.; Ford, P.; Yavatkar, R.; Baker, F.; Zhang, L.; Speer, M.; Braden, R.; Davie, B.; Wroclawski, J. and Felstaine, E., "A Framework for Integrated Services Operation over Diffserv Networks", RFC 2998, IETF, 2000. Black, D., "Differentiated Services and Tunnels", RFC 2983, IETF, 2000. Black, D.; Brim, S.; Carpenter, B. and Faucheur, F. L., "Per Hop Behavior Identification Codes", RFC 3140, IETF, 2001. Blake, S.; Black, D.; Carlson, M.; Davies, E.; Wang, Z. and Weiss, W., "An Architecture for Differentiated Service", RFC 2475, IETF, 1998. Bless, R., "Dynamic Aggregation of Reservations for Internet Services", Proceedings of the Tenth International Conference on Telecommunication Systems - Modeling and Analysis (ICTSM 10), Vol. 1, pp. 26-38, Monterey, 2002, [Online]. Available: http://www.tm.uka.de/doc/2003/ictsm-daris-journal-crc-web.pdf Bless, R.; Nichols, K. and Wehrle, K., "A Lower Effort Per-Domain Behavior (PDB) for Differentiated Services", RFC 3662, IETF, 2003. Blunk, L.; Damas, J.; Parent, F. and Robachevsky, A., "Routing Policy Specification Language next generation (RPSLng)", RFC 4012, IETF, 2005. Bohge, M., Renwanz, M., “A realisitic VoIP traffic generation and evaluation tool for OMNeT++”, First International OMNeT++ Workshop, 2008, [Online]. Available: http://www.tkn.tu-berlin.de/research/omnetVoipTool/ Boucadair, M., "QoS-Enhanced Border Gateway Protocol", draft-boucadair-qosbgp-spec-01 (work in progress), IETF, July 2005. Braden, R.; Clark, D. and Shenker, S., "Integrated Services in the Internet Architecture: an Overview", RFC 1633, IETF, 1994. Braden, R.; Zhang, L.; Berson, S.; Herzog, S. and Jamin, S., "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification", RFC 2205, IETF, 1997. Bradner, S., "IETF Working Group Guidelines and Procedures", RFC 2418, IETF, 1998. Brown, M., Underwood, T., Zmijewski, E., “The Day the YouTube Died”, Renesys Corp. at MENOG 3, [Online]. Available: http://www.renesys.com/tech/presentations/pdf/menog3-youtube.pdf Callon, R., "Use of OSI IS-IS for routing in TCP/IP and dual environments", RFC 1195, IETF, 1990. Callon, R., “Email list discussion on: [NSIS] FW: I-D Action:draft-ietf-nsis-ntlp20.txt”, IETF NSIS working group email archive, 11 June 2009, [Online]. Available: http://www.ietf.org/mail-archive/web/nsis/current/msg08563.html Carpenter, B. E., “Re: [Diffserv] A question”, email discussion on DiffServ working group list, [Online]. Available: http://www.ietf.org/mailarchive/web/diffserv/current/msg04257.html 156 17.11.2009 [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] Chan, K.; Babiarz, J. and Baker, F., "Aggregation of DiffServ Service Classes", RFC 5127, IETF, 2008. Chandra, R.; Traina, P. and Li, T., "BGP Communities Attribute", RFC 1997, IETF, 1996. Chen, E., "Route Refresh Capability for BGP-4", RFC 2918, IETF, 2000. Cisco, “An Introduction to IGRP”, ID: 26825, 2005, [Online]. Available: http://www.cisco.com/en/US/tech/tk365/technologies_white_paper09186a00800c8a e1.shtml Cisco, “BGP Best Path Selection Algorithm”, ID: 13753, 2006, [Online]. Available: http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431 .shtml Cisco, “Enhanced Interior Gateway Routing Protocol”,ID: 16406, 2005, [Online]. Available: http://www.cisco.com/en/US/tech/tk365/technologies_white_paper09186a0080094c b7.shtml Cisco, “Cisco 12000 Series Internet Router Architecture: Switch Fabric”, ID: 47240, 2005, [Online]. Available: https://www.cisco.com/en/US/products/hw/routers/ps167/products_tech_note09186 a00801e1da7.shtml Cisco, “Network Infrastructure for Ensuring Predictable Business Service Delivery”, ID: C11-397769-00, 2007, [Online]. Available: http://www.cisco.com/en/US/prod/collateral/routers/ps6342/prod_white_paper0900a ecd805f62b1.html Cisco, “Network Infrastructure – Chapter 3”, ID: OL-13817-04, [Online]. Available: http://www.cisco.com/en/US/docs/voice_ip_comm/cucm/srnd/6x/netstruc.html Cisco, “Congestion Avoidance Overview”, ID: QC-75, [Online]. Available: http://www.cisco.com/en/US/docs/ios/12_0/qos/configuration/guide/qcconavd.html Cisco, “Configuring QoS”, ID: 78-13490-01, [Online]. Available: http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.1/8aew/configuratio n/guide/qos.html Cisco, “Evolving Data Center Architectures: Meet the Challenge with Cisco Nexus 5000 Series Switches”, ID: C11-473501-01, [Online]. Available: http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns783/white_p aper_c11-473501.html Clos, C., “A study of non-blocking switching networks”, Bell System Technical Journal, vol. 32 issue 2, pp. 406–424, 1953. Colitti, L., “A strategy for IPv6 adoption”, RIPE 57, October 2008, [Online]. Available: http://www.ripe.net/ripe/meetings/ripe-57/presentations/ColittiA_strategy_for_IPv6_adoption.Z8ri.pdf Cristallo, G.; Jacquenet, C., "The BGP QOS_NLRI Attribute", draft-jacquenet-bgpqos-00 (work in progress), IETF, February 2004. Davie, B.; Charny, A.; Bennet, J.; Benson, K.; Boudec, J. L.; Courtney, W.; Davari, S.; Firoiu, V. and Stiliadis, D., "An Expedited Forwarding PHB (Per-Hop Behavior)", RFC 3246, IETF, 2002. DE-CIX, “DE-CIX topology 2009”, 2009, [Online]. Available: http://www.decix.net/content/network/topology.html DE-CIX, “DE-CIX yearly traffic graph”, 2009, [Online]. Available: http://www.decix.de/content/network.html Deering, S., Hinden, R., "Internet Protocol, Version 6 (IPv6) Specification", RFC 1883, IETF, 1995. Deering, S., Hinden, R., "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, IETF, 1998. 157 17.11.2009 [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] [83] [84] [85] [86] [87] [88] Delgrossi, L. & Berger, L., "Internet Stream Protocol Version 2 (ST2) Protocol Specification - Version ST2+", RFC 1819, IETF, 1995. Demers, A.; Keshav, S.; Shenkar, S.; ”Analysis and simulation of a fair queuing algorithm”. Proceedings of SIGCOMM '89, pages 1-12, 1989. Dijkstra, E. W., “A note on two problems in connexion with graphs”, Numerische Mathematik, 1, pp. 269-271, 1959, [Online]. Available: http://wwwm3.ma.tum.de/twiki/pub/MN0506/WebHome/dijkstra.pdf Djernaes, M., Appanna, C., Ward, D., “Context updates in BGP”, draft-djernaessimple-context-update-00 (work in progress), IETF, 2006. DSL Forum, “Migration to Ethernet-Based DSL Aggregation“, DSL Forum Technical Report TR-101, 2006, [Online]. Available: http://www.broadbandforum.org/technical/download/TR-101.pdf Eardley, P., "Metering and marking behaviour of PCN-nodes", Internet-Draft draftietf-pcn-marking-behaviour-03, IETF, Work in progress, 2009. Evans, J., Filsfils, C., "Deploying IP and MPLS QoS for multiservice networks: Theory and practice", Morgan Kaufmann/Elsevier, Amsterdam, 2007. Farinacci, D.; Li, T.; Hanks, S.; Meyer, D. and Traina, P., "Generic Routing Encapsulation (GRE)", RFC 2784, IETF, 2000. Farrel, A.; Ayyangar, A. and Vasseur, J., "Inter-Domain MPLS and GMPLS Traffic Engineering -- Resource Reservation Protocol-Traffic Engineering (RSVP-TE) Extensions", RFC 5151, IETF, 2008. Farrel, A.; Vasseur, J.-P. and Ayyangar, A., "A Framework for Inter-Domain Multiprotocol Label Switching Traffic Engineering", RFC 4726, IETF, 2006. Faucheur, F. L.; Wu, L.; Davie, B.; Davari, S.; Vaananen, P.; Krishnan, R.; Cheval, P. and Heinanen, J., "Multi-Protocol Label Switching (MPLS) Support of Differentiated Services", RFC 3270, IETF, 2002. Feher, G., Nemeth, K., Maliosz, M., et.al., "Boomerang A Simple Protocol for Resource Reservation in IP Networks", IEEE RTAS, 1999. Floyd, S., Jacobson, V., "Random early detection gateways for congestion avoidance", IEEE/ACM Transactions on Networking, V.1 N.4, p. 397-413, 1993. Ford, L. R. Jr., and Fulkerson, D. R., "Flows in Networks", Princeton University Press, Princeton, N.J., 1962. Franke, K., “Lecture material: Digital Communication Networks”, Chemnitz University, 2006. Fu, X., Schulzrinne, H., Bader, A., et. al., "NSIS: A new extensible IP signaling protocol suite," IEEE Communications Magazine, vol. 43, pp. 133 - 141, 2005. Fuller, V.; Li, T.; Yu, J. and Varadhan, K., "Classless Inter-Domain Routing (CIDR): an Address Assignment and Aggregation Strategy", RFC 1519, IETF, 1993. Fuller, V., Li, T., "Classless Inter-domain Routing (CIDR): The Internet Address Assignment and Aggregation Plan", RFC 4632, IETF, 2006. Gamer, T., Scharf, M., “Realistic Simulation Environments for IP-based Networks”, First International OMNeT++ Workshop, 2008, [Online]. Available: http://doc.tm.uka.de/2008/omnet2008.pdf Golestani, S.: "A Self-Clocked Fair Queueing Scheme for Broadband Applications". Proceedings of IEEE Infocom '94, p. 636-646, 1994. Grossman, D., "New Terminology and Clarifications for Diffserv", RFC 3260, IETF, 2002. Hawkinson, J., Bates, T., "Guidelines for creation, selection, and registration of an Autonomous System (AS)", RFC 1930, IETF, 1996. Hedrick, C., "Routing Information Protocol", RFC 1058, IETF, 1988. Heinanen, J.; Baker, F.; Weiss, W. & Wroclawski, J., "Assured Forwarding PHB Group", RFC 2597, IETF, 1999. 158 17.11.2009 [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99] [100] [101] [102] [103] [104] [105] [106] Heinanen, J., Guerin, R., "A Single Rate Three Color Marker", RFC 2697, IETF, 1999. Heinanen, J., Guerin, R., "A Two Rate Three Color Marker", RFC 2698, IETF, 1999. Herzog, S.; Boyle, J.; Cohen, R.; Durham, D.; Rajan, R. & Sastry, A., "COPS usage for RSVP", RFC 2749, IETF, 2000. Hinden, R., Deering, S., "IP Version 6 Addressing Architecture", RFC 4291, IETF, 2006. Huston, G., “BGP reports”, [Online]. Available: http://bgp.potaroo.net/ Hwang, J.; Altmann, J.; Oliver, H.; Suarez, A., “Enabling dynamic market-managed QoS interconnection in the next generation internet by a modified BGP mechanism”, ICC 2002, IEEE International Conference on Communications, 2002, [Online]. Available: http://ieeexplore.ieee.org/Xplore/login.jsp?url=/iel5/7828/21517/00997325.pdf?arnu mber=997325 IANA, “BGP Extended Communities Types”, IANA Protocol Registries, [Online]. Available: http://www.iana.org/assignments/bgp-extended-communities IANA, “IANAifType-MIB”, [Online]. Available: http://www.iana.org/assignments/ianaiftype-mib IEEE, "IEEE Standard for Local and metropolitan area networks Media Access Control (MAC) Bridges", IEEE 802.1D, 2004. IEEE, "IEEE standard for local and metropolitan area networks virtual bridged local area networks", IEEE 802.1Q, p. 1-285, 2006. IEEE, "IEEE Standard for Information technology-Telecommunications and information exchange between systems-Local and metropolitan area networksSpecific requirements - Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications", IEEE Std 802.11-2007 (Revision of IEEE Std 802.11-1999) , C1-1184, 2007. IEEE, "IEEE Std 802.1ad - 2005 IEEE Standard for Local and metropolitan area networks - virtual Bridged Local Area Networks, Amendment 4: Provider Bridges", IEEE Std 802.1ad-2005 (Amendment to IEEE Std 8021Q-2005) , p. 1-60, 2006. IEEE, "IEEE Standard for Local and metropolitan area networks¿Virtual Bridged Local Area Networks Amendment 7: Provider Backbone Bridges", IEEE Std 802.1ah-2008 (Amendment to IEEE Std 802.1Q-2005) , C1-109, 2008. IEEE, "IEEE Standards for Local and Metropolitan Area Networks: Supplements to Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications - Specification for 802.3 Full Duplex Operation and Physical Layer Specification for 100 Mb/s Operation on Two Pairs of Category 3 or Better Balanced Twisted Pair Cable (100BASE-T2)", IEEE Std 802.3x-1997 and IEEE Std 802.3y-1997 (Supplement to ISO/IEC 8802-3: 1996; ANSI/IEEE Std 802.3, 1996 Edition) , p. 1-324, 1997. IEEE, “IEEE 802 Tutorial: Data Center Bridging”, IEEE 802, 2007, [Online]. Available: http://www.ieee802.org/802_tutorials/07-November/Data-CenterBridging-Tutorial-Nov-2007-v2.pdf IEEE, ” Virtual Bridged Local Area Networks — Amendment: Congestion Notification”, IEEE P802.1Qau/D2.1, 2009. IEEE, "IEEE Standard for Local and metropolitan area networks Part 16: Air Interface for Broadband Wireless Access Systems", IEEE Std 802.16-2009 (Revision of IEEE Std 802.16-2004) , C1-2004, 2009. ISO, “Information technology – Telecommunications and information exchange between systems – Intermediate System to Intermediate System intra-domain routeing information exchange protocol for use in conjunction with the protocol for 159 17.11.2009 [107] [108] [109] [110] [111] [112] [113] [114] [115] [116] [117] [118] [119] [120] [121] [122] [123] [124] [125] [126] [127] [128] [129] providing the connectionless-mode network service (ISO 8473)”,ISO/IEC 10589:2002, second edition, 2002. ITU-T, “Traffic control and congestion control in IP based networks”, ITU-T Y.1221, 2002. ITU-T, “Network performance objectives for IP-based services”, ITU-T Y.1541, 2006. ITU-T, “NGN FG Proceedings Part II”, ITU-T NGN Focus Group, 2005. ITU-T, “One-way transmission time”, ITU-T G.114, 2003. ITU-T, “Gigabit-capable passive optical networks (GPON): General characteristics”, ITU-T G.984.1, 2008. ITU-T, “Asymmetric digital subscriber line (ADSL) transceivers”, ITU-T G.992.1, 1999. ITU-T, “Splitterless asymmetric digital subscriber line (ADSL) transceivers”, ITU-T G.992.2, 1999. ITU-T, “Asymmetric digital subscriber line transceivers 2 (ADSL2)”, ITU-T G.992.3, 2005. ITU-T, “Splitterless asymmetric digital subscriber line transceivers 2 (splitterless ADSL2)”, ITU-T G.992.4, 2002. ITU-T, “Very high speed digital subscriber line transceivers”, ITU-T G.993.1, 2004. ITU-T, “Very high speed digital subscriber line transceivers 2 (VDSL2)”, ITU-T G.993.2, 2006. ITU-T, “End-user multimedia QoS categories”, ITU-T G.1010, 2001. ITU-T, “Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs”, ITU-T P.862, 2001, [Online]. Available: http://www.itu.int/rec/TREC-P862/en Jacquenet, C.; Bourdon, G. ; Boucadair, M., "Service Automation and Dynamic Provisioning Techniques in IP/MPLS Environments (Wiley Series on Communications Networking & Distributed Systems)", Wiley, 2008. Jacobson, V.,“Differentiated Services for the Internet”, Internet2 Joint Applications/Engineering QoS Workshop, 1998, [Online]. Available: ftp://ftp.ee.lbl.gov/talks/vj-i2qos-may98.pdf Jamoussi, B.; Andersson, L.; Callon, R.; Dantu, R.; Wu, L.; Doolan, P.; Worster, T.; Feldman, N.; Fredette, A.; Girish, M.; Gray, E.; Heinanen, J.; Kilty, T. and Malis, A., "Constraint-Based LSP Setup using LDP", RFC 3212, IETF, 2002. Klein, P., Sprecher, N., “Provider Ethernet VLAN Cross Connect”, Seabridgenetworks/NSN, 2006, [Online]. Available: http://www.ieee802.org/1/files/public/docs2006/new-sprecher-vlan-xc-ieee-0106.pdf Knoll, T. M., "BGP Extended Community Attribute for QoS Marking", draft-knoll-idrqos-attribute-04 (work in progress), IETF, 2009. Knoll, T. M., "BGP Class of Service Interconnection", draft-knoll-idr-cosinterconnect-03 (work in progress), IETF, 2009. Knoll, T. M., “Flow control + priority consideration -> PRIORITY_PAUSE”, NG Ethernet Forum post, 2006. Knoll, T. M., “QoS capable Internet Exchange Points”, 2009, [Online]. Available: http://www.bgp-qos.org/qos-ixp/list.php Kompella, K., Rekhter, Y., "Label Switched Paths (LSP) Hierarchy with Generalized Multi-Protocol Label Switching (GMPLS) Traffic Engineering (TE)", RFC 4206, IETF, 2005. Lee, S., Gahng-Seop, A., Zhang, X. and Campbell, A., "INSIGNIA: An IP-Based Quality of Service Framework for Mobile Ad Hoc Networks". Journal of Parallel and Distributed Computing (Academic Press), Special issue on Wireless and Mobile Computing and Communications, Vol. 60, Number 4, pp. 374-406 April, 2000. 160 17.11.2009 [130] Malkin, G., "RIP Version 2", RFC 2453, IETF, 1998. [131] Manner, J; Fu, X., “Analysis of Existing Quality-of-Service Signaling Protocols”, RFC 4094, IETF, 2005. [132] Manner, J.; Karagiannis, G. & McDonald, A., "NSLP for Quality-of-Service Signaling", Internet-Draft draft-ietf-nsis-qos-nslp-16, IETF, Work in progress, 2008. [133] Manning, B., "Registering New BGP Attribute Types", RFC 2042, IETF, 1997. [134] Manns, D., “Simulative Untersuchung von klassenbasiertem Inter-AS IP-Forwarding mit Ethernet IXP”, Diplomarbeit – TU Chemnitz, Chemnitz, 2009. [135] Marques, P.; Sheth, N.; Raszuk, R.; Greene, B.; McPherson, D., "Dissemination of flow specification rules", draft-ietf-idr-flow-spec-09 (work in progress), IETF, May 2009. [136] Menth, M., Lehrieder, F., “Pre-Congestion Notification: Lightweight Admission Control and Flow Termination for the Future Internet”, ICC 2009, Dresden, 2009. [137] Merit, “Internet Routing Registry”, Merit Network Inc., 2009, [Online]. Available: http://www.irr.net/ [138] Mills, D., "Exterior Gateway Protocol formal specification", RFC 904, IETF, 1984. [139] Morand, P., Boucadair, M., Asgari, H., Egan, et al., “D1.4: Issues in MESCAL InterDomain QoS Delivery: Technologies, Bi-directionality, Inter-operability, and Financial Settlements”, MESCAL Consortium, 2004, Online. Available: http://www.istmescal.org/deliverables/MESCAL-D14-final-v2.pdf [140] Moy, J., "OSPF Version 2", RFC 2328, IETF, 1998. [141] Nichols, K., “An Opinionated View of the Current State of IP Differentiated Services”, UC Berkeley MIG Seminar, 1999, [Online]. Available: http://bmrc.berkeley.edu/courseware/cs298/fall99/nichols/kmn_ucbmm.pdf [142] Nichols, K.; Blake, S.; Baker, F. and Black, D., "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, IETF, 1998. [143] Nichols, K. and Carpenter, B., "Definition of Differentiated Services Per Domain Behaviors and Rules for their Specification", RFC 3086, IETF, 2001. [144] Nipper, A., “VLAN User Priority test on DE-CIX platform”, DE-CIX, 2009. [145] NS2 team, “Network Simulator 2 – ns2”, NS2 webpage, 2009, [Online]. Available: http://www.isi.edu/nsnam/ns/ [146] Ould-Brahim, H.; Fedyk, D. & Rekhter, Y., "BGP Traffic Engineering Attribute", RFC 5543, IETF, 2009. [147] Pan, P., Hahne, E. and Schulzrinne, H., "BGRP: A Tree-Based Aggregation Protocol for Inter-domain Reservations", Journal of Communications and Networks, Vol. 2, No. 2, pp. 157-167, 2000. [148] Pan, P., Schulzrinne, H.,"YESSIR: A Simple Reservation Mechanism for the Internet". Proceedings of NOSSDAV, Cambridge, UK, 1998. [149] Parekh, A.K.; Gallager, R.G.;”A generalized processor sharing approach to flow control in integrated services networks”. Proceedings of IEEE Infocom ’92, p. 915924, 1992. [150] Park, H., "Systematic QoS Class Mapping Framework for Application Requirement over Heterogeneous Networks" Telecommunications Network Strategy and Planning Symposium, Networks 2008, 2008. [151] PeeringDB, “Peering Database - PeeringDB”, PeeringDB.com, 2009, [Online]. Available: http://www.peeringdb.com [152] Perkins, C., "IP Encapsulation within IP", RFC 2003, IETF, 1996. [153] Postel, J., "Internet Protocol", RFC 791, IETF, 1981. [154] Quagga, “Quagga Routing Software Suite”, 2009, [Online]. Available: http://www.quagga.net [155] Rajahalme, J.; Conta, A.; Carpenter, B. and Deering, S., "IPv6 Flow Label Specification", RFC 3697, IETF, 2004. 161 17.11.2009 [156] Ramakrishnan, K.; Floyd, S. and Black, D., "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, IETF, 2001. [157] Rekhter, Y.; Li, T. and Hares, S., "A Border Gateway Protocol 4 (BGP-4)", RFC 4271, IETF, 2006. [158] Rice, L., “The inter-colonial telegraph station at Eucla.”, [Online]. Available: http://members.iinet.net.au/~oseagram/eucla.html [159] RIPE, “Representation of IP Routing Policies in a Routing Registry”, Réseaux IP Européens, 1994, [Online]. Available: ftp://ftp.ripe.net/ripe/docs/ripe-181.txt [160] Rosen, E.; Tappan, D.; Fedorkow, G.; Rekhter, Y.; Farinacci, D.; Li, T. and Conta, A., "MPLS Label Stack Encoding", RFC 3032, IETF, 2001. [161] Sangli, S.; Tappan, D. and Rekhter, Y., "BGP Extended Communities Attribute", RFC 4360, IETF, 2006. [162] Sara, “Peering policy for SARA (AS1126)”, SARA Computing and Networking Services, 2009, [Online]. Available: http://www.as1126.net/ [163] Seaman, M.; Smith, A.; Crawley, E. & Wroclawski, J., "Integrated Service Mappings on IEEE 802 Networks", RFC 2815, IETF, 2000. [164] Schulzrinne, H. and Stiemerling, M., "GIST: General Internet Signalling Transport", Internet-Draft draft-ietf-nsis-ntlp-20, IETF, Work in progress, 2009. [165] Schwabe, T., “IP-Netze mit Interdomain-BGP-Routing: Konvergenzverhalten, Dienstqualität und Dimensionierung”, Ph.D. dissertation, TU München, 2007. [166] Shenker, S.; Partridge, C. and Guerin, R., "Specification of Guaranteed Quality of Service", RFC 2212, IETF, 1997. [167] Smith, P., “BGP Scaling Techniques”, AfNOG workshop, 2006, [Online]. Available: http://ws.edu.isoc.org/data/2006/153397902444822943b7611/bgpscal.ppt [168] Sofia, R., Guerin, R. and Veiga, P., “SICAP, a Shared-segment Inter-domain Control Aggregation Protocol”, High Performance Switching and Routing, HPSR, Turin, 2003. [169] Spenneberg, R., "Linux-firewalls mit Iptables& Co.", Pearson Education, 2006. [170] Suter, B.; Lakshman, T.V.; Stiliadis,D.; Choudhury, A.K.; "Buffer Management Schemes for Supporting TCP in Gigabit Routers with Per-Flow Queueing". IEEE Journals in Selected Areas in Communications, 1999. [171] Suzuki, M., “Per-priority Flow Control”, IEEE 802.1 meeting Portland, 2004, [Online]. Available: http://www.ieee802.org/1/files/public/docs2004/Perpriority%20Flow%20Control1.pdf [172] Traina, P.; McPherson, D. & Scudder, J., "Autonomous System Confederations for BGP", RFC 5065, IETF, 2007. [173] Trick, U., Weber, F., "SIP, TCP/IP und Telekommunikationsnetze: Anforderungen Protokolle - Architekturen", Oldenbourg, München, 2004. [174] Verizon, “Verizon Business Policy for Settlement-Free Interconnection with Internet Networks”, Verizon Business, 2009, [Online]. Available: http://www.verizonbusiness.com/terms/peering/ [175] Villamizar, C.; Chandra, R. & Govindan, R., "BGP Route Flap Damping", RFC 2439, IETF, 1998. [176] Vohra, Q., Chen, E., "BGP Support for Four-octet AS Number Space", RFC 4893, IETF, 2007. [177] Westerlund, M., “Email list discussion on: [NSIS] GIST updated from todays IESG call”, IETF NSIS working group email archive, 9 April 2009, [Online]. Available: http://www.ietf.org/mail-archive/web/nsis/current/msg08534.html [178] Westerlund, M., “Email list discussion on: [NSIS] GIST updated from todays IESG call”, IETF NSIS working group email archive, 21 April 2009, [Online]. Available: http://www.ietf.org/mail-archive/web/nsis/current/msg08543.html [179] Wikipedia, “Eucla, Western Australia”, [Online]. Available: http://en.wikipedia.org/wiki/Eucla,_Western_Australia 162 17.11.2009 [180] Wikipedia, “Network neutrality”, [Online]. Available: http://en.wikipedia.org/wiki/Network_neutrality [181] Wireshark, “Wireshark network protocol analyzer”, Wireshark design team, 2009, [Online]. Available: http://www.wireshark.org [182] Wroclawski, J., "The Use of RSVP with IETF Integrated Services", RFC 2210, IETF, 1997. [183] Wroclawski, J., "Specification of the Controlled-Load Network Element Service", RFC 2211, IETF, 1997. [184] Yavatkar, R.; Hoffman, D.; Bernet, Y.; Baker, F. & Speer, M., "SBM (Subnet Bandwidth Manager): A Protocol for RSVP-based Admission Control over IEEE 802-style networks", RFC 2814, IETF, 2000. [185] Zhang, Z., "ExtCommunity map and carry TOS value of IP header", draft-zhang-idrbgp-extcommunity-qos-00 (work in progress), IETF, November 2005. List of Figures Fig. 1 Fig. 2 Fig. 3 Fig. 4 Fig. 5 Fig. 6 Fig. 7 Fig. 8 Fig. 9 Fig. 10 Fig. 11 Fig. 12 Fig. 13 Fig. 14 Fig. 15 Fig. 16 Fig. 17 Fig. 18 Fig. 19 Fig. 20 Fig. 21 Fig. 22 Fig. 23 Fig. 24 Fig. 25 Fig. 26 Fig. 27 Fig. 28 Fig. 29 Fig. 30 Fig. 31 Fig. 32 Fig. 33 IP version 4 datagram structure..................................................................................................................... 4 IPv4 address class system - [22] ................................................................................................................... 5 CIDR example network mask......................................................................................................................... 5 IP version 6 datagram structure..................................................................................................................... 6 Differentiated Services (DS) field in IPv4 and IPv6 datagram headers ................................................... 7 IP routing and forwarding functionality.......................................................................................................... 8 Internet routing hierarchy ................................................................................................................................ 9 Internet routing architecture.......................................................................................................................... 10 IP routing protocols – classified by applicability ........................................................................................ 10 IP routing protocols – classified by working principle ........................................................................... 12 Internet Exchange Point - IXP.................................................................................................................. 12 BGP Best Path Selection Algorithm - [49] ............................................................................................. 14 BGP message structure ............................................................................................................................ 15 BGP path attribute classification [46], [161] ........................................................................................... 15 BGP UPDATE message structure – after [157]..................................................................................... 16 BGP UPDATE message structure with Extended Community attribute ............................................ 17 BGP Route Reflector topology ................................................................................................................. 18 Autonomous System Confederations for BGP ...................................................................................... 18 IP router block diagram ............................................................................................................................. 19 IP router internal structure -> route processing ..................................................................................... 20 IP router with non-blocking fabric and virtual output queues............................................................... 22 Router internal forwarding path per hop behaviour............................................................................... 26 Drop-Tail queue dropping strategy .......................................................................................................... 27 Random Early Detection (RED) for congestion avoidance................................................................. 28 Longest Queue Drop (LQD) of virtually separated flows...................................................................... 29 Round Robin scheduling ........................................................................................................................... 30 Strict Priority scheduling............................................................................................................................ 30 Weighted Round Robin scheduling ......................................................................................................... 31 Symbolized fair queuing in an idealized GPS = Fluid-Flow Queuing ................................................. 31 Fluid-flow approximated queuing in WFQ .............................................................................................. 32 VoQ with 8 classes CoS support (scheduling and dropping) .............................................................. 33 Per hop forwarding behaviour composition in relaying nodes............................................................. 33 Leaky bucket algorithm ............................................................................................................................. 35 163 17.11.2009 Fig. 34 Fig. 35 Fig. 36 Fig. 37 Fig. 38 Fig. 39 Fig. 40 Fig. 41 Fig. 42 Fig. 43 Fig. 44 Fig. 45 Fig. 46 Fig. 47 Fig. 48 Fig. 49 Fig. 50 Fig. 51 Fig. 52 Fig. 53 Fig. 54 Fig. 55 Fig. 56 Fig. 57 Fig. 58 Fig. 59 Fig. 60 Fig. 61 Fig. 62 Fig. 63 Fig. 64 Fig. 65 Fig. 66 Fig. 67 Fig. 68 Fig. 69 Fig. 70 Fig. 71 Fig. 72 Fig. 73 Fig. 74 Fig. 75 Fig. 76 Fig. 77 Fig. 78 Fig. 79 Fig. 80 Fig. 81 Fig. 82 Fig. 83 Fig. 84 Fig. 85 Fig. 86 Fig. 87 Fig. 88 Fig. 89 Fig. 90 Fig. 91 Fig. 92 Fig. 93 Fig. 94 Fig. 95 Fig. 96 Token bucket algorithm............................................................................................................................. 36 QoS-based IP lookup variants ................................................................................................................. 37 Best Effort interconnection example........................................................................................................ 38 QoS-based forwarding interconnection example .................................................................................. 39 QoS-based path selection in BGP........................................................................................................... 40 QoS-based routing interconnection example......................................................................................... 40 Tunnelling scope ........................................................................................................................................ 42 QoS-based tunnelling interconnection example.................................................................................... 43 Differentiated Services regions, domains and nodes ........................................................................... 48 Behaviour aggregate classification and DSCP marking....................................................................... 48 Logical View of a Packet Classifier and Traffic Conditioner ................................................................ 49 PHB ÅÆ DSCP mapping......................................................................................................................... 49 PHB encoding [31] ..................................................................................................................................... 50 Encoding of Assured Forwarding PHBs ................................................................................................. 51 RSVP flow descriptor structure ................................................................................................................ 53 RSVP message flow diagram................................................................................................................... 54 RSVP support block diagram – after [39] ............................................................................................... 54 Cisco’s two RSVP operation models: IntServ and IntServ/DiffServ [53]............................................ 55 Ethernet frame format................................................................................................................................ 56 IEEE 802.1p User Priority marking in 802.1q (VLAN) tagged frames................................................ 57 VLAN Cross Connect / VLAN XC [123] .................................................................................................. 58 Q-in-Q / stacked VLAN / Provider Bridges - IEEE 802.1ad [100] ....................................................... 59 MAC-in-MAC / Provider Backbone Bridges (PBB) – IEEE 802.1ah [101]......................................... 59 Priority Flow Control [56]........................................................................................................................... 60 Congestion spreading [103]...................................................................................................................... 60 MPLS shim header structure and hierarchy usage............................................................................... 62 MPLS Label stack structure...................................................................................................................... 62 MPLS LSP signalling: contiguous, nested, stitched.............................................................................. 63 GMPLS label representations .................................................................................................................. 64 GMPLS LSP hierarchy .............................................................................................................................. 64 ATM cell structure ...................................................................................................................................... 68 Functional layering structure for the Ethernet data service [111] ....................................................... 69 AS interconnection options ....................................................................................................................... 71 DE-CIX topology 2009 [61] ....................................................................................................................... 73 Internet hierarchy ....................................................................................................................................... 74 Route Flap Dampening [167] ................................................................................................................... 79 MESCAL - Cascaded Approach [139] .................................................................................................... 79 Components of a NSIS node - [80].......................................................................................................... 83 GIST protocol change to “Experimental“ status [164].......................................................................... 83 GIST protocol objections explained by Ross Callon [43] ..................................................................... 84 PCN working principle - [136]................................................................................................................... 85 DE-CIX yearly traffic graph - [62]............................................................................................................. 86 Cross-Domain CoS marking concept...................................................................................................... 90 IANA registry for BGP Extended Community type numbers ............................................................... 91 BGP Extended Community Attribute structure with type 0x40 or 0x44.............................................. 91 Structure of the QoS Marking Community.............................................................................................. 92 CoS enabled AS interconnection example topology............................................................................. 94 QoS Marking Extended Community signalling example ...................................................................... 96 Class overload limitation concept ............................................................................................................ 97 CoS Capability Extended Community Structure.................................................................................... 98 Per-Hop-Behaviour Identification Codes implied by CoS Capability .................................................. 98 CoS Parameter Attribute structure .......................................................................................................... 99 Classification of the Mapping scope...................................................................................................... 101 User/Subscriber Service Classes Grouping - [16]............................................................................... 105 Service Class Characteristics - [16] ...................................................................................................... 106 DSCP to Service Class Mapping - [16]................................................................................................. 107 QoS Mechanisms Used for Each Service Class - [16] ....................................................................... 107 Treatment Aggregate / Service Class Performance Requirements - [45] ....................................... 108 Treatment Aggregate Behaviour - [45].................................................................................................. 109 MPLS E-LSP mapping of Treatment Aggregates - [45] ..................................................................... 109 QoS Class Mapping framework - [150]................................................................................................. 111 Mandatory L-LSP encoding rules - [75] ................................................................................................ 113 Scenario 1: single node interconnection............................................................................................... 118 164 17.11.2009 Fig. 97 Fig. 98 Fig. 99 Fig. 100 Fig. 101 Fig. 102 Fig. 103 Fig. 104 Fig. 105 Fig. 106 Fig. 107 Fig. 108 Fig. 109 Fig. 110 Fig. 111 Fig. 112 Fig. 113 Fig. 114 Fig. 115 Fig. 116 Fig. 117 Fig. 118 Fig. 119 Fig. 120 Fig. 121 Fig. 122 Fig. 123 Fig. 124 Fig. 125 Fig. 126 Fig. 127 Fig. 128 Fig. 129 Fig. 130 Fig. 131 Fig. 132 Fig. 133 Fig. 134 Fig. 135 Fig. 136 Fig. 137 Fig. 138 Fig. 139 Fig. 140 Fig. 141 Fig. 142 Fig. 143 Fig. 144 Fig. 145 Fig. 146 Fig. 147 Fig. 148 Fig. 149 Fig. 150 Fig. 151 Fig. 152 Fig. 153 Fig. 154 Fig. 155 S1: 9-f-cbwfq............................................................................................................................................. 118 S1: 9-a-no-priority .................................................................................................................................... 118 S1: 9-f-cbwfq............................................................................................................................................. 119 S1: 9-a-no-priority .................................................................................................................................... 119 S1: 9-f-strict-priority.................................................................................................................................. 119 Scenario 2: AS Interconnection – Single AS........................................................................................ 120 S2: 9-f-cbwfq............................................................................................................................................. 120 S2: 9-a-no-priority .................................................................................................................................... 120 S2: 9-b-cbwfq............................................................................................................................................ 121 S2: 9-e-cbwfq............................................................................................................................................ 121 Scenario 3: AS interconnection – Multi-AS .......................................................................................... 121 S3: 9-f-cbwfq............................................................................................................................................. 122 S3: 9-a-no-priority .................................................................................................................................... 122 Scenario 4: AS interconnection – Multi-AS 2 ....................................................................................... 122 S4: CBWFQ / no priority.......................................................................................................................... 123 S4: no priority / CBWFQ.......................................................................................................................... 123 Scenario 5: AS interconnection – Multi-AS 3 ....................................................................................... 123 S5: 2x CBWFQ / no priority .................................................................................................................... 124 S5: no priority / 2x CBWFQ .................................................................................................................... 124 S5: 2x CBWFQ / EF&BE......................................................................................................................... 124 S5: EF&BE / 2x CBWFQ......................................................................................................................... 124 Scenario 6: AS interconnection – Multi-AS 4 ....................................................................................... 125 S6: 2 classes w/o remark........................................................................................................................ 125 S6: 2 classes with remark....................................................................................................................... 125 S6: 1 class w/o remarking....................................................................................................................... 126 S6: 1 class with remarking...................................................................................................................... 126 Scenario 7: AS interconnection – Cross-Layer.................................................................................... 126 S7: with Ethernet QoS............................................................................................................................. 127 S7: without Ethernet QoS ....................................................................................................................... 127 Single node structure with token bucket filtering ................................................................................. 128 Single node TB 4->4................................................................................................................................ 129 Single node TB 4->3................................................................................................................................ 129 Single node TB 4->2................................................................................................................................ 130 Single node TB 4->1................................................................................................................................ 130 Quagga Routing Suite structure............................................................................................................. 132 Example setup for 4 QoS Marking Ex. Communities for IP-DiffServ .............................................. 133 Example setup for a CoS Capability Ex. Community.......................................................................... 133 Example setup for a CoS Parameter Attribute..................................................................................... 134 Wireshark screenshot with captured Extended Communities ........................................................... 136 Reception example of Extended Communities in commercial routers............................................. 137 Decoding result of the online form......................................................................................................... 137 Implementation test setup....................................................................................................................... 138 Router1: show ip bgp sum – full feed.................................................................................................... 139 Router 1: show ip bgp sum – single prefix with 4 communities......................................................... 140 Router 1: show ip bgp sum – 10 prefixes with 4 communities .......................................................... 140 Router1: show ip bgp sum – full feed with single community attached to all .................................. 140 Completely processed Extended Community attribute example ...................................................... 141 VLAN User Priority test at DE-CIX [144] ............................................................................................. 142 Major European IXPs with VLAN User Priority support...................................................................... 143 Active BGP entries over time [year] - [93] ............................................................................................ 144 Unique ASes over time [year] - [93] ...................................................................................................... 144 Hourly Average of Updated and Withdrawn Prefix Rate - [93].......................................................... 145 CoS signalling UPDATE message overhead – single prefix case.................................................... 146 Wireshark screenshot with 173 Extended Communities UPDATE................................................... 147 Structure of the QoS Marking Community............................................................................................ 149 Memory usage estimates for up to 8 classes and four technologies ............................................... 150 Memory usage for ext. communities sent within one UPDATE message ....................................... 151 Memory usage for large quantities of sent extended communities .................................................. 151 Linux remote control of existing commercial AS border router ......................................................... 154 165 17.11.2009 List of Tables Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Table 14 Table 15 Table 16 Table 17 Table 18 Table 19 Table 20 Table 21 Transfer demand matrix – after [79] ___________________________________________________ 46 Assured Forwarding DSCP encoding __________________________________________________ 51 Currently specified PHBs ____________________________________________________________ 52 Excerpt of IP QoS class definitions and performance objectives [108] ______________________ 56 Ethernet traffic types [97] ____________________________________________________________ 57 Mapping of traffic types to available queues [97] ________________________________________ 58 Chemnitz University applied Ethernet-priority-to-DSCP mapping __________________________ 58 UMTS QoS classes [1] ______________________________________________________________ 65 UMTS Bearer Service Attributes [1] ___________________________________________________ 66 LTE QoS class attributes [2]________________________________________________________ 67 Overview of available layer 2 and 3 quality of service classes ___________________________ 70 Technology Type Enumeration _____________________________________________________ 93 CoS Capability Attribute – binary class encoding ______________________________________ 98 Queue mapping reuse for priority mapping __________________________________________ 110 Cisco’s default CoS-to-DSCP mapping [55]__________________________________________ 112 Cisco’s default DSCP-to-CoS mapping [55]__________________________________________ 112 Chemnitz University applied CoS-to-DSCP mapping__________________________________ 112 Traffic source configuration parameters _____________________________________________ 116 Class and traffic type variations in simulations _______________________________________ 117 Simulation parameter settings _____________________________________________________ 128 Extended command line syntax for CoS configurations________________________________ 135 166 17.11.2009 Versicherung Hiermit versichere ich, dass ich die vorliegende Arbeit ohne unzulässige Hilfe Dritter und ohne Benutzung anderer als der angegebenen Hilfsmittel angefertigt habe; die aus fremden Quellen direkt oder indirekt übernommenen Gedanken sind als solche kenntlich gemacht. Bei der Auswahl und Auswertung des Materials sowie bei der Herstellung des Manuskripts habe ich Unterstützungsleistungen von folgenden Personen erhalten: Prof. Dr.-Ing. Thomas Bauschert ................................................... Prof. Dr.-Ing. habil. Klaus Franke ................................................... Simon Ehnert ................................................... Daniel Manns ................................................... Uwe Steglich ................................................... Brian Schaefer ................................................... Weitere Personen waren an der Abfassung der vorliegenden Arbeit nicht beteiligt. Die Hilfe eines Promotionsberaters habe ich nicht in Anspruch genommen. Weitere Personen haben von mir keine geldwerten Leistungen für Arbeiten erhalten, die im Zusammenhang mit dem Inhalt der vorgelegten Dissertation stehen. Die Arbeit wurde bisher weder im Inland noch im Ausland in gleicher oder ähnlicher Form einer anderen Prüfungsbehörde vorgelegt. Chemnitz, 17.11.2009 ........................................... ................................................. Ort, Datum Unterschrift 167 17.11.2009 Theses 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. The Internet has become increasingly popular in recent years and has a steadily growing user base. The resulting traffic load, especially due to rapidly increasing Internet access speeds, will lead to high traffic volumes in the core of the network. A rising usage for time and loss critical services, such as voice over IP (VoIP), video streaming (IPTV) and online gaming, across the Internet can be observed, together with high user expectations of the service quality. This will inevitably require quality of service (QoS) handling procedures in provider networks. The current Internet structure consists of about 30,000 interconnected service provider networks. Those interconnections are based on the Internet Protocol (IP) and do not distinguish the mixed traffic types within the transported traffic load. Link capacity over-provisioning is the easiest and most sustainable way to provide high quality of transmission service and will always be used. Over-provisioning of interconnection links results in low link capacity utilization and frequent speed upgrades due to the traffic growth rate. The resulting hardware upgrade cost for faster router interfaces will evolve into a financial burden for service providers, who only apply overprovisioning in their network operation. Today, service providers are already making use of the QoS concept of “Differentiated Services (DiffServ)” within their network domains - Autonomous Systems (AS). Its deployment is increasing and is expected to be universally available within ASes. The Internet’s default packet forwarding behaviour, Best Effort (BE), will not be sufficient in the future on interconnection links. AS interconnections need to support at least simple traffic separation and separate traffic queues for an enhanced interconnection quality support. The setup and operation of AS interconnections is a fundamental element in any provider’s network. Any inter-domain QoS solutions will therefore need to be simple for community acceptance. The thesis’ work identified two fundamental design requirements for a simple QoS concept. They are simplicity in design and simplicity in QoS support. QoS in this approach therefore refers to primitive traffic separation into several classes, which will experience differently prioritized forwarding behaviour in relaying nodes. Enqueueing in separate queues is thereby aspired to. The always performed link capacity over-provisioning combined with a simple traffic class separating inter-domain QoS concept will enable classbased over-provisioned interconnections. The signalling of available traffic classes is required and mutual (Service Level Agreement (SLA) based) solutions can be manually set up. However, 168 17.11.2009 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. the designed new class of service signalling procedures will inform all globally interconnected service providers about the available traffic separation support as well as automate the CoS enabled AS interconnection setup procedure. Because of the currently missing CoS support, service providers often perform multi-layer ingress classification on incoming traffic in order to make a good guess on which traffic is entering the domain. This costly classification procedure can be waived, if the new CoS signalling informs about the available class sets and provides inter-domain mapping information. Simulations have shown, that even the interconnection of differently set up CoS enabled ASes leads to a considerable increase in successfully transferred high priority traffic across a chain of several transit ASes in the path as compared to the BE only interconnections style. Talks to service providers have revealed a strong request for “Lower Effort (LE)” class of service support. De-prioritization support by means of signalled LE CoS is included in the concept’s specification. The support of BE and LE traffic the simplest recommended class set combination in the specification. The simple CoS support concept claims that the support of Expedited Forwarding (EF), one commonly used Assured Forwarding (AF) group, BE and LE traffic classes will suffice at AS interconnections for most service providers. A generally available 2 class or recommended 4 class CoS enabled global Internet, is aspired to. The advertised availability of higher priority traffic class support will potentially lead to misuse. Furthermore, high class forwarding quality can only be supplied to a limited share of the link capacity. Therefore, class-overload protection is required and will optionally be provided by the new concept. The class of service support allows for higher link utilization without noticeable service degradation. This way, the concept allows postponement of interface speed upgrades until a higher utilization threshold is crossed. The deferral is expected to deliver an easily achievable economical benefit to service providers. Quality of service support is not confined to the IP layer, but is offered on several packetized networking technologies. Multi-protocol Label Switching and Ethernet with virtual LAN (local area network) support are the two most common QoS capable tunnelling technologies for IP transport. The harmonization of IP QoS and lower layer QoS is essential and is manually cared for in the intra-domain case. Inter-domain signalling of cross-layer QoS support is a novel feature and is provided in the new CoS concept. Even upcoming tunnelled interconnection can thus be automatically harmonized. Simulations have shown that the preservation of class of service markings is vital for a successful traffic separation. ASes, which remark packets on their way through CoS domains with very limited class support virtually destroy the separation along the remaining AS forwarding chain and finer grained class sets can no longer be utilized for separation. The transparent transport of customer traffic is strongly recommended by the concept. Marking preservation is automatically achieved by traffic encapsulation and tunnelled transport. Current trends are observable, where Ethernet and MPLS based interdomain tunnelling is arising. The new CoS concept already provides the signalling means for the harmonized CoS interconnection. 169 17.11.2009 26. For the practical usage of the new concept, a Linux implementation, the implementation in the official release of the network analysing tool Wireshark and an online decoding form for decoding of raw signalling data is available. 27. Linux remote control of commercial routers via command line sessions is planned as an intermediate deployment solution of the concept with legacy routing equipment. The transitive design of the required signalling elements allows for the passive bidirectional signalling relay through existing routers without hardware or software update requirements. 28. The concept’s integration in commercial equipment is expected due to its simplicity and ease of implementation. 29. The current discussion about network neutrality reveals a fundamental objection to any traffic separation scheme. However, because of the new concept’s universal applicability and the resulting generally available CoS support to all network users, the concept is likely to be regarded as nondiscriminating. 30. Further techno-economic studies on the cost reduction potential of the concept will need to be carried out to guide the device upgrade and CoS deployment decision process. 31. A new BGP Community based signup procedure for new services and concepts has recently been proposed by the company Google. Depending on the outcome, this CoS support concept can even be used as contractual base for inter-provider class of service support agreements. 170 17.11.2009 Lebenslauf 171