* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Compensation of Asymmetrical Latency for Ethernet Clock
Survey
Document related concepts
Point-to-Point Protocol over Ethernet wikipedia , lookup
Zero-configuration networking wikipedia , lookup
Low-voltage differential signaling wikipedia , lookup
Computer network wikipedia , lookup
Internet protocol suite wikipedia , lookup
Distributed operating system wikipedia , lookup
Airborne Networking wikipedia , lookup
Network tap wikipedia , lookup
Recursive InterNetwork Architecture (RINA) wikipedia , lookup
Power over Ethernet wikipedia , lookup
Transcript
Compensation of Asymmetrical Latency for Ethernet Clock Synchronization Natasa Simanic∗ , Reinhard Exel∗ , Patrick Loschmidt∗ , Thomas Bigler∗ , and Nikolaus Kerö† ∗ Institute for Integrated Sensor Systems, Austrian Academy of Sciences, Wiener Neustadt, Austria Email: {Natasa.Simanic, Reinhard.Exel, Patrick.Loschmidt, Thomas.Bigler}@oeaw.ac.at † Oregano Systems - Design Consulting GesmbH, Vienna, Austria Email: [email protected] Abstract—Clock synchronization has become an indispensable service in most distributed systems as it allows to sort events on a common time scale and coordinate collaborative actions. With the demand for even higher synchronization accuracy, new challenges and barriers have to be tackled to fulfill these requirements. One of them, the inevitable signal propagation time between the devices, is compensated in many state-of-the-art synchronization protocols by round-trip measurements, neglecting any form of delay asymmetry of the communication link. In this paper, we analyze the impact of asymmetry in networks based on the physical layer of copper-based Ethernet and compare different approaches on how to mitigate the impact of asymmetry. We propose a non-invasive system performing asymmetry measurements on a link basis and show that such a system can integrate into existing synchronization solutions. Keywords: physical layer clock synchronization, asymmetry mitigation, Ethernet, high accuracy I. I NTRODUCTION Clock synchronization is a concept originating from the observation that internal computers’ clocks are differing and from the necessity to have them aligned. As the desired synchronization accuracy level rises, so does the complexity of the clock synchronization and the time distribution mechanisms. A simple clock synchronization scheme is found in radiocontrolled clocks. For example, the DCF77 [1] synchronization mechanism is based on long-wave radio transmitters broadcasting timing information sourced from an atomic clock to receivers at a distance greater than 2000 km. These socalled radio-controlled clocks adjust their time according to the received timing information. As no propagation time compensation is performed, the clocks with larger distance to the transmitter are delayed in a range of few milliseconds. However, for most applications in our daily life this offset is practically irrelevant. In order to achieve and maintain higher accuracy synchronization, the measurement and compensation of the propagation time of the signal is compulsory. State-of-the-art synchronization protocols such as Network Time Protocol (NTP) [2] or Precision Time Protocol (PTP, IEEE 1588) [3] perform round-trip measurements to mitigate the impact of the inevitable signal propagation time. By exchanging messages This work was partly financed by the province of Lower Austria, the European Regional Development Fund, the FIT-IT project Ætas under contract 825904 and Oregano Systems Design and Consulting. 978-1-61284-893-8/11/$26.00 ©2011 IEEE 0 T0 t2 t3 Node B Node A 0 t1 TAB TBA t4 Fig. 1. A typical round-trip delay measurement scheme for state-of-the-art clock synchronization protocols and timestamping them accurately, first the propagation time and then also the offset between the clocks are estimated and compensated for. Current implementations are able to reach an accuracy in sub-millisecond (NTP) or sub-microsecond range (PTP). The limiting factor for achieving even higher accuracy with these kind of measurements is the fact of message’s latency not being symmetric, i. e. transmit and receive paths do not have the same network transit time. Typically, this fact is neglected either to simplify the synchronization process or because the impact of asymmetry can be considered low with respect to the requested accuracy limit. Asymmetry does not impair the precision, but if it is ignored it creates a clock offset. As synchronization is getting even better, thanks to hardware support, the offset caused by asymmetry is a significant factor for the accuracy of a clock synchronization system. The asymmetry problem is described in more detail and narrowed down to a physical level in section II. Sources and causes are stated and its scale and influence are explained. In section III, the state-of-the-art methods of dealing with the asymmetry compensation are described. An insight into the ideas of dynamical methods of dealing with this problem in 100 Base-TX Ethernet is presented in section IV. In section V we present possible solutions how to integrate the proposed method into current synchronization networks. Finally, section VI summarizes the findings and gives an outlook for future work in this area. II. D ELAY A SYMMETRY IN C LOCK S YNCHRONIZATION To identify the problem of asymmetry, recall how state-ofthe-art communication protocols, such as PTP, exchange peer delay measurement messages for round-trip measurements, as highlighted in figure 1. The clock offset and the mean link delay of messages exchanged between the nodes are typically calculated out of the timestamps collected in node A (i. e. the node being synchronized). For the sake of simplicity we assume the clock rate in both nodes is identical and only the offset T0 between the nodes exits. The offset can be estimated based on the propagation delays from node A to node B, TAB , and reverse, TBA . These delays can be expressed by: TAB = (t2 − T0 ) − t1 , (1) TBA = t4 − (t3 − T0 ), (2) where t1 and t3 are the message sending time instances, and t2 , t4 are the message receiving time instances, in the corresponding nodes. Regardless if the link is asymmetric (TAB 6= TBA ) or not, the offset estimation, T0 , is calculated as follows: t2 − t1 − (t4 − t3 ) TAB − TBA − , (3) 2 2 where TAB − TBA represents the channel asymmetry. In the round-trip based latency measurements, the asymmetry is neglected, which results in a clock synchronization offset. In addition, if node A and B have a significant frequency offset, the frequency offset must be compensated before the measurement in order to scale the timespan t3 − t2 to the timescale of node A. Communication networks can be classified as asymmetric for variety of reasons. Some characteristic that can influence it are bandwidth (as in ADSL networks), media access (as in cellular networks), link bit error rate (networks with different link quality), or the way of data transfer (i. e. uni- or bidirectionality) [4]. In this paper we will keep focus on the asymmetry occurring on the physical level only. There, the asymmetry itself may originate from different sources: one is the physical property of the transmission channel, the other the design and implementation of the physical layer device. In the following a summary of the sources in copper-based Ethernet is given. T0 = A. PHY Device Induced Asymmetry For practically every sophisticated modulation scheme, the receiver module in a PHY device is a complex design, which is performing parameter estimation methods on the sampled signal and traceback decoding to optimally decode the received signal. Hence, the receiver generates a much higher processing delay than the transmitter of the same PHY, rendering the device per se asymmetric. Note that this in fact would not be an issue if all PHY devices within the system were to have the same receive-to-transmit asymmetry. Since every transmitter is paired to a receiver on the opposite side, the asymmetry would therefore ideally cancel out. But in a realistic network this is not the case, considering that receive and transmit delay values are vendor specific, as shown by the measurements in [5]. In case of 100 Mbit/s Ethernet, another peculiarity of many commercial-off-the-shelf (COTS) PHYs contributes significantly to the asymmetry. 100 Base-TX transmits with 125 MBaud/s, which is recovered as a 125 MHz clock at the receiver. As the Media Independent Interface (MII) operates at 25 MHz, the receive clock must be divided by a factor of five. If the clock division is not aligned to the symbol boundaries, the receive processing may be variable by multiples (0 up to 4) of 8 ns [6]. This creates an asymmetry of up to 32 ns per link. The measurements presented in [7], using a COTS physical layer device and a 100 m CAT 5 cable, have shown that the link delay between the nodes remains constant once the physical connection is established, but varies with each new re-establishment. Although the delay is constant for an active link, it still induces asymmetry as the delay at which the PHY locks in is unknown. B. Cable Induced Asymmetry The cabling between the PHYs also contributes to the asymmetry. In most wired communication systems, either the receive and transmit paths are split by different frequencies or dedicated wires in each direction are used to allow for simple separation of receive and transmit information. The former case is typical for single fiber optical communication and only generates asymmetry if the fiber is dispersive (different propagation speeds for different wavelengths). The latter case is typical for, e. g., 100 BASE-TX Ethernet where a full-duplex link uses two cable pairs, one for each direction. As the twisted pairs can have a different twist rate and be of a different length, it is obvious that the pairs may have a different delay. According to the ANSI/TIA/EIA 568-B cabling standards [8], a standard compliant CAT 5 cable may have a delay skew of up to 50 ns for a 100 m cable. III. S TATE O F T HE A RT As already outlined, sub-nanosecond accuracy in clock synchronization requires a solution able to measure and mitigate link delay asymmetries. As every link may introduce different delays in each direction, the asymmetry should be ideally compensated on a link basis. The more common end-to-end measurements would theoretically yield the same outcome, if it can be ensured that the measurement is actually performed on the same wire pair. This would imply using a simple halfduplex communication or crossing out all links connecting the nodes. Hence, in complex network topologies the links between core switches or routers must be altered whenever an asymmetry measurement is performed. As a side effect, the communication of the entire network is impaired during this time. With the second version of PTP, IEEE 1588-2008 [3], a step forward is indirectly made by enabling asymmetry compensation. Aiming for the simpler reconfiguration after network topology change, this protocol enabled the delay to be measured on a per link basis, using peer-to-peer transparent clocks (P2P TC). In addition, the protocol reserves the delayAsymmetry field for the asymmetry value. Yet, how the asymmetry is measured is not defined within the protocol. Hence, the PTP is in principle capable to deal with asymmetry if the value is sourced from an external measurement. The pitfall in measuring the asymmetry is that it cannot be measured within the system if all the available communication links are not mandatory symmetric. Algorithmic approaches like the ones presented in [9] are not feasible. In their particular approach the master must know the asymmetry and adjust its inter-packet gap exactly to this value. In practice, the value of the asymmetry is not known to either side of the link. From a more general perspective, asymmetry can be solved rather easily: it just requires an unbiased external observer monitoring the instant of a common event among all nodes, e. g. a 1 PPS pulse. As installing an observer infrastructure for the asymmetry issue only is in most cases not a tractable solution, we focus on solutions able to measure and compensate for asymmetry within the communication system. The success for asymmetry measurements is not only bound by the abilities to make the link (permanently or temporarily) symmetric, but also by the ability to draw accurate timestamps. If the timestamp accuracy is worse than the expected asymmetry, then the asymmetry compensation might in fact increase the offset. A. Manual Calibration The easiest and simplest solutions to mitigate the cable asymmetry is to manually measure and calibrate all cables used within the network. Despite the simplicity of this approach there are a number of drawbacks: the measurement itself is inconvenient if the cables are already installed within the system. Depending on the measurement method, the length delay measurement tool must either connect to both ends of the cable under test or perform a reflectometric measurement with an open or shorted termination. Labor costs for performing such measurements are considerable. Even if the link delays are all correctly measured, the method is rather error-prone as every slight change in the network requires a re-measurement. Consider a swapped cable in a patch panel or just a device which erroneously selects the auto-cross feature in a PHY changing the cable pairs without notification to the user. As asymmetry changes are not detectable within the system any deliberate or accidental modification of the network may potentially change the delays and therefore create a synchronization bias. B. Communication Line Swapping Since delay measurements on the same line for both communication directions are a precondition for adequate asymmetry compensation, twisted pair line exchange is a reasonable consideration. There are several measurement scenarios possible. China Mobile, a company owning the world’s largest mobile telecommunication network, operates a large heterogeneous backhaul network with many different kinds of physical links, such as optical fibres and copper-based connections. Accurate synchronization is a major key in telecommunication networks to ensure the quality of the network for the subscribers. As the constraints in terms of accuracy are getting even tighter, two solutions measuring asymmetry are considered to be implemented in the core network using PTP [10]: 1) Line Swapping Method – Half-duplex: The first method proposes setting the receivers and transmitters to half-duplex mode, forcing the traffic through the same physical line and measuring the delay precisely in both directions. After the measurement, the link is reverted to full-duplex mode again and the traffic flows conventionally, as shown in the figure 2a. If the measurement is performed periodically and not only during start-up, the communication link is interrupted and causing connection timeouts or congestion. 2) Line Swapping Method – Line Exchange: The second mechanism is based on exchanging the transmission and reception lines, either manually or automatically, after the first set of the PTP timestamps is exchanged. The protocol is then run again, with the exchanged lines, for the second timestamps set, as depicted in figure 2b. With the information of both round-trip measurements, it is possible to calculate the difference of the delays in both directions and finally the oneway delays. This method as well suffers from interrupted links and therefore the same restrictions for dynamic measurements apply as for the half-duplex approach. Both, half-duplex operation or line-swapping in Ethernet can be triggered by the PHY management interface (MDIO). IV. DYNAMIC A SYMMETRY C OMPENSATION In case of a unidirectional link, for any form of asymmetry measurement it is mandatory to swap the direction of the link to measure both directions of exactly the same wire or wire pair. If this is only done at startup of the network, the boot period gets extended. In the general case, where both sides of the link are equipped with local, independent oscillators, the frequency skew has to be compensated first. Depending on the network type, this may take from several seconds up to several minutes. For networks with variable delay (e. g. due to diurnal delay variations), it is desirable to measure the asymmetry during operation. However, a swap of transmission lines imposes a discontinuity in data transfer. A. Line Swapping Method: Line Iteration As an extension to the previous methods, we suggest the possibility of adding one hot-spare communication link between two nodes (figure 2c). In case an asymmetry measurement is performed, the spare line is used to allow for uninterrupted communication. As the spare pair constantly transmits data, the clock of the far side of the link is locally available and frequency skew estimation is not required. The method can be extended to more than two links increasing the efficiency of this approach. If there are n total links, n − 1 can be physically used, hence the efficiency is n−1 n . Approach Advantage Disadvantage Manual calibration Line swapping HD Line exchange Dedicated in-system comp. accurate simplicity simplicity continuous conn. cost, error-prone interrupted conn. interrupted conn. special HW Complexity Accuracy low low low medium medium – high low low high TABLE I C OMPARISON OF THE DIFFERENT APPROACHES Node A Node B (a) Line swapping approach; latency measurement over half-duplex communication Node A Node B (b) Line swapping approach; full-duplex communication, measuring by exchanging the two cable lines Node A Node B (c) Line swapping approach; exchanging the cable lines and measuring on the additional line set Node A Node B (d) Dedicated compensation system, Ætas; same media used, clock synchronization by out-of-band signaling Fig. 2. Approaches overview; the dotted arrows signify the data paths used for the clock synchronization measurements B. Dedicated In-System Compensation The approach considered in framework of the Ætas project assumes that syntonization, i. e. common clock frequency distribution in the network, is executed by the physical layer of Ethernet. In a nutshell, a receiver recovers the transmitter’s clock and provides it as an output. This approach is similar to the one of Synchronous Ethernet. Removing the inevitable offset between the nodes, i. e. phase alignment or synchronization, is performed through a periodic, automatic one-way delay measurements on the link using synchronization messages similar to standard PTP. The innovation of this approach is that the synchronization between the link partners is done by an out-of-band communication scheme, orthogonal to Ethernet, creating two logical links. One of them is used for the normal user traffic and is driven by a standard PHY. The other logical link is created by a custom IC dealing with synchronization only, while using the same physical cable pair. Ethernet traffic is in this case not interrupted with synchronization packets, since they are sent over a separate channel (as shown in figure 2d) and fed to the dedicated clock synchronization hardware. This is accomplished by applying a modulation scheme to the synchronization data which is highly orthogonal to the Multilevel Transmit3 (MLT-3) encoding of 100 Base-TX Ethernet. Considered modulation schemes are Direct Sequence Spread Spectrum (DSSS) and m-ary Ternary Orthogonal Keying (MTOK) due to their configurable low cross-correlation properties. As the PAM5 modulation of GBit/s Ethernet is very similar to MLT3, the modulation can be designed to be orthogonal to 10, 100 and 1000 MBit/s and therefore forward compatible with GBit/s Ethernet. More detail about the generic structure is given in our previous paper [11]. The clock synchronizing link can perform line swapping during operation without affecting the standard data communication. Hence, asymmetry measurements are possible without interruption of the user communication similar to the approach with a spare link. The out-of-band link can also be used to transfer synchronization messages, thus enabling a complete synchronization through this link. All the techniques for asymmetry measurements considered in section III and section IV are summarized and results are presented in the table I. The main benefits and drawbacks are listed, as well as complexity of implementation and accuracy of achieved compensation in case of dynamic asymmetry. Low refers to tens of ns, medium to the ns-range and high to sub-ns accuracy per link. V. Æ TAS I NTEGRATION AND C OMPATIBILITY WITHIN C LOCK S YNCHRONIZATION S YSTEMS Strictly speaking, the Ætas approach is a proprietary one. However, the demand for low jitter in clock synchronization systems requires a break with traditional approaches. For instance, transparent clocks in switches are, in a strict sense, not allowed to modify data as they are not gateways. In about the same way the Ætas approach violates the IEEE 802.3 standard. To address this possible breaches with the standard, we consider some scenarios to clarify that our approach can Fig. 3. Connection of Ætas equipped nodes; entire synchronization protocol performed by the Ætas hardware module Ætas switch VCO MII Switching controller Ætas Port 1 Port 2 Ætas Node 1 Fig. 4. Ætas Ætas Node 2 Ætas Port 3 Ætas Node 3 Communication over an Ætas equipped switch be integrated into existing solutions without being entirely proprietary. In the figure 3, the simplest use case is shown, where two Ætas devices communicate on a peer-to-peer basis. The Ætas modules consist of an integrated PHY, which is responsible for the communication over the Ethernet, the Ætas channel, and additional integrated parts, like a hardware timestamper and a microcontroller for the protocol processing. Furthermore, different inputs (e. g. a trigger input) and outputs (e. g., a pulse per second (PPS) output, a trigger output and a configurable clock output) are available. The two Ætas supplied nodes in figure 3 are able to communicate and synchronize without any help of PTP, neither hardware nor software. The dotted line outlines that a logical channel with an orthogonal modulation scheme is used for the asymmetry compensation on the same cable pair as Ethernet. Placing Ætas in a bigger frame, figure 4 demonstrates the scenario where the the integration of Ætas functionality into a regular Ethernet switch is shown, performing frame switching on layer 2 (data link) of the OSI model. In this scenario an Ætas switch and three Ætas-enabled devices are linked. Compared to a regular switch, switching for the logical asymmetry compensation channel has additionally to be done on the physical layer (layer 1 of the OSI model). Here, the main components of the switch include Ætas modules for every port and a switching controller, which are mutually linked using standardized MII. To be independent of the selected switching controller, the asymmetry information is stored in additional registers of the Ætas PHY and can be read out by the MDIO interface of the MII. This way the Ætas modules can be used like standard PHYs. As the goal is to build a synchronous network, one of the Ætas devices has to be elected as master which consequently provides the timebase for the overall Ætas network. The election of the master is done via a hierarchical concept similar to stratum in NTP, where the devices belong to a fixed stratum class and the switch has to run a master selection algorithm to obtain the most suitable connected device. Stating that node 3 was selected as master, its clock is then recovered in the associated module on port 3 of the switch. The clock frequency and phase then have to be distributed to all other Ætas modules inside the switch and finally to node 1 and 2. To enable this, node 3 drives the voltage-controlled oscillator (VCO) in a way to achieve phase lock to the master node 3. The VCO clock is then fed as the 25 MHz reference clock to all Ætas modules (represented with the dotted line in figure 4). Another option is to directly use the derived receive clock from the master port. This has the drawback that an outage of the master device may lead to a possible clock outage on the switch and a failure of all switch ports. Hence, this approach would need a fallback clock generator on the switch and a clock monitoring circuit. Still this might cause an abrupt change of frequency and phase, as these clocks are not synchronized. The use of a VCO addresses these issues. A master failure breaks the synchronization loop with the local VCO, but if the guidance for the VCO is handed over to the local clock within a reasonable time span, the synchronization impairment can be limited. The serial connection between the Ætas modules in the switch (solid line) is required to exchange clock synchronization information like the selected master port of the switch or delays and asymmetries. Backward compatibility with IEEE 1588 is considered, as shown in figure 5, for a scenario where only one node is equipped with an Ætas module, and another one is able to use PTP compatible hardware timestamping (or even software timestamping only). The system is still able to provide clock synchronization, but in this case the Ætas node operates like an ordinary PTP node and has to include the PTP stack in the host system. The usage of PTP is not restricted to this simple peer-to-peer case, but can also be implemented in the Ætas switch (figure 4). To achieve decent accuracy, a PTP enhanced switching controller is required. It is extended by a PTP protocol engine to provide transparent and boundary clock We propose an out-of-band signaling method, using the same cable pair as Ethernet, able to measure asymmetry and perform all clock synchronization related tasks. The advantage of this approach is that it can measure the asymmetry during operation, enabling nanosecond accurate clock synchronization, while still staying standard compliant. In addition, we outline that such a system seamlessly integrates into existing clock distribution solutions (IEEE 1588), even in case of network topologies consisting of multiple cascaded switches. Future research in this area includes engineering the out-ofband communication method to achieve the desired timestamping accuracy and finally the integration of the proposed system. R EFERENCES Fig. 5. Backward compatibility with PTP features. Some vendors (e. g. Zarlink [12]) already provide such controllers, natively supporting the IEEE 1588 protocol. As the Ætas devices are connected point-to-point and therefore do not need packet switching, it induces a certain amount of processing overhead when using the User Datagram Protocol (UDP), the underlying protocol of PTP. Consequently, a simpler approach is to use the layer 2 PTP protocol, standardized in Annex F of IEEE 1588-2008 [3]. This protocol directly operates on Ethernet frames in a point-to-point way and can thus be implemented without the usage of complex addressing schemes. In fact, it is also possible to implement this protocol on the hardware level, so that the host system can just read out the current time of day (TOD) from the Ætas devices. VI. C ONCLUSION Due to the introduction of hardware timestamping, clock synchronization systems are getting more and more accurate. While compensation for the propagation delay can be considered as sufficient for achieving accuracies below 100 ns in Ethernet networks, accuracies in the nanosecond range demand a method able to measure and compensate for the asymmetry. Cable induced asymmetry is in particular difficult to tackle, as the same cable pair must be measured in both directions. [1] Linum Software GmbH, “Alles über das DCF77 Signal,” May 2011. [Online]. Available: http://www.dcf77.de/ [2] D. Mills, J. Martin, J. Burbank, and W. Kasch, Network Time Protocol Version 4: Protocol and Algorithms Specification, Internet Engineering Task Force (IETF) Std., Jun. 2010. [Online]. Available: http://tools.ietf.org/html/rfc5905 [3] IEEE 1588 (tm) 2.1 Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems. IEEE, March 2006. [4] W. Al-Khatib and K. Gunavathi, “A Novel Mechanism to Improve Performance of TCP Protocol over Asymmetric Networks,” The International Arab Journal of Information Technology, vol. 5, no. 1, pp. 66–74, Jan. 2008. [5] T. Müller, A. Ockert, and H. Weibel, “PHYs and Symetrical Propagation Delay,” in Conference on IEEE 1588 Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems, 2004. [6] D. Rosselot, “DP83848 and DP83849 100MB Data Latency,” Application Note 1507, Aug 2006. [7] P. Loschmidt, “On Enhanced Clock Synchronization Performance Through Dedicated Ethernet Hardware Support,” Ph.D. dissertation, Vienna University of Technology, Vienna, Austria, Dec. 2010. [8] Commercial Building Telecommunications Cabling Standard 568-B, TIA/EIA Std. [9] S. Lv, Y. Lu, and Y. Ji, “An Enhanced IEEE 1588 Time Synchronization for Asymmetric Communication Link in Packet Transport Network,” in IEEE Communications Letters, vol. 14, no. 8. IEEE Press Piscataway, NJ, USA, Aug. 2010, pp. 764–766. [10] L. Huang, “Compensation for Asymmetry of Physical Line,” Mar 2011, 802.1 AVB, 201103 IEEE 802 plenary. [11] R. Exel, G. Gaderer, and N. Kerö, “Physical Layer Ethernet Clock Synchronization,” Dec. 2010. [12] Zarlink Semiconductor Inc., ZL30310 - Combined Synchronous Ethernet and IEEE1588 Network Synchronization - Datasheet, June 2009.