Download Wide-Area Ethernet Network Configuration for System

Georgia Tech Protective Relay Conference, April 2012
Wide-Area Ethernet Network Configuration for System Protection Messaging
Jun Wen
Craig Hammond
Southern California Edison Company
Pomona, CA
Eric A. Udren
Quanta Technology, LLC
Pittsburgh, PA
Introduction
Southern California Edison Company (SCE) is now designing and installing one of the world's most
extensive high-speed wide area protection systems - the Centralized Remedial Action Scheme (CRAS) [1]. C-RAS uses protective relays installed in transmission substations across the SCE
service territory to monitor critical transmission line flows and other electrical measurements, as
well as relay or breaker operations that remove lines from service and might trigger rapid
transmission system collapse. These monitoring relays transmit high speed data to a pair of
redundant central controller arrays using IEC 61850 GOOSE messages over a wide area Ethernet
network (WAN) comprised of dual-redundant T1 and Ethernet data links. The central controller
arrays decide how to remediate a line loss within milliseconds of receiving the line trip message,
and trip loads or generation to maintain system stability using WAN links, GOOSE messaging, and
mitigation relays at shedding substation sites.
As opposed to conventional special-purpose RAS installations, C-RAS performs holistic protective
actions based on the central controller's communications-based view of the entire SCE system.
SCE programs in these controllers the C-RAS decision logic for dozens, and eventually hundreds,
of contingencies. System planners can readily update the programmed logic to handle new
contingencies as they arise.
Critical to the operation of C-RAS are the Layer 3 WAN paths carrying the GOOSE messages.
These are fully as critical to protecting the SCE transmission grid as pilot relaying channels for fault
protection. Since GOOSE over an Ethernet WAN is just now being investigated by the industry,
the SCE C-RAS technical team has been conducting successive steps of analysis and testing of this
messaging design. The paper describes data rate analysis, actual tests of GOOSE messaging
through routers and T1 links with heavy traffic flow, and experience with configuration of routers
to optimize GOOSE reliability.
As the industry begins its journey of applying mission critical protection functions over Ethernet
WAN infrastructure, this practical experience shows how one critical application is achieving
successful performance.
1
Overview of SCE C-RAS Project Requirements and System Architecture
Southern California Edison (SCE) is the third largest Investor Owned Utility (IOU) in the United
States. It serves power to more than 13 million people in an area of about 50,000 square miles in
the southern part of state of California, excepting cities of Los Angeles and San Diego.
The basic architecture of SCE’s network includes a major load center surrounded by long lines to
remote generation or to other utilities. In order to deliver power from the remote locations to the
load center under all circumstances, the network requires redundancy to transmit the power when an
outage of an import transmission line occurs. Where this redundancy is not available, a Remedial
Action Scheme (RAS) is required to handle such a line loss by tripping excessive generation or load
to maintain system stability and reliability.
R
R
R
R
R
R
R
R
R
R
R
R
R
R
R
R
R
Figure 1.
Existing RASs on SCE transmission grid
A RAS, also called a Special Protection Scheme (SPS), is an automatic protection system designed
to detect abnormal or predetermined system conditions, and take corrective actions other than
and/or in addition to the isolation of faulted components to maintain system reliability. It used to be
a rare situation for SCE that the tripping of generation using a RAS was preferable to new
transmission construction. However, since deregulation, customer reliability-of-service obligations
were decoupled from generation providers. With the recent cost of building new transmission lines
2
and the difficulty of siting them within a reasonable time frame, RAS has become an economical
and timely choice. As of today, SCE has 17 RASs, and most are deployed over major import
transmission corridors, as indicated in Figure 1.
After several cycles of abundance and shortage of energy resources following deregulation,
California established a Renewable Portfolio Standard (RPS) requirement for IOUs in 2002 that
required 33% of the utilities’ retail sales come from renewable resources by the year 2020 [12]. As
a result, over the last few years, grid interconnection requests from new renewable resources have
escalated dramatically. In order to enable the interconnection of the new generation, there will be a
proliferation of new RASs in the next few years. Figure 2 shows the growth of RAS deployments
over the past four decades in SCE and a prediction of new RASs in the next decade. It is clear from
this chart that the increase in the number of RASs is not linear over time, and the increase in the
current decade will be greater than ever, bringing great challenges for RAS implementation and
operation [13].
40
36
Number of RAS Additions
35
30
25
20
15
12
10
5
3
3
1
0
1971-1980 1981-1990 1991-2000 2001-2010 2011-2020
Figure 2.
Statistics and prediction of RAS deployments in SCE
The existing isolated custom RAS technology used at SCE won’t be able to accommodate future
system needs because of its several key deficiencies:
1. Each existing RAS operates in an isolated environment without having information about
broader system conditions, including the actions and arming status of other RASs within the
same region. The lack of information could result in uncoordinated operations, and this will
become a more severe problem when there are more individual RASs in service.
2. The logic controllers are dispersed across the service territory. Managing and upgrading
numerous remotely located distributed RASs requires travel to the physical location of the
logic controller, which is typically located in the relay room of the substation that is most
central to the RAS. The staff maintenance and travel time will become an issue with the
proliferation of new RASs.
3
Centralization of the control logic is desirable from both operational and maintenance perspectives
in light of the above limitations. SCE is now actively pursuing development of a Centralized-RAS
(C-RAS) system, which, in comparison to the existing RASs that work in local isolated
environments, will collect data from all the major substations across the entire service territory and
process all the logic at the control center, in lieu of relay rooms in substations.
Figure 3 gives a high-level overview of the C-RAS system architecture. The system is fully
redundant with duplicated A and B subsystems operating in parallel. Each A or B subsystem will
have its own central controller system, monitoring relays, mitigation relays, and telecommunication
circuits. The central controller for each A or B subsystem is designed with triple redundancy (trimode redundant or TMR) and installed in secure and geographically separated locations: Grid
Control Center (GCC) and Alternate Grid Control Center (AGCC). There will be approximately
100 substations, most of which are 500kV and 230kV substations, to be equipped with monitoring
relays or mitigation relays. Each substation will have two sets of relays, one for C-RAS A, and the
other for C-RAS B. Each substation will have two redundant and diversely routed
telecommunication circuits with at least T-1 capability, respectively to GCC and AGCC. Between
GCC and AGCC, there will be two redundant and diversely routed Gigabit Ethernet links to
exchange System A and System B information coming from the substations.
Figure 3.
High-level overview of C-RAS architecture
Relays at the substations are either monitoring or mitigating relays. Monitoring relays report
loading of critical lines to the central controllers every few seconds. They also report trips of these
lines – normally due to relay action – within milliseconds so that the controller can implement a
strategy to mitigate the resulting overload on the remaining lines and preserve system stability.
Mitigating relays at substations or generating locations receive control commands from the central
4
controllers to shed loads or generation. The relays in a substation have their communications tied
together in System A and System B isolated communications networks which are interfaced to SCE
system-wide communications infrastructure as described in detail in upcoming sections. A
particular substation may have both monitoring and mitigating relays. However, in a typical lineloss situation, the central controller calls for mitigation action at substations far from the one from
which the line loss report comes.
C-RAS is essentially a Wide Area Monitoring, Protection, Automation, and Control (WAMPAC)
system, in which the communication design is the key to reliable operation. The next several
sections will describe the communications architecture, protocol services, and reliability design for
the communications infrastructure in the C-RAS system environment. The use of Ethernet has led
the authors to investigate the design and configuration of communications equipment and facilities
to achieve the high-speed mission critical requirements of C-RAS.
Overview of Ethernet Wide Area Network Use
Required communications connectivity
The prior section described how C-RAS will eventually tie together 100 or more substations for
monitoring and mitigation with redundant central controllers. The information exchange will use
Ethernet communications paths. At each substation, dual redundant Ethernet local area networks
(LANs) will connect monitoring and mitigating relays to a pair of redundant substation Ethernet
routers, which in turn interface with isolated, separately routed, redundant SCE wide area
telecommunications paths. At the GCC and AGCC, more Ethernet routers interface these wide area
paths to System A and System B Ethernet LANs. Three tri-mode redundant central controller
processors connect to each of these LANs, along with other servers and secured interfaces to
support operators and information sharing between C-RAS and the SCE organization of users and
support personnel.
The only sharing between isolated central controllers A and B is via the bidirectional crosslink
between the control centers. The crosslink traffic from A to B consists solely of the aggregated
incoming values from all the substations tied to LAN A, made available to central controller B so it
can remain in service for situations of maintenance or of failure of a substation source somewhere
in System A. The crosslink from control center B to A serves the same backup substation data
sharing function in a symmetric way.
SCE communications infrastructure to support C-RAS requirements
The wide area networking is carried out over SCE’s network of SONET paths operating on an
optical fiber ring serving many major transmission substations, with fiber branches to other
substations. There are some remote locations without fiber service – SCE provides redundant
5
digital microwave channels to these sites. While SCE has a strong data communications backbone,
the deployment of C-RAS is leading to massive upgrading or addition of new SONET and digital
communications infrastructure capacity not previously required between substations and control
center locations.
All the redundant links between substations and the control center systems are implemented with
TDM connections operating with at least DS1/T1 capacity (1.544 Mb/s). Substations with large
relay counts are considered for multiple DS1 channel bandwidth allocation. The bidirectional
crosslinks between Control Center A and Control Center B are Gigabit (1 Gbps) Ethernet
connections.
The C-RAS substation LANs and the control center LANs carry Ethernet packets of specific types
we will discuss. However, we note that the network communications, mostly SONET, are all time
division multiplexed (TDM) formats, which operate in a deterministic or preordered data format
and do not inherently carry Ethernet message packets. Fortunately, readily available Ethernet
routers for substations and for the GCC and AGCC networks have interfaces for T1 wide area
network paths, and handle the conversion of Ethernet packets to and from the TDM network
format. Since routers are typically available to support multiple types of wide area connections,
SCE has the option of eventually connecting a wide area Ethernet-based network, sometimes called
an IP network (for Layer 3 Internet Protocol on Ethernet) for all of its wide area data services in lieu
of TDM links and rings. With either of these wide area networking approaches, the link with
terminal routers is able to convey every type of Ethernet packet, with some selective handling based
on configuration (setting parameters) of the routers. In general, Ethernet services and packet types
can be mixed on the TDM link just as they can be for a direct Ethernet connection.
Choice of Ethernet messaging
As explained in the introduction, the very need for C-RAS is driven by the dynamically growing
number of critical loading situations on the SCE transmission system. SCE has taken this
dramatically new design approach to achieve the flexibility of central programming in adding or
modifying large numbers of RASs that must operate across the system, as well as to achieve holistic
coordinated behavior of so many RAS calculations running in parallel.
The use of Ethernet networking provides inherent flexibility and scalability of the C-RAS network.
As new monitoring and mitigating connections are added, communications of data is achieved by
one pair of new relay connections to substation networks. Even new substations are added to the
wide area scheme by connecting one pair of isolated T1 network connections to ports on the
existing control center routers. While configuration of the entire network and its data flows is
unavoidably a large task, Ethernet’s convenient physical connections, flexible data flows, and
configuration with standardized or automated tools and databases greatly simplifies the task. Any
solution based on fixed-configuration data frames, protocols, and processors would require a great
deal more hand configuration work and expense for the initial installation and for each change.
6
Ethernet messaging with properly designed application protocols also supports certain types of
modifications to the system even as it continues in service – critical because C-RAS enables
unavoidable heavy transmission system loading, and thus cannot be routinely be shut down for
upgrades and modifications once it is commissioned.
Because of the total volume of data flowing in and out of the central controllers, the complexity of
many RAS analytics or computations, and the required speed of response, the central controller
processors need high computational capacity. The C-RAS design uses data-center grade server
blades with vastly more processing power than familiar substation IEDs. These server blades are
designed to exchange and process large data volumes via Ethernet network connections; serialcommunicating devices would struggle with such a load. The controllers are able to subscribe to
the full volume of substation IEC 61850 GOOSE traffic published on the network in parallel, as
described in the next subsection. Each central controller processor is able to receive or subscribe to
all incoming field data on the LAN, and perform its own processing without individual data
connections for each controller.
Use of IEC 61850 GOOSE messaging
The mission-critical messaging for RAS triggers (usually line breaker trips) and for mitigation trip
commands is IEC 61850-8-1 GOOSE messaging [7] as described in [2] and many other references
on IEC 61850. The simple design of GOOSE, applied directly on top of only standard Ethernet
physical and network layers, allows reliable publication and recognition of protection information
in milliseconds on a LAN – not practical with familiar 7-layer internet protocol stacks. Another
major benefit of GOOSE is that message publishers and subscribers (substation relays and central
controllers) can be configured according to the IEC 61850 XML-based substation configuration
language (SCL) through a standard tool-based process as described in IEC 61850-6 [6]. In these
tools, application requirements and connections or topology are entered by the engineer in userfriendly formats, along with information on the configuration capabilities of the relays and
controllers connected to the network. The engineering process of IEC 61850-6 generates
configuration files that are loaded into the relays and controllers. The tool-based process aims to
replace the laborious and error-prone point-by-point manual configuration of information to be
exchanged among devices on the network.
GOOSE is a multicast layer 2 messaging service originally intended for use only within a LAN
environment, and having no built-in facility for routing of GOOSE messages over a wide area
network. In C-RAS, router configuration is used to transfer the GOOSE messages between
substations and control center LANs, as explained further below.
GOOSE use by relays and controllers
Monitoring and mitigating relays publish IEC 61850-8-1 GOOSE messages. The central controller
arrays subscribe to all of the GOOSE packet streams from all of the substations. GOOSE Ethernet
packets from monitoring relays carry payload data bits representing the states of circuit breakers
7
(reporting an opening due to a relay trip) for critically loaded lines. Other bits may indicate the
relay operation that triggers a trip (to speed up the reporting of the change), test modes, relay
maintenance states, alarms, and other status indicators. The same relays publish other GOOSE
packets with analog line loading values, required by central controllers to determine when to arm
particular remedial actions that must be carried out if that line is suddenly lost.
Mitigation relays also publish GOOSE packets for the control centers. Status bits indicate the state
of load breakers, as well as feedback of receipt of trip requests from central controllers. Load or
generation analog values may also be published by mitigation relays – the central controllers can
use these values to preplan an adequate load shedding or generation shedding solution, to be carried
out when a critically loaded line trips.
Central controllers publish GOOSE messages to which relays in substations subscribe. The most
critical of these are for speedy mitigation actions such as load shedding trips or generation trips by
targeted mitigation relays, as well as feedback of test or status states initiated at the substation or
requested by the central controller.
All GOOSE message packets convey information specified in IEC 61850-8-1 packet specifications
that are important for system security and monitoring – source identifier, packet time tag, packet
sequence number, time the packet is allowed to live, quality of information, test mode, and revision
number of the current relay setting configuration (ConfRev) among others.
In the case of C-RAS, a separate administrative network connection through a gateway on the
substation LAN, not shown in Figure 3, provides access for communications with relays in
substations for TCP/IP application traffic such as relay configuration software, or uploading of
event and oscillographic records after operations. This traffic thus does not consume any
bandwidth required by the mission-critical GOOSE packet traffic on the redundant T1 C-RASspecific links. The gateway also provides additional security for controlling access to relays on the
substation LANs.
Role of Ethernet routers
At every substation, each of LAN A and LAN B includes an Ethernet router with combined and
integrated (or separate) Ethernet switch. The Ethernet switch functionality provides multiple ports
for the relays in the substations to connect together in a single LAN – the relays can send packets to
each other and to the router function. Each substation router has a single connection for the T1
WAN link, and connects the relatively small group of relays operating in the hostile electrical
environment of the substation. All LAN connections use optical fibers, and the switches and router
are hardened for substation installation – they comply with IEEE 1613 [4] and IEC 61850-3 [5]
environmental standards for interference immunity, temperature range, operating conditions, and
power supply.
8
Control center routers, by contrast, are data-center types that have a large number of WAN ports
and are designed to handle far larger volumes of data than the substation router/switches. They
include sophisticated processing and management of network packets and services. They are
installed in the protected environment of the control center, and are not designed for the hostile
environment of a substation.
The principal functions of the routers at both ends include:
•
Tunneling GOOSE messages between the substation and the control center LAN
IEC Technical Report 61850-90-1, “Use of IEC 61850 for the communication between substations”
[8], Section 8.2.2 describes GOOSE communications between two substations (or any two
physically separated sites) as an application of a transparent tunnel of relatively high bandwidth that
connects the two LANs into one large functional LAN. In this case, the pair of routers converting
packets to T1 TDM format and back provide such a tunnel. The tunnel can covey any traffic of
types it is configured for – in this case GOOSE messages, as well as TCP/IP traffic for relay event
data collection.
Published and multicast 61850 GOOSE messages have no routing layers or destination addresses.
The router subscribes to the GOOSE messages of interest on its LAN, wraps them in a transport
layer, and sends them over the WAN link to the other router which unpacks and republishes the
GOOSE message, as though it had been published right on the receiving network. The only major
difference is that the bridging process introduces unavoidable time delay for processing and for
transport over the T1 link.
•
Firewalls and Encryption
C-RAS relays and controllers are located in physically-secured sites and protected against
inappropriate access; the routers permit only limited access to LANs through firewalls with
management of data flow types and directions. To further secure C-RAS communications between
physically secured sites in substations and control centers, all traffic between the routers is
encrypted. Encryption and decryption processing by routers takes extra milliseconds at each end.
For C-RAS the maximum time budget for routing, encryption, and link transport delay in one
direction is 19 ms. Recent router products stay within this time budget if configured properly for
heavy loads – see test results at the end of this paper. The technology of router design is advancing
rapidly and newer generations will be faster.
System response speed and GOOSE message loading
C-RAS must respond to line trips in no more than 50 ms from breaker opening to initiating the
tripping of load/generator shedding breakers. This includes 38 ms already budgeted for GOOSE
communications (19 ms in each direction) plus all of the relay and controller processing.
9
During quiescent times, GOOSE messages are continuously streamed at a relatively low rate (one
packet every 1 to 2 s), to monitor the functioning of the communications paths between relays and
controllers, and also to provide state reports needed by any new device that wakes up on the
network.
However, when a breaker trips, the monitoring relay watching that breaker must generate a new
GOOSE packet within milliseconds that reports the state change to the central controller array. The
IEC 61850 standard [7] and the description in [2] illustrate how the publishing relay will typically
send more than just one quick message reporting the change – it sends a burst of repeated GOOSE
messages within 4 to 8 ms, only gradually slowing back down to a heartbeat rate like 1 message per
second again.
As a result, the network is very lightly loaded during calm times, but a substation event can
generate a flurry of GOOSE packets that fill packet queues in routers, as the packets wait for their
turns to be encrypted and sent along the T1 link that operates at a fixed and limited speed. GOOSE
packets that are late getting into the queue may be delayed by many milliseconds, depending on
how the routers handle the queues.
It is important that C-RAS perform with adequate speed in situations where many power system
events happen at once or in close succession. In such situations many GOOSE message packets are
generated around the same time. This could be from an unexpected multiple line trips; in California
there is particular concern with an earthquake that causes multiple line trips within seconds. The
architects of C-RAS have modeled the production by monitoring relays of GOOSE messages in
these stress situations, along with the network data loading they produce, to help in developing the
network configuration strategies described in the next section.
Networking Technology Overview
We explained how IEC 61850 GOOSE messaging is a logical choice for exchanging breaker states,
analog load flow values, trip commands, and system status points between monitoring or mitigating
relays in substations and the dual redundant central controller arrays. However, the
communications of high volumes of GOOSE message traffic between substations and the central
controller arrays through secure encrypting routers and T1 links had not been attempted in any
publicized installation in the past. SCE needed to configure the communications and networking
equipment to achieve the following performance specifications:
• Latency – for critical line loss situations, C-RAS must initiate mitigation breaker tripping
within 50 ms of the opening of the breaker on a critical monitored transmission line in a
heavily loaded corridor. Allowing for relay and central controller response times, the
GOOSE uplinks and downlinks could not take longer than 19 ms in either direction. This
upper limit includes encryption/decryption of Ethernet packets, and delay of GOOSE
10
messages through the T1 bandwidth limited link when a burst of GOOSE from multiple
relays fills the router queues with Ethernet packets to transfer.
•
Packet loss – if a flood of GOOSE packets triggered by a major event overflows any
message queue in a router or switch, there is risk that a critical GOOSE message may be
lost. The authors developed models to predict rates of data packet accumulation for
substation configurations with GOOSE burst events queuing packets in the routers and
switches, and the packet delays in these situations.
•
Scalability – for each of the two isolated redundant System A and System B LANs in one
substation, there could be up to 12 monitoring and/or mitigating relays, each of which can
send sudden GOOSE bursts. 6 relays is a typical complement. C-RAS central controller
arrays and their Ethernet networks eventually support router connections from 100 or more
such substations. All networks and Ethernet equipment – notably hardened routers at
substations and high capacity routers at the control centers – must be sized and validated for
the worst case traffic generated by a catastrophic system-wide event.
As explained above, 61850 GOOSE is based on a layer 2 multicast frame, which is intended for use
only on a local area network (LAN). This multicast frame floods to every live network port within
the LAN environment. GOOSE messages are best effort publisher-subscriber transmissions,
meaning that there is no confirmation that a published frame has been received by intended
subscribers. To help ensure delivery of a frame, publishers send multiple copies of a critical
message reporting a change at a decaying rate just in case one is lost: 4ms, 8ms, 16ms, 50 ms etc.
This results in a very “chatty” network.
By default, this traffic will not be forwarded by layer 3 routers because the traffic does not have a
layer 3 header. In fact, a router’s reason for existence is to prevent the forwarding of this type of
traffic. Consequently, for a router to transport GOOSE messages over a layer 3 wide area network
(WAN) special configuration is required.
Each router has a long list of configuration settings which impacts its ability to meet the
performance specifications for this specialized application. The most important among these are
discussed in the following.
Use of VLANs
If we were to configure switches and routers to bridge network connectivity for layer 2 GOOSE
among many substations and the control centers, and use GOOSE messages, every message from
every relay would be published to every other relay in the remote substations even if those relays
had no need to hear the traffic. This is an extreme waste of network bandwidth and it puts
unnecessary processing overhead on networked IEDs. In C-RAS the need is to get relays to talk
primarily with central controllers. To accomplish this it is helpful to put GOOSE messages into
controllable network segments through the use of VLANs (virtualized LANs). This creates multiple
11
logical LANs that selectively include various specific paths or parts of the full local and wide area
networking arrangement. In this application, each substation can have a unique VLAN
identification and association.
A layer 2 GOOSE packet contains a standard Ethernet field defined in the IEEE 802.1q subpart of
the full Ethernet standards. This field, sometimes called the 802.1q tag, contains a VLAN identifier
(one of up to 4096 VLANs) and an 8-level priority tag (more discussion later). Relays and
controllers have the ability to communicate using IEEE 802.1q tagging. The Ethernet switch
functions (discrete switches, or switch functions within switch/router combined units) can sort
messages to eliminate the unnecessary traffic flows. Each redundant System A and System B LAN
in a substation has unique dedicated VLANs which contain the monitoring and mitigation relays.
Controllers on the control center LAN for GCC System A or AGCC System B need to subscribe to
messages from System A or System B in all substations. The control center switches merge the
VLANs from all substations onto the control center LAN, without allowing them to pass from one
substation link to another. The controller itself must have applications that can subscribe to VLANs
from all the substations at once. When a controller sends GOOSE to particular mitigating relays, it
uses only the required VLANs, again limiting the GOOSE propagation to the substation LANs
where action will be taken. This VLAN control keeps GOOSE messages from flowing to any place
where they are not needed.
Router transport options for GOOSE
By default GOOSE messages are stopped at a router interface. For GOOSE messages to be
transported over the WAN a layer 2 transport technologies must be employed. These technologies
either bridge or tunnel the layer 2 GOOSE frames over the WAN. Bridging and tunneling are
techniques by which routers wrap and unwrap GOOSE packets at each end of the WAN link. Each
of the following standards-defined techniques for transporting GOOSE over a layer 3 WAN has
pros and cons in terms of latency, scalability, security, and quality of service (QoS) management for
the GOOSE transport mechanism:
1. EoMPLS (Ethernet over MPLS) as defined in Standard RFC 4448 of the Internet
Engineering Task Force has the least amount of overhead and is typically faster to transport
as compared to the layer 2 tunneling protocols, which apply layer 3 headers. This transport
method does not apply a layer 3 header so it is both difficult to encrypt and more
challenging to apply quality of service policies.
2. L2TPv2 (layer 2 tunneling protocol version 2) and L2TPv3 (layer 2 tunneling protocol
version 3) as defined in Standards RFC 2661 and RFC 3931 of the Internet Engineering
Task Force, respectively, are comparable. Although they have slightly more overhead than
EoMPLS, they have much simpler configuration. They are also easier for the router to
encrypt and are more manageable in terms of quality of service policies.
12
3. VPLS (virtual private LAN services) is not supported on any hardened equipment for
substation use at the time of this writing, but it could be a powerful transport mechanism if
ever integrated.
These are defined industry-standard services which must be provided by the router suppliers as
configuration settings in routers selected for the application at both ends of the WAN links. (1) and
(2) are point to point tunneling solutions, meaning they can connect a substation to a control center,
but not connect two substations and a control center over the same tunnel. VPLS, by contrast,
allows for multipoint connections, but this technology is typically only available on high end
routers and has yet to be offered in a hardened substation router platform.
In the next section, we summarize results of lab tests on selected transport services.
Priority and quality of service (QoS)
Queuing delays are a significant problem for the C-RAS network. One technique to control
queuing delays is to separate out different types of data and give priority to critical data through the
use of quality of service (QoS) policies. This is especially critical for low bandwidth WAN
interfaces such as T1, which are easily congested during major power system events. Relays and
controllers use the IEEE 802.1q tag priority field described above to tag critical GOOSE messages
with a high priority class of service (CoS) value from among 8 priority tag values ranging from 0
(lowest class) to 7 (highest class). Less important GOOSE messages can be tagged with a lower
class of service value. This allows critical data to receive preferential treatment by jumping to the
front of the router’s packet sending queue when the network is congested. The ultimate link
bandwidth is fixed, so if QoS gives better treatment to one data type, another data type is receiving
lower priority and greater delays and may be dropped when the buffer fills.
It is important to ensure that some high priority queue access is allocated for routing protocol traffic
that routers use continuously to maintain the state of their WAN connection. This routing protocol
traffic is small in volume, and unrelated to the C-RAS GOOSE application traffic, but is critical if
the link between routers is to continue in operation. If this isn’t considered, it is possible to bring
the communications path down during congestion because the routers’ keep-alive packets will fail
to be exchanged. QoS policy can allocate a specified portion of the available bandwidth for this
traffic that must never be pushed aside.
During testing documented below we noticed that when a circuit becomes completely congested,
delays can exceed the 19ms one way transport time budget limit even with QoS. These excessive
delays were resolved with traffic shaping as described after the test results.
A T1 router interface has a serialization delay as packets go through the transmit ring and are placed
onto the WAN media. The larger the packet is, the longer the delay. GOOSE messages are
typically 200 to 300 bytes long so they can be serialized quickly if they don’t get stuck behind a
larger 1500 byte packet of some other function. Due to this challenge of limited bandwidth, the
13
network design also splits off administrative traffic from mission-critical GOOSE traffic. GOOSE
traffic is transported over the dedicated C-RAS network paths on which these descriptions have
focused, while administrative TCP/IP traffic such as configuration changes, firmware upgrades,
event data gathering, and configuration archiving are carried out over a separate IT-grade multiprotocol label switching (MPLS) enabled network. This physical separation of non-GOOSE traffic
helps ensure the primary C-RAS network operates without interference.
Security
To ensure that the C-RAS network is secure data traffic is encrypted over the WAN link for data
confidentiality using IP Secure (IPsec), an IT-standard encryption specification. IPsec also helps
provide data integrity by preventing a man in the middle attack over the WAN path.
Using a separate administrative network path to reduce unwanted traffic on the GOOSE links also
provides barriers to enhance network security design. A gateway at each substation controls and
authenticates communications coming in from the administrative network. Access is controlled to
each of the relays and controllers through the use of access control lists, which limits access to
network segments. Access controls also prevent unauthorized access to the routers and switches.
In addition, two routers which are to communicate must exchange authorized encrypted keys when
they negotiate to establish neighbor adjacencies – in other words, to exchange information on
routing tables and routes between them.
Performance management
Application functions monitor the latency of GOOSE messages (via time tag checking) and keep
track of outages for specific GOOSE sources. This data is compiled for management reporting of
the performance statistics for communications of GOOSE messages in both directions under all
conditions over time, and the availability of individual communications links.
Router/switch selection
Hardened routers and switches, which comply with IEEE 1613 [4] and IEC 61850-3 [5]
environmental standards, are used in each substation. This hardware has field replaceable redundant
power supplies that support both high and low voltage inputs, with dc battery powering capability.
Since the data centers are controlled environments, we do not require hardened equipment in these
locations. This control center hardware provides high network performance and is extremely
scalable. Although this equipment doesn’t need to be hardened, it still requires redundancy.
Transport Service Test Results
The SCE IT team performed laboratory tests to evaluate performance of various network
technologies. This helped us to select the best-fit technology and allowed us to identify any gaps in
the overall solution.
14
The team knew from the outset that QoS and IPSec were going to be required so these were
constants in our tests. Transport options, segmentation of GOOSE, spanning tree variations
(network integrity detection and path failure recovery), and routing protocols were the variables.
The tests typically consisted of the following sequences:
1.
2.
3.
4.
5.
6.
GOOSE tunneling without advanced features.
GOOSE tunneling with QoS.
GOOSE tunneling with QoS and IPSec without any network load.
GOOSE tunneling with QoS and IPSec during minor network load.
GOOSE tunneling with QoS and IPSec during heavy congestion.
GOOSE tunneling without QoS or IPSec during heavy congestion.
A laboratory test packet generator was used to generate traffic and simulate congestion on the
network. Latency performance was measured by the relays, which examined time tags and were
connected to a GPS clock for synchronization. In addition, traffic captures were taken with an
analyzer to see how long it took for packets to traverse the network.
It is important to note that the tests below focus on the Ethernet network components – especially
routers with simulated T1 connection – relay and controller processing adds to these times. Also,
network latency over real T1 circuits is slightly higher due to propagation delays. Table 1 shows
the test results when using Ethernet over MPLS (EoMPLS) as the WAN transport protocol; Tables
2 and 3 show results for L2TPv2 and v3 respectively. Note that EoMPLS does not provide the
ability to encrypt data via IPSec so there are no results for those tests. In all test results, shaded
boxes indicate delays exceeding the 38 ms (19 ms * 2 for round trip) budget.
Event 1
Test
ID
Transport
Type
Testing Features
Event 2
Event 3
Event 4
Event 5
Event 6
All time values are round trip times
GOOSE Transport Only
1
EoMPLS
2
EoMPLS
3
4
5
6.25ms
6.26ms
6.25ms
6.25ms
6.25ms
6.25ms
GOOSE Transport + QoS
(No Load)
6.25ms
6.25ms
6.24ms
6.25ms
6.25ms
6.25ms
EoMPLS
GOOSE Transport + QoS
(Minor Load)
8.33ms
8.33ms
8.34ms
8.33ms
8.33ms
8.33ms
EoMPLS
GOOSE Transport + QoS
(Heavy Congestion)
31.2ms
27.0ms
6.25ms
6.25ms
22.9ms
31.2ms
EoMPLS
GOOSE Transport
Without QoS (Heavy
Congestion)
272.9ms
233.3ms
45.8ms
6.25ms
52.09ms
85.4ms
Table 1 - EoMPLS latency test results
15
Event 1
Test
ID
Transport
Type
Testing Features
Event 2
Event 3
Event 4
Event 5
Event 6
All time values are round trip times
GOOSE Transport Only
6
L2TPv2
7
L2TPv2
8
9
10
11
8.33ms
8.33ms
8.33ms
10.4ms
8.33ms
8.33ms
GOOSE Transport + QoS
(no load)
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
10.4ms
L2TPv2
GOOSE Transport + QoS
+ IPSEC (no load)
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
L2TPv2
GOOSE Transport + QoS
+ IPSEC (Minor load)
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
L2TPv2
GOOSE Transport + QoS
+ IPSEC (Heavy
Congestion)
35.4ms
49.9ms
27.0ms
37.5ms
39.5ms
33.3ms
L2TPv2
GOOSE Transport without
QoS or IPSEC (Heavy
Congestion)
246.8ms
310.2ms
8.33ms
49.7ms
376.1ms
92.9ms
Event 4
Event 5
Event 6
Table 2 - L2TPv2 latency test results
Event 1
Event 2
Event 3
Test
ID
Transport
Type
12
L2TPv3
GOOSE Transport Only
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
13
L2TPv3
GOOSE Transport + QoS
(No Load)
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
14
L2TPv3
GOOSE Transport + QoS +
IPSEC (No Load)
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
8.33ms
15
L2TPv3
GOOSE Transport + QoS +
IPSEC (Minor Load)
8.33ms
10.4ms
8.33ms
8.33ms
8.33ms
8.35ms
16
L2TPv3
GOOSE Transport + QoS +
IPSEC (Heavy Congestion)
37.5ms
45.8ms
35.4ms
33.3ms
8.33ms
33.3ms
17
L2TPv3
GOOSE Transport Without
QoS (Heavy Congestion)
760.4ms
216.7ms
191.67ms
133.3ms
47.9ms
8.33ms
Testing Features
All time values are round trip times
Table 3 - L2TPv3 latency test results
16
QoS traffic shaping
The testing results showed that when a T1 interface became extremely congested by GOOSE
bursting of many relays, one way data times across the network would exceed the 19 ms limit even
when QoS policy places GOOSE traffic in the high-priority queue. We saw times as high as 25 ms
one way (50 ms round trip; table rows 10 and 16), which is unacceptably slow. To overcome this
delay, we applied a traffic shaper to the QoS policy which limits the maximum traffic through the
router to 1.3 Mbps. This traffic shaping limit is a router setting that buffers data packets exceeding
1.3 Mbps and reduces congestion between the router and the T1 link. Shaping made a critical
difference - although we effectively reduced the available bandwidth by 200 kbps, we saw 20 ms to
30 ms reduction in round-trip latency from the typical cases of table rows 10 and 16 for the test case
of intense GOOSE traffic, with a much more stable jitter delay (less variance in delays among
measurements). With the proper dose of traffic shaping in the router configuration, the latency is
significantly lower than the maximum limit even with unrealistically high levels of GOOSE traffic
congestion. We found that critical GOOSE messages always reached their destination well within
the required times while lower priority traffic at unrealistic volumes was delayed or in some cases
discarded.
Communications Technology Trends
We stated in the introduction that C-RAS has the dual-redundant high-availability prototypical
architecture for a generic wide-area monitoring, protection, automation, and control (WAMPAC)
system in which the utility industry is becoming intensely interested – the center of the
Transmission Smart Grid [3]. While C-RAS is aimed at implementing remedial action schemes, the
design approach can be used for any sort of system monitoring, control, or protection with
appropriate applications and processing at the control center, and streaming of required data points
from around the system.
In the general case, wide area monitoring (WAM) and WAMPAC systems are conceived in the
context of synchrophasor transmission as described in IEEE C37.118-2005 [10]. With
synchrophasor measurements, analog values from around the system are time-correlated to within
less than a microsecond using GPS or Ethernet network coordination of timing across the region.
Along with synchrophasor definitions and measurement techniques, C37.118-2005 describes a
streaming communications protocol for synchrophasor values based on manual configuration and
serial data paths. Data streams can also be encapsulated and sent as Ethernet packets. Because
synchrophasors are intended for sharing over wide area networks, and because of how the standards
development processes of IEEE and IEC have progressed, these packets are not IEC 61850
GOOSE, nor are they directly compatible with IEC 61850 design or configuration tools.
To deal with this situation, IEEE has split the C37.118-2005 synchrophasor standard into a new
measurement-only part C37.118.1-2011 and a communications-only part C37.118.2-2011. In
parallel, the IEC 61850 development working group has created a new transport protocol described
in IEC Technical Report 61850-90-5 [9] that is compatible with IEC 61850 systems and
17
configuration methods. The measurements can be synchrophasors or any other streamed data types.
The transport mechanism is a new pair of services – a form of wide-area Ethernet network routable
GOOSE (R-GOOSE) and a form of routable Sampled Values (R-SV) service. Whereas C-RAS
uses special router configuration and encryption to transfer GOOSE over T1 WAN links, IEC
61850-90-5 provides a service that naturally and easily transports such information across the WAN
in an IEC 61850 format. Among the features of IEC-61850-90-5 transport:
•
Layer 3 UDP/IP multicast passes readily through routers and across WANs.
•
Using the IT-standard router service known as Internet Group Management Protocol
(IGMP) Version 3, subscribing IEDs and their routers can automatically locate the
publishers they seek even when separated by an arbitrary WAN having multiple hops.
Today, this is a strictly manual configuration of point-to-point bridging in routers.
•
A standard encryption technique is defined.
•
R-GOOSE and R-SV message packets are enhanced with a key-based high security (Secure
Hash Algorithm or SHA-256, also called SHA-2) authentication signature to foil spoofing
or substitution disruptions.
•
Management of the ongoing distribution of security keys to approved publishers for use in
creating authentication signatures employs the IT-standard Group Domain of Interpretation
(GDOI) key distribution center process.
With convenient auto-configure routing using standard IT equipment, leading-edge security
features, and compatibility with the IEC 61850-6 configuration process of [6], IEC 61850-90-5 is
an excellent solution for a system like C-RAS; R-GOOSE provides exactly the required capability.
However, the standard is just now being published, and IED manufacturers will need time to create
new communications processing platforms that can handle the complex authentication hash code
calculations at high speed. Thus, 61850-90-5 is on the roadmap for use in C-RAS when available
in practical commercial versions in the future. The architecture of C-RAS allows for a full mixture
of GOOSE and R-GOOSE substation links, so that the new technology can be implemented
incrementally over time. Central controllers can process messages in both GOOSE and R-GOOSE
formats.
Conclusions
Faced with demands on its transmission grid from new generation providers, SCE is now deploying
its Centralized Remedial Action Scheme (C-RAS) – among the largest wide-area control systems
applied to an electric transmission grid anywhere. The C-RAS architecture is the prototype for
future wide area monitoring, protection, automation, and control (WAMPAC) systems that collect
data from across the grid to carry out the full range of utility transmission control functions,
including those with high speed and mission-critical reliability.
18
C-RAS uses Ethernet networking and IEC 61850 GOOSE messaging for flexibility, scalability,
speed of operation, and inherent performance monitoring. Wide area application is beyond the
original intended application of GOOSE, and requires special attention to the arrangement of the
Ethernet network paths to assure fast, reliable, secure transmission. The paper has described how
the communications infrastructure has been designed and configured to meet C-RAS requirements.
Experience from C-RAS development may be helpful for other industry applications of IEC 61850
GOOSE messaging between sites connected by a WAN link.
The first phase of C-RAS will be fully deployed in 2014. Over the years of C-RAS growth,
ongoing relay development will bring implementations of new services using the protocol of IEC
61850-90-5, which is specifically designed for applications like C-RAS and can be incrementally
introduced into individual substations and communications links.
Acknowledgement
The authors gratefully acknowledge the technical contributions of Patricia Arons, Edward Lee,
Howard Ham, Anthony Karian, and Anthony Johnson of SCE.
References
[1] Arons, Patricia, “SCE pilots the next level of grid protection,” T&D World Magazine,
December 2007, http://tdworld.com/overhead_transmission/power_sce_pilots_next/.
[2] Udren, E.A., “IEEE (ANSI) Device Number 16 – Ethernet Switches and Routers”, 61st Annual
Georgia Tech Protective Relay Conference, May 3-5, 2007.
[3] Udren, E.A., Novosel, D., “Wide Area Monitoring and Control”, PACWorld Conference invited
paper, Dublin, Ireland, June 2010; published in PACWorld Book, Chapter 6, 2012.
[4] IEEE Standard 1613, “Environmental and Testing Requirements for Communications
Networking Devices in Electric Power Substations”, 2003; Amended, June 2010.
[5] IEC 61850-3, “Communication networks and systems in substations - Part 3: General
requirements, Edition 1, 2002. Edition 2 now in preparation, under title “Communication
networks and systems for power utility automation – Part 3: General requirements.”
[6] IEC 61850-6, “Communication networks and systems for power utility automation – Part 6:
Configuration description language for communication in electrical substations related to
IEDs”, Edition 2, 2010.
[7] IEC 61850-8-1, “Communication networks and systems for power utility automation – Part 8-1:
Specific communication service mapping (SCSM) – Mappings to MMS (ISO 9506-1 and ISO
9506-2) and to ISO/IEC 8802-3”, Edition 2, 2011.
[8] IEC 61850-90-1/Technical Report, “Use of iec 61850 for the communication between
substations”, Edition 1, 2010.
19
[9] IEC 61850-90-5 Technical Report, “Communication networks and systems for power utility
automation – Part 90-5: Use of IEC 61850 to transmit synchrophasor information according to
IEEE C37.118”, Edition 1, 2012.
[10]
IEEE C37.118-2005, “IEEE Standard for Synchrophasors for Power Systems”
[11] North American Electric Reliability Corporation (NERC), “Glossary of terms used in
reliability standards,” February 2005, http://www.nerc.com/docs/standards/dt/Glossary.pdf
[12] California Senate Bill No. 1078, Chapter 516, September 2002,
http://www.energy.ca.gov/portfolio/documents/SB1078.PDF
[13] 2009 SCE General Rate Case Testimony on Centralized Remedial Action Scheme, vol. 3,
Part 5.
Biographies
Jun Wen, Senior Power System Engineer in Transmission Strategy and Special Assessment Group at
Southern California Edison (SCE) Company, has over 10 years of industry and academia experience in
power system and power electronics. She is currently leading the development of the Centralized-RAS
project – an IEC 61850 based Wide-Area Monitoring, Protection and Control system. She is also a member
of the WECC Modeling and Validation Working Group. Before she joined SCE, she was advisor for OneCycle Control, Inc, and consultant for Aegis Technology, Inc. She received her MSEE from Tsinghua
University and PhD in Electrical Engineering from University of California, Irvine, has published over
twenty technical articles and holds one US and International Patent.
Craig Hammond currently works for the network design and implementation team for Southern California
Edison (SCE). Craig has over 13 years of industry experience in the networking field. Craig holds a CCIE
along with a number of other certifications. Prior to SCE he worked at Fox Entertainment Group, EarthLink,
and a large bank where he performed large scale integrations and provided high level network design.
Eric A. Udren has a 42 year distinguished career in design and application of protective relaying, utility
substation control, and communications systems. He programmed the world’s first computer based
transmission line relay, led development of the world’s first LAN-based substation protection and control
system, and managed development and application at Westinghouse, ABB, and Eaton Electrical. He has
worked with utilities to develop new substation protection and control designs as a consultant since 2004.
He is now Executive Advisor with Quanta Technology and is based in Pittsburgh.
Eric is IEEE Fellow, Chair of the Relaying Communications Subcommittee of IEEE Power System Relaying
Committee, and chairs two PSRC working groups. He is US Technical Advisor for IEC TC 95 relay
standards; and is member of IEC TC 57 WG 10 that develops IEC 61850 power system communications
protocol and modeling. Eric serves on the North American Electric Reliability Corporation (NERC) System
Protection and Control Subcommittee, and Protection System Maintenance and Test Standard Drafting
Team. He has written and presented over 80 technical papers and book chapters. He can be reached at
[email protected].
20