Download Owner has selected a trial area for the System Acceptance Test

EXHIBIT H (11.8)
SYSTEM ACCEPTANCE TEST PLAN
1.
Background
Owner is planning to implement an Advanced Metering Infrastructure (AMI) system for its operation
in Albany, GA. The system will be used to access meters and load control devices from selected
locations using RF-based, two-way communications. AMI is intended to improve Owner’s services
by increasing meter accuracy, providing additional load data for engineering analysis, improving
responsiveness for restoring unplanned outages and adding customer service offerings such as
demand response.
AMI systems constitute a substantial financial investment for utilities. These complex systems are
comprised of software, hardware and communication components that have been developed and
enhanced over the last 20 years. Owner has selected AMI Vendor’s technology as the means to
communicate from Owner’s central offices to a variety of field devices including meters and other
operational components.
Due to the size of the investment, the adaptation of AMI Vendor’s technology to Owner’s operations
and requirement of adequate system performance and interfaces to Owner’s existing business
systems, a System Acceptance Test (SAT) will be used to ensure that Owner receives a system that
meets its needs and meets the verbal and written commitments made by AMI Vendor. The goal of
the acceptance testing described in this document is to verify the overall quality, correct operation,
scalability, completeness, usability, portability, performance and robustness of the functional
components and integrated system supplied by AMI Vendor.
2.
System Specification
The AMI functionality required by Owner and committed to by AMI Vendor is defined by Owner’s
Request for Proposal entitled, “FileName.ext”, and dated MMM DD, YYYY. Modifications and
clarifications to this functionality are included in the following documents:
AMI Vendor’s Table of Compliance included in its proposal entitled, “FileName.ext” and dated MMM
DD, YYYY.
Minutes and Agreements from AMI Project Planning meetings and discussions between AMI Vendor
and Owner entitled, “Minutes to Owner-AMI Vendor Project Planning Meeting.doc” (if applicable).
3.
Test Strategy
The overall strategy for Owner’s acceptance testing is to verify that the AMI system provided by AMI
Vendor meets the requirements and commitments defined per Exhibit A of the Owner’s RFP. Owner
will develop and document the formal test procedure and tools to be applied for each application or
function. The testing will be carried out by Owner personnel. If a test failure occurs, AMI Vendor will
be notified and allowed the opportunity to validate results, suggest alternate methods, remedy the
failure or take other mutually agreeable actions. Once a system has been commissioned and all test
meters are installed, the testing is expected to last 45 days or less.
The AMI testing will consist of four types of tests and/or inspections:
Sample System Acceptance Test Plan
1
Proprietary and Confidential
3.1
Inspections
Inspections will be used to verify the physical condition of the received system
components. Inspections will include ensuring that equipment/components supplied
matches what was offered by AMI Vendor.
3.2
Functional Testing
These tests will be used to verify the functionality of the delivered system. The tested
functions include end-to-end system testing for basic applications such as transfer of
billing data determinants, meter readings on-demand, remotely disconnecting and
connecting meters, etc. Testing will be carried out by Owner personnel or by assigned
agencies.
3.3
Integration Testing
These tests will be used to verify that the delivered system(s) exchange data with each
other and with the systems that are already in place at the Owner’s operations. Data
exchanges will be tested for a variety of factors including data integrity and reliability.
Testing will be carried out by Owner personnel or by assigned agencies.
3.4
Performance Testing
Where required, the testing will document the performance of the AMI system under
circumstances approximating normal operating conditions for Owner. These tests will be
conducted and measured by varying time periods as required to meet the specifications
and Owner’s intended use of the system.
4.
Test Process
Owner or its assigned test/inspection agencies will perform all tests and inspections after AMI Vendor
has commissioned a mutually agreeable SAT system. The test period will commence no later than 60
days after Successful System Commissioning. Successful System Commissioning is defined as
having end-to-end communications from the Server to a small sample of the SAT meters. The test
period is expected to last 45 days or less dependent upon satisfactory implementation of the network
communications technology and software interfaces.
It is suggested that AMI Vendor participate as a witness for the last week of testing to support Owner
and to increase probability of success. This participation is at AMI Vendor’s expense. Notice of interest
for witness testing should be given to Owner in writing at least two weeks in advance of the test to
accommodate AMI Vendor’s desires.
4.1
Procedure
The acceptance test results shall be evaluated using the following procedure:
Owner will provide AMI Vendor detailed Acceptance Test Procedures that are accepted
or modified to mutual satisfaction.
Owner notifies AMI Vendor of upcoming test.
Owner produces an Acceptance Test Report and submits it to AMI Vendor for review.
4.2
Actions upon success
Sample System Acceptance Test Plan
2
Proprietary and Confidential
If a system component or function successfully passes the Acceptance Test, written
results will be provided by Owner to AMI Vendor advising of the completion.
4.3
Actions upon failure
If a system component or function fails the Acceptance Test Procedure, AMI Vendor will
be notified and have the opportunity to correct the deficiency or deficiencies. Retests will
be scheduled for mutually agreeable dates. Revised versions will be re-tested per the
originally agreed Acceptance Test Procedure. Owner reserves the right to reject meter
shipments that fail statistically sampled accuracy tests.
Should the system component subsequently fail to perform its function during the testing
period, Owner will advise AMI Vendor of the failure. Owner will have sole discretion as
to whether the Acceptance Test Procedure will be repeated.
5.
High Level Test Plan
Owner or its assigned test/inspection agencies will perform all tests after AMI Vendor has
commissioned the SAT system. Inspections will be performed as required and in a manner convenient
to Owner. The test duration is expected to be approximately 45 days dependent upon satisfactory
implementation of the network communications technology.
AMI Vendor is invited to participate as a test witness for selective tests or for the entire test process.
A notice of interest for witness testing should be given to Owner in writing at least two weeks in
advance of selected test(s) to accommodate AMI Vendor’s desires.
Owner has selected a trial area for the System Acceptance Test based upon:

Accessibility

Availability of suitable bulk-communications channel

Cross section of customers

Geography and topology

Density
The SAT is expected to last approximately 45 days. AMI Vendor is expected to deliver applicable
system components required to perform the SAT for this system in accordance with these dates.
Owner is responsible for providing all test equipment.
5.1
Inspections
Table 1 defines inspections that may be performed by Owner or its representatives.
These will be performed at the convenience of Owner to ensure conformance to the
project schedule. The inspections will not be limited to the components of the SAT system,
they will continue through delivery of 100% of the AMI system
Sample System Acceptance Test Plan
3
Proprietary and Confidential
Table 1
Inspections
Component
Function
Residential and C/I gas, water
and electric meters where
applicable
Visual inspection for:
 Nameplate data
 Bar Code data
 Form and Voltage (electric)
 Manufacturing quality
 Production and test data
available and in
conformance to Owner
requirements
 Shipping damage
Responsible Party
Owner and/or installation
subcontractor
Validation measurements for:
 kWh accuracy
 Accuracy of voltage
measurements
 Verification of power quality
tolerances
Network communication
components
5.2
Visual inspection for:
 Nameplate data
 Bar code data
 Manufacturing quality
 Production and test data
available and in
conformance to Owner
requirements
 Shipping damage
Owner and/or installation
subcontractor
Functional Tests
Table 2 defines Functional Testing required for verifying that the system meets the Owner
specification. Timing of these tests will be determined by Owner in order to achieve an
effective implementation process. AMI Vendor is encouraged to make suggestions
regarding timing and/or order of tests based upon prior experiences. Many of these tests
will be run multiple times over the test period to verify communications reliability.
The tests that follow may be supplemented with additional tests based on Owner research
of AMI Vendor reference systems. As mutually agreed, supplemental tests may be added
to verify system performance. Tests may not be added for functions that were not part of
these reference documents unless mutually agreed to by AMI Vendor and Owner.
Sample System Acceptance Test Plan
4
Proprietary and Confidential
Table 2
Functional Tests
System
Functions
Responsible
Requirement
Purpose
On-demand meter
reading (single
device)
Retrieve measurement readings
(e.g., kWh/kW, reactive, gas,
water or other meter readings)
from single devices on-demand.
Verify communications
continuity and availability
Data Processing
On-demand meter
reading (multiple
devices)
Simulate on-demand retrieval of
measurement readings (e.g.,
kWh/kW, reactive, gas water or
other meter readings) from
devices based on customer
request and data availability.
Emulates multiple,
simultaneous customer
calls for hi-bill
complaints.
Customer Service
Scheduled meter
reading (single
device)
Retrieve measurement readings
(e.g., kWh/kW, reactive, or other
meter readings including gas and
water) from single devices based
on a predetermined schedule.
Verify communications
continuity and
availability.
Data Processing
Scheduled meter
reading (multiple
devices)
Retrieve scheduled measurement
readings (e.g., kWh/kW, reactive,
gas water or other meter readings
including propane) from selected
groups of meters based on device
availability, Owner meter routes
and cycle days,
Verifies batch/group data
acquisition to replace
meter readers.
Data Processing
On-demand
voltage readings
(multiple devices)
Simulate on-demand retrieval of
voltage reads from selected
electric meters.
Emulates engineering
and operations
evaluation of near real
time system voltage.
Operations and
Engineering
Programmed
polling for meter
status
Simulate on-demand acquisition of
meter status from strategically or
randomly located devices.
Emulates outage
identification scenarios.
Operations and
Engineering
Scheduled and ondemand retrieval
of load profile data
(single device)
Retrieve load profile measurement
readings (e.g., kWh/kW, reactive,
phase voltage, amps, gallons,
cubic feet or other meter readings
including propane) from single
devices based on a predetermined
schedule or on-demand. Varying
interval lengths of 15, 30 and 60
minutes will be used, in
accordance with AMI Vendor
specifications for the particular
module being tested. Collection
periods to vary from a few hours
up to 45 days.
Verifies that profile data
can be stored and
collected as needed.
Storage in meter
precludes large amounts
of data storage in AMI
Server.
Data Processing,
Operations and
Engineering
Sample System Acceptance Test Plan
5
Party / Dept.
Proprietary and Confidential
System
Functions
Responsible
Requirement
Purpose
Scheduled and ondemand retrieval
of load profile data
(multiple devices)
Retrieve load profile measurement
readings (e.g., kWh/kW, reactive,
phase voltage, amps, gallons,
cubic feet or other meter readings
including propane) from predefined groups of devices. Varying
interval lengths of 5, 15, 30 and 60
minutes will be used, in
accordance with AMI Vendor
specifications for the particular
module being tested. Collection
periods to vary from a few hours
up to 45 days.
Verifies that profile data
can be stored and
collected as needed for
meters groups. Groups
will simulate engineering
requirements for loss or
load analyses of circuits
or partial circuits.
Data Processing,
Operations and
Engineering
On-demand
pinging of selected
meter groups
Simulate on-demand acquisition of
meter status from selected groups
of devices based on engineering
and operations requirements.
Emulates outage
management scenarios
for location and
restoration
Operations and
Engineering
Remote
disconnect and
reconnect
Perform remote disconnect and
reconnect of numerous residential
customer loads.
Verify
disconnect/reconnect
functionality
Customer Service
Meter diagnostic
and error codes
Retrieve and process meter
diagnostic codes according to AMI
and MDM Vendor stated
functionality.
Verify that meter
diagnostic codes can be
retrieved and processed
Operations and
Engineering
Field
programmability
Remotely change parameters for
data retention and retrieval, in
accordance with AMI Vendor
specifications for the particular
module being tested.
Verify that added or
changed functionality
does not require field
visits
Data Processing
Tamper detection
Implement numerous traditional
diversion techniques and monitor
performance for detection. This
includes meter-based functions
(outage, reverse rotation, etc.) and
software applications such as
usage monitoring with scheduled
reads no load conditions, etc.
Current implementation is a report
to run for manual verification.
Verify system
performance at detecting
energy theft.
Operations and
Engineering / Data
Processing /
Customer Service
Sample System Acceptance Test Plan
6
Party / Dept.
Operations and
Engineering
Proprietary and Confidential
System
Functions
Responsible
Requirement
Purpose
Load Control
Control of Irrigation loads at select
sites as well as Water Heater and
Air Conditioning loads at
residential locations according to
numerous load control scenarios.
These scenarios are used to: test
the ability to control assorted
batches of devices, cancel a
scenario, extend a scenario, verify
the opt-out functionality, ensure
that the AMI system can be used
to verify interruption at an
individual site, etc.
Verify that load control
functionality meets the
requirements of the RFP.
Marketing
Power quality
Monitor and evaluate ability of
devices to accurately measure
power quality events, in
accordance with AMI Vendor
specifications for the particular
module being tested Testing may
be laboratory or field settings at
Owner discretion.
Verify power quality
performance per
specification and EPRI
reliability indices.
Operations and
Engineering
Net metering
Monitor and evaluate the ability of
residential meters to accurately
measure, retain and transmit bidirectional power flow, in
accordance with AMI Vendor
specifications for the particular
module being tested.
System must meet new
energy bill provision that
potentially allows
residential customers to
implement on-site energy
supply and sell excess
power back to Owner.
Data Processing /
Sample System Acceptance Test Plan
7
Party / Dept.
Operations and
Engineering
Proprietary and Confidential
5.3
Integration Tests
Table 3 defines Integration Testing required for verifying that the system meets the Owner
specifications and AMI Vendor Table of Compliance. Owner understands that software
integration requires collaboration among multiple suppliers. AMI Vendor commits to
provide software that meets current integration requirements and to assist Owner with
the support to achieve integration goals. The timing of integration tests will be at Owner’s
discretion in order to meet project milestones. AMI Vendor is encouraged to make
suggestions regarding timing and/or order of tests based upon prior experiences. Many
tests may be run multiple times to verify communications reliability.
The tests that follow may be supplemented with additional tests based on Owner research
of AMI Vendor’s reference systems. As mutually agreed, supplemental tests may be
added to verify system performance. Tests may not be added for functions that were not
part of these reference documents unless mutually agreed to by AMI Vendor and Owner.
Table 3
Integration Tests
Integration
Activity
Responsible
Requirement
Purpose
Inventory
management
Verify that vendor supplied meter
test file has appropriate data and
file structure to import in to
Owner’s inventory record system.
Ensure that all data is
complete and requires
no manual data entry.
Data Processing
Input meterchange data
records into
Owner billing
system during
system
implementation
Verify that meter change data
including: old meter #, new meter
#, old meter read new meter read,
Account #, Date, Multiplier and
pole # can be automatically
transferred into CIS system during
meter change-out process.
Manual entry of data
adds labor costs and
increases chance of
error. System must
manage process with
minimal operator
intervention.
Data Processing
Input meterchange data
records into AMI
Vendor platform
during system
implementation
Verify that meter change data
including: meter #, device serial
number, Account #, AMI route and
template option for device type
can be automatically transferred to
AMI system during meter changeout process.
Manual entry of data
adds labor costs and
increases chance of
error. System must
manage process with
minimal operator
intervention.
Data Processing
Input meterchange data
records for longterm read-out and
read-in of meters
Enter meter change records into
the Billing system and transfer
these entries to the other system
without duplicate keystrokes. If
CIS Vendor does not support
MultiSpeak, Flat File data transfer
will be tested. Assumes day to day
swap-out of meters which works
similarly to change records during
system implementation.
Verify that double entries
are not required for AMI
and CIS systems.
Data Processing
Sample System Acceptance Test Plan
8
Party / Dept.
Proprietary and Confidential
Integration
Activity
Integration of AMI
to the meter data
management and
billing system
Responsible
Requirement
Purpose
Test transfer of profile and billing
data from AMI system to billing
system to:
Verify that AMI system
manages billing data
transfer as defined in the
specifications.
 Validate profile data
 Input billing determinants
 Verify that the MDM system
can initiate and manage the
remote disconnect functionality
 Other elements per the Owner
AMI, MDM and LC
requirements.
Sample System Acceptance Test Plan
9
Party / Dept.
Data Processing
Proprietary and Confidential
5.4
Performance Tests
Table 4 defines Performance Testing required for the AMI system. These tests will be
measured and quantified over critical times during the SAT or later as needed to verify
system suitability for Owner’s purposes.
Table 4
Performance Tests
Basic System
Functions
Responsible
Requirement
Purpose
Support of
typical volume of
Owner user’s
and data traffic
on network
Monitor performance of system on
Owner network to ensure that all users
have adequate system response.
Ensure the Ownersupplied network and
servers are properly sized
for the AMI Vendor
software and Owner’s
volume of usage.
Customer
Service
Support of
typical volume of
Owner online
data
requirements
Monitor performance of software
application on Owner network to ensure
that all users have adequate system
response.
Ensure that storage
capacity of the Ownersupplied server(s) is
adequate for the AMI
Vendor software and
Owner’s volume of
usage.
Customer
Service
Meter reading
response time
Measure meter reading response times
for a variety of data acquisition
scenarios. These scenarios will include:
 On-demand and scheduled reads of
single unit
 On-demand and scheduled reads of
groups of meters approximating
Owner cycle day and engineering
requirements.
 On-demand and scheduled pinging
of meters for outage evaluation
 On-demand and scheduled retrieval
of load profile data for single unit or
groups of units
Ensure that system
meets specifications and
promises of vendor and
that system can meet
Owner’s objectives
Data
Processing
Communications
availability
Measure overall communications
success rate during the duration of the
SAT and during select time periods of
the SAT. Select time periods will include
times of peak loads for Owner.
Ensure that the AMI
system communications
performance meets the
objectives and
specifications of Owner.
Operations
and
Engineering
Failure rate
Measure failure rate of devices during
the SAT
Ensure that product does
not have out-of-box
failure rate in excess of
specifications.
Operations
and
Engineering
Sample System Acceptance Test Plan
10
Party / Dept.
Proprietary and Confidential