Download Attachment A: Related Project: Link Cotton CRC to ATNF Narrabri

ITS BUSINESS CASE
High Speed Network Links for
CSIRO Observatories
Last Updated:
29-Jun-17
Version:
0.5
Role
Representative
Business Owner
Ray Norris
ATNF Marsfield
Ron Beresford
Chris Phillips
Neil Killeen
Vince McIntyre
Tasso Tzioumis
ATNF Parkes
John Reynolds
ATNF Narrabri
Bob Sault
CTIP Marsfield
Shaun Amy
Michael Homsey
ICT Centre Marsfield
Terry Percival
IT Services
Ron Baxter
John Morrissey
Rob Thomsett
AARNet
Keith Burston
NSW RNO
Alan Cowie
ITS BUSINESS CASE
Network Links to Observatories
Change Record
Date
Author
Version
Change Reference
25-Mar-04
Ron Baxter
0.1
No previous document.
31-Mar-04
Ron Baxter
0.2
Some risks and costs added,
distributed for Planning
Meeting on April 2
23-Apr-04
Ron Baxter
0.3
Incorporating outcomes from
Planning Meeting
28-May-04
John Morrissey
0.4
Revised sections 6.1 and 6.4
Chris Phillips
0.5
ATNF revisions
Tasso Tzioumis
Table of Contents
Change Record ..................................................................................................................... 2
Table of Contents .................................................................................................................. 2
1.
Executive Summary ..................................................................................................... 3
2.
Project Introduction ..................................................................................................... 3
2.1
Background ........................................................................................................................... 3
2.2
Overview ............................................................................................................................... 4
2.3
3.
Project Success ...................................................................................................................... 4
Project Scope and Objectives ........................................................................................ 5
3.1
Key Objectives & Outcomes ..................................................................................................... 5
3.2
Out of Scope .......................................................................................................................... 6
4.
Constraints ................................................................................................................. 6
5.
Related Projects .......................................................................................................... 6
6.
Key Activities and Deliverables ...................................................................................... 7
6.1
“Lighting up” the Regional Network ........................................................................................... 7
6.2
Fibre Links to CSIRO Sites ....................................................................................................... 8
6.3
Production Network – Connect Site LANs ................................................................................... 9
6.4
Research Network – for high speed data transfer ...................................................................... 10
6.5
Key Milestones ..................................................................................................................... 11
7.
Risks ........................................................................................................................ 11
7.1
Project Risks ........................................................................................................................ 11
7.2
Business Risks ..................................................................................................................... 12
8.
Benefits/Value Analysis .............................................................................................. 12
8.1
Position CSIRO as a World Leader in VLBI ................................................................................ 12
8.2
Developing a Data Grid ......................................................................................................... 13
8.3
Intensify the Pulsar Timing Partnership ................................................................................... 14
8.4
Efficient Service Delivery ....................................................................................................... 14
Last updated: 29-Jun-17
v0.5
2 of 23
ITS BUSINESS CASE
8.5
8.6
9.
Network Links to Observatories
Research Network: Enhance CSIRO’s Reputation ...................................................................... 15
Returns and Benefits ............................................................................................................. 15
Cost Analysis............................................................................................................. 15
9.1
10.
Costs- Future Value .................................................................... Error! Bookmark not defined.
Staff Impact/Change Management ............................................................................... 16
Attachment A: Related Project: Link Cotton CRC to ATNF Narrabri ............................................ 17
Attachment B –Notes on Reduced Costs and Benefits of Network Links to Observatories .............. 18
Attachment C - Quantifying the gains from increased VLBI data rates & sensitivity...................... 21
1. Executive Summary
This project will build on the Australian government’s investment in the
Australian Research and Education Network (AREN) to provide fibre network
connections to the three CSIRO observatories and to the Tidbinbilla Tracking
Station.
It is motivated by CSIRO’s goals to enhance our global reputation and to
intensify partnerships.
A new dedicated research network will connect the observatories to selected
partner sites, and will use SX Transport to connect internationally. The benefits
will include an increased return on investment from CSIRO’s observatories, an
early demonstration of the benefits of grid computing, and opportunities to lift
CSIRO’s profile both with government and internationally.
For CSIRO’s production network, the local networks of the observatories will
connect to the CSIRO Wide Area Network at gigabit speeds. The benefits will
include a bigger improvement from One-IT Service delivery, better data
integrity through automated off-site backups and better collaboration facilities
that will be used more widely.
The capital cost is about $2.5 million, and the running costs will be about
$500K annually. This annual cost is similar to the cost of leasing 2 megabit
links commercially, but will deliver more than 1,000 times the bandwidth.
2. Project Introduction
2.1
Background
In the CSIRO Strategic Plan for 2003-2007, the second goal is Delivering
world-class science – enhancing our global reputation for science excellence.
An objective under this goal is
2.4 – Help Australia play a leadership role in major international science
facilities such as the SKA.
The Square Kilometre Array (SKA) will be an array of radio telescope antennas
that will be at least 100 times more sensitive than current arrays, and may be
located in Australia.
This year CSIRO has an opportunity to demonstrate this leadership by building
on the infrastructure of the Australian Research and Education Network (AREN)
which is being funded by the Australian Government. This will enable high-
Last updated: 29-Jun-17
v0.5
3 of 23
ITS BUSINESS CASE
Network Links to Observatories
speed interconnection of the three observatories of ATNF and the Tidbinbilla
Tracking Station, and will enhance their capabilities.
The third goal of the CSIRO Strategic Plan is Partnering for Community Impact
with objectives
3.1 Focus and intensify collaboration with universities, CRCs and other
agencies
3.4 Partner with other agencies to advance Australia’s global development
contributions
This project will facilitate ATNF partnerships with Swinburne University,
University of Tasmania and Australian National University for the projects
discussed here; and will assist the other ATNF partners in Australia and
internationally.
Currently, Wide Area Network links to the ATNF telescopes are provided by
leased phone lines with speeds as shown in the table.
Telescope
Location
Nearby town
Leased Line
Speed
Connects to
ATNF Parkes
Observatory
Parkes
512K
Radiophysics Laboratory,
Marsfield
ATNF Mopra
Observatory,
(near
Coonabarabran)
Coonabarabran
128K
ATNF Paul Wild
Observatory, Narrabri
ATNF Paul Wild
Observatory,
Narrabri
Narrabri
512K
Radiophysics Laboratory.
Marsfield
Tidbinbilla Deep
Space Tracking
Station
Canberra
128K
ITS Canberra
Because these links are slow, two of these sites must operate email, file
servers and backup facilities locally.
2.2
Overview
As part of AREN, AARNet is implementing the third generation for the national
backbone (AARNet3) as described in these briefing slides. AARNet3 will use the
NextGen network which was acquired by Leighton Contractors in December
2003.
The AARNet3 NextGen fibre will be used as the backbone to interconnect the
observatories. Scientific drivers for high speed network connections to the
telescopes include Very Long Baseline Interferometry (VLBI), the Virtual
Observatory, and Pulsar Timing.
2.3
Project Success
Successful completion of this Project will be measured as follows:
Success Measure
Last updated: 29-Jun-17
Relativity
v0.5
[Key Reports]
4 of 23
ITS BUSINESS CASE
1
Network Links to Observatories
2
3
4
5
6
7
Satisfied Client Group/s
8
9
10
X
Meet all the project's objectives/requirements
X
Meet the agreed budget
X
Deliver the key product/service on time
X
Add value to the organisation
X
Meet quality requirements
X
Sense of professional satisfaction for the Team
X
3. Project Scope and Objectives
3.1
Key Objectives & Outcomes
3.1.1 An Advanced Regional Network for AARNet
The objective is to collaborate with AARNet on the implementation of a
regional network design that can deliver strategic requirements for research
and education in regional locations. In this collaboration, CSIRO will assist with
RFT, review network architecture, provide environment for equipment on
CSIRO sites, and install tail-end equipment at sites.
The outcomes will be
1. an impetus to research and education in regional Australia through
advanced networking
2. a new competitive force in the regional network market.
3.1.2 A Research Network for High Speed Transfer of Observed Data
The objective is to design and implement a separate network with dedicated
circuits interconnecting the three CSIRO observatories, Tidbinbilla tracking
Station, and the Southern Cross Trans-Pacific Testbed (SX Transport).
The outcome will be a network that can provide dedicated high-speed data
paths for VLBI within 3 years (at least 10 Gb, possibly 40 Gb) that will
stimulate scientific advances in projects such as eVLBI.
3.1.3 Production Network – Gigabit Connections to Regional Sites
The objective is to provide gigabit connections from the local networks at
Parkes, Mopra and the Narrabri Compact Array to the AARNet shared regional
network backbone and thus to the CSIRO WAN, and to also provide dial-up
ISDN links as a backup.
The outcomes will be transfer of observed data in “near real time”, and
consolidated IT management for these sites under the One-IT Service Delivery
model.
Last updated: 29-Jun-17
v0.5
5 of 23
ITS BUSINESS CASE
3.2
Network Links to Observatories
Out of Scope
Objectives out of scope for this Project include:
Related Project Objectives (Is Not)
Stakeholder/RP

Connection to Mt Pleasant (near Hobart) telescope
for eVLBI

Future project with ATNF as
stakeholder

Connection to Ceduna telescope for eVLBI

Future project with ATNF as
stakeholder

Research network connection to Swinburne Univ
for eVLBI and Pulsar Timing

Steven Tingay, Swinburne
Uni

Grangenet

Demonstrate high-speed networks as an enabling
technology for educational and social projects in
regional Australia (Dubbo)

Terry Percival, CeNTIE

Network connections for Siding Spring and Mt
Stromlo telescopes

ANU

Implement off-site backups over the network

Data Storage Management
Project

Consolidate Exchange mail servers

ITS Operations

Service Delivery Efficiencies from connection of
Cotton CRC to ATNF Narrabri

ITS Networks project, ITS
Operations
4. Constraints
Publicity in 2004 – sufficient progress to enable significant announcements by
Aug/Sep 2004 is highly desirable.
Significant VLBI achievements by 2006 will assist CSIRO’s case for SKA.
5. Related Projects
Other projects that this Project is critically dependent upon are:
Project
Relationship/Dependencies
Services Involved
One-IT Service Delivery
Improved service delivery over
gigabit connections is a key
benefit
Enterprise service desk,
desktop support, server
management
Data Storage
Management Project
Provision of off-site backups over
the network is a key benefit
Enterprise Data Storage &
Management
Last updated: 29-Jun-17
v0.5
6 of 23
ITS BUSINESS CASE
ATNF eVLBI
Network Links to Observatories
Real-time correlation of highspeed data streams is a key
objective
Development of high-speed
correlator
6. Key Activities and Deliverables
The east coast section of the NextGen network is a self-healing loop, and the
western side of the loop passes close to the ATNF observatories. Currently the
NextGen network has one pair of fibres in operation, and AARNet will “light up”
a second pair to provide these regional links. A third pair could be used later
for regional network links other than research and education.
Figure 1 - The NextGen Network
6.1
“Lighting up” the Regional Network
At present the NextGen regional loop consists of “dark” fibre and as such
requires the installation of equipment every 80km to activate or “light” the
network. It is hoped that the base costs of activating the network will be
provided by a request for additional AREN funding. It will cost between 8 and
10 million dollars to light the base network. If this funding is not provided the
member institutions will be required to jointly fund the lighting of the network.
This would double the costs of the whole project and is therefore a major
project risk. This is an AARNet activity and CSIRO will assist with planning,
design and implementation.
The second stage in activating the network requires the installation of Wave
Division Multiplexing equipment on the second pair of NextGen fibres. Two
options exist:
1. Coarse Wave Division Multiplexing (CWDM), which can provide about 8
circuits with data-speeds of 1 Gb. This equipment is relatively
inexpensive.
2. Dense Wave Division Multiplexing (DWDM), which can provide up to 80
circuits and offering bandwidths greater than 1 Gb – such as 10 Gb or
40Gb
The VLBI project wants to use 10 Gb or 40 Gb links by 2007, so CSIRO is keen
to see DWDM deployed on this network. The cost of DWDM is about 40%
higher, so the contributions of all the AARNet members using this network will
be higher. Although the base costs of DWDM are higher the equipment lifecycle
Last updated: 29-Jun-17
v0.5
7 of 23
ITS BUSINESS CASE
Network Links to Observatories
is also significantly higher at greater than 10 years compared to 3-4 years for
CWDM.
Early estimates of CSIRO’s startup costs for a DWDM network are between
$1.6 million and $2.4 million with the annual cost between $300K and $500K.
This startup cost includes a provision for a single 1 Gb connection to each site
on the network.
6.2
Fibre Links to CSIRO Sites
NDC has surveyed the sites and provided indicative estimates of costs for the
tail-end fibre connections to the CSIRO sites. The maps show both the closest
point on the cable, and the closest Controlled Environment Vault (CEV). (To
see the full detail of these maps, zoom in and view in colour to see the blue
paths.)
6.2.1 The Parkes Observatory
Figure 2 - Connection to Parkes
6.2.2 The Mopra Observatory
The Siding Spring Observatory is operated by ANU and houses a number of optical
telescopes. ANU will contribute so that this link will extend beyond ATNF-Mopra to Siding
Spring.
Last updated: 29-Jun-17
v0.5
8 of 23
ITS BUSINESS CASE
Network Links to Observatories
Figure 3 - Connection to Mopra
6.2.3 The Paul Wild Observatory at Narrabri
This link also passes the Cotton CRC (CSIRO and NSW Department of Agriculture) and the
Wheat Research Institute (Sydney University). Connection of the Cotton CRC is related
project with a low added cost – see Attachment A.
Figure 4 - Connections at Narrabri
6.3
Production Network – Connect Site LANs
One component of AARNet3 will be a shared network backbone provided on the
second pair of NextGen fibres. This network will provide regional links for
Last updated: 29-Jun-17
v0.5
9 of 23
ITS BUSINESS CASE
Network Links to Observatories
Universities and CSIRO in NSW. This will be shared by AARNet members, with
each member providing the tail links. Preliminary estimates indicate a total
budget of between $900K and $1.4 million is required for these tail circuits.
ANU, Siding
Springs
ATNF Mopra
(Coonabarab
ran)
ATNF
Narrabri
Cotton
Research
Narrabri
ATNF Parkes
AARNet Regional Network
Sydney
Access
Node, UTS
Figure 5 - Production Network Tail Links
If additional circuits are required on the network above those provided in the
base costs listed in section 6.1 CSIRO will be required to spend $40K per link
for the installation of interfaces at each end of the link.
6.4
Research Network – for high speed data transfer
The AARNet3 Regional Network may use Coarse Wave Division Multiplexing
(CWDM) or Dense Wave Division Multiplexing (DWDM) to provide the backbone
as discussed above. The research network will be built from dedicated
wavelengths.
ATNF
Narrabri
ATNF
Mopra
ATNF
Parkes
ATNF
Epping
SX
Transport
Tidbinbilla
This Project
Swinburne
Univ
Ceduna
Last updated: 29-Jun-17
Related (future) Projects
Mt
Pleasant,
Tas
v0.5
10 of 23
ITS BUSINESS CASE
Network Links to Observatories
Figure 6 - Research Network - a possible design
When the DWDM network needs to be upgraded for additional bandwidth over
the AARNet3 Regional Network the cost per 10 Gigabit link is about $72K for
distance under 600 Km, and $144K for longer distances. The design above
shows 2 shorter links (Mopra and Parkes) and one longer link (Epping). The
project has the option of configuring 10Gb circuits on day one if required.
The Tidbinbilla link is more complex as it involves connecting via the Canberra
based ICON network and obtaining bandwidth on the AARNet3 metropolitan
backbone. The proposed Canberra-Tidbinbilla interconnect passes Mt Stromlo.
The cost to CSIRO for the Tidbinbilla link is dependant on some negotiations
with Telstra for access to spare fibre pairs on an existing cable to the site. If
these negotiations are unsuccessful we estimate that it will cost approximately
$400K to install a new cable between the site and the Canberra ICON network.
Future links will include Mount Pleasant Radio Telescope (near Hobart), the
Ceduna Radio Telescope,the Swinburne University Centre for Astrophysics and
Supercomputing, international connections (e.g. Japan, USA) and possibly to
SKA pathfinder projects in WA.
6.5
Key Milestones
Key milestones for this Project are:
Event/Milestone
Deliverable
Date
AARNet secures AREN funding
Agreement with Department of
Education, Science and Training
May-04
AARNet completes RFT for
equipment
Supplier selected and equipment
costs known
May-04
AARNet secures member sign-offs
Agreement from participating
universities and CSIRO
Jun-04
AARNet provides physical
connections to CSIRO sites
Connections into active equipment in
site computer rooms
Aug-04
Production network operational with
gigabit links
Cutover from existing leased lines to
gigabit backbone
Aug-04
Research network links (first phase)
operational
Dedicated links operate at 1 Gb
Nov-04
7. Risks
7.1
Project Risks
Ability to Deliver Networks to Required Specifications
The technology being deployed is well tested, and the suppliers well known.
Our network designs are not pushing specifications to their limits. So the risk
that we will not be able to deliver the specified network performance is low.
Ability of ATNF, ITS and AARNet Staff to Deliver
Last updated: 29-Jun-17
v0.5
11 of 23
ITS BUSINESS CASE
Network Links to Observatories
The teams in the three groups are well qualified and highly motivated, so this is
also a low risk.
Stakeholder Commitment
Commitment from ATNF scientists and ATNF partners (Swinburne, University of
Tasmania, ANU, …) is high. AARNet is motivated to deliver good outcomes in
response to Federal Government funding. However, it is not clear yet if the
AARNet regional members will all be in a position to support the initiative
promptly, so this risk is assessed as moderate.
Summary
This project is not a high risk project.
7.2
Business Risks
Level of AREN Funding
The current expectation is that the Department of Education, Science (DEST)
and Training will fund $5 million, and the total cost of equipment required is
around $10 million. In Costs Analysis, we present costs to CSIRO based on this
expectation. Any substantial decrease in DEST funding would put the project at
risk. This risk is assessed as low.
Financial commitment from AARNet members
The project depends on all the participating universities providing a share of
the costs, so any financial difficulties of members could present a project risk.
This risk is assessed as medium.
Loss of Reputation if unable to deliver
The Department of Education, Science is supporting the project with the
expectation that it will lead to scientific advances. If CSIRO is not able to
demonstrate these advances, our reputation as a leading science agency would
suffer. This risk is assessed as low.
Summary
The business risks are not high.
8. Benefits/Value Analysis
8.1
Position CSIRO as a World Leader in VLBI
VLBI (Very Long Baseline Interferometry) currently requires data to be
recorded at each telescope and then transported to a central point for
correlation. For eVLBI, data will be recorded at each telescope, and at the
same time will be transferred across the network to be correlated in real time
(or near real time in the early stages) at a central location. The eVLBI project
will include the ATNF telescopes and the Tidbinbilla tracking station near
Canberra initially.
Other research groups working on real-time correlation for VLBI are aiming at
data rates of 1-2 Gb/sec. CSIRO aims to operate with data rates of at least 2 x
8 Gb/sec within 3 years and this would provide the world's best VLBI
sensitivities. This will open new avenues of astronomical research and has the
potential to revolutionise the VLBI field and lead to significant new discoveries.
Last updated: 29-Jun-17
v0.5
12 of 23
ITS BUSINESS CASE
Network Links to Observatories
Attachment B and Attachment C provide details of the costs and benefits
summarised below.
8.1.1 Current Costs and Outputs
Costs for the VLBI project are approximately:
o
5% usage of equipment valued at $240 million (ATNF and Univ of Tas)
o
2% usage of equipment valued at $100 million (Tidbinbilla)
o
Exclusive use of correlation equipment and tapes valued at $1 million
o
$500,000 per year in operating costs including staff for observing
Annual output about 8 papers/year (= 5% of ATNF papers (156 in 2003))
8.1.2 With the First Stage of Research Network
When the research network is in place with dedicated 1 Gb links, the sensitivity
will be increased 2 to 3 times. The benefits will be:
o
the science achievable is enhanced by the increased sensitivity which
enables study of fainter astronomical objects
o
greater sensitivity and ease of use which will attract more astronomers
o
automated data transfer will save 20 person weeks per year
These benefits will effectively double the value of the ATNF infrastructure
utilised by VLBI (currently 5% of $240 million).
8.1.3 With the Final Stage of the Research Network
When the research links are operating at 10 Gb or better, the sensitivity will be
increased 11 times. This will:
o
open up new avenues of research by greatly increasing the number of
astronomical objects for study. This has the potential for many
significant discoveries in high resolution radio astronomy.
o
enhance and facilitate access to VLBI observations for the whole
astronomical community and it is expected to bring many new users to
VLBI. Thus VLBI in Australia will cease to be a niche technique and will
join the mainstream of radio astronomy observing.
o
increase the proportion of VLBI time by increasing the scientific
capabilities and attracting new users.
o
provide savings in operating costs by effectively eliminating the specific
VLBI costs of the current network and integrating operations with the
ATCA at Narrabri.
These benefits will provide ATNF with infrastructure that has an effective value
of $2.7 billion if the increases in sensitivity were achieved by constructing
larger antennas. The 5% utilised by VLBI will effectively increase in value from
$10M to $135M.
8.2
Developing a Data Grid
The Australian Virtual Observatory aims to provide an interface to data archives
from Australian Telescope Compact Array. This project, together with other
Last updated: 29-Jun-17
v0.5
13 of 23
ITS BUSINESS CASE
Network Links to Observatories
international projects, will facilitate data mining projects by astronomers here
and internationally. The ICT Centre is running a case study for data grids that
use a computer cluster to manage about 2 terabytes of observed data. This
cluster is in Canberra and data is physically transported. Gigabit connections
will allow the data to be transferred across the network and the collection will
probably be moved to somewhere on the Research Network – perhaps Sydney.
8.3
Intensify the Pulsar Timing Partnership
The Pulsar Timing Array is a collaborative effort between the ATNF, Swinburne
University of Technology and international collaborators. It also uses computer
clusters – currently a small cluster at Parkes, but the aim is to use a more
powerful cluster at Swinburne. High speed links will allow transport in near
real time, rather than shipping disk packs.
8.4
Efficient Service Delivery
This project will improve service delivery on the local networks at Parkes and
the Narrabri Compact Array. A Related Project will provide a similar
improvement for the local network at Cotton Research.
One-IT will re-organize service delivery to use a 3 tier model with only tier-2
being on-site, and the impact of this re-organisation will be greater on sites
with gigabit connections.
To illustrate, consider some services provided over slow links compared to
high-speed links, as shown in the table.
Service
Slow speed site
Gigabit Site
Desktop support
Most tasks by local
support staff
Many tasks solved by
central support
PC software builds
Performed locally using
CDs
Done using downloads from
a central site, and can be
policy driven and
automated.
Management of
servers for files,
DHCP, DNS, etc
Done locally, need to
maintain expertise for a
range of servers at each
site, so the expertise
available is medium
level
Done remotely, using highly
experienced staff. Also
saves on training costs.
Off-site backups
Tapes written locally
and physically
transported to another
site
Backups can be over the
network to another data
centre
Video conference
support
Need to use ISDN,
require expertise locally,
bridging is complex
Only require IP on-site,
technical support can
mostly be provided by
remote specialists
Last updated: 29-Jun-17
v0.5
14 of 23
ITS BUSINESS CASE
Network Links to Observatories
The outcome is that part of the time of local IT support staff is freed up from
the generic IT tasks and can be re-directed to science-linked requirements of
projects at that site.
8.5
Research Network: Enhance CSIRO’s Reputation
The Government is funding AREN with $42.5 million. This project is one
potential high profile outcome that can demonstrate benefits from the
government investment. If CSIRO is able to use this project to enhance its
reputation in the eyes of government and the international research
community, it should improve prospects for future government funding to
CSIRO.
8.6
Returns and Benefits
Key Project Objectives
Output
Measure/TYPE
(IRACIS)
A Regional Network for AARNet in
NSW
Fibre links to CSIRO sites
IS
A Research Network to connect
observatories
Dedicated high-speed links
for observed data
IR, IS
A Production network with gigabit
links
One IT Service Delivery
AC, IS
Outcome(s)
Measure/TYPE (IRACIS)
Accountable
Stakeholder
Enhanced value of observatory
infrastructure

IR
ATNF
Improved efficiency for consolidated
service delivery

AC
One-IT
9. Cost Analysis
9.1
Infrastructure Costs
CWDM Option
The CSIRO initial cost would be between $1 million and $1.5 million with annual
maintenance between $150,000 and $350,000.
DWDM Option
The CSIRO initial cost would be between $1.7 and $2.5, with annual maintenance between
$300K and $500K
Last updated: 29-Jun-17
v0.5
15 of 23
ITS BUSINESS CASE
9.2
Network Links to Observatories
Staff Costs
Time and travel for ITS Networks project staff - $150K
10. Staff Impact/Change Management
There will be changes to the way ATNF scientific staff time is used during
observations for VLBI.
There will be changes to the job descriptions of IT support staff at Parkes, ATNF
Narrabri and the Cotton CRC. These changes will come as part of the One-IT
project, but the One-IT changes will have more impact on sites with gigabit
connectivity (compared to sites on slow links).
Last updated: 29-Jun-17
v0.5
16 of 23
ITS BUSINESS CASE
Network Links to Observatories
Attachment A: Related Project: Link Cotton CRC to ATNF
Narrabri
The AARNet connection at Narrabri will be into the computer room at the ATNF site. The
fibre laid from the AARNet backbone to ATNF will have at least 12 pairs, so that Myall Vale
can be connected to ATNF using one of the spare pairs. The Myall Vale site will then be
configures as two or more VLANs of the local network based at ATNF.
The cost to provide
1. tail-end fibre from the road into Myall Vale
2. active equipment at each end of the ATNF-Myall Vale link
will be $50,000.
The saving from discontinuing the current leased line to Myall Vale will be $30,000 per
year.
This project will enhance collaboration of the Cotton CRC and its 11 core participants
around Australia.
Last updated: 29-Jun-17
v0.5
17 of 23
ITS BUSINESS CASE
Network Links to Observatories
Attachment B –Reduced Costs and Increased Benefits of
Fast Network Links to Radio Astronomy Observatories
A. Tzioumis
20/4/2004
This note provides estimates of current costs and attempts to quantify the gains that will
result from fast network links. The most significant aspect is the greatly enhanced
performance of the network and the consequent enabling of new science.
Position CSIRO as a World Leader in e-VLBI
Other research groups working on real-time correlation for VLBI are aiming at data rates
of 1-2 Gb/sec. CSIRO aims to operate with data rates of at least 2 x 8 Gb/sec within 3
years and this would provide the world's best VLBI sensitivities. This will open new
avenues of astronomical research and has the potential to revolutionise the VLBI field and
lead to significant new discoveries.
An interim 1 Gb/s network would be useful scientifically and a technologically necessary
step towards the higher data rates. The ATNF would be able to utilise shared Gb/s links
but dedicated multiple 1 Gb/s links would provide much better opportunities for significant
scientific improvements.
Current Costs and Outputs
Costs for the VLBI project at ATNF and partners in Australia. Values are approximate and
rounded off.

The components are
o ~5% of time on ATNF telescopes, total value of about $200M (ATCA $100M,
Parkes $80M, Mopra $20M)
o + 5% of time on UTas antennas, valued at $40M (Hobart $20M, Ceduna $20M)
o + 2% of Tidbinbilla time, total value of ~$100M

Special correlation equipment (correlator, PTs, tapes) valued at $1M

~5% of cost of operating & maintaining the telescopes and general ATNF
facilities. ATNF operations budget is ~$10M (out of total of $20M). Hence, prorata operating cost of VLBI is about $0.5M per year. This includes VLBI specific
components i.e.
o
Staff for operations support and correlation at ATNF ~ 1 FTE
o
Support for VLBI observations at ATNF of at least 20-25 person weeks/year (23 people/antenna x 3 ATNF antennas x 3x1-week periods/year)
+ Operating costs at UTas & Tidbinbilla.

Annual output about 8 papers/year (= 5% of ATNF papers (156 in 2003))
With Shared Gigabit Links
When all the telescopes and correlator facilities have gigabit connections via a shared
backbone, there are a number of options to use the network to:
 streamline current operations, with real productivity gains
 achieve a modest increase in network sensitivity and performance.
To achieve any of these gains a substantial proportion of the network links will need to be
devoted to e-VLBI during observing sessions.
Last updated: 29-Jun-17
v0.5
18 of 23
ITS BUSINESS CASE
Network Links to Observatories
However, it should be noted that shared 1 Gb/s links will only approach the performance
already achieved with disk-based VLBI systems. To improve on that the ATNF could use
multiple dedicated 1 Gb/s links if these became available. The advantages will be:
A. The ability to transfer the current data rates (128 Mb/s) in real time to the
correlator facility in Sydney would greatly benefit operations, i.e.
 save most of the 20-25 person weeks/year needed to support observations.
2-3 people could run most of the network remotely, and some of this
support could also come from current correlator operations support. Up to
20 person weeks/year may be saved.
 provide near real-time results, which will greatly enhance the reliability of
the VLBI network and provide the ability to react to transient events
 attract new astronomers to VLBI by greatly simplifying the effort currently
needed.
B. Increase the data rates (2, 4 or 8 times) to increase VLBI sensitivity (Attach. C).
 256Mb/s are available now at the telescopes and could be handled with a
little effort at the Sydney correlator. Sensitivity increase ~40%.
 512 Mb/s are available but need a modest effort to interface to the network.
Will have to be processed by the Swinburne super-computer by storing on
disks and correlating more slowly. Sensitivity increased x 2.
 1 Gb/s may be achievable for special projects by duplicating some
equipment at the telescopes. Again only the Swinburne supercomputer
could handle these rates. Sensitivity increased ~ x3.
The increases in data rates and sensitivity substantially enhance the achievable science.
The largest impact is increasing the numbers of fainter astronomical objects that could be
studied. These are relatively moderate for the 2-3 times increase in sensitivity offered by
this network and comparable to what is achievable with disk systems.
The value of these increases could be quantified by comparisons with equivalent increases
in observing time or antenna size. Details are given in Attachment C. In summary, the
faster network rates are effectively equivalent to:
 utilising 10-40% of the network while only using 5% of time
 increased productivity by "saving" equivalent costs of $1-4M/year
 increasing the effective value of the ATNF facilities from $200M to $260-500M.
Thus the 5% VLBI time effectively increases in value from $10M to $12-25M.
With Dedicated multiple 10 Gb/s Links
When the new correlator and the dedicated 2 x 8 Gb/s data paths are in use (in about 3
years), the correlation will be done in real time, and the sensitivity will be enhanced about
10 times. This will:
 open up new avenues of research by greatly increasing the number of
astronomical objects for study. This has the potential for many significant
discoveries in high resolution radio astronomy.
 provide savings in operating costs by effectively eliminating the specific VLBI
costs of the current network and integrating operations with the ATCA at
Narrabri.
 enhance and facilitate access to VLBI observations for the whole astronomical
community and it is expected to bring many new users to VLBI. Thus VLBI in
Australia will cease to be a niche technique and will join the mainstream of
radio astronomy observing.
 increase the proportion of VLBI time by increasing the scientific capabilities and
attracting new users.
 generate equivalent savings by greatly increasing the productivity of the
telescopes (see Attachment C for details). In summary these are effectively
equivalent to:
Last updated: 29-Jun-17
v0.5
19 of 23
ITS BUSINESS CASE
o
o
o
Network Links to Observatories
the 5% of time is increased 128-fold! 1-week then will be equivalent
to 2.5 years of continuous observing on the current systems. Or 1 day
on the new system would be equivalent to the combined rates of the
previous 6 years on the current VLBI system!
equivalent operational budget of $64M at current data rates
equivalent to having telescopes worth ~$2.7B The 5% VLBI time then
effectively increases from ~$10M to ~$135M.
The development of the networks, new correlators and the e-VLBI operation are all key
technologies for the SKA. Thus these developments will facilitate demonstration of SKA
capabilities in Australia and enhance Australia's prospects to host the ~US$1B SKA.
Last updated: 29-Jun-17
v0.5
20 of 23
ITS BUSINESS CASE
Network Links to Observatories
Attachment C - Quantifying the gains from increased VLBI
data rates & sensitivity
A. Tzioumis
20/4/2004
Introduction
It is difficult to accurately quantify the gains that may be achieved by increasing the data
recording rates The approaches here often oversimplify the situation and are meant only
as a guide. They give an idea of the cost of achieving similar increases by other available
means. Alternatively, they provide an indicative dollar value of the proposed
improvements in data rates.
Some concepts
Radio telescopes record extremely faint signals, many millions of times below the radio
noise level. Hence, high sensitivity and special correlation techniques are used to detect
the signal from the general radio noise.
The sensitivity of a radio telescope or array, is determined primarily by:
 The size of the antenna. Sensitivity is proportional to the dish Area (A), which
 2
is determined by the dish Diameter (D). Area A   D

The sensitivity of the receivers, as measured by the system Temperature
(Tsys). State-of-the-art receivers achieve Tsys = 20K. Sensitivity is
proportional to



2
1 Tsys .
The Bandwidth (B) of the receiving system in MHz or GHz. This determines the
output data rate in Mb/s or Gb/s. Usually, output data rates are 4 x B.
The integration time T i.e. the time that raw data is appropriately averaged
together to increase the signal-to-noise-ratio (SNR) of the data.
Sensitivity is proportional to
B T
The sensitivity of the telescope provides a measure of the weakest radio signals that can
be detected. The vast majority of astronomical objects emit very weak radio signals.
Hence, sensitivity increases open to study a much greater population and wider range of
astronomical objects. This enhanced "parameter space" of study most often provides the
most exciting discoveries in astronomy.
Sensitivity enhancement
The sensitivity of a radio telescope or array can be increased by:
 increasing the size of antenna. This can be very expensive as the cost of an
2.7



antenna scales as D
(See Thompson, Moran, and Swenson, p. 163).
decreasing the system temperature Tsys. Improvements are made in this area
all the time but changing receiver hardware can also be expensive.
increasing the Bandwidth B, and hence the data rates. This is the area that fast
network speeds have an impact.
increasing the integration time T. This is also very expensive as telescopes are
expensive to operate. In some cases like VLBI, the utility of longer integration
is often limited by other technical factors like phase stability and coherence.
The improved sensitivity resulting from increases in Bandwidth (made possible by higher
network rates) can be quantified at a simple level by comparing with the equivalent costs
of changing other parameters in the system like Time (T) and antenna Area (A).
Last updated: 29-Jun-17
v0.5
21 of 23
ITS BUSINESS CASE
Network Links to Observatories
A. Comparison with Time T
As sensitivity is proportional to B  T , any increase in Bandwidth B while Time
remains constant is directly equivalent to keeping B constant but increasing Time
by the same amount. For the proposed e-VLBI systems the Bandwidth increases
have the same effest as the data rate increases. Hence:
 The 1 Gb/s shared network proposed will allow increases in data rates from the
current 128 Mb/s to 256 or 512 Mb/s or even up to 1 Gb/s. Thus data rates
increase by factors of 2-8. This is equivalent to increasing the time of operation
2-8 times. As VLBI uses 5% of time on CSIRO facilities valued at $200M,
increases by factors of 2-8 provide that many times increase in productivity of
these assets. It may be thought of as equivalent to utilising 10-40% of the
facilities while only really using 5% of the time. Equivalently, the 5% of VLBI
time can be considered to cost 5% of the ATNF operating budget of $10M/year
i.e. $0.5M/year. An effective time increase of 2-8 times can then be thought to
provide equivalent productivity increases in saved operating costs of $14M/year! It would cost that much if one could buy the extra time on the
telescopes.

The 2x8 Gb/s e-VLBI network that would be made possible with the advanced
10 Gb/s network links, achieves an 128-fold increase in data rates or
equivalently in bandwidth. Translating these to Time gives a 128-fold increase
in effective time utilisation. Thus the 5% on the $200M facilities becomes
640% i.e. it is equivalent to using full-time 6 such facilities or using the
facilities exclusively for VLBI for 6 years! Similarly, the $0.5M/year pro-rata
operational budget in ATNF for VLBI becomes an equivalent $64M saving!
CAUTION: These need to be treated carefully, as in many cases the increases in
bandwidth cannot actually be substituted by increases in time. The above figures
are intended to give a feel for equivalent costs.
B. Comparison with telescope Area A
Sensitivity is also directly proportional to the size of the antennas and the costs of
these are relatively well known. So, the increases in sensitivity due to the increase
data rates and hence Bandwidth can also be translated to equivalent increases in
telescope collecting area.
Increases in Bandwidth increase sensitivity as
increase in Area also scales as
the Diameter
increase of
B and hence the equivalent
(increase _ in _ B) . The telescope Area A scales as
D 2 . Thus an increase in B would correspond to a Diameter D
1
4
(increase _ in _ B) .
However, it has been well established that antenna building costs scale as
Thus antenna costs would scale as
(increase _ in _ B)
1
( 2.7 )
4
D 2.7 .
.
The ATNF antennas comprise 7 x 22m antennas (ATCA and Mopra) and the 64m
antenna at Parkes. Mechanical construction costs of these are about $100M ($4m
for each 22m antenna, $70M for Parkes).
 The 1 Gb/s shared network proposed will allow increases in data rates from the
current 128 Mb/s to 256 or 512 Mb/s or even up to 1 Gb/s. Thus data rates
increase by factors of 2-8. This is equivalent to increasing the area of the
antenna by 1.4-2.8 times and the antenna cost by 1.6-4 times. Effectively, the
Last updated: 29-Jun-17
v0.5
22 of 23
ITS BUSINESS CASE

Network Links to Observatories
equivalent value of the ATNF facilities increases from $200M to $260-500M.
The 5% VLBI share increases accordingly (from $10M to $12-25M).
The 2 x 8 GB/s e-VLBI network that would be made possible with the advanced
10 Gb/s network links, achieves an 128-fold increase in data rates or
equivalently in bandwidth. The increase in sensitivity or equivalently in area is
11 times and the effective antenna cost increase is 26 times! Hence, the
equivalent value of the ATNF facilities increases to $2.7B!! The 5% VLBI time
effectively increases in value from $10M to $135M.
CAUTION: Again, these numbers are meant only to demonstrate that increases in
data rates achieve increases in telescope sensitivities that could be achieved by
building bigger telescopes costing many $10Ms or even $100Ms.
Comparison with other planned facilities
The only planned high data rates facility is the e-MERLIN in the UK. It is a similar
instrument but covers shorter distances from 5-200km and has less overall collecting
area. It will use dedicated fibre links for rates up to 24 Gb/s but the effective Bandwidth
will be the same as the ATNF proposal. The extra bits are to be used for RFI mitigation
which is a lesser problem in Australia. However, we can achieve similar RFI mitigation
effects by increasing our rates as our correlator can handle a phenomenal 128 Gb/s from
each telescope!
This project has been funded as an upgrade of the existing MERLIN radio-linked
instrument and will cost about A$20M. (Details in http://www.merlin.ac.uk/e-merlin/)
Summary
The proposed increases in data rates for VLBI that will be provided by the new network
connections will provide very significant increases in observational capacity. These can be
roughly quantified by comparison with equivalent increases that could be achieved by
increasing observational time or the size of the telescopes. In both cases, the equivalent
savings in operational costs or the equivalent increase in the value of the facility are many
millions of dollars.
However, the main impact of the increased sensitivity is the opening of new areas of
study in radio astronomy and the huge scientific impact that these can have.
Last updated: 29-Jun-17
v0.5
23 of 23