Download Elder Matias - Invited Talk at Canadian Information Process Society

eScience – The CLS Case Study
E. Matias and S. Lomas
For the Project Team
BigBANGWIDTH  CLS  IBM  University of Alberta  University of Western Ontario
Agenda
•
•
•
•
•
•
•
What is eScience
The Canadian Light Source Facility
Traditional User Access Model
CLS Remote Access Project
User Configurable Light Paths
Lessons Learned
Conclusion
What is eScience?
• Providing a means for scientists to do better
science, faster and at lower cost using
advanced networking.
In addition to CLS remote access are there
other examples?
– SETI
– Radio Telescopes
– Neptune
– Grid Computing….
Examples…. Neptune
Neptune @ University of Victoria
• Seafloor observatory
• Remote Telemetry System (800km ring)
• Joint Canada/US Project
Examples …. Grid Computing
• What is a grid
– Clusters of clusters linked together
– Generally geographically distributed
• Some examples:
– WestGrid (covering major universities in western
Canada is an example)
– CERN - LHC (27km accelerator) is an example
of an experimental setup where no one county
has sufficient processing capabilities
Examples …. LHC
– LHC at CERN is an example of an grid application where
no one county has sufficient processing capabilities
– 15 million gigabytes of data per year
– In 2006 LHC Tier 1 Grid was tested
– TRIUMF is a Canadian Tier 1 Centre for LHC Experiments
Courtesy TRIUMF
Examples….. WestGrid
WestGrid and WestGrid II
•
•
30 Terabyte disk storage
360 Terabyte tape storage
Partners Institutions:
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
$50M Project
Diverse cluster of machines and
resources for high-performance
computing
Example Resources:
• 1680 processor IBM eServer Blade
Centre
• 256 processor HP XC Cluster
• 144 processor HP AlphaServer
• 20 – 4 processor HP ES40s
• 256 processor SGI Origin
• Two 64 processor IBM p595
• Cray XD1 system
• 56 processor IBM JS20
•
•
•
•
University of Victoria
University of British Columbia
Simon Fraser University
University of Northern British Columbia
TRIUMF
University of Lethbridge
The Banff Centre
University of Calgary
University of Alberta
Athabasca University
University of Saskatchewan
University of Regina
University of Manitoba
University of Winnipeg
Brandon University
Industry Partners:
–
HP, IBM and SGI
Agenda
•
•
•
•
•
•
•
What is eScience
The Canadian Light Source Facility
Traditional User Access Model
CLS Remote Access Project
User Configuration Light Paths
Lessons Learned
Conclusion
Canada’s Synchrotron Facility
• Located on the University of Saskatchewan
Campus
• One of the largest national science projects in a
generation
• Accelerator Complex Consists of:
–
–
–
–
Linac Accelerator (200 MeV)
Booster (200 MeV to 2.9 GeV) Ring
Storage Ring (170.88 m - 2.9 GeV)
Two diagnostic beamlines
• Seven beamlines in operation or commissioning
• Six beamlines under construction
• Six beamlines being proposed (not-funded)
Machine Layout
Beamlines
• Operate in parallel harnessing synchrotron light
generated by the circulating electron beam
• Each beamline is tailored to a class of experiments
• Beam-teams of users provide scientific direction
• Average beamline contains 50 to 150 motors
• Data rates are different from beamline to beamline
CMCF 2 Beamline
• Operate in parallel harnessing synchrotron light
generated by the circulating electron beam
• Each beamline is tailored to a class of experiments
• Beam-teams of users provide scientific direction
• Average beamline contains 50 to 150 motors
• Data rates are different from beamline to beamline
Applications: Proteins & Disease
Applications: Medical Studies
Applications: Mining & Petroleum
Applications: Better Satellites,
Planes and Cars
Applications: Nano-Technology
Agenda
•
•
•
•
•
•
•
What is eScience
The Canadian Light Source Facility
Traditional User Access Model
CLS Remote Access Project
User Configurable Light Paths
Lessons Learned
Conclusion
Traditional User Access Model
• Academic Users
–
–
–
–
–
The user applies for beam-time once every six months
Peer review (Independent Committee)
Safety review (CLS)
Technical feasibility review (CLS)
Time scheduled in 8 hour slices usually multiple shifts
• Commercial Users
– Straight fee-for-service arrangement without peer review
A good target for web based automation…..
Automation?
• This sounds like an electronic document
management problem?
• Yes, it is.
• Good target for outsourcing.
• The selected platform:
– Cronus Technologies “C-Factor” and
– Oracle.
• Hosted off-site by Cronus
Managed by our User Services Office
Next Project?
• This time gain the flexibility of SOA
• Look at developing the infrastructure for
remote access and to interweave different
users.
• We have three beamlines that by their
nature are good targets for remote access.
Agenda
•
•
•
•
•
•
•
What is eScience
The Canadian Light Source Facility
Traditional User Access Model
CLS Remote Access Project
User Configurable Light Paths
Lessons Learned
Conclusion
Project Terms of Reference
• Major project funding from CANARIE with additional
funding from:
–
–
–
–
•
•
•
•
IBM
Canadian Light Source
Bigbangwidth
University of Western Ontario
UP & UML software engineering approach
All custom developed software will be open source
Project duration 14 months
Project is completed before the target beamlines
– Therefore concentration is on reusable components
• Service Oriented Architectures and Web Services
Project Delivery Team
•
CANARIE Funded Project
•
Canadian Light Source
–
–
–
–
–
–
–
–
–
•
IBM Canada
–
–
•
Marina Fuller (Requirements and Testing)*
Stewart McIntyre (User Champion VESPERS)
Gary Good (System Support)
Alberta Synchrotron Institute
–
•
Chris Armstrong (System Architect)*
John Haley (System Analyst/Architect)*
University of Western Ontario
–
–
–
•
Dionisio Medrano (System Analyst)*
Daron Chabot (System Analyst)*
Jason Chan (Intern)*
Elder Matias (Project Leader/Manager)
Michel Fodje (CMCF Beamline Scientist)
Renfei Feng (VESPERS Beamline Scientist)
Jason Cyrenne (Networking)
Bob Harvey (Networking)
Russ Berg (EPICS/CMCF)
Ernst Bergman (User Champion CMCF)
Big Bangwidth
–
–
Stuart Lomas (Networking)
Steve Hyatt (UCLP WebServices Software)
* Full Time
High Level Functional
Requirements (Long Term)
Beam Lines
Experiment Configuration
- Control beam line
- Manage experiments (add, edit, delete)
- View experiment results/data
- Save experiment results
- Run experiments on simulated. Beam Line
Experiment
setup/results
ELF
- Analyzed results (PX)
Analyze and Process Results
- Send result to be analyzed
- Analyze results (ie. Sample scoring)
- Process results/data
Protein Solvers
- Analyzed results (PX)
Create 3D Models
External Systems
Remote
Access &
Control
- Analyzed results (PX and Vespers)
- interface into other systems
Quartz
Sample Management
- Analyzed results (Vespers)
- Enter sample
- Track sample
- Sample Management
User Office
- View user details
- Create new project
- View current projects
- View completed projects
- Schedule projects
- Login
- Logout
Australian Map Code
Sample
Management
- Analyzed results (Vespers)
BigBangWidth
- Access to networking web service
Kronos System
- User accounts
- Experiments accounts
- Beam time schedule
Beamline Scheduling
- Schedule Beamline
Component Overview
Metadata
Network
UI
Services
Message
Services
Users
EPICS
Hardware Controllers
User
Interfaces
Performance Metrics and Diagnostic Logging
• Challenge:
–
–
–
–
Experiment
Results
Support dynamic reconfiguration
Support on-line changes in a 24/7 environment
Provide flexibility and ease in reconfiguring the environment
Separation of meta data from presentation
• Solution
– XML based configuration information instead of hard-coding
Beam Line
Component Overview
Metadata
Network
UI
Services
Message
Services
Users
EPICS
Hardware Controllers
User
Interfaces
Beam Line
Performance Metrics and Diagnostic Logging
•
Challenge:
–
–
–
•
Experiment
Web-standards are still immature,
Results
Must have a real-time feel to the user
Diverse client hardware/software independently selected at each university
Solution
–
–
–
Thin-client browser (Java Script)
AJAX used to provide real-time like interface with Spring Framework
Identified a single supported browser (Fire-fox) all others at users own risk
Component Overview
Metadata
Network
UI
Services
Message
Services
Users
EPICS
Hardware Controllers
User
Interfaces
Beam Line
Performance Metrics and Diagnostic Logging
•
Challenge:
–
–
–
•
Experiment
Requires secure data transfer
Results
Real-time performance – Guaranteed Quality of Service
Users located at major research Universities, Institutes in Canada and Australia
Solution
–
–
LightPath and LighPath Accelerator Technology
CANet4 with International connections
Component Overview
Metadata
Network
UI
Services
Message
Services
Users
EPICS
Hardware Controllers
User
Interfaces
Beam Line
Performance Metrics and Diagnostic Logging
•
Challenge:
–
–
–
–
•
Experiment
Results
Robust Reliable
User performance requirements are unknown
Support on-line changes, since the Facility Operates 24/7 with limited outage periods
Common interface presented to the user
Solution
– Websphere Hosted
– Provides Services for Managing Users and Presentation of Data to the User
– Spring Framework and Custom Java Classes
Component Overview
Metadata
Network
UI
Services
Message
Services
Users
EPICS
Hardware Controllers
User
Interfaces
Beam Line
Performance Metrics and Diagnostic Logging
•
Challenge:
Experiment
Results
– The “glue” that ties things together
– Ability to Interface to both Internal and External Services
– Reliable, flexible, ability to deal with services connecting and disconnecting gracefully
•
Solution
– Provides internal and external services to communicate with other systems, analysis
codes etc.
– Web-services for diverse and distributed services
Component Overview
Metadata
Network
UI
Services
Message
Services
Users
EPICS
Hardware Controllers
User
Interfaces
Beam Line
Performance Metrics and Diagnostic Logging
•
Challenge:
Experiment
Results
– Understand where the system fails
– Achieve performance objectives
– Allocate resources to performance only where there is a clear measurable benefit
•
Solution
– Build in some basic auditing to determine bottlenecks and trace faults
Component Overview
Metadata
Network
UI
Services
Message
Services
Users
EPICS
Hardware Controllers
User
Interfaces
Beam Line
Performance Metrics and Diagnostic Logging
•
Challenge:
– Several Terra bytes of data
– Several megabytes per experiment
– Provide good performance
•
Experiment
Results
Solution
– Storage Area Network (SAN)
– Light-paths to permit the rapid transfer of data to the user home institution
Component Overview
Metadata
Network
UI
Services
Message
Services
EPICS
Users
Hardware Controllers
User
Interfaces
Beam Line
Performance Metrics and Diagnostic Logging
•
Challenge:
Experiment
Results
– Control diverse hardware
– Implement motion control and data acquisition algorithms
– Support both local and remote access
•
Solution
– EPICS (framework extensively used at synchrotrons around the world)
– Integrate vendor and other libraries as needed
•
Legacy System?
EPICS Overview
Metadata
EDM
Matlab
Spec
IDL
Labview
Interface
Remote
Access
Network
Channel
Access
Scanner
Channel
Access State
Server Machine
Engine
Network
Device
Drivers
Network
Hardwired Driver
VME, Firewire
Serial Drive
VME, Firewire
Hardware Controllers
Application
Software
Fault Logging
EPICS Provides:
Supervisory Control and Data Acquisition (SCADA) or
(Distributed Control System) DCS functions
• Originally written by the US Department of Energy in 1980s
• Widely used at large scale science facilities around the world
EPICS
Profibus
TCP/IP
Operator Workstation
User Applications
CA
Touch
Panels
CA
State Machine
Engine
CA
IOC
VME
CA
Single Board
Computer
Channel Access Protocol
CA
CA
IOC
Siemens
S7/300 PLC
Modbus
TCP/IP
CA
IOC
CA
Telemecanique
Momentum PLC
GPIB
IOC
CA
RS-232
IOC
Logical Component Diagram
Experiment
Metadata
jdbc
Collaboration
Service
TBD
Experiment
Data
file
Channel (CAJ)
Browser
UI
XML/HTTP(S)
UI Presentation
Services
Experiment
Services
tcp
Epics
Beamline
Services
Channel (Adaptor)
(various)
(Adaptor)
XML/file
XML/file
Other Control
Software
(various)
Presentation
Config. Data
Services
Beamline
Config. Data
- analysis
- 3rd party analysis
- system services (e.g. LDAP)
Functional
Component
Data
Component
Dependency:
protocol
Layout
Detector
Scan
Start Scan
Stop Scan
Exposure
(seconds)
Estimated
Scan Time
Current
Zoom
Future
Zoom
Step Across
(microns)
Number of
Steps Across
Current
Focus
Future
Focus
Step Down
(microns)
Number of
Steps Down
Camera
Height
(microns)
Source
Sample
Position
Move
Y
H
Z
X
45º
Stop
Current X
Future X
Current H
Future H
Y is a function of H
Z is a function of H
On
Off
Source
Status
Remote Beamline Access
Prototype Architecture
J2EE - SPRING
Web Server
UI - jaxscript
rest
Spring MVC
EPICS  Spring
Bridge
Beamline - EPICS
Network
Browser
Javascript/Ajax
Application
Logic
POJOs
SOAP
Other Services
- SOAP, etc.
Persistence
- Spring DB
RDBMS
(DERBY)
Simplified Business Model for
Prototype
Project
ProjectRole
-name
-organizationName
-startDate
-attributes*
-notes
-person
-role
*
1
1
1*
Experiment
Sample
*
-name
-id
-shippedDate
-receivedDate
-attributes*
-notes
1
*
1
*
BeamlineControl
-stateAttributes
-currentState
+setSamplePos()
+setSlitSize()
+setScanDuration()
+startScan()
+stopScan()
1
*
*
1
-startDateTime
-endDateTime
-attributes*
-notes
1
ExperimentData
-startCollectionTimestamp
-endCollectionTimestamp
-fileName
-notes
-stateValue
*
Application Screen Shots: Meta
Data
Application Screen Shot:
Experiment
Agenda
•
•
•
•
•
•
•
The Canadian Light Source Facility
Traditional User Access Model
What is eScience
CLS Remote Access Project
User Configurable Light Paths
Lessons Learned
Conclusion
Network Architecture
request
Domain
Manager
control
LPA816e
UBC
control
Control
(UCLP)
Canadian Light Source
Saskatoon
LPA816e
SFU
LPA816e
U of Lethbridge
LPA816e
U of Calgary
CMCF
Beamline
Computer
LPA816e
CANARIE
and ORAN
Lightpath Network
LPA816e
U of Alberta
LPA816e
UWO
LPA816e
U of Toronto
VESPERS Beamline
Computer
LPA816e
Lightpath Accelerator network.
This solution allows any one research
workstation, at right, to connect to one of
the two beamline computers at the CLS.
This solution can be extended to allow
multiple connections through each
LPA816e Lightpath Director.
October, 2006
McGill
LPA816e
NRC-BRI
BigBangwidth Lightpath
Accelerator
BigBangwidth Lightpath Accelerator extends and
automates Advanced Networking Lightpaths
 Software and hardware system
 Extends Advanced Network Lightpaths across the LAN
• “Last Mile Solution” for lightpaths
 Interfaces to CANARIE UCLP
• Provides an automated front end, no IT
 Also works without UCLP, in LAN or WAN
LPA816e lightpath cross-connect
Lightpath Accelerator Operation
This graph shows the traffic each second as a stream is
recognized and moved from the LAN (blue) to a lightpath
(red).
Integrating LPA with UCLP
Lightpath Accelerator
controls a software
virtual crossconnect that
commands UCLP.
real device
In effect, CA*Net4 is
treated as a single
lightpath cross-connect
UCLP
commands
virtual device
domain manager
process
real device
Agenda
•
•
•
•
•
•
•
The Canadian Light Source Facility
Traditional User Access Model
What is eScience
CLS Remote Access Project
User Configurable Light Paths
Lessons Learned
Conclusion
Lessons Learned
• The technology is powerful but very complex
• Web-services and the associated technology are
not mature this was a problem in this project
– Standards are a moving target
• When evaluating frameworks, the products were
evolving faster than the selection process
• Engaged consultants and collaborators with
previous experience (IBM, and BigBangwidth in our
case)
• Concepts are not completely new
The End
Thank you.