Download G2 - Paul Roe, Associate Professor

G2 and ActiveSheets
Paul Roe
QUT
[email protected]
Yes Australia!
1
Overview: Two Projects
1. G2: .NET framework which makes cycle
stealing easy
2. Active sheets: Spreadsheets for cluster /
grid computing
•
Underlying theme: Databases are cool!
2
G2 Cycle Stealing Framework
3
Cycle Stealing Requirements
•Security
•No extras to install
•No extra sys admin
•No interference
•Eye candy!
System must be transparent
•Simplicity
•No new libraries or prog models
•No complex deployment
•No volatile workstations to
consider
•Maximise volunteers (minimise
volunteer requirements)
System must be transparent
G2
framework
Application
programmer/
user
.NET CLR and
Frameworks
Volunteer
(user / sys admin)
4
G2 Virtual Web Service Server
G2 Virtual Server
volunteers
Client
Code
G2
Data
SOAP
Internet
HTTP
Code
Data
Job
DB
Code
Data
Internet
Code
Data
Code
Data
•
•
•
•
Functional (dataflow) model of computation based on WS
Virtual high performance web service server
G2 programming model identical to .NET web services
Code and data automatically and transparently distributed5
G2 leverages .NET
• .NET CLR provides sandbox security
– Jobs run within IE
• Use Windows scheduler to detect idle PC
• Standard web service communications, extended to support:
– G2 intermediary server
– Web service caching / persistence
– Reliable messaging
• Code deployment (download)
– Extended to support automatic code upload to server
• Isolated storage, extended to support:
– Automatic and transparent file caching
– Job checkpointing
• Visual studio plug-in to support writing G2 components “add G2
reference” vs “add web reference”
• SQL server database to store jobs (persisted web services) on
server
– DB provides reliability, assume volunteers and clients may fail
6
G2 Family
• G2 Classic
• G2 P2P
–
–
–
–
Peer to peer
Distributed objects
Uses remoting rather than web services
Custom remoting channel
• G2X
– XML/Yukon version of G2
– Supports querying of database
• G3
– Adaptive G2
– Combines P2P, G2X and dedicated cluster
7
Status
• ‘Local’ applications
– Data mining
– Code cracking – brute force cipher analysis
– GA: VLSI layout
• Lot of performance tuning, scheduling etc.
• G2 classic: general release soon, seeking
early users
• Formal model of performance
• g2.fit.qut.edu.au
8
ActiveSheets Vision
•
•
•
•
Spreadsheet driven HPC = accessible HPC
Spreadsheets powerful and popular end user tools
No parallel programming or grid expertise required
Good pre and post processing of data, charting etc.
9
Parallel Computing for Users
• Spreadsheets: dataflow and data parallelism
A
B
C
D
E
1
1
2
3
4
=sum(A1:D1)
2
=A1+B1
3
=A2*C2
=C1+D1
• Parallelism from data and independent functions
– Natural parallelism a user can understand
10
Excel
• Complex and very sophisticated
• Extensible
– Can define custom functions and plug-ins
– Custom functions (stubs) represent backend
functionality
• Sequential evaluation mechanism
– Extend Excel to support parallel evaluation (plug-in)
– Peacefully coexist with Excel don’t re-implement
evaluation or break existing spreadsheets
11
Architecture
• Recently rebuilt using .NET
• Support for WS & NetSolve
• Auto generate Excel stub funs
from metadata e.g. WSDL
• Local database cache
• Pluggable backend adapters
Excel
Add-in
Adapters
NetSolve
AS Engine
(.NET)
Job
Manager
(Result
Cache)
…
…
Web Services
12
Windows service
Status and Plans
• Currently testing, hope to make a
distribution available soon (July)
• Looking for beta testers
• Future plans
– Remote data
– Greater job control
– Workflow
– More support for WS (e.g. WS-security) and
OGSA
13
Questions?
• If you are interested in either project
please contact me: [email protected]
14
G2 System Architecture
Web services
G2
proxy
Web
Server
Web
Browser
Client
Program
G2
Middleware
Web
Browser
Job
DB
Web
Browser
Client
Key:
Server
G2
Framework
Parallel
App.
Parallel
App.
Parallel
App.
Volunteers
Programmer
supplied
Off the shelf
.NET
15
G2 Code Distribution
G2
Proxy
Client
Component
Refers to
Volunteer
Component
Client
SOAP/
HTTP
Code
Upload
Web
Server
HTML/
HTTP
Web
Browser
Embeds
Volunteer
Host
SOAP/
HTTP
G2
Middleware
Manages
Volunteer
assembly
cache
Server
Server
Code
Download
.NET
Runtime
Loads
Manages
Download
assembly
cache
Volunteer
16
G2 File Staging
Web service
file request
G2
Server
Files
3
Volunteer
2
1
2
Files on the
LAN/Internet
read via HTTP or
NTFS
Files
Files
Isolated
storage
Isolated
storage
17
File caches