Download Scaling Up Data Intensive Scientific Applications to Campus Grids

Scaling Up Data Intensive
Scientific Applications to
Campus Grids
Douglas Thain
University of Notre Dame
LSAP Workshop
Munich, June 2009
1
Overview
Challenges in Using Campus Grids
Solution: Abstractions
Examples and Applications
– All-Pairs: Biometrics, Data Mining
– Wavefront: Genomics and Economics
– Some-Pairs: Genomics
Abstractions, Workflows, and Languages
2
What is a Campus Grid?
A campus grid is an aggregation of all
available computing power found in an
institution:
– Idle cycles from desktop machines.
– Unused cycles from dedicated clusters.
Examples of campus grids:
– 600 CPUs at the University of Notre Dame
– 2000 CPUs at the University of Wisconsin
– 13,000 CPUs at Purdue University
Cluster, cloud grid are all similar concepts.
3
4
5
6
Campus grids can give us access
to more machines than we can
possibly use.
But are they easy to use?
7
Example: Biometrics Research
Goal: Design robust face comparison function.
F
F
0.97
0.05
8
Similarity Matrix Construction
1.0
0.8
0.1
0.0
0.0
0.1
1.0
0.0
0.1
0.1
0.0
1.0
0.0
0.1
0.3
1.0
0.0
0.0
1.0
0.1
Challenge
Workload:
60,000 iris images
1MB each
.02s per F
833 CPU-days
600 TB of I/O
1.0
9
I have 60,000 iris images acquired in
my research lab. I want to reduce
each one to a feature space, and then
compare all of them to each other. I
want to spend my time doing science,
not struggling with computers.
I own a few machines.
We have access to a campus grid.
I have a laptop.
What should I do?
10
We said:
Try using our campus grid!
(How hard could it be?)
11
Non-Expert User Using 500 CPUs
Try 1: Each F is a batch job.
Failure: Dispatch latency >> F runtime.
CPU
F CPU
F CPU
F CPU
F CPU
F
HN
Try 3: Bundle all files into one package.
Failure: Everyone loads 1GB at once.
Try 2: Each row is a batch job.
Failure: Too many small ops on FS.
F
F
F
F
F
F
F
F
F
F
F
F
F
F
F
CPU
F CPU
F CPU
F CPU
F CPU
F
HN
Try 4: User gives up and attempts
to solve an easier or smaller problem.
F
F
F
F
F
F
F
F
F
F
F
F
F
F
F
CPU
F CPU
F CPU
F CPU
F CPU
F
HN
12
Why are Grids Hard to Use?
System Properties:
–
–
–
–
Wildly varying resource availability.
Heterogeneous resources.
Unpredictable preemption.
Unexpected resource limits.
User Considerations:
– Jobs can’t run for too long... but, they can’t run too
quickly, either!
– I/O operations must be carefully matched to the
capacity of clients, servers, and networks.
– Users often do not even have access to the necessary
information to make good choices!
13
Overview
Challenges in Using Campus Grids
Solution: Abstractions
Examples and Applications
– All-Pairs: Biometrics, Data Mining
– Wavefront: Genomics and Economics
– Some-Pairs: Genomics
Abstractions, Workflows, and Languages
14
Observation
In a given field of study, many people
repeat the same pattern of work many
times, making slight changes to the data
and algorithms.
If the system knows the overall pattern in
advance, then it can do a better job of
executing it reliably and efficiently.
If the user knows in advance what patterns
are allowed, then they have a better idea of
how to construct their workloads.
15
What’s the Most Successful
Parallel Programming Language?
OpenGL:
– A declarative specification of a workload.
– Ported to a wide variety of HW over 20 years.
– The graphics pipeline is very specific:
Transform points to coordinate space.
Connect polygons to transformed points.
Stretch textures across polygons.
Sort everything by Z-depth.
– Can we apply the same idea to grids?
16
Abstractions
for Distributed Computing
Abstraction: a declarative specification
of the computation and data of a workload.
A restricted pattern, not meant to be a
general purpose programming language.
Uses data structures instead of files.
Provide users with a bright path.
Regular structure makes it tractable to
model and predict performance.
17
Abstractions as
Higher-Order Functions
AllPairs( set A, set B, function F )
– returns M[i,j] = F( A[i], B[j] )
SomePairs( set A, list(i,j) L, function F )
– returns list of F( A[i], A[j] )
Wavefront( matrix R, function F )
– returns R[i,j] = F( R[i-1,j], R[I,j-1] )
18
Working with Abstractions
A1
A2
An
A1
A2
Bn
F
AllPairs( A, B, F )
Compact Data Structure
Custom
Workflow
Engine
Campus Grid
19
Overview
Challenges in Using Campus Grids
Solution: Abstractions
Examples and Applications
– All-Pairs: Biometrics, Data Mining
– Wavefront: Genomics and Economics
– Some-Pairs: Genomics
Abstractions, Workflows, and Languages
20
All-Pairs Abstraction
AllPairs( set A, set B, function F )
returns matrix M where
M[i][j] = F( A[i], B[j] ) for all i,j
A1
A1
An
B1
B1
Bn
A1
A2
A3
B1
F
F
F
B2
F
F
F
B3
F
F
F
allpairs A B F.exe
AllPairs(A,B,F)
F
21
How Does the Abstraction Help?
The custom workflow engine:
– Chooses right data transfer strategy.
– Chooses the right number of resources.
– Chooses blocking of functions into jobs.
– Recovers from a larger number of failures.
– Predicts overall runtime accurately.
All of these tasks are nearly impossible for
arbitrary workloads, but are tractable (not
trivial) to solve for a specific abstraction.
22
23
Distribute Data Via Spanning Tree
24
Choose the Right # of CPUs
25
Conventional vs Abstraction
26
All-Pairs in Production
Our All-Pairs implementation has provided over
57 CPU-years of computation to the ND
biometrics research group over the last year.
Largest run so far: 58,396 irises from the Face
Recognition Grand Challenge. The largest
experiment ever run on publically available data.
Competing biometric research relies on samples
of 100-1000 images, which can miss important
population effects.
Reduced computation time from 833 days to 10
days, making it feasible to repeat multiple times for
27
a graduate thesis. (We can go faster yet.)
28
Overview
Challenges in Using Campus Grids
Solution: Abstractions
Examples and Applications
– All-Pairs: Biometrics, Data Mining
– Wavefront: Genomics and Economics
– Some-Pairs: Genomics
Abstractions, Workflows, and Languages
29
Wavefront( matrix M, function F(x,y,d) )
returns matrix M such that
M[i,j] = F( M[i-1,j], M[I,j-1], M[i-1,j-1] )
M[0,4]
F
x
d
M[0,3]
M
y
M[0,2]
y
F
d
d
y
F
x
F
x
F
x
d
M[0,1]
x
F
x
d
Wavefront(M,F)
M[2,4] M[3,4] M[4,4]
y
y
F
d
d
x
y
F
d
y
F
x
M[3,2] M[4,3]
y
F
x
d
M[4,2]
y
F
x
d
y
M[0,0] M[1,0] M[2,0] M[3,0] M[4,0]
30
Applications of Wavefront
Bioinformatics:
– Compute the alignment of two large DNA strings in
order to find similarities between species. Existing
tools do not scale up to complete DNA strings.
Economics:
– Simulate the interaction between two competing
firms, each of which has an effect on resource
consumption and market price. E.g. When will we run
out of oil?
Applies to any kind of optimization problem
solvable with dynamic programming.
31
Problem: Dispatch Latency
Even with an infinite number of CPUs,
dispatch latency controls the total
execution time: O(n) in the best case.
However, job dispatch latency in an
unloaded grid is about 30 seconds, which
may outweigh the runtime of F.
Things get worse when queues are long!
Solution: Build a lightweight task dispatch
system. (Idea from Falkon@UC)
32
worker
worker
worker
worker
worker
worker
queue
tasks
wavefront
engine
tasks
done
work
queue
1000s of workers
dispatched via
Condor/SGE/SSH
put F.exe
put in.txt
exec F.exe <in.txt >out.txt
get out.txt
In.txt
worker
F
out.txt
33
Problem: Performance Variation
Tasks can be delayed for many reasons:
– Heterogeneous hardware.
– Interference with disk/network.
– Policy based suspension.
Any delayed task in Wavefront has a cascading
effect on the rest of the workload.
Solution - Fast Abort: Keep statistics on task
runtimes, and abort those that lie significantly
outside the mean. Prefer to assign jobs to
machines with a fast history.
34
500x500 Wavefront on ~200 CPUs
35
Wavefront on a 200-CPU Cluster
36
Wavefront on a 32-Core CPU
37
Performance Prediction is Possible
Often, users have no idea whether a task will
take one day or one year -> better to find out at
the beginning!
Allows the system to choose automatically
whether to run locally or on the campus grid.
Of course, performance prediction is technically
and philosophically dangerous: we simply argue
that abstractions are more predictable than
general programs.
38
Overview
Challenges in Using Campus Grids
Solution: Abstractions
Examples and Applications
– All-Pairs: Biometrics, Data Mining
– Wavefront: Genomics and Economics
– Some-Pairs: Genomics
Abstractions, Workflows, and Languages
39
The Genome Assembly Problem
AGTCGATCGATCGATAATCGATCCTAGCTAGCTACGA
Chemical Sequencing
AGTCGATCGATCGAT
TCGATAATCGATCCTAGCTA
AGCTAGCTACGA
Millions of “reads”
100s bytes long.
Computational Assembly
AGTCGATCGATCGAT
TCGATAATCGATCCTAGCTA
AGCTAGCTACGA
40
Sample Genomes
Sequential
Pairs
Time
Reads
Data
A. gambiae
scaffold
101K
80MB
738K
12 hours
A. gambiae
complete
180K 1.4GB
12M
6 days
S. Bicolor
simulated
7.9M 5.7GB
84M
30 days
41
Genome Assembly Today
Several commercial firms provide an
assembly service that takes weeks on a
dedicated cluster, costs O($10K), and is
based on human genome heuristics.
Genome researchers would like to be able
to perform custom assemblies using their
own data and heuristics.
Can this be done on a campus grid?
42
Some-Pairs Abstraction
SomePairs( set A, list (i,j), function F(x,y) )
returns
list of F( A[i], A[j] )
A1
A1
A1
An
(1,2)
(2,1)
(2,3)
(3,3)
F
A1
A2
A3
F
SomePairs(A,L,F)
A2
A3
F
F
F
43
Distributed Genome Assembly
A1
A1
An
(1,2)
(2,1)
(2,3)
(3,3)
F
queue
tasks
somepairs
master
tasks
done
100s of workers
dispatched to
worker
Notre Dame,
worker
worker
Purdue, and
worker
worker
Wisconsin
worker
detail of a single worker:
work
queue
put align.exe
put in.txt
exec F.exe <in.txt >out.txt
get out.txt
in.txt
worker
F
out.txt
44
Small Genome (101K reads)
45
Medium Genome (180K reads)
46
Large Genome (7.9M)
47
From Workstation to Grid
48
What’s the Upshot?
We can do full-scale assemblies as a routine
matter on existing conventional machines.
Our solution is faster (wall-clock time) than the
next faster assembler run on 1024x BG/L.
You could almost certainly do better with a
dedicated cluster and a fast interconnect, but
such systems are not universally available.
Our solution opens up research in assembly to
labs with “NASCAR” instead of “Formula-One”
hardware.
49
Overview
Challenges in Using Campus Grids
Solution: Abstractions
Examples and Applications
– All-Pairs: Biometrics, Data Mining
– Wavefront: Genomics and Economics
– Some-Pairs: Genomics
Abstractions, Workflows, and Languages
50
Other Abstractions for Computing
Directed Graph
Map-Reduce
Bag of Tasks
R
M
R
51
Partial Lattice of Abstractions
Robust
Performance
All-Pairs
Some Pairs
Bag of Tasks
Wavefront
Map Reduce
Directed Graph
Expressive
Power
Lambda Calculus
52
Two Abstractions Compared
AllPairs( set A, set B, F(x,y) )
SomePairs( set S, list L, F(x,y) )
Assuming that A = B = S…
Can you express AllPairs using SomePairs?
– Yes, but you must enumerate all pairs explicitly. It is not
trivial for SomePairs to minimize the amount of data
transferred to each node.
Can you express SomePairs using AllPairs?
– Yes, but only by doing excessive amounts of work, and
then winnowing the results.
53
Abstractions as a Social Tool
Collaboration with outside groups is how we
encounter the most interesting, challenging, and
important problems, in computer science.
However, often neither side understands which
details are essential or non-essential:
– Can you deal with files that have upper case letters?
– Oh, by the way, we have 10TB of input, is that ok?
– (A little bit of an exaggeration.)
An abstraction is an excellent chalkboard tool:
– Accessible to anyone with a little bit of mathematics.
– Makes it easy to see what must be plugged in.
– Forces out essential details: data size, execution time.
54
Conclusion
Grids, clouds, and clusters provide enormous
computing power, but are very challenging to
use effectively.
An abstraction provides a robust, scalable
solution to a narrow category of problems; each
requires different kinds of optimizations.
Limiting expressive power, results in systems
that are usable, predictable, and reliable.
Is there a menu of abstractions that would
satisfy many consumers of grid computing?
55
Acknowledgments
Cooperative Computing Lab
– http://www.cse.nd.edu/~ccl
Faculty:
–
–
–
–
Patrick Flynn
Nitesh Chawla
Kenneth Judd
Scott Emrich
Grad Students
–
–
–
–
–
Chris Moretti
Hoang Bui
Li Yu
Mike Olson
Michael Albrecht
Undergrads
–
–
–
–
–
Mike Kelly
Rory Carmichael
Mark Pasquier
Christopher Lyon
Jared Bulosan
NSF Grants CCF-0621434, CNS-0643229
56