Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Scaling Up Data Intensive Scientific Applications to Campus Grids Douglas Thain University of Notre Dame LSAP Workshop Munich, June 2009 1 Overview Challenges in Using Campus Grids Solution: Abstractions Examples and Applications – All-Pairs: Biometrics, Data Mining – Wavefront: Genomics and Economics – Some-Pairs: Genomics Abstractions, Workflows, and Languages 2 What is a Campus Grid? A campus grid is an aggregation of all available computing power found in an institution: – Idle cycles from desktop machines. – Unused cycles from dedicated clusters. Examples of campus grids: – 600 CPUs at the University of Notre Dame – 2000 CPUs at the University of Wisconsin – 13,000 CPUs at Purdue University Cluster, cloud grid are all similar concepts. 3 4 5 6 Campus grids can give us access to more machines than we can possibly use. But are they easy to use? 7 Example: Biometrics Research Goal: Design robust face comparison function. F F 0.97 0.05 8 Similarity Matrix Construction 1.0 0.8 0.1 0.0 0.0 0.1 1.0 0.0 0.1 0.1 0.0 1.0 0.0 0.1 0.3 1.0 0.0 0.0 1.0 0.1 Challenge Workload: 60,000 iris images 1MB each .02s per F 833 CPU-days 600 TB of I/O 1.0 9 I have 60,000 iris images acquired in my research lab. I want to reduce each one to a feature space, and then compare all of them to each other. I want to spend my time doing science, not struggling with computers. I own a few machines. We have access to a campus grid. I have a laptop. What should I do? 10 We said: Try using our campus grid! (How hard could it be?) 11 Non-Expert User Using 500 CPUs Try 1: Each F is a batch job. Failure: Dispatch latency >> F runtime. CPU F CPU F CPU F CPU F CPU F HN Try 3: Bundle all files into one package. Failure: Everyone loads 1GB at once. Try 2: Each row is a batch job. Failure: Too many small ops on FS. F F F F F F F F F F F F F F F CPU F CPU F CPU F CPU F CPU F HN Try 4: User gives up and attempts to solve an easier or smaller problem. F F F F F F F F F F F F F F F CPU F CPU F CPU F CPU F CPU F HN 12 Why are Grids Hard to Use? System Properties: – – – – Wildly varying resource availability. Heterogeneous resources. Unpredictable preemption. Unexpected resource limits. User Considerations: – Jobs can’t run for too long... but, they can’t run too quickly, either! – I/O operations must be carefully matched to the capacity of clients, servers, and networks. – Users often do not even have access to the necessary information to make good choices! 13 Overview Challenges in Using Campus Grids Solution: Abstractions Examples and Applications – All-Pairs: Biometrics, Data Mining – Wavefront: Genomics and Economics – Some-Pairs: Genomics Abstractions, Workflows, and Languages 14 Observation In a given field of study, many people repeat the same pattern of work many times, making slight changes to the data and algorithms. If the system knows the overall pattern in advance, then it can do a better job of executing it reliably and efficiently. If the user knows in advance what patterns are allowed, then they have a better idea of how to construct their workloads. 15 What’s the Most Successful Parallel Programming Language? OpenGL: – A declarative specification of a workload. – Ported to a wide variety of HW over 20 years. – The graphics pipeline is very specific: Transform points to coordinate space. Connect polygons to transformed points. Stretch textures across polygons. Sort everything by Z-depth. – Can we apply the same idea to grids? 16 Abstractions for Distributed Computing Abstraction: a declarative specification of the computation and data of a workload. A restricted pattern, not meant to be a general purpose programming language. Uses data structures instead of files. Provide users with a bright path. Regular structure makes it tractable to model and predict performance. 17 Abstractions as Higher-Order Functions AllPairs( set A, set B, function F ) – returns M[i,j] = F( A[i], B[j] ) SomePairs( set A, list(i,j) L, function F ) – returns list of F( A[i], A[j] ) Wavefront( matrix R, function F ) – returns R[i,j] = F( R[i-1,j], R[I,j-1] ) 18 Working with Abstractions A1 A2 An A1 A2 Bn F AllPairs( A, B, F ) Compact Data Structure Custom Workflow Engine Campus Grid 19 Overview Challenges in Using Campus Grids Solution: Abstractions Examples and Applications – All-Pairs: Biometrics, Data Mining – Wavefront: Genomics and Economics – Some-Pairs: Genomics Abstractions, Workflows, and Languages 20 All-Pairs Abstraction AllPairs( set A, set B, function F ) returns matrix M where M[i][j] = F( A[i], B[j] ) for all i,j A1 A1 An B1 B1 Bn A1 A2 A3 B1 F F F B2 F F F B3 F F F allpairs A B F.exe AllPairs(A,B,F) F 21 How Does the Abstraction Help? The custom workflow engine: – Chooses right data transfer strategy. – Chooses the right number of resources. – Chooses blocking of functions into jobs. – Recovers from a larger number of failures. – Predicts overall runtime accurately. All of these tasks are nearly impossible for arbitrary workloads, but are tractable (not trivial) to solve for a specific abstraction. 22 23 Distribute Data Via Spanning Tree 24 Choose the Right # of CPUs 25 Conventional vs Abstraction 26 All-Pairs in Production Our All-Pairs implementation has provided over 57 CPU-years of computation to the ND biometrics research group over the last year. Largest run so far: 58,396 irises from the Face Recognition Grand Challenge. The largest experiment ever run on publically available data. Competing biometric research relies on samples of 100-1000 images, which can miss important population effects. Reduced computation time from 833 days to 10 days, making it feasible to repeat multiple times for 27 a graduate thesis. (We can go faster yet.) 28 Overview Challenges in Using Campus Grids Solution: Abstractions Examples and Applications – All-Pairs: Biometrics, Data Mining – Wavefront: Genomics and Economics – Some-Pairs: Genomics Abstractions, Workflows, and Languages 29 Wavefront( matrix M, function F(x,y,d) ) returns matrix M such that M[i,j] = F( M[i-1,j], M[I,j-1], M[i-1,j-1] ) M[0,4] F x d M[0,3] M y M[0,2] y F d d y F x F x F x d M[0,1] x F x d Wavefront(M,F) M[2,4] M[3,4] M[4,4] y y F d d x y F d y F x M[3,2] M[4,3] y F x d M[4,2] y F x d y M[0,0] M[1,0] M[2,0] M[3,0] M[4,0] 30 Applications of Wavefront Bioinformatics: – Compute the alignment of two large DNA strings in order to find similarities between species. Existing tools do not scale up to complete DNA strings. Economics: – Simulate the interaction between two competing firms, each of which has an effect on resource consumption and market price. E.g. When will we run out of oil? Applies to any kind of optimization problem solvable with dynamic programming. 31 Problem: Dispatch Latency Even with an infinite number of CPUs, dispatch latency controls the total execution time: O(n) in the best case. However, job dispatch latency in an unloaded grid is about 30 seconds, which may outweigh the runtime of F. Things get worse when queues are long! Solution: Build a lightweight task dispatch system. (Idea from Falkon@UC) 32 worker worker worker worker worker worker queue tasks wavefront engine tasks done work queue 1000s of workers dispatched via Condor/SGE/SSH put F.exe put in.txt exec F.exe <in.txt >out.txt get out.txt In.txt worker F out.txt 33 Problem: Performance Variation Tasks can be delayed for many reasons: – Heterogeneous hardware. – Interference with disk/network. – Policy based suspension. Any delayed task in Wavefront has a cascading effect on the rest of the workload. Solution - Fast Abort: Keep statistics on task runtimes, and abort those that lie significantly outside the mean. Prefer to assign jobs to machines with a fast history. 34 500x500 Wavefront on ~200 CPUs 35 Wavefront on a 200-CPU Cluster 36 Wavefront on a 32-Core CPU 37 Performance Prediction is Possible Often, users have no idea whether a task will take one day or one year -> better to find out at the beginning! Allows the system to choose automatically whether to run locally or on the campus grid. Of course, performance prediction is technically and philosophically dangerous: we simply argue that abstractions are more predictable than general programs. 38 Overview Challenges in Using Campus Grids Solution: Abstractions Examples and Applications – All-Pairs: Biometrics, Data Mining – Wavefront: Genomics and Economics – Some-Pairs: Genomics Abstractions, Workflows, and Languages 39 The Genome Assembly Problem AGTCGATCGATCGATAATCGATCCTAGCTAGCTACGA Chemical Sequencing AGTCGATCGATCGAT TCGATAATCGATCCTAGCTA AGCTAGCTACGA Millions of “reads” 100s bytes long. Computational Assembly AGTCGATCGATCGAT TCGATAATCGATCCTAGCTA AGCTAGCTACGA 40 Sample Genomes Sequential Pairs Time Reads Data A. gambiae scaffold 101K 80MB 738K 12 hours A. gambiae complete 180K 1.4GB 12M 6 days S. Bicolor simulated 7.9M 5.7GB 84M 30 days 41 Genome Assembly Today Several commercial firms provide an assembly service that takes weeks on a dedicated cluster, costs O($10K), and is based on human genome heuristics. Genome researchers would like to be able to perform custom assemblies using their own data and heuristics. Can this be done on a campus grid? 42 Some-Pairs Abstraction SomePairs( set A, list (i,j), function F(x,y) ) returns list of F( A[i], A[j] ) A1 A1 A1 An (1,2) (2,1) (2,3) (3,3) F A1 A2 A3 F SomePairs(A,L,F) A2 A3 F F F 43 Distributed Genome Assembly A1 A1 An (1,2) (2,1) (2,3) (3,3) F queue tasks somepairs master tasks done 100s of workers dispatched to worker Notre Dame, worker worker Purdue, and worker worker Wisconsin worker detail of a single worker: work queue put align.exe put in.txt exec F.exe <in.txt >out.txt get out.txt in.txt worker F out.txt 44 Small Genome (101K reads) 45 Medium Genome (180K reads) 46 Large Genome (7.9M) 47 From Workstation to Grid 48 What’s the Upshot? We can do full-scale assemblies as a routine matter on existing conventional machines. Our solution is faster (wall-clock time) than the next faster assembler run on 1024x BG/L. You could almost certainly do better with a dedicated cluster and a fast interconnect, but such systems are not universally available. Our solution opens up research in assembly to labs with “NASCAR” instead of “Formula-One” hardware. 49 Overview Challenges in Using Campus Grids Solution: Abstractions Examples and Applications – All-Pairs: Biometrics, Data Mining – Wavefront: Genomics and Economics – Some-Pairs: Genomics Abstractions, Workflows, and Languages 50 Other Abstractions for Computing Directed Graph Map-Reduce Bag of Tasks R M R 51 Partial Lattice of Abstractions Robust Performance All-Pairs Some Pairs Bag of Tasks Wavefront Map Reduce Directed Graph Expressive Power Lambda Calculus 52 Two Abstractions Compared AllPairs( set A, set B, F(x,y) ) SomePairs( set S, list L, F(x,y) ) Assuming that A = B = S… Can you express AllPairs using SomePairs? – Yes, but you must enumerate all pairs explicitly. It is not trivial for SomePairs to minimize the amount of data transferred to each node. Can you express SomePairs using AllPairs? – Yes, but only by doing excessive amounts of work, and then winnowing the results. 53 Abstractions as a Social Tool Collaboration with outside groups is how we encounter the most interesting, challenging, and important problems, in computer science. However, often neither side understands which details are essential or non-essential: – Can you deal with files that have upper case letters? – Oh, by the way, we have 10TB of input, is that ok? – (A little bit of an exaggeration.) An abstraction is an excellent chalkboard tool: – Accessible to anyone with a little bit of mathematics. – Makes it easy to see what must be plugged in. – Forces out essential details: data size, execution time. 54 Conclusion Grids, clouds, and clusters provide enormous computing power, but are very challenging to use effectively. An abstraction provides a robust, scalable solution to a narrow category of problems; each requires different kinds of optimizations. Limiting expressive power, results in systems that are usable, predictable, and reliable. Is there a menu of abstractions that would satisfy many consumers of grid computing? 55 Acknowledgments Cooperative Computing Lab – http://www.cse.nd.edu/~ccl Faculty: – – – – Patrick Flynn Nitesh Chawla Kenneth Judd Scott Emrich Grad Students – – – – – Chris Moretti Hoang Bui Li Yu Mike Olson Michael Albrecht Undergrads – – – – – Mike Kelly Rory Carmichael Mark Pasquier Christopher Lyon Jared Bulosan NSF Grants CCF-0621434, CNS-0643229 56