* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Slides
Survey
Document related concepts
Entity–attribute–value model wikipedia , lookup
Serializability wikipedia , lookup
Oracle Database wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Functional Database Model wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Concurrency control wikipedia , lookup
Relational model wikipedia , lookup
Versant Object Database wikipedia , lookup
ContactPoint wikipedia , lookup
Transcript
Efficient Data Handling in Large-Scale Sequence Database Searches Heshan Lin (NCSU) Xiaosong Ma (NCSU and ORNL) Wu-chun Feng (LANL VT) Al Geist (ORNL) Nagiza Samatova (ORNL) 1 Outline Sequence database search Parallel BLAST background mpiBLAST & pioBLAST New release: mpiBLAST-pio GreenGene: search NT against NT practice (SC|05, StorCloud Demo) 2 Sequence Database Search is Critical for Biomedical Science Routinely used in many biomedical researches Search similarities between query sequences and sequence database Predict structures and functions of new sequences Analogous to web search engines (e.g. Google) Web Search Engine Sequence DB Search Input Key word(s) Query sequence(s) Search space Internet Known sequence database Output Related web pages DB sequences similar to the query Sorted by Closeness & rank Score (Similarity) 3 Challenge for Sequence DB Search Sequence DB Search is Hampered by the Growing Gap between Sequence Growth and Processor Memory Sequence databases are growing exponentially in size Because of this gap: there is a lot of repeated I/O introduced by loading sequence data back and forth from the file system to the memory. This adversely affects the performance. 4 BLAST at the Core of Sequence DB Search Widely used search tool: But, it is: Approximately 75%-90% of all compute cycles in life sciences are devoted to BLAST searches Computationally demanding, O(n2) Requires huge database to be stored in memory Generates gigabytes of output file for large database searches Parallel Blast as a means to address computational challenge 5 BLAST Parallelization: Query Segmentation Queries >Perilla Frutescens CDS 0001 TTGGTATCCACGGAAGAGAGAGAAAATGTTGGGAATTTTCAGCGGAC GTATAGTATCATTGCCGGAAGAGCTGGTGGCTGCCGGGAACC Worker Nodes >Perilla Frutescens CDS 0002 GGAGGGTGGCTGGTGGGTATTGGCGGCCCGACCGATCTGCCCCGAC CGACGGCTCCTGCCACCCGAACATGTGATAGAAAGGAQQQQQQQQ >Perilla Frutescens CDS 0003 TTTTTTTCTTGATGCTGAAATCTATCCAAACATCACCAGTCCTCACGAG TCCTTGACCAAATTCTTGCTTTCTGGCACAATCTGAAGCCCAAAGGC Database >gi|3123744|dbj|AB013447.1|AB013447 TTGGTATCCACGGAAGAGAGAGAAAATGTTGGGAATTTTCAGCGGAC GTATAGTATCATTGCCGGAAGAGCTGGTGGCTGCCGGGAACC >gi|221778|dbj|D00026.1|HS2HSV2P4 GGAGGGTGGCTGGTGGGTATTGGCGGCCCGACCGATCTGCCCCGAC CGACGGCTCCTGCCACCCGAACATG >gi|7328961|dbj|AB032155.1|AB032154S2 TTTTTTTCTTGATGCTGAAATCTATCCAAACATCACCAGTCCTCACGAG TCCTTGACCAAATTCTTGCTTTCTGGCACAATCTGAAGCCCAAAGGC W. Feng et al. “mpiBLAST on the GreenGene Distributed Supercomputer”, SC|05 6 Pros and Cons of Query Segmentation Advantages Low parallelization overhead Linear speedup when database fits into single processor memory Disadvantages Suffers repeated I/O when database cannot fit into main memory Resource under-utilization / load imbalance when #queries smaller than or comparable to #processors 7 BLAST Parallelization: Database Segmentation Queries >Perilla Frutescens CDS 0001 Worker Nodes TTGGTATCCACGGAAGAGAGAGAAAATGTTGGGAATTTTCAGCGGAC GTATAGTATCATTGCCGGAAGAGCTGGTGGCTGCCGGGAACC >Perilla Frutescens CDS 0002 GGAGGGTGGCTGGTGGGTATTGGCGGCCCGACCGATCTGCCCCGAC CGACGGCTCCTGCCACCCGAACATGTGATAGAAAGGAQQQQQQQQ >Perilla Frutescens CDS 0003 TTTTTTTCTTGATGCTGAAATCTATCCAAACATCACCAGTCCTCACGAG TCCTTGACCAAATTCTTGCTTTCTGGCACAATCTGAAGCCCAAAGGC Database >gi|3123744|dbj|AB013447.1|AB013447 TTGGTATCCACGGAAGAGAGAGAAAATGTTGGGAATTTTCAGCGGAC GTATAGTATCATTGCCGGAAGAGCTGGTGGCTGCCGGGAACC >gi|221778|dbj|D00026.1|HS2HSV2P4 GGAGGGTGGCTGGTGGGTATTGGCGGCCCGACCGATCTGCCCCGAC CGACGGCTCCTGCCACCCGAACATG >gi|7328961|dbj|AB032155.1|AB032154S2 TTTTTTTCTTGATGCTGAAATCTATCCAAACATCACCAGTCCTCACGAG TCCTTGACCAAATTCTTGCTTTCTGGCACAATCTGAAGCCCAAAGGC W. Feng et al. “mpiBLAST on the GreenGene Distributed Supercomputer”, SC|05 8 Pros and Cons of Database Segmentation Advantages Disadvantages Fitting large database into aggregate memory Able to utilize large machines regardless of #queries Higher parallel search overhead, local results need to be merged globally Challenge Reduce result merging & processing overhead 9 mpiBLAST: A Specific Implementation of Database Segmentation Open-source parallel BLAST developed at LANL: http://mpiblast.lanl.gov or http://www.mpiblast.org Increasingly popular: more than 40,000 downloads in 2½ years Integrated with NCBI BLAST Based on database segmentation Performance Achieves super linear speedup when using small # processors Problem: overhead in data handling limits scalability 10 mpiBLAST System Design Master-slave model: one master, p-1 workers Searching done in workers Search all queries against a subset of DB frags Generate partial results – meta data of alignments (ASN.1 format, include seq id, scores, etc.) Output processing done in master Merge partial results from all workers Fetch correspondent sequence data Compute and output alignments 11 mpiBLAST 1.2 Input Databases partitioned statically before search Inflexible execution time sensitive to # fragments re-partitioning required to use different # procs Management overhead generating large number of small files, hard to manage, migrate and share Execution Time Vs. # Fragment # Fragments sensitivity test - Search 150k queries against nr database - Using 32 processors Total Execution Time 4500 4000 3500 3000 2500 2000 1500 1000 500 0 0 50 100 150 200 Number of Fragments 12 mpiBLAST 1.2 Output Master must cache all results Master result 1 Worker1 Seq id result 1 result 2 result 3 …. DB Frag Seq data sent over network result 2 Worker2 DB Frag result 3 Worker3 Alignment1 Seq data Alignment2 DB Frag Alignment3 Serialized by the master Global output file 13 mpiBLAST 1.2 Scalability Consequence of inefficient data handling: rapidly growing non-search overhead as No. of procs increases Output data size increases Execution Time vs. # Procs 4500 Other time 4000 Search time -Search 150k queries against nr - Vary number of processors - Database evenly partitioned according to # processors 3500 Time (Seconds) 3000 2500 2000 1500 1000 500 0 4 8 16 Number of Processors 32 64 14 pioBLAST: Research Prototype With Data Access Optimizations Research prototype of efficient parallel BLAST developed at ORNL & NCSU Built on top of mpiBLAST1.2 Apply parallel/collective I/O techniques Enable dynamic partitioning Parallel database input and result output Highly efficient result processing Workers compute alignments in parallel Workers buffer and write local output in parallel Enhanced worker-master communication for reducing data transfer volume 15 Dynamic Partitioning of pioBLAST No static pre-partitioning Virtual fragments generated dynamically at run time One single database image to search against Workers read inputs in parallel with MPI-IO interface Fragment size configurable at run time Easily supports dynamic load balancing Worker1 Worker2 Worker3 Worker n Frag1 Frag2 Frag3 FragN Frag1 Frag2 Frag3 FragN Global Sequence Data 16 Output Processing of pioBLAST Reduce communication Master Worker1 result 1.1 result 1.2 result 1.3 Worker2 DB Frag result 2.1 result 2.2 result 2.3 Worker3 DB Frag result 3.1 result 3.2 DB Frag Computing alignments result 3.3 and buffering them in parallel 1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3 Collective writing: I/O in parallel 2.1 1.1 3.1 3.2 1.2 2.2 2.3 Global output file 17 mpiBLAST 1.2 vs. pioBLAST: Node Scalability Platform: SGI Altix at ORNL Search 150k NR queries on different #procs Search time Other time Program-No. of processes mp i-6 4 pio -64 4500 4000 3500 3000 2500 2000 1500 1000 500 0 mp i-3 2 pio -32 mpiBLAST: non-search overhead increases fast pioBLAST: non-search time remains low mp i-1 6 pio -16 mp i-8 pio -8 Database: nr (1GB) Node scalability mp i-4 pio -4 256 processors (1.5GHz Itanium2), 8GB memory/proc, XFS Execution time (s) 18 mpiBLAST 1.2 vs. pioBLAST: Output Scalability Search different #query seqs on 64 procs 4000 3500 Search Other 3000 2500 2000 1500 1000 500 Program-Output Size m pi -1 53 M pi o15 3M m pi -9 6M pi o96 M m pi -4 7M pi o47 M 0 m pi -1 1M pi o11 M Same platform and database Varied query size to generate different output size Execution Time (s) 19 mpiBLAST Evolves: v1.4 Exact e-value statistics Improved result processing Not ready for the large DB search Reduce worker-master communication by packing needed biosequences data along with partial results Alleviate master bottleneck with query pipe-lining Output processing still serialized Partial results and correspondent sequences data for a single query could be huge (gigabytes) Performance Efficient in handling queries with small output Hang or perform slow for queries with large output 20 mpiBLAST + pioBLAST = mpiBLAST-pio Highly efficient, open source parallel BLAST (available at http://mpiblast.lanl.gov/) Joint effort between mpiBLAST and pioBLAST research teams Current release based on mpiBLAST 1.4 Exact e-value statistics Keep scheduling (query pipelining) and data distribution Efficient parallel output processing from pioBLAST Worker compute and buffer local output in parallel Non-collective parallel write to better support query pipelining Modifications on NCBI BLAST less than 30 lines Support all but anchor output formats 21 mpiBLAST-pio Meets a Grand Challenge: Searching NT vs. NT SC|05 StorCloud demo (Nov. 13 - Nov. 17) Team Institutions: LANL, NCSU, U. Utah, and Virginia Tech Vendors: Intel, Panta Systems, and Foundry Networks Sequencing NT against itself (16GB raw size) Why? Provide insightful knowledge to catalog NT database Demonstrate scalability of mpiBLAST(pio) to larger problem Meet the computational challenge with power of distributed parallel computing How? GreenGene Distributed Supercomputer > 3000 processors from 4 distributed sites of super computers 22 GreenGene Distributed Supercomputer How? SC2005 Showroom Floor Intel (Dupont) U.Utah Va Tech 23 W. Feng et al., “mpiBLAST on the GreenGene Distributed Supercomputer”, SC|05 Combining Query Segmentation and Database Segmentation The whole query file (NT) does not fit in memory Improve memory utilization SuperMaster GroupMaster GroupMaster GroupMaster NT Replica NT Replica NT Replica 24 Lessons Learned from NT vs NT Search Results Single supercomputer not enough Database segmentation is necessary to deal with “Hard Queries” – not just for reducing I/O but also for parallelizing the computation Finish 526,000 sequences (1/7 NT) in one day Case 1: 122k single query, take 64 procs 7 hours to finish, 1.8G output size (448hrs on single processor) Case 2: 2M single query, not finished on 128 procs within 12 hours mpiBLAST-pio demonstrate capability of conducting large database against database sequence alignment 25 Acknowledgements The work of pioBLAST was funded in part or in full by the US Department of Energy’s Genomes to Life program under the ORNL-PNNL project, ”Exploratory Data Intensive Computing for Complex Biological Systems”. The work of integrating data access optimizations of pioBLAST into mpiBLAST-pio was supported through LANL contract W-7405ENG-36. Other mpiBLAST-pio development contributors Jeremy Archuleta (LANL), Avery Ching (NWU), Pavan Balaji (OSU) 26 References “Efficient Data Access for Parallel BLAST,” 19th Int’l Parallel & Distributed Processing Symp., April 2005. “The Design, Implementation, and Evaluation of mpiBLAST” Best Paper: Applications Track, 4th Int’l Conf. on Linux Clusters, Jun. 2003. “mpiBLAST: Delivering Super-Linear Speedup with an Open-Source Parallelization of BLAST,” Pacific Symp. on Biocomputing, Jan. 2003. 27