Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
The St. Jude Children’s Research Hospital/Washington University Pediatric Cancer Genome Project: A CIO’s Perspective Clayton W. Naeve, Ph.D. Endowed Chair in Bioinformatics SVP & CIO St. Jude Children’s Research Hospital St. Jude Data: The First 50 Years 2000 2 1/2 Years (1000 TB) 1800 PCGP Data: 917 TB, 148 million files 1600 1400 Terabytes 1200 1000 800 48 Years (800 TB) 600 400 200 0 Admin/Clinical The Data Deluge Research St. Jude/WashU Pediatric Cancer Genome Project • • • • • • Launched Feb. 2010 St. Jude/WashU collaboration WGS on 600 patients (leukemia, brain tumors, solid tumors) Matched germline and tumor samples 1200 genomes (~90 billion bp/genome) in 36 months ~2 Petabytes of data The PCGP Project Challenges to Information Sciences • • • • • • Moving data Data workflow Data analysis Computational horsepower Data storage Data sharing PCGP Challenges • Multi-Terabyte data transit across networks is not trivial • DNA sequence raw data reads, contig assembly, alignment to reference, variants, etc. shipped to SJCRH as binary BAM files: ~100 GB • 24 hrs to infinity to send via commodity internet • Internet2 connectivity (10 Gbs via MRC) to transfer files from WashU to SJCRH • Evaluated 5 different fast data transfer algorithms….selected FDT (developed at CalTech to transfer LHC data at Cern) • Developed a pipeline to facilitate transfer • Today: ~5 hour transit time/file Moving Data HPCF IBM iDataplex 1008 cores/4 TB RAM Data Transfer Node (dtn01) 505 IBM SoNAS 734 TB (usable) IBM BladeCenter Cluster 810 cores/3 TB RAM x6 x84 x4 x4 x4 Mellanox Grid Director 4036 X4 Internal Data Transfer Node (datamover) Mellanox Grid Director 4036 Mellanox IS5200 Chasis Switch x2 Mellanox IS5200 Chasis Switch x4 ESX Cluster 29 Servers x2 COMPACT xSeries 335 COMPACT xSeries 335 COMPACT X4 xSeries 335 COMPACT xSeries 335 PVFS Servers SGI IS5500 60TB (usable) Moving Data Mellanox BridgeX BX5020 Mellanox BridgeX BX5020 10 GE Campus Network 10 GE Campus Network SGI Altix UV1000 640 cores/5 TB RAM Moving Data • Began work on PCGP 9 months prior to launch • Developed a LIMS system for Validation Lab • Developed a PCGP SharePoint site to facilitate collaboration internally • Developed a bioinformatics workflow engine: PALLAS • • • • • • • • • • Security management Data provenance management Intermediate and final result tracking Flexible workflow design Rapid new analytical algorithms/tools configuration Web-based LSF job submission and monitoring Support a range of protocols to connect to other web application systems, databases, file systems, and etc. Integrated with applications, such as SRM, Genome Browser and etc. Data integration with tissue sample, clinical, and research data Vision: parse each algorithm to the appropriate computing environment Data Workflow Jinghui Zhang and CompBio Team • • • • • BAM Quality Assurance: • Tumor Purity Algorithm (SJCRH) • Not Disease/Genomic Swap (SNP checks) • Xenograft Filter (Remove Contaminating Mouse Reads) • Gene Exon and Genome Coverage algorithms (Gang Wu) BAM file work: • Bam file extraction and visualization • Samtools and C++/bioperl api’s • Bambino • IGV Single Nucleotide Variation: • Freebayes • In-house PCGP Copy Number Variation: • Stan’s Copy Number Algorithm • Regression Tree Algorithm Structural Variation: • One End Anchored Inference: • CREST • ViralTopology Data Analyses • • • • • • Fusion Detection: • In-house (Michael Rusch) RNAseq: • RNAseq mysql/Cufflinks ChipSeq: • ChiPseq mysql/in house (John Obenauer) viralScan • in-house (McGoldrick) Integration: • GFF intersect • Gff2fasta • gffBuilders • Cancer warehouse Visualization: • Circos maker • BED GFF Tracks maker • • • • • • • • • IBM BladeCenter (810 cores/3TB RAM) IBM iDataplex (1,008 cores/4TB RAM) – April 2010 SGI Altix UV1000 (640 cores/5TB RAM/60TB storage using Lustre v2.2) – December 2011 IBM SoNAS (780 TB) – March 2011 Data Transfer Node (10 Gbps I2 connection) – April 2011 Internal Data Transfer Node (10 Gbps x2) – June 2011 QDR Infiniband (40 Gbps for all HPC equipment) – January 2012 Software (Platform LSF, Intel Parallel Studio) Total: 2,366 cores, 13TB RAM (estimated 11.6 Tflops) • 2010: 365,000 cpu hours • 2011: 712,000 cpu hours Computational Horsepower (HPCF) • IBM SoNAS (780 TB) – March 2011 • • • • Scales to 21PB; 1 billion files/filesystem; 7,200 drives Current total on campus: 3.8 Petabytes (3,800,000 Gb) PCGP uses 917 TB (<- +500TB on tape), 148 million data files IBM TSM systems for backup/archive (Tiered) • • • • • 240 SAS (15k) drives 480 SAS-NL (7.2k) drives Current 7,900 tape capacity, up to 1.6TB/tape; 12.6+ PB total 734 TB usable under one file system High speed/low latency backend interconnect (QDR InfiniBand 20Gb per port and 100ns latency) Data Storage >356 Patients/712 Complete Genomes Gene sequencing project identifies potential drug targets in common childhood brain tumor Nature June 20, 2012 Researchers studying the genetic roots of the most common malignant childhood brain tumor have discovered missteps in three of the four subtypes of the cancer that involve genes already targeted for drug development. The most significant gene alterations are linked to subtypes of medulloblastoma that currently have the best and worst prognosis. They were among 41 genes associated for the first time to medulloblastoma by the St. Jude Children's Research Hospital – Washington University Pediatric Cancer Genome Project. World's largest release of comprehensive human cancer genome data helps researchers everywhere speed discoveries Nature Genetics May 29, 2012 To speed progress against cancer and other diseases, the St. Jude Children's Research Hospital – Washington University Pediatric Cancer Genome Project today announced the largest-ever release of comprehensive human cancer genome data for free access by the global scientific community. The amount of information released more than doubles the volume of high-coverage, whole genome data currently available from all human genome sources combined. This information is valuable not just to cancer researchers, but also to scientists studying almost any disease. Genome sequencing initiative links altered gene to age-related neuroblastoma risk Journal of the American Medical Association March 13, 2012 St. Jude Children’s Research Hospital – Washington University Pediatric Cancer Genome Project and Memorial Sloan-Kettering Cancer Center discover the first gene alteration associated with patient age and neuroblastoma outcome. Researchers have identified the first gene mutation associated with a chronic and often fatal form of neuroblastoma that typically strikes adolescents and young adults. The finding provides the first clue about the genetic basis of the long-recognized but poorly understood link between treatment outcome and age at diagnosis. Cancer sequencing initiative discovers mutations tied to aggressive childhood brain tumors Nature Genetics January 29, 2012 Findings from the St. Jude Children's Research Hospital – Washington University Pediatric Cancer Genome Project (PCGP) offer important insight into a poorly understood tumor that kills more than 90 percent of patients within two years. The tumor, diffuse intrinsic pontine glioma (DIPG), is found almost exclusively in children and accounts for 10 to 15 percent of pediatric tumors of the brain and central nervous system. Cancer sequencing project identifies potential approaches to combat aggressive leukemia Nature January 11, 2012 Researchers with the St. Jude Children's Research Hospital - Washington University Pediatric Cancer Genome Project (PCGP) have discovered that a subtype of leukemia characterized by a poor prognosis is fueled by mutations in pathways distinctly different from a seemingly similar leukemia associated with a much better outcome. The work provides the first details of the genetic alterations fueling a subtype of acute lymphoblastic leukemia (ALL) known as early T-cell precursor ALL (ETP-ALL). The results suggest ETP-ALL has more in common with acute myeloid leukemia (AML) than with other subtypes of ALL. Gene identified as a new target for treatment of aggressive childhood eye tumor Nature January 11, 2012 New findings from the St. Jude Children's Research Hospital – Washington University Pediatric Cancer Genome Project (PCGP) have helped identify the mechanism that makes the childhood eye tumor retinoblastoma so aggressive. The discovery explains why the tumor develops so rapidly while other cancers can take years or even decades to form. The finding also led investigators to a new treatment target and possible therapy for the rare childhood tumor of the retina, the light-sensing tissue at the back of the eye. Progress http://www.pediatriccancergenomeproject.org http://explore.pediatriccancergenomeproject.org Data Sharing • Data Integration is critical: platform data (expression, WGS, methylation, etc.) and processed data (“genomics” data with phenotype data (clinical care, clinical research)) Data Sharing 19 Academic Departments Computational Biology Information Sciences Shared Resources 2 PhD 2 Support Enterprise Informatics 8-10 Faculty 50-60 Support Staff 10 PhD Bioinformatics 2 developers PCGP 5 PhD 127 FTEs Research Informatics 56 FTEs 1 Dev. Clinical Informatics 81 FTEs HPC Offshore Developers 15 FTEs Total=>150 FTEs with “research informatics” skills Key: Staff • Project total cost: $65M (11 Illuminas @ WashU and 4 @ SJCRH, sequencing costs, staffing, IT, etc.) • New “IT” staff @ SJCRH: 10 FTEs in CompBiol, 0 FTEs in IS • Capital IT investment: ~$7.2 M at SJCRH, $9M at WashU • IT is ~25% of overall project costs (doesn’t include costs of other participating SJ FTEs) Information Sciences PCGP Team • • • • • • • • • • • • • • Ashish Pagare David Zhao Dan Alford Stephen Espy Kiran Chand Bobba Scott Malone Dr. Antonio Ferreira Bill Pappas James McMurry Dr. Jianmin Wang Dr. John Obenauer Jared Becksfort Pankaj Gupta Dr. Suraj Mukatira Key: Staff • • • • • • • • • • • • • • • Simon Hagstrom Sundeep Shakya Asmita Vaidya Swetha Mandava Bhagavathy Krishna Manohar Gorthi Sandhya Rani Kolli Sivaram Chintalapudi Roshan Shrestha Irina McGuire PJ Stevens Thanh Le John Penrod Pat Eddy Dr. Dan McGoldrick Questions? cluster Contig assembly SV PALLAS Data Workflow large memory CNV INDELS GPU SNV CIRCOS