Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Entity–attribute–value model wikipedia , lookup
Oracle Database wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Relational model wikipedia , lookup
Functional Database Model wikipedia , lookup
Concurrency control wikipedia , lookup
Versant Object Database wikipedia , lookup
Database model wikipedia , lookup
Component Infrastructure of CQoS and Its Application in Scientific Computations Li Li1, Boyana Norris1, Lois Curfman McInnes1, Kevin Huck2 , Joseph Kenny3 , Meng-Shiou Wu4 1Argonne National Laboratory, Argonne, IL. 2University 3Sandia 4Ames of Oregon National Laboratories, California Laboratory CCA meeting Jan. 2009 Outline Motivation CQoS introduction Database component design Application examples Ongoing and future work 2 Overall Goals Automate the configuration and runtime adaptation of highperformance component applications, through the so called Computational Quality of Service (CQoS) infrastructure – Instrumentation of component interfaces – Performance data gathering – Performance analysis – Adaptive algorithm support Motivating application examples – Quantum Chemistry challenges: How, during runtime, can we make the best choices for reliability, accuracy, and performance of interoperable QC components? • When several QC components provide the same functionality, what criteria should be employed to select one implementation for a particular application instance and computational environment? • How do we incorporate the most appropriate externally developed components? (e.g., which algorithms to employ from numerical optimization components?) 3 Motivating Application Examples (cont.) Overall simulation times for nonlinear (time-dependent) PDE-based models often depend to a large extent on the robustness and efficiency of sparse linear solvers – Properties of linear system change during runtime – No single method is best because of the complexity of long-running applications Efficient parallel structured adaptive mesh refinement (SAMR) applications depend on load-balancing algorithms – Computational resources are dynamically concentrated to areas in need of a high accuracy – Application and computer state change at runtime – Dynamic resource allocation requires the workload partitioning algorithm be selected at runtime according to state change 4 Outline Motivation CQoS introduction Database component design Application examples Ongoing and future work 5 CQoS Analysis Infrastructure Performance monitoring, problem/solution characterization, and performance model building Instrumented Component Application Cases Performance Databases (historical & runtime) Interactive Analysis and Model Building Scientist can analyze data interactively CQoS Control Infrastructure Interpretation and execution of control laws to modify an application’s behavior Control System (parameter changes and component substitution) Substitution Assertion Database Scientist can provide decisions on substitution and reparameterization CQoS-Enabled Component Application Component Substitution Set Component A Component B Component C 6 Database Needs for the Scientific Application Adaptation Performance analysis of candidate solver/algorithm – Large number of performance runs – Store, manage, and search performance data Store and manage hardware, compiler, and application metadata – Information essential to algorithm selection, e.g., system configurations, problem properties, application states Optimal algorithm determination – Input data (or problem features) – Algorithmic parameters – Performance models (or hints) 7 Database Needs for Scientific Application Adaptation (cont.) Database use cases: – Store historical performance data and application metadata – Facilitate offline performance analysis – Match the current application state against historical data through DB queries during runtime – Search for optimal algorithm w.r.t. current application state – Retrieve settings associated with the optimal algorithm so we can apply it immediately to the application during runtime 8 Outline Motivation CQoS introduction Database component design Application examples Ongoing and future work 9 CQoS Database Component Design Designed C++ and SIDL interfaces for CQoS database management Implemented prototype database management components – Description and software: http://wiki.mcs.anl.gov/cqos/index.php/CQo S_database_components_version_0.0.0 – Based on PerfDMF performance data format and PERI metadata formats – Comparator interface and corresponding component for searching and matching parameter sets 10 …… CQoS Database Component Design Perf. data: query/store Perf. Database Metadata: query/store Meta-Database Metadata: compare/match Meta-Comparator Perf. data: compare/match Perf. Comparator Adaptive Heuristic : component …… : component connection Fig.1. Connect database and comparator components to adaptive heuristics component. There can be multiple database and comparator components that deal with different data types. 11 Use DB interfaces in 2D driven-cavity /* instantiate parameter 1 */ ierr = ComputeQuantity(matrix,"icmk","splits",&res,&flg); CHKERRQ(ierr); MatrixProperty param1("splits", "matrix_meta", res.i); /* instantiate parameter 2 */ ierr = ComputeQuantity(matrix,"structure","nnzeros",&res,&flg); CHKERRQ(ierr); MatrixProperty param2("nnzeros", "matrix_meta", res.i); /**** Store matrix property set into database. ***/ int myRank; ierr = MPI_Comm_rank(PETSC_COMM_WORLD, &myRank); CHKERRQ(ierr); if (myRank == 0){ int localID; int trialID; string conninfo("dbname = perfdb"); /* Generate a runtime database manager. It connects to a PostgreSQL database through DB interfaces. */ RunTimeRecord *R = RunTimeRecord::instance(); R->Connect2DB(conninfo); trialID = R->getTrialID(); localID = R->getCurEvtID(cflStr); /* instantiate a parameter set */ PropertySet aSet; /* add parameter 1 and 2 into the set */ aSet.addAParameter(¶m1); aSet.addAParameter(¶m2); /* store the parameter set into database, */ R->loadParameterSet(trialID, localID, aSet); } CQoS Performance and Metadata Performance (general) – Historical performance data from different instances of the same application or related applications: • Obtained through source instrumentation, e.g., TAU (U. Oregon) • Binary instrumentation, e.g., HPCToolkit (Rice U.) Ideally, for each application execution, the metadata should provide enough information to be able to reproduce a particular application instance. Examples: – Input data (reduced representations) • E.g., molecule characteristics,matrix properties – Algorithmic parameters • E.g., convergence level, maximum number of iterations – System parameters • Compilers, hardware – Domain-specific • Provided by scientist/algorithm developer 13 Outline Motivation CCA and CQoS introduction Database component design Application examples Ongoing and future work 14 Example: CQoS in Quantum Chemistry Initial focus: parallel application configuration of QC applications so that these can run effectively on various high-performance machines – Eliminate guesswork or trial-and-error configuration Future work: more sophisticated analysis to configure algorithmic parameters for particular molecular targets, calculation approaches, and hardware environments 1J. Steensland and J. Ray, "A Partitioner-Centric Model for SAMR Partitioning Trade-Off Optimization 15 : Part I," International Journal of High Performance Computing Applications, 2005, 19(4):409-422. Interactions of the Quantum Chemistry Components With the Database and Comparator CQoS Components CQoS/QC Component Wiring CQoS Component Usage in Quantum Chemistry CQoS database usage – Application metadata • Molecule characteristics: atom types, topology, moments of inertia • Algorithm parameters: tunable parameters, convergence level – System parameters • Compilers • Machine info, e.g., number of nodes, threads per node, network – Historical performance data • Execution times, etc. • Obtained through source instrumentation, e.g., TAU • Can guide configuration of related new simulations CQoS comparator components – Compare sets of parameters within the performance database – Quantum chemistry applications can match the current application state against historical data through database queries during runtime. – Use metadata to guide parameter selection and application configuration • Match molecule similarity, basis set similarity, electronic correlation approach, etc. 18 Ongoing and Future Work (Incomplete List) Integration of ongoing efforts in – Performance tools: common interfaces and data representation (leverage PerfExplorer, TAU performance interfaces, PERI tools, and other efforts) Support training experiment design – To perform an empirical search for selecting the optimal solver components/parameters Incorporate more offline performance analysis capabilities (machine learning, statistical analysis, etc.) Apply to more problem domains, implementing extensions as necessary 19 Acknowledgements to Collaborators TAU Performance Tools group, University of Oregon Victor Eijkhout, the University of Texas at Austin CCA Forum members Funding: – Department of Energy (DOE) Mathematical, Information, and Computational Science (MICS) program – DOE Scientific Discovery through Advanced Computing (SciDAC) program – National Science Foundation 20 Main Database Component Interfaces interface DB extends gov.cca.Port{ bool connect(); bool disconnect(); bool isClosed(); void setConnectionInfo(in string info); string getConnectioninfo(); int executeQuery(in string commd, out Outcome res); interface Comparator extends gov.cca.Port { /* Comparison operations between parameter sets */ void setLHS(in ParameterSet lefthand); void setRHS(in ParameterSet righthand); ParameterSet getLHS(); ParameterSet getRHS(); void storeParameter(in int trialID, in int iterNo, in Parameter aParam); // store a parameter into DB void storeParameterSet(in int trialID, in int iterNo, in ParameterSet aParamSet); // store a set of parameter into DB int getDimension(); Parameter getLHSParameterAt(in string paraName); Parameter getRHSParameterAt(in string paraName); void getParameter(in int trialID, int iterNo, inout Parameter aParam); // retrieve parameter value void getParameterSet(in int trialID, int iterNo, inout ParameterSet aParamSet);// retrieve parameter set value void setToleranceAt(in string name, in double epsilon); double getToleranceAt(in string name); int getMatchingTrialsBetween(in ParameterSet lower, in ParameterSet upper, out Outcome trialIDs); // retrieve trials from database, whose parameter set value is within [lower, upper] int getMatchingTrials(in ParameterSet lower, in vector e psilons, out Outcome trialIDs);// retrieve trials from database, whose parameter set value is within [lower-epsilons, lower+epsilons] } void setRelationAt(in string name, in int aRelation); int getRelationAt(in string name); bool doCompare(); } Database Component Usage – Example 1: 2D Driven Cavity Flow1 Linear solver: GMRES(30), vary only fill level of ILU preconditioner Adaptive heuristic based on: – Matrix properties (which change during runtime) computed with Anamod (Eijkhout, http://sourceforge.net/projects/salsa/) T. S. Coffey, C.T. Kelley, and D.E. Keyes. Pseudo-transient continuation and differential algebraic equations. SIAM J. Sci. Comp, 25:553–569, 2003. 1 22 How are the Database Components Used? During runtime, the driver (e.g., linear solver proxy component) evaluates important matrix properties, and matches the properties to historical data in MetaDB through PropertyComparator interfaces. Linear solver performance data is retrieved and compared given the current matrix properties. This is accomplished by the PerfComparator component. The linear solver parameters resulting in the best performance, in this case fill level of ILU preconditioner, is returned back to the driver. The driver adapts accordingly to continue execution. 23