Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
CVS Reorganization, Installation Reorganization & A Simple Build System Steve Fischer October 24, 2002 Goals Make GUS portable Self-explanatory CVS structure Projects wait for other projects only when necessary Easily maintain released versions while development continues Support development on a unified file system (new file server) Improve collaborators’ ability to understand our code structure And ours (particularly new folks) Improved ability to release projects on a schedule Easily build and configure runnable instances of our projects Avoid conflicting with the pre-existing structure of external sites Don’t rely on local file systems having different installed versions Encourage a move away from individual ownership of projects CVS Reorganization CVS Structure At the root there are projects, which have components, which have parts: $CVSROOT/ Project1/ Component1/ bin/ lib/ Component2/ Project2/ Projectn/ parts Projects Units of software development having their own release schedule May depend on other projects Have their own release number, eg, GUS 3.1.2, Annotator 1.0.0 Either on a specific release or ‘latest’ No circular dependencies Dependency example: DoTS->GUS->CSP Proposed Projects (phase 1) AllGenes Annotator (new) CSP (CBIL Style Police: web form assistance) DJob (distributed job controller) DoTS (build, including DoTS related plugins) GLE (grammars for transcription) GUS ParaDBs PlasmoDB RAD (plugins & website) (or should these be separate?) Projects Contain Components For example: GUS/ Common/ DBAdmin Model/ A component is usually formed from one or a couple of package directories (eg, java packages). In addition to code, components have: Documentation TODO list Configuration Required data A component may depend on others within its project Proposed GUS/ Components SeqMatch/ Common/ DBAdmin/ GenBank/ GOPredictor/ Model/ ObjRelJ/ ObjRelP/ Pipeline/ PluginMgr/ SchemaBrowser/ Util/ WebDevKit/ Proposed GUS Components in Detail - 1 SeqMatch/ Common/ Data model for genbank parser GOPredictor/ Database admin scripts GenBank/ Commonly used schema dependent stuff, such as common plugins DBAdmin/ Blast, BLAST2, and BLAT object models Predict GO function Model/ Schema definition files Hand edited objects (eg, GUS::Model::DoTS::Assembly.pm.man) Proposed GUS Components in Detail - 2 ObjRelJ/ ObjRelP/ Web based schema browser Util/ GA and friends SchemaBrowser/ Pipeline management API PluginMgr/ Perl object layer: superclasses and code generator Pipeline/ Java object layer: superclasses Commonly used schema-independent utilities WebDevKit/ Servlet and/or JSP based Components Contain Standard Parts As needed: Component1/ bin/ cgi-bin/ config/ data/ doc/ htdocs/ lib/ src/ test/ Component Parts in Detail - 1 Bin Cgi-bin/ Cgi executables Config/ executables Properties files for the component Files typically named component.prop Doc/ Specifications User guides TODO Component Parts in Detail - 2 Data/ Htdocs/ Extra data needed by component, eg, matrices, Static web pages Test/ Test data Component Parts in Detail - 3 Lib/ Contains linkable object files Perl includes full package path (Project::component::module) lib/ perl/ Project/ component/ *.pm xml/ *.xml Component Parts in Detail - 4 Src/ Contains source that needs to be compiled or otherwise transformed before it can be installed src/ c/ java/ edu/ cbil/ project/ component/ org/ gusdb/ component/ Process to Migrate CVS Structure copy origCVS/ GUS/ perl/ … www/ perl/ lib/ Fasta.pm CSP/ orig fromCVS/ GUS/ perl/ … www/ perl/ lib/ Transform script Fasta.pm CSP/ pruned newCVS/ DoTS/ Common/ lib/ perl/ Common/ Fasta.pm CSP/ GUS/ new Process to Migrate CVS Structure Do transform Check out old to origCVS/ Copy to fromCVS/ Run script to transform from fromCVS/ to newCVS/ moves files (thus pruning fromCVS/) Validate transform by: Examing transform script (see where YOUR stuff is going to be) Examine fromCVS/ to see what I left behind Create new CVSROOT with newCVS/ (losing history) How can I get everybody’s OK on the transform?? Releases A project is the unit of release When it is released, it is tagged in cvs with a release number, eg 2.1.1 At that time, other projects can declare a dependency on that particular release (discussed in detail later) Bug fixes are applied to the tagged branch Installation Reorganization Objectives Install into a single relocatable location: $GUS_HOME Be able to find (almost) all GUS installed resources there: Executables Libraries Documentation Configuration Some data Third party resources Support multiple running instances on a machine, eg dev, beta Make explicit the versions of each included resource Don’t conflict with the site’s pre-existing structure Easy to install and uninstall Avoid path and classpath conflicts In a file called $GUS_HOME/versions Also have installation targets for websites and appl. servers Installing Project1 to $GUS_HOME P1/ C1 C2 $GUS_HOME/ Bin/ Lib/ Doc/ P2/ C1 C2 P3/ C1 C2 = dependencies Understanding $GUS_HOME The home for a single GUS related installation Contains one or more projects, and the projects they depend on Including third party resources (such as BioJava or BioPerl) May contain projects that are running separately, as long as they don’t have conflicting dependencies (May even contain the entire set of GUS projects) It contains a version file, eg: DoTS 1.2.1 GUS 2.0.0 CSP 3.0.1 BioJava 1.1.1 Sample $GUS_HOME $GUS_HOME/ bin/ dotsbuild extractSeqs config/ dotsbuild.prop csp.prop gus.prop data/ matrices/ doc/ DoTS/Dotsbuild/TODO GUS/Model/uml/*.uml CSP/UserGuide.html (cont’d)… Sample $GUS_HOME (cont’d) $GUS_HOME/ … lib/ java/ gusmodel.jar guswdk.jar perl/ DoTS/DotsBuild/*.pm GUS/Model/*.pm GUS/ObjRelP/*.pm GUS/WebDevKit/*.pm CSP/*.pm xml/ *.xml Installation Targets - Website Contains cgi-bin and htdocs $/world/www.allgenes/ cgi-bin/ *.pl htdocs/ *.html Installation Targets – Application Server EG, Tomcat Will copy .jar files and other stuff as needed Configuration Code never refers to absolute file paths Many resources are conveniently located relative to $GUS_HOME, so don’t need configuration Otherwise, code relies on property files for configuration We provide a perl and java PropertySet object to make this easy Property files are found in… $GUS_HOME/config Named after the component that is using them: gusmodel.prop Consider using properties instead of macro substitutions when possible A Simple Ant-based Build System Objectives Install projects to installation targets Serves two kinds of installs: From $PROJECT_HOME to installation targets (developers) From .tar file to installation targets (external users) Support inter-project dependencies Support dependencies on third party resources Handle object layer code generation Be easy to use Be relatively easy to maintain Be generic across projects and components, but also flexible $PROJECT_HOME -> $GUS_HOME (developers) Install the build system Check out the project of interest Create $PROJECT_HOME Check out the install/ project Copy build to a local bin/ build DoTS install $GUS_HOME –co Now $PROJECT_HOME is an image of CVS containing your project and all projects it depends on. Edit away Install for testing build DoTS install $GUS_HOME .tar -> $GUS_HOME (external users) Untar the download file cd install/ setenv $GUS_HOME /usr/local/somewhere ./build DoTS install $GUS_HOME Also works for installing to a website ./build GeneDB installweb /world/www.genedb The Mechanics Uses Ant (http://jakarta.apache.org/ant/) Each project contains its own build.xml file The install/ directory contains the main build.xml file Uses xml to declaratively specify the build Used extensively in the java community Calls the appropriate project’s build.xml Houses “subroutines” The project build.xml file Specifies the project’s dependencies on other projects Specifies the project’s component’s dependencies on each other Calls default project and component build routines, unless custom ones supplied The Mechanics (cont’d) Java compilation Copying from $PROJECT_HOME to target (eg, $GUS_HOME) The build compiles java into the classes/ directory of a component It places .jar files into lib/java/ directory of a component Does compilation in dependency order Copies and merges all dirs, such as bin/, lib/ doc/ Can perform macro substitution on the way Creates version file Because Ant is kind of slow, build can do just a component instead of a whole project Special case: object layer code generation Generates into: Perl: Java: GUS/Model/DoTS/Assembly.pm.man Only generates if files are older than “schema definition” file: $PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/dots $PROJECT_HOME/GUS/Model/src/java/org/gusdb/model/core Etc Hand created files have .man suffix, eg: $PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/DoTS $PROJECT_HOME/GUS/Model/lib/perl/GUS/Model/Core Etc $PROJECT_HOME/GUS/Model/schema/definition.sql This file is generated by schema modification process (and lives in CVS) Making it All Happen Goals Getting it done quickly With a minimum of disruption Overarching strategy Work on projects and components one at a time Start with ones that are most central and not in development GUS/ObjRelP GUS/Model/lib/perl/Model/DoTS GUS/PluginMgr GUS/Common DoTS DJob Get that core working with new build system and $GUS_HOME In parallel, convert to GUS 3.0 In parallel, bring in GUS/ObjRelJ and Annotator (Dave) Upgrade WebDevKit install apparatus Bring in Web sites one at a time Inconveniences During the time that a project is being upgraded: Everybody will need to commit their work to cvs Will need to maintain 2 copies in cvs: Old New Will need to merge changes to both Hopefully this will be short for most projects