Download Bockholt_Software_Quality

Software Quality Assurance in
Neuroinformatics
H Jeremy Bockholt
20080630 NITRC Grantee Meeting
QuickTime™ and a
decompres sor
are needed to s ee this picture.
Outline
•
•
•
•
Background and Examples
Software Process
Software Quality Assurance
Recommendations
Background
• Neuroinformatics is an emerging discipline which attempts to
integrate neuroscientific information from the level of the
genome to the level of human behavior.
• Neuroinformatics produces digital capabilities for a web-based
information management system in the form of interoperable
databases and associated data management tools.
• Neuroinformatics tools include:
– software for querying and data mining, data manipulation
and analysis, scientific visualization, biological modeling
and simulation, and electronic communication and
collaboration
Examples of Neuroinformatics
Tools
•
•
•
•
MRN has over 5500 human subjects, with
6500 scanning sessions using MR, EEG,
MEG
– We are now actively sequencing
1,000,000 SNPs arrays on subjects
as well as collecting vast amounts of
clinical, neuropsychological,
behavioral, and physiological data
– Using MRN tools, over 200 users at 7
sites collaborate on 150 active
research studies with access to
nearly 200 TB of data and growing…
BIRN has a vast set of multisite tools
with even larger user base, set of data
and tools and has adopted a federated
model
XNAT multi-site tool for collaborative
research data management
Analysis tools such as Slicer, FSL, AFNI,
SPM, BRAINS, Freesurfer, etc.
QuickTime™ and a
TIFF (U ncompressed) decompressor
are needed to see this picture.
Goal
•
Just like we are converging on standards and recommendations for
data storage, data exchange, data access, we should explore the
possibility of having recommendations for software process and
software quality assurance for the field of neuroinformatics
Software Process
• What tools and set of procedures and policies should be
followed to create software.
– Source Code Revision Control
– Compilation
– Testing
– Documentation
– Enhancement Tracking
– Project Management
– Communication and Collaboration
Software Process Challenges
• The largest challenge in neuroinformatics is that usually there
is a resistant to following a software process as the perception
is that it take too much time
• It is common for developers and users can be at separate sites
• Many of our neuroinformatics developers are not software
engineers
• Time constraints and funding constraints sometimes limit
neuroinformatics tool development just to functional
prototypes that have limited or no testing or documentation
associated with them
The NA-MIC Software Process
•
NA-MIC has a publication on software process and follows an
excellent example of a software process optimized for
Neuroinformatics
– Adaptation of the distributed, open source software process
– Targeted to the data-driven approach.
– Utilize eXtreme Programming techniques
• generating user stories, using simple design, writing tests first,
refactoring, using continuous integration, and using coding
standards are employed
• some aspects such as having on-site customers, and pairprogramming are not used
– Suggests using CVS, DART, DOXYGEN, Bugzilla, Cmake, to
support software development process
– Encourages software developers to share code and tools and
encourages participation in two programming weeks per year to
permit face to face time for rapid development
Software Quality Assurance
• The purpose of a software quality assurance plan is to define
the formal processes and procedures for software
development, a release review process, software development
standards, and testing standards for a organization to follow
across all software projects.
• The plan should formally define communication strategies,
Software Testing Plan, Software Measurements (calibration and
user satisfaction), Risk Assessment, Corrective action process,
and Training for developers
• Software Quality should be measured and tracked at each
development iteration
Software Quality Challenges
• Most Neuroinformatics software development is for users at
multiple sites, much of the software is deployed into multi-site
research consortia
• Most Neuroinformatics software development occurs under
limited resources, shortened development cycles, large time
constraints
• A formal quality plan seems to be rarely followed,
• Documentation, training, and thorough testing are typically cut
or truncated to meet the constraints of budget and time
• While the field is starting to share tools and source code
(thanks to NITRC), we’re not sharing software process and
strategies for maintaining quality
A Software Quality Plan for
Neuroinformatics
•
•
•
•
We have developed a software quality plan that has been optimized to
avoid some of the challenges found in the field of neuroinformatics.
Some of the highlights are using DART and Bugzilla to automatically
perform and track software quality metrics
– programmer productivity (bugs/module)
– module defect density (code size/programming time)
– requirements stability (number of initial requirements / total number
of requirements)
– system spoilage (effort fixing bugs / total project effort)
Conduct user satisfaction surveys at the onset and during use of each
software product release
We have also generated recommendations on software development
standards (IEEE) and specifically recommend following General
Principles of Software Validation (US FDA, 2002) for neuroinformatics
tools
Future Work
• How can a software process be shared?
• How can a software quality plan be shared?
• Can the community decide on a preferred software process and
software quality plan
• We should publish a lessons learned manuscript on software
quality in multi-site neuroinformatics
Conclusions
• NITRC is a well-positioned to guide the field on software
process and software quality
• Having tool developers share their software process and
quality plans would assist adopters of the tools better assess
tools for integration with other components