Download WMG WTST5 talk - Testing Education

Teaching about Software Testing
that is NOT for Defect Detection
W. Morven Gentleman
Dalhousie University
[email protected]
Conventional Perspective
• Software is tested on behalf of developer
• Purpose of testing is to find (and fix) bugs
This is an important perspective,
but not the only one
Other perspectives:
how unusual are they?
• Testing to meet regulatory or contractual
requirements
• Testing a software artifact to measure and
model quantitative attributes of the software
• Testing to measure human behaviour when
interacting with the software under test
• Testing where the investigation is openended, to find questions to ask
Measuring quantitative attributes
• Benchmarking
• Sizing and capacity planning
• Tuning performance to observed load
Measuring human behaviour
•
•
•
•
Usability testing
Assessing training requirements
Assessing change management activities
Studying tool adoption
Open-ended investigation
•
•
•
•
Validating a system model or specification
Studying competitors’ products
Interoperability testing
Identifying ways to use a product
Observations on teaching
• Students have great difficulty assimilating
experimental techniques and experimental
evidence. They cannot
–
–
–
–
–
–
–
conceive of interesting questions.
propose experiments.to answer such questions
suggest analyses of experimental data.
interpret experimental results.
recognize assumptions contradicted empirically
notice anomalies that indicate the unsuspected .
manage an investment of experimental effort.
Diagnosis
• CS/SE/IS students get no exposure to the culture
of empirical science
– the fundamental precept that all asserted truth must
ultimately be founded on experimental observation
– theory should be seen as an approximation
summarizing experimental evidence
• They are given a mathematical model of a
problem, and they are unable to distinguish that
model from physical reality
• Remedy?
Some topics students find novel
• Practical instrumentation
– Choosing what to measure and what scale to use
– Characterizing load
– Systematic sources of measurement artifact
• Practical application of statistics
–
–
–
–
Statistical vs.. practical significance
Exploratory data analysis
Modeling
Experimental design
Real Industrial Example
Time to update in-memory
Oracle Database – 1 try
• Specification requires < 0.5 seconds
• Vendor demonstration 0.472 seconds
Time to update in-memory
Oracle Database – 100 tries
6.00E-01
5.00E-01
4.00E-01
Series1
3.00E-01
2.00E-01
1.00E-01
0.00E+00
0
20
40
60
80
100
120
Time to update in-memory
Oracle Database – 5000 tries
7.00E-01
6.00E-01
5.00E-01
4.00E-01
Serie s1
3.00E-01
2.00E-01
1.00E-01
0.00E+00
0
1000
2000
3000
4000
5000
6000
Another real example
Time to open an Excel file
View of all software testing as
searching for defects
• Does it help in designing experiments and
analysis?
• Are defect detection strategies appropriate
in other situations?
Definition of defect
• “Operationally, it is useful to work with 2
definitions of a defect:
– From the producers viewpoint: a product
requirement that has not been met or an
attribute of a product that is not in the statement
of requirements that define the product.
– From the customers viewpoint: anything that
causes customer dissatisfaction, whether in the
statement of requirements or not”
Conclusion
A broader perspective on software testing can
be applicable to CS/SE/IS students in
careers with soft6ware developers and
software customers