Download Choosing a Specific Focus from the Experimental Framework

Experiences Gained from
Experimentation in Software Courses
at the University of Maryland
Vic Basili
University of Maryland
Issues
• Courses available for experimentation
• Benefits and Issues with classroom
experiments
• Training in experimentation
Software Engineering Courses
• There are two major software engineering classes
– CMSC 435 - A software engineering project course taught by
various people in various ways (graduate/undergraduate
credit available)
– CMSC 735 – A software engineering graduate course in
modeling` measurement, and process improvement
• And one HCI course
– CSMC 435 – A course in Computer Human Interfaces that
also teaches and uses experimental methods as part of the
curriculum.
• There are no research methods courses in the CS department,
nor is there likely to be in the near future
• Many other courses have projects but they are not used to run
experiments
CMSC 435
• Students work in teams (3 or more) to use the technology to build
some kind of software product
• Have been running experiments since 1975, beginning with Bob
Rieter’s thesis
• Early methodology analysis (top down design, chief programmer
teams, …) to specific technology experiments (reading vs. testing,
requirements reading, OO reading)
•
Experiment team is usually two to five people, of various levels of
experience, interacting and reviewing the design and
implementation, and one or two people analyzing the results.
CMSC 735
• Course is directly about model building and measurement – and
learning and experimentation
• Graduate students work on building models rather than software
systems
• Run experiments here also but mostly in the small
– study effects of technique,
– play with experimental methods (observation protocols,
grounded theory, …)
Running Classroom Experiments
• Goals for experimenting with the class:
– Teaches students in the class
• The study of SE follows an experimental paradigm
• The concepts behind experimentation
• That we do not know the effects of a process and the
effects of context variables and are learning
• How to learn more about the reffects
– Gives graduate students a chance to run an experiment
– Gives experimental software engineers
• a set of data points for analysis for mostly novice subjects
• a chance to debug protocols and improve the process
definition
• More confidence and credibility when running of a similar
experiment with a company
Running Classroom Experiments
• Problems with classroom experiments
– Limiting the context of the study
– Selecting process that is relevant to both students and
professionals
– Dealing with the learning curve
– Dealing with student/classroom conflicts
• Subject experience
• Multiple motivations
• Limited time frame, nature of the problem
• Design options
• Cheating
Running Classroom Experiments
•
Controlled experiments have limits when done in classroom or
training situations
– You must train everyone (effects the control group issue)
– You cannot teach a new technique and not expect people to use it,
especially when it is in comparison to a non-technique
– You face a variety of threats to validity
– There is high risk – spend the time and do not get sufficient
information/analysis
•
This effects the design
– Consider the PBR experimental style design (partial factorial) as
opposed to the reading vs. testing design
•
You need to supplement the approach with other evaluation
techniques
– e.g., a mix of quantitative and qualitative analysis, observations,
grounded theory
Teaching CMSC735
•
Besides models and metrics, cover aspects of the experimental discipline
•
Experimental Paradigms:
– Scientific, engineering, empirical, analytic methods
•
Experimental Classifications
– Level of variable relationship
– Experience of Subjects
– Experimental Setting
– Type of Study
– Types of Analysis
•
Factors Jeopardizing Validity
– internal validity, external validity
•
Building laboratory Manuals
•
Picking a dissertation topic in SE
The Experimental Discipline
•
Experimental Classifications
– Level of variable relationship
• Descriptive, Correlational, Cause-effect:
– Experience of Subjects
• Expert -> Novice
– Experimental Setting
• in vivo, in vitro
– Type of Study
• experimental (controlled experiment, quasi-experimental
design, case study, field study
– Types of Analysis
• Quantitative Analysis, Qualitative Analysis
Writing an Experimental SE Dissertation
Definitions
RESEARCH:
Diligent search or inquiry; scientific investigation and study to discover facts
.
SCIENCE:
Systematic knowledge of natural or physical phenomena;
Facts ascertained by observation, experiment, and introduction;
Ordered arrangement of facts known under classes or heads;
Theoretical knowledge as distinguished from practical;
Knowledge of principle and rules of invention, construction, mechanism, etc.,
As distinguished from art.
THEORY/MODEL:
A system for explaining a set of phenomena by specifying constructs and the
laws that relate these constructs to each other.
Writing an Experimental SE Dissertation
Definitions
Fact:
Hypothesis:
Experiment:
Theory:
Law:
Model:
Paradigm:
Truth:
information obtained through direct observation
educated guess, precedes an experiment
operation carried out (sometimes under controlled
conditions) to discover unknown effect/law, test/establish
hypothesis, illustrate a known law
possible explanation based upon many facts/reason
description/observation of behavior used for prediction
based upon facts and reason
simplified representation of a system/phenomenon
can be a theory or a law
conceptual filter, how we perceive/interpret
what really is
RESEARCH APPROACHES
ESTABLISHED FIELD:
Easier to answer questions
Areas better defined
More consensus on the importance of an area
Standard methods of study
METHODOLOGICAL APPROACHES:
ANALYSIS:
Build a theory
Derive properties
Show boundary conditions and limits
EXPERIMENTATION:
Formulate hypotheses
Deduce empirical consequences
Test the hypotheses by collecting data
QUESTIONS for EVALUATING RESEARCH
IS THERE NEW KNOWLEDGE?
Were the methods used to obtain the knowledge scientifically sound?
ARE THE RESULTS SIGNIFICANT?
Do they improve our ability to describe, predict, control or explain?
PICKING A TOPIC:
Build on prior theories
Fill in gaps in theories
Create new theories that explain better than old
Disprove a commonly held “proven” theory
CHARACTERISTICS:
Can be neatly packaged
Focused
Consistent methodology
QUESTIONS for EVALUATING RESEARCH
What would make the following dissertation research?
–
–
–
–
–
Building a descriptive model/theory
Building a predictive model/theory
Improving an existing model/theory
Verifying properties of a model/theory
Implementing/automating a model/theory
THINKING ABOUT THE RESEARCH PROCESS
THEORY AND RESEARCH PERSPECTIVES
PROBLEM
STATEMENT/
QUESTIONS
WHAT IS
KNOWN
GAP IN
KNOWLEDGE
RATIONALE/
FUNDAMENTAL
REASONS
SIGNIFICANCE/
ASSUMPTIONS
IMPLICATIONS
THINKING ABOUT THE RESEARCH PROCESS
PROCEDURAL PERSEPECTIVE
SEARCH
QUESTIONS
POPULATION
DATA NEEDED/
VARIABLES
DATA SOURCE/
INSTRUMENT
DATA
COLLECTION
PROCEDURE
ANALYSIS
SAMPLE EXPERIEMTNAL DISSERTATION
CHAPTER 1:
Introduction:
A.
General statement of the problem
B.
Statement of the hypotheses, objectives, or questions
C.
Definitions of terms (assumptions/limitations/significance)
CHAPTER 2:
Review of the Literature:
A.
Review of previous research
B.
Pertinent opinion
C.
Summary of the state-of-the-art (tie it all together)
CHAPTER 3:
Method:
A.
Description of the subjects (how chosen)
B.
Research design and procedures
(overview of statistical procedures)
C.
Description of measures employed
SAMPLE EXPERIMENTAL DISSERTATION
CHAPTER 4:
Findings:
A.
Description of finding pertinent to each hypothesis, objective, or
question
B.
Other findings
CHAPTER 5:
Summary and Discussion:
A.
Summary of research problem, method, and finding
B.
Conclusions
C.
Implications
D.
Suggestions for further research