Download Mutation Testing: An Artificial Immune System

Mutation Testing: An Artificial Immune System
Approach
Peter May, Keith Mander and Jon Timmis
Computing Laboratory
University of Kent
Canterbury, Kent, CT2 7NF.
Tel: +44 (0)1227 764000
{psm4,kcm,jt6}@kent.ac.uk
Abstract. Through time, many software engineering fundamentals change
such as programming languages, paradigms and development teams.
These changes affect test suites; if these suites do not evolve, they will
wear out. Mutation testing is a computationally expensive, fault-based
testing method. By using high quality test data however, only a minimal
number of mutations are required, allowing changes in the development
environment. Initially however, test data is rarely of high quality. To
improve it necessitates more mutations. The focus of this conceptual paper is the adoption of an immune system metaphor as the basis for an
alternative mutation testing system. It is envisaged that such a system,
through careful monitoring of program development, will be able to provide a minimal set of effective mutations and test cases; a vaccine which
can be applied to the software development process to help protect it
from errors.
1
Introduction
Mutation testing is a powerful fault-based unit testing method [16, 17, 19, 20]. It
aims to give confidence in the system under test (SUT) by iteratively improving test data. The competent programmer hypothesis implies that programmers
create programs close to correct [7, 13]. Based on this theory, tests that cause
simple mutations (single lexeme differences, for example) of the SUT to produce
incorrect outputs, indicate the relative adequacy of the tests and the correctness
of the program.
Unfortunately, mutation testing experiences several problems. Notably, most
traditional systems are interpretive [19] implying a large computational expense
to generate and execute the many mutants required. This could also create subtle
changes in a program’s behaviour as the programs are not being executed in their
intended operational environment [20]. This leaves unit testing techniques, such
as mutation testing, too expensive for use in industry [16]. Three main reasons
for industry’s failure to adopt mutation testing strategies were suggested by
Offutt and Untch [19] as: a lack of economic incentives to use rigorous testing
methodologies; a failure to incorporate unit testing into the development cycle;
and difficulties with automating testing. The motivation for this paper is to
overcome these hurdles by introducing a new approach to mutation testing using
a biological metaphor.
Artificial immune systems (AIS) are adaptive systems, inspired from immunology, and applied to problem solving [4]. They use the human immune
system as a metaphor to develop new systems, such as for computer network security [8]. The human immune system is designed to protect a body from invading organisms by identifying, learning and attacking such cells. They therefore
demonstrate useful features such as pattern recognition, learning and memory.
An additional advantage is the ability to improve the immune system by the use
of vaccines. A vaccine allows antibodies to be developed to quicken the response
to an infection by the associated pathogen (invading organism). This presents a
useful analogy; the SUT is comparable to a body we wish to protect from infections - from errors. A vaccination is needed to protect the software development
process from these errors.
In section 2 we provide background information on mutation testing. This
is complemented by a new view of mutation testing in section 3. Artificial immune systems are discussed in section 4, followed by the combination of the two
disciplines in section 5.
2
Mutation Testing
DeMillo et. al. [13] introduced the idea of mutation testing as a means to improve
the quality of test data and to test programs. It is based on the notion of a
competent programmer, who creates programs close to being correct [7, 13]. If
such a programmer creates an incorrect program, then the correct version may
differ by only a small amount - a single lexeme perhaps.
Mutation testing works on the principle of generating and testing large numbers of mutant programs (mutants). Mutants differ only by a small amount (a
single lexeme) and are produced by mutation operators (mutagens) which alter
the semantics of a program. The coupling effect implies that tests that can discern small errors (single lexeme differences for example) are implicitly able to
recognise complex errors too [13]. Finding the small errors in our programs will
also find more complex errors in them. Mutation testing can therefore be used
to measure test data quality.
The mutation testing process works by first assuming that executing the original program with the test data produces the desired outputs even if it contains
an error. The original program must be developed until this assumption is true.
Mutations of this program are then created and tested on the same test data
set. If a mutant’s outputs differ from the original programs, then that particular
test case has detected an error in that mutant. This mutant is of no value and is
killed, but the test case is important (as it can distinguish incorrect programs)
and is kept.
Once all test cases have been executed on all mutants, those mutants that
remain alive are indistinguishable from the original (they produce the same
output). Some of these programs may be semantically equivalent to the original
program, producing the same outputs irrespective of the test case, i.e. no test
case will kill these mutants. These equivalent mutants are removed from the
mutation testing process before proceeding. The proportion of mutants killed
(discounting equivalent mutants) gives an indication of the quality of the test
data used to test the program. This is known as the Mutation Score. Increasing
this percentage to as near to 100% as possible will increase our confidence in the
test data.
The remaining living, non-equivalent mutants produce the same outputs as
the original, and may correspond to the correct program if the original has an
error. Further testing is required to distinguish if any of these mutants contain
errors. This is done by developing new fault-finding test cases.
Unfortunately, mutation testing suffers from a number of problems. First,
there are execution costs from running tests on the original program and all its
mutations. In addition to this, finding the mutations is also expensive. To tackle
these issues, most research has focussed on one of three strategies: do fewer (i.e.
N-Selective Mutation [18]), do faster (i.e. Mutant Schema Generation [20]), or
do smarter (i.e. Weak Mutation [9]) [19].
3
A New Mutation Testing View
The traditional view of mutation testing quality is based on the number of
mutants killed by the test data set. Obtaining a high mutation score can consequently be achieved in two ways: (i) using a high quality test set, or (ii) having
a low quality mutagen set that produces distinguishable mutants regardless of
the test data used. The second case is undesirable as this implies little testing
has occurred. Mutation testing quality therefore needs to be related to both the
quality of the test set and the mutagen set.
Considering test set quality, perfect test data would greatly simplify testing
because only correct programs would execute successfully. With such a test set,
the mutation quality is not as vital, only a minimal set of mutagens need be
applied - an N-selective approach. As the test set quality decreases though, the
mutation quality needs to improve in order to help improve the test cases. A
minimal set of good quality mutagens is required.
The competent programmer hypothesis states that programs are created close
to correct [7, 13], implying that mutagens should generate mutations reflecting
the common errors made by the programmer. Such errors will likely differ between programming teams, application languages and systems being developed.
In order to achieve this, a system that can learn the appropriate mutagens for
a particular team and application is desirable. These mutagens can be used to
improve the test data quality. By representing the most likely errors, the mutation testing system is customised to detect these faults, thus providing the basis
for improving mutation testing quality.
Combining the need for high quality mutagens and test data, results in a
useful analogy to this mutation testing system. The program can be viewed as
a human who we wish to protect from infections. A vaccination is required that
can be applied to the system to improve the health of the software development process. Minimal high quality test and mutagen sets provide this vaccine,
enabling the system to learn and adapt to the common mistakes made by the
programming team.
4
Immune Systems
Biology offers many interesting solutions to problems that can be applied to a variety of different applications in the computing field. One such metaphor that has
had increasing interest is that of the human immune system. It provides features
such as learning, memory, pattern recognition, predator-prey relationships, and
mutation abilities. Qualities, such as these, are often particularly desirable and
so immunological ideas are utilised to help develop various computing systems
[5].
Learning and memory are two features which are of particular interest to our
system. Both the mutagen and test data set must reflect the current operational
environment - the application, programming team, language for example. Such
environments vary considerably, and so the mutation testing system must be
adaptive and able to learn. Additionally, mutation testing itself contains two
distinct populations: mutagen and test cases. Immune systems also contain two
populations (immune cells and invading cells) which dynamically change and
this allows a convenient analogy to be derived.
De Castro and Timmis define artificial immune systems (AIS) to be adaptive
systems, inspired by theoretical immunology and observed immune functions,
principles and models, which are applied to problem solving [4]. Rather than a
creating a comprehensive model, these are systems developed using the immune
system as inspiration in an attempt to capture the features it provides. In order
to usefully apply AIS techniques though, it is pertinent to study some basic
immunology.
4.1
Human Immune System
The human immune system is a multi-layered defence mechanism designed to
protect our bodies from infectious organisms, or pathogens, such as viruses and
bacteria. It is composed of physical barriers, such as skin; biochemical barriers,
such as saliva and other fluids containing destructive enzymes; and two complementary forms of immunity: innate and adaptive. The innate immune system
is an unchanging mechanism that has two roles. It provides an initial response
to a wide range of invading pathogens and is also responsible for initiating and
controlling the adaptive immune response. The adaptive (or aquired) immune
system is more sophisticated, allowing it to direct its attack to previously unseen pathogens that cannot be removed by the innate system. Because of its
adaptability, the acquired immune system (or simply, the immune system) is the
system most often focussed on for metaphors.
The immune system is directed towards microbes, known as antigens, found
on the surface of invading organisms. Exposure to antigens modifies the adaptive
system to allow it to deal more effectively with a repeat infection by the same
antigen. Such exposures could occur naturally by infections, or artificially by
vaccines. The response is the same, resulting in the immune system learning to
recognise and attack the invading pathogens.
The two main components to the immune system response are B-cell and
T-cell lymphocytes (white blood cells). Both these types of cell originate in the
bone marrow, but differ in their site of maturation (where lymphocytes learn to
differentiate harmful cells). B-cells maturate in the bone marrow, whereas T-cells
migrate to the thymus for maturation. If the lymphocytes cannot differentiate
harmful cells they may bind to our own (self) molecules, causing an immune
response to be triggered. Such occurrences are called autoimmune diseases and
mean our immune system is fighting our own bodies.
T-cells display receptors on their surface which recognise and bind to antigens
that are processed and bound to a molecule known as major histocompatibility
complex (MHC). A cell that has ingested (or been infected by) an antigen, digests the antigen causing it to fragment into peptides. These then bind to MHC
molecules, forming a complex which is displayed on the surface of the cell. Tcells recognising this complex can directly attack infected self cells. In addition
to this, T-cells also provide functions such as regulation of other cells (providing
co-stimulatory signals to B-cells, for example).
B-cells are responsible for producing single specificity receptors to bind to
antigen, known as antibodies, and indicate for the invading cell to be destroyed.
Each antibody (and antigen) has a shape; a structure that determines which
antigens it will detect - its specificity. A B-cell only produces antibodies of
one specificity. Diversity in antibodies, and so in the antigens recognisable, is
obtained through the large number of lymphocytes producing receptors of differing specificity. Jerne [10] states that the number of lymphocytes in an adult
human is approximately 1012 . To recognise a pathogen, a complementary binding has to be made between antibody and an antigen on the pathogens surface.
The strength of this binding is defined as their affinity. The higher the affinity,
the stronger the binding. Recognition of an invading microorganism, based on
the strength of this affinity, triggers an adaptive immune response to kill the
pathogen.
There are two distinct theories regarding how the adaptive immune system
learns and remembers antigens which are worth examining. These are clonal
selection, and immune network theory. Clonal selection is based on a memory
pool, where as immune network theory is based on interactions between immune
cells.
Clonal Selection Theory Clonal selection [3] provides a theory to explain the
adaptive immune response to invading pathogens. When an antigen is recognised
by an antibody, the associated B-cell is selected to proliferate. This results in
a large number of cloned B-cells that match in specificity to the parent. The
number of clones produced is proportional to the affinity, or binding strength,
between the antibody and antigen. During reproduction, B-cells undergo mutation with high rates, termed somatic hypermutation. The effect of this is to alter
the shape of the receptors. As this rate is inversely proportional to the affinity,
mutating the receptors attempts to improve the affinity to the antigen; a local
search.
Furthermore, B-cells can not only proliferate into antibody secreting cells,
but they can differentiate into long-lived memory cells. These memory cells have
receptors that recognise the antigen that caused the B-cell to proliferate. Memory
cells probably do not produce antibodies themselves [4], but upon binding to an
antigen, proliferate into B-cells capable of producing antibodies.
Network Theory The clonal selection algorithm above views the immune system as a set of molecules that are at rest unless there is an invading pathogen.
An alternative view proposed by Jerne [10] is called immune network theory. It
proposes a set of interactions between network molecules irrespective of whether
there are any antigens present in the system. Properties such as learning and
memory now become a result of molecular interactions as opposed to a specific
memory pool.
In general, regions on an antigen’s surface called epitopes are recognised
by paratopes on antibodies. However, antibodies also display epitopes of their
own, known as idiotopes. A binding can now occur between antibodies, via the
same mechanisms for binding with antigens. This allows interactions, creating
a network structure, and thus stimulation, even in the absence of antigen. If
a paratope recognises an epitope (whether located on antigen or antibody), it
receives a stimulatory signal. A suppression signal is received if it is recognised
by another antibody. This controls the responses that cells make; a stimulatory
signal prompts the lymphocyte to proliferate for example, where as a suppression
signal may cause cell death.
When an antigen is encountered in the system, it is recognised by a set of
paratopes on antibodies. This same set of paratopes, through idiotypic interactions, will also recognise a set of epitopes on other lymphocytes. As this second
set is matched by the same set of paratopes, it forms an internal image of the
antigen. This is how new antigens are learnt and remembered.
Having an understanding of basic immunology aids in making analogies. Artificial immune systems use this immunological knowledge to provide a structural
basis for computing systems in an attempt to gain some of the features displayed
by immune systems.
4.2
Artificial Immune Systems
Artificial immune systems are adaptive systems applied to problem solving, that
are inspired by theoretical and observed immunology [4]. From this biological
metaphor, they hope to extract and utilise the features that natural immune
systems possess. Features such as recognition, learning, memory, and predatorprey response patterns.
An artificial immune system must embody three central principles: (i) a basic
model of an immune component, (ii) be designed by encompassing ideas from
theoretical and/or experimental immunology, and (iii) be aimed at problem solving [4]. A fundamental idea or model is required, meaning that simply using
immunological terminology is not sufficient. Such a model is typically antibodyantigen interactions coupled with antibody mutation (somatic hypermutation)
[4].
Given a problem domain, a framework is needed to bridge the gap to the
solution. By studying other biologically inspired algorithms, a generalised framework can be conceived. This consists of three layers: (i) representations for the
components of the system, (ii) measures of affinity between component interactions, and (iii) immune algorithms regulating system dynamics [4]. The first
layer to be constructed on the problem is a representation of the system components - for example, details of how to represent a particular element as an
antibody. The representation chosen will give rise to a selection of appropriate
affinity (the strength of interactions between immune cells and invading cells)
measures (Euclidean distance for example), forming the second layer. Utilising
these two layers, a third layer controlling the systems population dynamics can
be adopted. For example, clonal selection, immune network theory, or negative
selection algorithms.
The combination of these layers provides an elegant bridge from problem
to solution, providing a methodology to create an application that utilises AIS
techniques to capture some (or all) of the immune systems features. However, in
the context of software mutation testing, only limited use has been made of other
biologically inspired techniques: namely genetic and bacteriological algorithms
[1]. The following section, outlines this approach.
4.3
Biological Approaches to Mutation Testing
Baudry et. al. [1] describe two significant approaches to optimising test cases:
(i) Genetic Algorithms (GAs); and (ii) Bacteriological Algorithms (BAs).
Genetic algorithms operate similarly to the clonal selection algorithm mentioned above, except that (a) reproduction and mutation are not proportional
to the affinity; and (b) a reproduced component is generated from the combination (technically, crossover) of two parents. For more information on GAs,
consult [14]. With respect to mutation testing, each individual in the population represents a set of test cases. This population undergoes a standard GA
algorithm to improve the quality (the mutation score) of a member. The results
obtained from this experiment show potential. A 10% mutation rate improves
the mutation score to the 85-90% range.
The authors then compared this work to a bacteriological algorithm approach. BAs are similar to GAs except that (a) they maintain a memory set
consisting of the best individual per generation, and (b) each individual of the
population is atomic - it cannot be divided. Therefore, an individual can only
be reproduced and mutated. In mutation testing terms, the population represents the set of test cases needing improvement and not an individual as in GAs.
Their results are an improvement over the GA approach. The mutation score
achieved 95% and demonstrate the potential that can be achieved by hybridising
mutation testing with biological metaphors.
However, their work only focussed on optimising test cases for a specific subset of systems; namely those systems that translate input data from one format
to another, i.e. a parser or compiler. They did not address the simultaneous
optimisation of mutagens. Fortunately, the immune system shows predator-prey
responses between two populations - antigen and antibody. Both populations
evolve in an attempt to survive with respect the other. This is intrinsic behaviour
of immune systems. Additionally, there are subtle differences in the algorithms.
Clonal selection does not perform crossover, unlike a GA, and reproduction and
mutation are relative to the fitness of an individual, unlike a BA. Using the notion of an immune systems, it is proposed to create a vaccine that can be applied
to software development to protect it from errors.
5
Improving the Health of Software Development by
Vaccination
Figure 1 demonstrates the effective paths through which AIS techniques can be
applied to mutation testing.
Perfect test data
Ideal
quadrant
Too
expensive
No
mutation
testing
Complete
mutation
testing
Ineffective
testing
Safe
starting
position
Worthless test data
Fig. 1. The Mutation Testing/AIS approach
Of the four regions indicated, two are undesirable. The bottom left quadrant
represents an ineffective state; we have worthless test data (not distinguishing
correct programs from incorrect ones) and are doing no effective mutation testing
to improve it. Conversely, the top right quadrant is too expensive; we have
perfect test data distinguishing all programs but we still test all mutations. The
ideal situation would be to have a minimal amount of high quality test data to
distinguish correct programs, together with a minimal amount of well focussed
mutation testing to improve low quality test data (thus overcoming problems
associated with test suites wearing out) - the top left quadrant.
The vaccine is simply:
– The smallest set of test cases that kill the largest proportion of mutants,
– The smallest set of mutagens that deliver the largest set of consistently living
mutants.
However, it is difficult to know if we have reached the top left quadrant. The
only safe starting point is to assume we are in the bottom right quadrant and
allow test data and mutagen quality to improve.
5.1
General System Outline
The general framework described earlier consists of three layers. Using this structure, the proposed system is detailed. This is summarised in table 1 with a
mapping between mutation testing and immune systems.
Layer 1: Representation for the System Components. A system being
developed that we wish to protect is analogous to a (human) body. Code from
this system being tested will therefore be represented as a B-cell, and a mutation
will be an antibody. Mutation testing applies test data to kill a percentage of the
mutants; test cases are harmful to mutants, and are therefore termed antigens.
Layer 2: Affinity Measures between Component Interactions. There
are two competing populations interacting within this system: antibodies and
antigens - or mutants and test data. Test data (antigens) evolves to kill as
many of the mutants (antibodies) as possible. A mutation score represents the
affinity measure for the test data (antigens). Conversely though, the mutants
(antibodies) want to survive - the mutants want to be the correct program. Test
data executed on a mutant produces a specific output. When compared with
the output from the original program, a scale can be introduced detailing how
suitable (how close its outputs are to the originals) one mutant is compared with
the others. This is the mutant’s (antibody’s) affinity measure.
Layer 3: Immune Algorithms Regulating System Dynamics. Test data
(antigens) applied to the system cause it to respond, similar to antigenic recognition in the clonal selection algorithm. Using a mutant’s (antibody’s) affinity
measure (comparative correctness of its outputs), a selection of mutants (antibodies) will undergo clonal selection: they will clone proportional to, and mutate inversely proportional to their affinity. This process can be seen as a local
search, refining the mutagens (via selected mutations) that constantly produce
living mutants (they want to evolve to the correct program producing the correct
outputs). The refined mutagens can be added to a mutagenic immune memory,
forming half the required vaccine.
The other desire from the vaccination is to obtain a high quality test data
(antigen) set. The test data (antigen) affinity measure (mutation score) can be
used in a similar clonal selection method to reproduce and mutate the test data.
Tests that constantly kill mutant programs form good quality tests and are added
to the test set immune memory, forming the second half of the vaccination.
These clonal selection processes can be iteratively repeated until a suitable
vaccine is derived.
Table 1. Mapping between mutation testing and immunology
Mutation Testing
Immune Systems
Program
B-cells
Mutants
Antibodies
Test Data
Antigens
Producing and testing mutants Clonal Selection
Evolving test data
Clonal Selection
5.2
Outline of Software Vaccination Procedure
Using the mapping detailed in table 1, there are two populations interacting and
evolving using clonal selection algorithms. The resultant memory sets form the
vaccine. To obtain this, the procedure can be envisaged to work as follows, using
a combination of the two clonal selection algorithms - one controlling mutagen
quality, the other test data quality. The separation of these two algorithms allows
separate adaption of each population. For simplicity, a single program, P, is
considered as opposed to a multi-class system.
Mutagen Adaption
1. Initialisation: Mutate program P to create a set of initial mutants, P’
2. Mutagen Adaption: Execute test data on P’
(a) Affinity Evaluation: Evaluate how well the mutant survives the test data
(b) Clonal Selection: Clone high affinity mutants proportional to their affinities
(c) Affinity Maturation: Mutate a selection of the clones inversely proportional to their affinities
(d) Memory: Add the appropriate mutagens to a mutagen memory set
3. Cycle: Repeat step 2 until a suitable mutagen set is identified.
Test Data Adaption
1. Initialisation: Mutate program P to create a set of initial mutants, P’
2. Test Data Adaption: Execute test data on P’
(a) Affinity Evaluation: Evaluate the percentage of mutants a test case kills
(b) Clonal Selection: Clone high affinity test cases proportional to their
affinities
(c) Affinity Maturation: Mutate the clones inversely proportional to their
affinities
(d) Memory: Add the best test cases to a memory set
3. Cycle: Repeat step 2 until a suitable test set is found.
5.3
Practical Issues
As with traditional mutation testing systems, mutant creation and execution
require significant time. To counter this situation, our efforts have initially consisted of generating a system based on Mutant Schema Generation (MSG) [20].
This creates a metamutant program containing all possible mutations, saving
compilation costs by only compiling once. Each mutant can be selected at runtime by selecting the appropriate mutagen from the metamutant.
The basis to MSG is that a program’s syntactic tree can be altered so that
mutagen functions replace certain nodes. In Java, for example, replacing i=3+4;
with i=arithOp(3,4,+);. The plus in this example could be replaced by a bit
string that selects an appropriate operation to perform from a list. Passing this
bit string to the program at runtime will select the mutation that is to be applied.
With all mutations compiled into the same source file, each mutant program is
now referenced by a long bit string. In AIS terms, each bit string (or mutant) is
an antibody. This makes the task a search problem; find the smallest set of bit
strings that constantly remain living.
An additional problem is how to determine the affinity of a program. Some
measure is required as to how close a program’s outputs are to the original
program outputs. Currently mutation testing is dichotomous - mutants live or
die. For an immune system approach, a scale needs to be introduced to allow
affinity calculations.
Further, some process is required, given a set of living mutants, to determine
what are the appropriate mutagens for the vaccine. The living mutants will,
hopefully, have common mutations in them representing commonly made errors.
These require strategic clustering to derive common mutagens.
6
Conclusion
Generation and execution time of many mutant programs is of great concern in
mutation testing. Traditional research in this field has concentrated on one of
three possible approaches: do fewer, do faster, or do smarter [19]. An initial proposal for an adaptive mutation testing system has been discussed that attempts
to combine all three areas, and overcome problems with test suites wearing out.
It is proposed that the immune system provides a good metaphor for such
a mutation testing system. It has features such as learning, memory, pattern
recognition, and predator-prey response patterns. Features that make it particulary suitable for the mutation testing domain. This metaphor has been used
to create a system that monitors software development to create a vaccine; a
vaccine that will deliver a minimal high quality test set (to distinguish correct
mutant programs) coupled with a minimal amount of well focussed mutation
testing (to deal with test suites wearing out). The vaccine can then be ’injected’
into the software development cycle, resulting in an adaptive do fewer, do faster,
do smarter testing phase.
This process presents a number of problems. Firstly, the mutagen adaption
clonal selection phase requires an affinity measure based on how well a program
executes with a specific set of test data. How is this achieved? Secondly, given a
mutated program that represents a possible error, how do we select the associated
mutation operator that generated it?
To summarise, we believe that applying artificial immune systems to mutation testing is a plausible concept, resulting in the concept of a vaccine to
improve the health of the software development process.
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable
comments.
References
1. B. Baudry, F. Fleurey, J.-M. Jzquel, and Y. Le Traon, ”Genes and Bacteria for
Automated Test Cases Optimization in the .NET Environment”, presented at ISSRE
02 (Int. Symposium on Software Reliability Engineering), Annaplois, MD, USA, 2002.
2. B. Beizer, ”Software Testing Techniques”, Van Nostrand Reinhold, 1990.
3. F.M. Burnet, ”The Clonal Selection Theory of Acquired Immunity”, Cambridge
University Press, 1959.
4. L.N. de Castro, and J. Timmis, ”Artificial Immune Systems: A New Computational
Intelligence Approach”, Springer, 2002.
5. L.N. de Castro, and F.J. Von Zuben, ”The Clonal Selection Algorithm with Engineering Applications”, Workshop Proceedings of GECCO, pp. 36-37, Las Vegas,
USA, July 2000.
6. R.A. DeMillo, ”Completely Validated Software: Test Adequacy and Program Mutation”, Proceedings of the Eleventh International Conference on Software Engineering,
Pittsburgh, Pennsylvania, USA, 1989.
7. R.G. Hamlet, ”Testing Programs with the Aid of a Compiler”, IEEE Transactions
on Software Engineering, 3(4), July 1977.
8. S.A. Hofmeyr, and S. Forrest, ”Architecture for an Artificial Immune System”,
Evolutionary Computation, 7, pp. 45-68, 2000.
9. W.E. Howden, ”Weak Mutation Testing and Completeness of Test Sets”, IEEE
Transactions on Software Engineering, 8, pp. 371-379, July 1982.
10. N.K. Jerne, ”Towards A Network Theory of the Immune System”, Annals of Immunology, 125C, pp. 373-389, 1974.
11. K.N. King, and A.J. Offut, ”A Fortran Language System for Mutation-Based Software Testing”, Software - Practice and Experience, 21(7), pp. 685-718, July 1991.
12. A.P. Mathur, ”Performance, Effectiveness, and Reliability Issues in Software Testing”, Proceedings of the Fifteenth Annual International Computer Software and Applications Conference, pp. 604-605, Tokyo, Japan, September 1991.
13. R. DeMillo, R. Lipton, and F. Sayward, ”Hints on Test Data Selection: Help For
The Practicing Programmer”, IEEE Computer. Vol. 11(4), pp. 34-41, 1978.
14. M. Mitchell, ”An Introduction to Genetic Algorithms”, The MIT Press, 1998.
15. T. Morrison, and U. Aickelin, ”An Artificial Immune System as a Recommender for
Web Sites”, Proceedings of the First International Conference on Artificial Immune
Systems, pp. 161-169, Canterbury, UK, September 2002.
16. A.J. Offutt, ”A Practical System for Mutation Testing: Help for the Common
Programmer”, Proceedings of the International Test Conference, pp. 824-830, 1994.
17. A.J. Offut, A. Lee, G. Rothermel, R.H.Untch, and C. Zapf, ”An experimental
determination of sufficient mutant operators”, ACM Transactions on Software Engineering Methodology, 5(2), pp. 99-118, April 1996.
18. A.J. Offut, G. Rothermel, and C. Zapf, ”An experimental evaluation of selective
mutation”, Proceedings of the Fifteenth International Conference on Software Engineering, pp. 100-107, Baltimore, MD. May 1993.
19. A.J. Offutt, and R.H. Untch, ”Mutation 2000: Uniting the Orthogonal”, Mutation
Testing for the New Century, W.E. Wong (Ed.) Kluwer 2001.
20. R.H. Untch, A.J. Offutt, and M.J. Harrold, ”Mutation Analysis Using Mutant
Schemata”, Proceedings of the 1993 International Symposium on Software Testing
and Analysis, (Cambridge MA.), pp. 139-148, June 1993.