Download Software Testing with Python

Software Testing with Python
Magnus Lyckå
Thinkware AB
www.thinkware.se
EuroPython Conference 2004
Chalmers, Göteborg, Sweden
© 2004, Magnus Lyckå
In the next 30 minutes you should...
●
●
●
●
Learn about different aspects of software testing
such as unit tests and acceptance tests.
See how the standard modules for unit tests are
used, learn about some alternatives and have a
grasp of their respective pros and cons.
Know about some options for acceptance tests.
Know about some standard Python modules and
third party tools that are helpful in software
testing and related activities.
What is software testing?
●
●
“Testing means verifying that your code is
running correctly by exercising the code under
known conditions and checking that the results
are as expected.” – Alex Martelli, Python in a Nutshell
If we are serious with our requirements on a
software system, we should make sure that we
verify them, and testing is the most common kind
of verification for software.
Software testing in a context
Software Quality Assurance
Software Verification
Software Testing
Automating tests
●
Writing automated tests is usually more work
than testing things manually once, but they make
it much easier to...
–
Work in a repeatable and predicable way
–
Run tests more often
–
Run tests unattended
–
Find regression bugs
–
Test more operating systems
–
Manage the test load as the system grows
Example of requirements and
tests on different levels
System Specification
API Specification
Detailed Design
Acceptance Test
Integration Test
Unit Test
Requirements and tests on different
levels in Extreme Programming (XP)
System specified through customer tests
API validated through continuous integration
Design by writing programmer tests
Consequences of the XP approach
●
Requirements are written in a verifiable way.
●
No discrepancies between requirements and tests.
●
●
We get unambiguous and repeatable ways of
verifying requirements.
Continuous picture of project progress
But...
●
Customers must be able to understand the tests.
●
We need test automation frameworks.
●
Automated tests → more software to maintain.
Software Testing with Python
Unit testing
Unit tests in Python
●
●
●
●
Unit tests, or programmer tests, are tests the
programmer writes to make sure that the code
does what the programmer intended it to do.
Unit tests can also help document how a piece of
code (e.g. a class) is supposed to be used.
Python has two standard modules for unit testing,
unittest and doctest.
There are several third party modules that aim to
complement or replace unittest.
The unittest module
●
●
●
●
The Python unittest module is modeled after
the unit test modules developed within the XP
community by Kent Beck and Erich Gamma.
It's intended for a test-first approach, where tests
are written before the actual code that it tests.
Always writing tests first is a bit like always
eating your meat before you eat your dessert...
When your tests pass, you're done. (XPers
combine this with refactoring.)
Concepts in unittest
●
●
●
●
A test fixture consists of the actions needed to
setup for a test, and clean up afterwards.
A test case is the smallest unit of testing. It might
for instance call a function and check the results.
A test suite is a collection of test cases and/or test
suites that should be executed together.
A test runner runs test suites (or individual test
cases), collects the results and presents it as text
or graphically to the user.
The TestCase class
●
●
●
Test cases sharing a common fixture will be
implemented as methods (whose names start with
test) in a sub class of unittest.TestCase.
The test runner will create one instance object per
test case, and run SetUp, followed by the test case
method and finally tearDown.
Checks are made with the methods fail, failIf,
assert_/failUnless, assertEqual/failUnlessEqual,
assertNotEqual/failIfEqual and
assertRaises/failUnlessRaises.
The test runner
●
●
In many cases, running tests is as simple as
executing unitest.main() in the file
containing the test cases.
Testing progress will be reported, and the test
runner makes a distinction between FAILURE
and ERROR.
–
FAILURE means that the check didn't produce the
expected result.
–
ERROR means that something else in the test went
wrong. (I.e. You got an exception.)
Unittest template
import unittest
import MyModule
class MyModuleTest(unittest.TestCase):
def testOneCase(self):
pass
if __name__ == '__main__':
unittest.main()
Unittest examples
●
divtest.py
–
●
Trivial example to demonstrate features of unittest.
container_ut.py
–
Real world example from SystemSpecifyer (see
http://sourceforge.net/projects/systemspecifyer)
The doctest module
●
●
●
The Python doctest module was written by
Tim Peters to check that coding examples in docstrings were correct, by testing them.
It has also found use as a more general unit
testing module, and from Python 2.3, this has
been made more convenient.
Using doctest helps you combine tests and
documentation in a more readable way than
unittest does.
Doctest example
def divide(a,b):
"""
Return a divided with b.
Divide will return the result of an integer division
>>> divide(6,3)
2
Division by zero is handled as expected...
>>> divide(1,0)
Traceback (most recent call last):
...
ZeroDivisionError: integer division or modulo by zero
"""
result = a / b
return result
if __name__ == '__main__':
import doctest, sys
doctest.testmod(sys.modules[__name__])
Unittest vs doctest
●
●
Unittest mantra: Test a little, code a little
–
The logical choice if you use extreme programming
or test-driven development of some other kind.
–
Maybe better for complex tests?
Doctest mantra: Code a little, test a little
–
The natural companion if you prefer to experiment
interactively with your classes and functions in the
Python interactive environment.
–
Probably better for adding tests to already written
code.
Unittest vs doctest on the web
●
Charming Python: Testing frameworks in Python
–
●
Literate Testing: Automated Testing with doctest
–
●
http://www-106.ibm.com/developerworks/linux/library/l-cptest.html
http://www.python.org/pycon/dc2004/papers/4/
DocTest at WikiWiki
–
http://c2.com/cgi/wiki?DocTest
Other contenders...
●
Sancho – a unit testing framework from MEMS
Exchange. Adds coverage analysis etc.
–
●
Peckcheck by Darius Bacon.
–
●
http://www.mems-exchange.org/software/sancho/
http://www.accesscom.com/~darius/software/clickcheck.html
The “second standard library” std.
–
Here, 11:00 today...
Unittest complements
●
●
Pester – the Python version of Jester.
–
It finds code that is not covered by tests, makes some
change to your code, runs your tests, and if the tests
pass it displays a message saying what it changed.
–
http://jester.sourceforge.net/
Test coverage support
–
E.g. http://www.garethrees.org/2001/12/04/python-coverage/
●
Mock objects: Here, 9:30
●
A better test runner: Here, 10:00
Software Testing with Python
Integration testing
Python Integration Test
●
●
●
No static linking. We must run code to test
interfaces! (On the other hand, compilers and
linkers fail to see lots of integration problems...)
Since building/linking isn't an issue in a pure
Python project, we can use either unit test tools or
acceptance test tools for integration tests.
Python is good at gluing things together, and thus
helpful in all sorts of integration work. See e.g.
http://www.thinkware.se/cgi-bin/thinki.cgi/UsingPythonWithOtherLanguages
Software Testing with Python
Acceptance testing
Test Frameworks using Python
●
QMTest – Uses Python for test expressions.
–
●
PyFIT – Python clone of Ward Cunningham's
Framework For Integrated Testing (FIT).
–
●
http://www.xprogramming.com/software.htm
Software Testing Automation Framework (STAF)
Big framework from IBM with Python API.
–
●
http://www.codesourcery.com/qmtest/
http://staf.sourceforge.net/
TextTest – See more here, today 11:30!
Software Testing with Python
Not quite software testing...
The usual suspects...
●
●
●
●
Python is excellent for analysis and manipulation
of data. A great tool for test related work.
The re library is useful but use special tools
when available, such as for parsing XML files.
For dealing with files you might use the modules
gzip, zipfile, codecs, filecmp,
struct, sgmllib, xml.* etc
Python interfaces well with internet services,
database, and all sorts of external programs.
difflib
●
●
●
More or less like the unix diff utility, but as a
Python module. (New in version 2.1.)
Useful when we want a more detailed response
than FAILED from a test – particularly if we are
trying to spot small changes in big amounts of
data.
Great for spotting differences in configurations,
for instance database schemas.
Difflib example
>>> import difflib; d=difflib.Differ()
>>> diff =d.compare(['Hello World', "This is the same.",
"Time flies like an arrow. Isn't that great?"],
['Hello World!', "This is the same.",
"Fruit flies like a banana. Isn't that great?"])
>>> print "\n".join(diff)
- Hello World
+ Hello World!
?
+
This is the same.
- Time flies like an arrow. Isn't that great?
? ^ ^^
- ^^^^
+ Fruit flies like a banana. Isn't that great?
? ^^^ ^^
+++ ^^
AT&T Graphviz
●
Tools to generate graphs from C-like text files.
–
Dot – for directed graphs.
–
Neato – for undirected graphs
–
Fairly clever algorithms for decent looking layout
–
Generates graphs in many file formats
–
Suitable for automatic generation of graphs
–
Some control over placement is possible
●
Great for dependency analysis!
●
Not Python, but good anyway... :)
Remember this?
System Specification
API Specification
Detailed Design
Acceptance Test
Integration Test
Unit Test
AT&T Graphviz example 1
digraph G {
label = "V-model";
sysspec
[label="System Specification"];
systest [label="Acceptance Test"];
apispec [label="API Specification"];
apitest [label="Integration Test"];
unitspec [label="Detailed Design"];
unittest [label="Unit Test"];
sysspec -> apispec -> unitspec;
unittest -> apitest -> systest;
sysspec -> systest;
apispec -> apitest;
unitspec -> unittest;
}
AT&T Graphviz example 2
digraph G {
label = "V-model";
node [shape=box, style=filled,
color="#CCCCFF"];
sysspec [label=
"System Specification"];
apispec [label="API Specification"];
unitspec [label="Detailed Design"];
node [shape=box, style=filled,
color="#FFFF99"];
systest [label="Acceptance Test"];
apitest [label="Integration Test"];
unittest [label="Unit Test"];
{rank = same; sysspec; systest};
{rank = same; apispec; apitest};
{rank = same; unitspec; unittest};
...
Useful Books
●
Python in a Nutshell, by Alex Martelli
●
Text Processing in Python, by David Mertz
●
Software Test Automation, by Fewster & Graham
●
●
●
Just Enough Software Test Automation, by
Daniel J. Mosley & Bruce A. Posey
Testing Extreme Programming, by Lisa Crispin
and Tip House
Test-Driven Development By Example, by Kent
Beck