Software Testing with Python Magnus Lyckå Thinkware AB www.thinkware.se EuroPython Conference 2004 Chalmers, Göteborg, Sweden 2004, Magnus Lyckå
In the next 30 minutes you should... Learn about different aspects of software testing such as unit tests and acceptance tests. See how the standard modules for unit tests are used, learn about some alternatives and have a grasp of their respective pros and cons. Know about some options for acceptance tests. Know about some standard Python modules and third party tools that are helpful in software testing and related activities.
What is software testing? Testing means verifying that your code is running correctly by exercising the code under known conditions and checking that the results are as expected. Alex Martelli, Python in a Nutshell If we are serious with our requirements on a software system, we should make sure that we verify them, and testing is the most common kind of verification for software.
Software testing in a context Software Quality Assurance Software Verification Software Testing
Automating tests Writing automated tests is usually more work than testing things manually once, but they make it much easier to... Work in a repeatable and predicable way Run tests more often Run tests unattended Find regression bugs Test more operating systems Manage the test load as the system grows
Example of requirements and tests on different levels System Specification Acceptance Test API Specification Integration Test Detailed Design Unit Test
Requirements and tests on different levels in Extreme Programming (XP) System specified through customer tests API validated through continuous integration Design by writing programmer tests
Consequences of the XP approach Requirements are written in a verifiable way. No discrepancies between requirements and tests. We get unambiguous and repeatable ways of verifying requirements. Continuous picture of project progress But... Customers must be able to understand the tests. We need test automation frameworks. Automated tests more software to maintain.
Software Testing with Python Unit testing
Unit tests in Python Unit tests, or programmer tests, are tests the programmer writes to make sure that the code does what the programmer intended it to do. Unit tests can also help document how a piece of code (e.g. a class) is supposed to be used. Python has two standard modules for unit testing, unittest and doctest. There are several third party modules that aim to complement or replace unittest.
The unittest module The Python unittest module is modeled after the unit test modules developed within the XP community by Kent Beck and Erich Gamma. It's intended for a test-first approach, where tests are written before the actual code that it tests. Always writing tests first is a bit like always eating your meat before you eat your dessert... When your tests pass, you're done. (XPers combine this with refactoring.)
Concepts in unittest A test fixture consists of the actions needed to setup for a test, and clean up afterwards. A test case is the smallest unit of testing. It might for instance call a function and check the results. A test suite is a collection of test cases and/or test suites that should be executed together. A test runner runs test suites (or individual test cases), collects the results and presents it as text or graphically to the user.
The TestCase class Test cases sharing a common fixture will be implemented as methods (whose names start with test) in a sub class of unittest.testcase. The test runner will create one instance object per test case, and run SetUp, followed by the test case method and finally teardown. Checks are made with the methods fail, failif, assert_/failunless, assertequal/failunlessequal, assertnotequal/failifequal and assertraises/failunlessraises.
The test runner In many cases, running tests is as simple as executing unitest.main() in the file containing the test cases. Testing progress will be reported, and the test runner makes a distinction between FAILURE and ERROR. FAILURE means that the check didn't produce the expected result. ERROR means that something else in the test went wrong. (I.e. You got an exception.)
Unittest template import unittest import MyModule class MyModuleTest(unittest.TestCase): def testonecase(self): pass if name == ' main ': unittest.main()
Unittest examples divtest.py Trivial example to demonstrate features of unittest. container_ut.py Real world example from SystemSpecifyer (see http://sourceforge.net/projects/systemspecifyer)
The doctest module The Python doctest module was written by Tim Peters to check that coding examples in docstrings were correct, by testing them. It has also found use as a more general unit testing module, and from Python 2.3, this has been made more convenient. Using doctest helps you combine tests and documentation in a more readable way than unittest does.
Doctest example def divide(a,b): """ Return a divided with b. Divide will return the result of an integer division >>> divide(6,3) 2 Division by zero is handled as expected... >>> divide(1,0) Traceback (most recent call last):... ZeroDivisionError: integer division or modulo by zero """ result = a / b return result if name == ' main ': import doctest, sys doctest.testmod(sys.modules[ name ])
Unittest vs doctest Unittest mantra: Test a little, code a little The logical choice if you use extreme programming or test-driven development of some other kind. Maybe better for complex tests? Doctest mantra: Code a little, test a little The natural companion if you prefer to experiment interactively with your classes and functions in the Python interactive environment. Probably better for adding tests to already written code.
Unittest vs doctest on the web Charming Python: Testing frameworks in Python http://www-106.ibm.com/developerworks/linux/library/l-cptest.html Literate Testing: Automated Testing with doctest http://www.python.org/pycon/dc2004/papers/4/ DocTest at WikiWiki http://c2.com/cgi/wiki?doctest
Other contenders... Sancho a unit testing framework from MEMS Exchange. Adds coverage analysis etc. http://www.mems-exchange.org/software/sancho/ Peckcheck by Darius Bacon. http://www.accesscom.com/~darius/software/clickcheck.html The second standard library std. Here, 11:00 today...
Unittest complements Pester the Python version of Jester. It finds code that is not covered by tests, makes some change to your code, runs your tests, and if the tests pass it displays a message saying what it changed. http://jester.sourceforge.net/ Test coverage support E.g. http://www.garethrees.org/2001/12/04/python-coverage/ Mock objects: Here, 9:30 A better test runner: Here, 10:00
Software Testing with Python Integration testing
Python Integration Test No static linking. We must run code to test interfaces! (On the other hand, compilers and linkers fail to see lots of integration problems...) Since building/linking isn't an issue in a pure Python project, we can use either unit test tools or acceptance test tools for integration tests. Python is good at gluing things together, and thus helpful in all sorts of integration work. See e.g. http://www.thinkware.se/cgi-bin/thinki.cgi/usingpythonwithotherlanguages
Software Testing with Python Acceptance testing
Test Frameworks using Python QMTest Uses Python for test expressions. http://www.codesourcery.com/qmtest/ PyFIT Python clone of Ward Cunningham's Framework For Integrated Testing (FIT). http://www.xprogramming.com/software.htm Software Testing Automation Framework (STAF) Big framework from IBM with Python API. http://staf.sourceforge.net/ TextTest See more here, today 11:30!
Software Testing with Python Not quite software testing...
The usual suspects... Python is excellent for analysis and manipulation of data. A great tool for test related work. The re library is useful but use special tools when available, such as for parsing XML files. For dealing with files you might use the modules gzip, zipfile, codecs, filecmp, struct, sgmllib, xml.* etc Python interfaces well with internet services, database, and all sorts of external programs.
difflib More or less like the unix diff utility, but as a Python module. (New in version 2.1.) Useful when we want a more detailed response than FAILED from a test particularly if we are trying to spot small changes in big amounts of data. Great for spotting differences in configurations, for instance database schemas.
Difflib example >>> import difflib; d=difflib.differ() >>> diff =d.compare(['hello World', "This is the same.", "Time flies like an arrow. Isn't that great?"], ['Hello World!', "This is the same.", "Fruit flies like a banana. Isn't that great?"]) >>> print "\n".join(diff) - Hello World + Hello World!? + This is the same. - Time flies like an arrow. Isn't that great?? ^ ^^ - ^^^^ + Fruit flies like a banana. Isn't that great?? ^^^ ^^ +++ ^^
AT&T Graphviz Tools to generate graphs from C-like text files. Dot for directed graphs. Neato for undirected graphs Fairly clever algorithms for decent looking layout Generates graphs in many file formats Suitable for automatic generation of graphs Some control over placement is possible Great for dependency analysis! Not Python, but good anyway... :)
Remember this? System Specification Acceptance Test API Specification Integration Test Detailed Design Unit Test
AT&T Graphviz example 1 digraph G { label = "V-model"; sysspec [label="system Specification"]; systest [label="acceptance Test"]; apispec [label="api Specification"]; apitest [label="integration Test"]; unitspec [label="detailed Design"]; unittest [label="unit Test"]; } sysspec -> apispec -> unitspec; unittest -> apitest -> systest; sysspec -> systest; apispec -> apitest; unitspec -> unittest;
AT&T Graphviz example 2 digraph G { label = "V-model"; node [shape=box, style=filled, color="#ccccff"]; sysspec [label= "System Specification"]; apispec [label="api Specification"]; unitspec [label="detailed Design"]; node [shape=box, style=filled, color="#ffff99"]; systest [label="acceptance Test"]; apitest [label="integration Test"]; unittest [label="unit Test"];... {rank = same; sysspec; systest}; {rank = same; apispec; apitest}; {rank = same; unitspec; unittest};
Useful Books Python in a Nutshell, by Alex Martelli Text Processing in Python, by David Mertz Software Test Automation, by Fewster & Graham Just Enough Software Test Automation, by Daniel J. Mosley & Bruce A. Posey Testing Extreme Programming, by Lisa Crispin and Tip House Test-Driven Development By Example, by Kent Beck