Automated Testing with Python assertequal(code.state(), happy ) Martin Pitt <martin.pitt@canonical.com>
Why automated tests? avoid regressions easy code changes/refactoring simplify integration design tool documentation
Unit test from Apport $ python problem_report.py -v basic creation and operation.... ok writing and re-decoding a big random file.... ok handling of CompressedValue values.... ok reading a report with binary data.... ok write_mime() with key filters.... ok [...] ----------------------------------------------------- Ran 25 tests in 4.889s
Unit tests smallest possible testable portion of the code noninteractive independent from each other no assumptions about environment or privileges small and fast
Integration test from pkgbinarymangler $ test/run -v Installed-Size gets updated... ok language packs are not stripped... ok OEM PPA for main package... ok OEM PPA for main package for blacklisted project... FAIL [...] ========================================================== FAIL: OEM PPA for main package for blacklisted project ---------------------------------------------------------- Traceback (most recent call last): File "test/run", line 379, in test_ppa_oem_main_blacklisted self.assert_( locale/fr/lc_messages/vanilla.mo in out) AssertionError ---------------------------------------------------------- Ran 12 tests in 91.783s FAILED (failures=1)
Integration tests end-to-end testing of larger scenarios more expensive reasonable assumptions about environment should still be mostly noninteractive
Python unittest structure 1 i m p o r t c o u n t e r 2 i m p o r t u n i t t e s t 3 4 c l a s s CounterTest ( u n i t t e s t. TestCase ) : 5 d e f setup ( s e l f ) : 6 s e l f. c = c o u n t e r. Counter ( ) 7 s e l f. w o r k d i r = t e m p f i l e. mkdtemp ( ) 8 9 d e f teardown ( s e l f ) : 10 s h u t i l. r m t r e e ( s e l f. w o r k d i r ) 11 12 d e f t e s t f o o ( s e l f ) : 13 #... 14 15 # main 16 u n i t t e s t. main ( )
Python unittest test cases 1 d e f t e s t i n i t ( s e l f ) : 2 Counter i s i n i t i a l i z e d w i t h 0 3 s e l f. a s s e r t E q u a l ( s e l f. c. count ( ), 0) 4 5 d e f t e s t i n c ( s e l f ) : 6 Counter. i n c ( ) adds +1 7 s e l f. c. i n c ( ) 8 s e l f. a s s e r t E q u a l ( s e l f. c. count ( ), 1) 9 10 d e f t e s t s t r ( s e l f ) : 11 Counter s t r i n g f o r m a t t i n g 12 s e l f. c. s e t ( 2 3 ) 13 s e l f. a s s e r t E q u a l ( x%s y % s e l f. c, x23y )
Errors and corner cases 1 d e f t e s t n o n i n t ( s e l f ) : 2 R e j e c t s non i n t e g e r s 3 s e l f. a s s e r t R a i s e s ( TypeError, s e l f. c. s e t, 1. 5 ) 4 5 d e f t e s t h u g e ( s e l f ) : 6 S u p p o r t s a r b i t r a r i l y l a r g e numbers 7 s e l f. c. s e t ( s y s. maxint ) 8 s e l f. c. i n c ( ) 9 s e l f. a s s e r t ( s e l f. c. count ( ) > s y s. maxint ) 10 s e l f. a s s e r t R a i s e s ( TypeError, s e l f. c. s e t, 1. 5 ) 11 12 d e f t e s t n o n n e g ( s e l f ) : 13 Can t count below z e r o 14 s e l f. c. i n c ( ) 15 s e l f. c. dec ( ) # s h o u l d be 0 now 16 s e l f. a s s e r t R a i s e s ( V a l u e E r r o r, s e l f. c. dec ) 17 # d e f i n e d v a l u e a f t e r e r r o r 18 s e l f. a s s e r t E q u a l ( s e l f. c. count ( ), 0)
Test-friendly code avoid hardcoded external addresses use proper abstractions (dbapi2, complete URLs, logic vs. GUI) break complex tasks into separate methods
Techniques JFDI! locality for unit tests: mock objects locality for integration tests: files: mini-chroots with mkdtemp() databases: sqlite :memory: network: SimpleHttpServer on localhost packages: $APT CONFIG devices: loop, $SYSFS PATH UI testing: event synthesis, xvfb
doctest 1 d e f f a c t o r i a l ( n ) : 2 Return t h e f a c t o r i a l o f n, an e x a c t i n t e g e r >= 0. 3 4 I f t h e r e s u l t i s s m a l l enough to f i t i n an i n t, 5 r e t u r n an i n t. E l s e r e t u r n a l o n g. 6 7 >>> f a c t o r i a l ( 0 ) 8 1 9 >>> f a c t o r i a l ( 5 ) 10 120 11 12 N e g a t i v e v a l u e s a r e not a l l o w e d : 13 14 >>> f a c t o r i a l ( 1) 15 Traceback ( most r e c e n t c a l l l a s t ) : 16... 17 V a l u e E r r o r : n must be >= 0 18 19 20 n = 0 #...
Need more power? Mock objects: python-mock Comprehensive UI testing: mago kvm
References http://en.wikipedia.org/wiki/unit testing http://docs.python.org/library/unittest.html test suites of Apport, Jockey, pkgbinarymangler, apt