Automated Testing with Python

Transcription

1 Automated Testing with Python assertequal(code.state(), happy ) Martin Pitt <[email protected]>

2 Why automated tests? avoid regressions easy code changes/refactoring simplify integration design tool documentation

3 Unit test from Apport $ python problem_report.py -v basic creation and operation.... ok writing and re-decoding a big random file.... ok handling of CompressedValue values.... ok reading a report with binary data.... ok write_mime() with key filters.... ok [...] Ran 25 tests in 4.889s

4 Unit tests smallest possible testable portion of the code noninteractive independent from each other no assumptions about environment or privileges small and fast

5 Integration test from pkgbinarymangler $ test/run -v Installed-Size gets updated... ok language packs are not stripped... ok OEM PPA for main package... ok OEM PPA for main package for blacklisted project... FAIL [...] ========================================================== FAIL: OEM PPA for main package for blacklisted project Traceback (most recent call last): File "test/run", line 379, in test_ppa_oem_main_blacklisted self.assert_( locale/fr/lc_messages/vanilla.mo in out) AssertionError Ran 12 tests in s FAILED (failures=1)

6 Integration tests end-to-end testing of larger scenarios more expensive reasonable assumptions about environment should still be mostly noninteractive

7 Python unittest structure 1 i m p o r t c o u n t e r 2 i m p o r t u n i t t e s t 3 4 c l a s s CounterTest ( u n i t t e s t. TestCase ) : 5 d e f setup ( s e l f ) : 6 s e l f. c = c o u n t e r. Counter ( ) 7 s e l f. w o r k d i r = t e m p f i l e. mkdtemp ( ) 8 9 d e f teardown ( s e l f ) : 10 s h u t i l. r m t r e e ( s e l f. w o r k d i r ) d e f t e s t f o o ( s e l f ) : 13 # # main 16 u n i t t e s t. main ( )

8 Python unittest test cases 1 d e f t e s t i n i t ( s e l f ) : 2 Counter i s i n i t i a l i z e d w i t h 0 3 s e l f. a s s e r t E q u a l ( s e l f. c. count ( ), 0) 4 5 d e f t e s t i n c ( s e l f ) : 6 Counter. i n c ( ) adds +1 7 s e l f. c. i n c ( ) 8 s e l f. a s s e r t E q u a l ( s e l f. c. count ( ), 1) 9 10 d e f t e s t s t r ( s e l f ) : 11 Counter s t r i n g f o r m a t t i n g 12 s e l f. c. s e t ( 2 3 ) 13 s e l f. a s s e r t E q u a l ( x%s y % s e l f. c, x23y )

9 Errors and corner cases 1 d e f t e s t n o n i n t ( s e l f ) : 2 R e j e c t s non i n t e g e r s 3 s e l f. a s s e r t R a i s e s ( TypeError, s e l f. c. s e t, 1. 5 ) 4 5 d e f t e s t h u g e ( s e l f ) : 6 S u p p o r t s a r b i t r a r i l y l a r g e numbers 7 s e l f. c. s e t ( s y s. maxint ) 8 s e l f. c. i n c ( ) 9 s e l f. a s s e r t ( s e l f. c. count ( ) > s y s. maxint ) 10 s e l f. a s s e r t R a i s e s ( TypeError, s e l f. c. s e t, 1. 5 ) d e f t e s t n o n n e g ( s e l f ) : 13 Can t count below z e r o 14 s e l f. c. i n c ( ) 15 s e l f. c. dec ( ) # s h o u l d be 0 now 16 s e l f. a s s e r t R a i s e s ( V a l u e E r r o r, s e l f. c. dec ) 17 # d e f i n e d v a l u e a f t e r e r r o r 18 s e l f. a s s e r t E q u a l ( s e l f. c. count ( ), 0)

10 Test-friendly code avoid hardcoded external addresses use proper abstractions (dbapi2, complete URLs, logic vs. GUI) break complex tasks into separate methods

11 Techniques JFDI! locality for unit tests: mock objects locality for integration tests: files: mini-chroots with mkdtemp() databases: sqlite :memory: network: SimpleHttpServer on localhost packages: $APT CONFIG devices: loop, $SYSFS PATH UI testing: event synthesis, xvfb

12 doctest 1 d e f f a c t o r i a l ( n ) : 2 Return t h e f a c t o r i a l o f n, an e x a c t i n t e g e r >= I f t h e r e s u l t i s s m a l l enough to f i t i n an i n t, 5 r e t u r n an i n t. E l s e r e t u r n a l o n g. 6 7 >>> f a c t o r i a l ( 0 ) >>> f a c t o r i a l ( 5 ) N e g a t i v e v a l u e s a r e not a l l o w e d : >>> f a c t o r i a l ( 1) 15 Traceback ( most r e c e n t c a l l l a s t ) : V a l u e E r r o r : n must be >= n = 0 #...

13 Need more power? Mock objects: python-mock Comprehensive UI testing: mago kvm

14 References testing test suites of Apport, Jockey, pkgbinarymangler, apt