Continuous Integration/Testing and why you should assume every change breaks your code

Transcription

1 MÜNSTER Continuous Integration/Testing and why you should assume every change breaks your code René Milk 4th November 2015

2 MÜNSTER Continuous Integration/Testing 2 /16 Not talking about C functions Continuous completely automatic triggering (nightly on branch updates...) automatic reporting simple visualization wide accessibility metrics

3 MÜNSTER Continuous Integration/Testing 3 /16 No quadrature rules contained in this talk either Integration does your software stack still build after changes? compile run link work as a dependency for other software? how about on system X?

4 MÜNSTER Continuous Integration/Testing 4 /16 and ofc no weak solutions for PDEs Testing automated repeatable deterministic reliable Unit Testing System Testing Performance Testing

5 MÜNSTER Continuous Integration/Testing 5 /16 assume your every change breaks everybody s code saves time no need to argue about possible effects of changes It s better to err on the side of caution. You will break code from time to time. software is modular codebases are huge dependency graphs are complex distributed development: multiple people branches features/fixes...

6 Tools for CI: Travis MÜNSTER Continuous Integration/Testing 6 /16 free for FOSS integrates with GitHub repositories explicitly supports C++ python ruby but if your test/software runs on a standard-ish Ubuntu you re good triggers on branch push visual feedback in pull requests and badges very easy to create different configurations

7 Tools for CI: Travis MÜNSTER Continuous Integration/Testing 7 /16 language: python python: - "2.7" script: - DISPLAY=:99.0 py.test -k "${PYTEST_MARKER}" install: - python setup.py build_ext -i notifications: on_success: change on_failure: change after_success: - coveralls branches: except: - gh-pages

8 Tools for CI: Travis MÜNSTER Continuous Integration/Testing 8 /16 sudo: false language: cpp matrix: include: - os: linux compiler: gcc addons: &gcc49 apt: sources: [ ubuntu-toolchain-r-test ] packages: [ g gcc-4.9 ] env: CXX_COMPILER=g COMPILER=gcc os: linux compiler: gcc addons: *gcc49 env: CXX_COMPILER=g COMPILER=gcc-4.9 DELETE="dune-istl"

9 Tools for CI: Travis MÜNSTER Continuous Integration/Testing 9 /16

10 Tools for CI: Travis MÜNSTER Continuous Integration/Testing 10 /16

11 MÜNSTER Continuous Integration/Testing 11 /16 Tools for CI: Buildbot Python framework to build CI systems not a ready made solution Master/Slave setup good for testing complex stacks on diverse architectures much more freedom than on Travis at a much higher maintenance and setup cost

12 MÜNSTER Continuous Integration/Testing 12 /16 Tools for CI: Buildbot - Example setup to deploy new masters with only 5 LOC buildbot code for that to work is 2300 LOC git repository tracks individual DUNE-modulesŕepositories -> dynamic build steps -> make test result log tied into redmine on buildstatus change filter by user in change

13 MÜNSTER Continuous Integration/Testing 13 /16 Tools for CI: Buildbot - Example (master config) from common import config passwords genconfig config.current_db = passwords.dune_db_url import redmine.db.connection redmine.db.connection.setup(config.current_db) BuildmasterConfig = genconfig.genconfig( " project="dune-stuff" base_port=11020 use_cmake=true)

14 MÜNSTER Continuous Integration/Testing 14 /16 Tools for CI: Buildbot - Example (buildbot.opts) [clang_3.5] file = config.opts/clang-3.5 cc = clang-3.5 [gcc_5.0] file = config.opts/gcc-snapshot cc = gcc-snapshot [variants] # name = semicolon seperated list of directory names # to delete prior to buildbot config no_fem = dune-fem minimal = dune-grid;dune-localfunctions;dune-fem; dune-typetree;dune-pdelab;dune-istl

15 Buildbot MÜNSTER Continuous Integration/Testing 15 /16

16 Testing: What? MÜNSTER Continuous Integration/Testing 16 /16 utopic Every line of code is unit tested with function level granularity. System tests cover all possible software configurations and inputs. Performance testing is run on a wide variety of benchmark problems in a restricted possibly virtualized environment. All of that on a very diverse set of computer architectures operating systems support library versions. Not viable in the real world. Configuration space is just too big. Compute time and available hardware is limited. Test cycles need to be short to not impede development.

17 Continuous Integration/Testing 17 /16 Testing: What? realistic Every non-trivial piece of code is unit tested with algorithm level granularity. System tests cover the most common configuration options. Benchmark runs are recorded.

18 Continuous Integration/Testing 18 /16 Testing: What? minimal The code is run for a benchmark problem with known/analytic solution.

19 Continuous Integration/Testing 19 /16 Testing: How? Use a test harness: google-test Boost.Test py.test Simplifies selective test running result gathering behavior expectation Writing a DUNE module? Use dune-testtools for simple test parameterization record algorithm results error norms and compare in system tests Make sure exceptions get raised on invalid data input integrate into build system (ie make test)

20 Testing: How? - Gtest Example Continuous Integration/Testing 20 /16 typedef testing::types<yaspgrid<1> YaspGrid<4> Grids; template <class T> struct CornerRangeTest : public ::testing::test { CornerRangeTest() : grid_prv(0. 1. level) {} void check() { for (auto it : grid_prv.grid().leafgridview()) check_range(it cornerrange(it->geometry())); for (auto v : DSC::valueRange(T::dimensionworld)) EXPECT_GE(v 0); } }; TYPED_TEST_CASE(CornerRangeTest Grids); TYPED_TEST(CornerRangeTest Misc) { this->check(); }

21 Testing: What tests are allowed to fail Continuous Integration/Testing 21 /16 NONE except those where the platform requirements cannot be met

22 Testing: What else is there? Continuous Integration/Testing 22 /16 Using cmake? Make ctest submit test results to cdash

23 Continuous Integration/Testing 23 /16 Thank you for your attention. Slides available at