Continuous integration and Minimization of dependencies. Tijs van der Storm

Size: px

Start display at page:

Download "Continuous integration and Minimization of dependencies. Tijs van der Storm"

Dominick Fletcher
8 years ago
Views:

1 Continuous integration and Minimization of dependencies Tijs van der Storm

2 Overview Continuous integration Background The Sisyphus system Benefits Minimizing compilation dependencies Build architecture of C programs Automatic dependency reduction Qualitative evaluation

dependencies Build architecture of C programs

3 Continuous integration

4 A Deliver Icon

5 Updating component based systems Example: ASF+SDF Meta Environment Have to maintain knowledge which versions of components work together which versions of components the user has Lowest level of it works : it builds Continuous integration continuous release

components work together which versions of components the user

6 Continuous Integration (CI) Extreme version of daily builds Frequent checkins Frequent builds Testing (unit, integration, acceptance,...) Key features Unified build script for automation Centralized source location (version control)

7 Sisyphus Component based CI Language & platform agnostic High configurability Accurate version tracking Update generation Work in progress

8 Sisyphus architecture Sun Builder changes Sisyphus Server results Knowledge Base notify Builder... Builder Linux Windows?

9 Selective recompilation Iteration n Iteration n+1 Iteration n+2 toolbus i toolbus i+1 toolbus i+1 toolbuslib toolbuslib aterm j aterm j+1

10 Flexible updates Iteration n Iteration n+1 Iteration n+2 toolbus i toolbus i+1 toolbus i+1 toolbuslib toolbuslib aterm j aterm j+1

11 Benefits of continuous integration Integration bugs detected early the effort of integration is exponentially proportional to the amount of time between integrations [Fowler] Frequent releases always a working executable available The more often the better Fast builds are a good thing

time between integrations [Fowler] Frequent releases always a working

12 Minimizing dependencies

13 Architecture influences build process Currently: dependencies between packages components are built from scratch Possible: dependencies between files dictates single language, e.g., C Optimization opportunities Reduce number of files read from disk Increase incrementality

dependencies between files dictates singl

14 Some facts... Some facts about a large industrial C program: 80 to 90% of unused but included program units on average, each file includes 60% of headers multiple times average size program file: 37Kb average size preprocessed image: 1.96 Mb Courtesy of Yu et. al.

15 Not my work: Precompilation Removing false code dependencies to speedup software build processes [Yu et. al. CASCON03] Build optimization in three steps: extract fine grained dependency graph remove unused program entities from it generate restructured headers Is just in time and lazy

CASCON03] Build optimization in three steps: extract fine grained

16 Program unit dependency graph typedef int NUMBER; struct node; typedef struct node { float value; struct node* next; list } *list; struct A { union { NUMBER value; node } u; }; extern int printf(char *format,...); enum { Satisfied, Denied }; int main(void) { list l, n; for (n = l; n; n = n->next) printf( %f, n->value); return (int)satisfied; } main printf Satisfied A NUMBER

17 Relational formalization Program units Unit = Decl Def RequirementsOf Unit Decl (= PUDG) UsersOf = RequirementsOf 1 Files File = Hfile Cfile UnitsOf File Unit FilesOf = UnitsOf 1

18 Reorganizing includes (example) 1.c 2.c foo, bar bar 1.c 2.c bar.h foo bar bar foo.h bar.h foo.h

19 Reorganizing includes Determine the definitions of a C file c Cfile U = UnitsOf[c] Determine the required declarations R = RequirementsOf + [U] Determine the header files that contain them I = FilesOf[R] Let c include I

declarations R = RequirementsOf + [U] Determine the

20 Restructuring headers (example) 1.c 2.c 3.c 1.c 2.c 3.c bar foo, bar foo bar bar foo foo foo_bar.h bar.h foo.h

21 Restructuring headers Start with an initial partitioning: put each declaration in its own header H = { {d} d Decl } Do a fixpoint computation over H: if any A, B in H with UsersOf + [A] = UsersOf + [B] merge declarations: H = H {A B} \ {A, B} Determine file dependencies as before

22 Evaluation Reported performance improvements: build size fresh build time incremental build time Nice features: independent of compiler orthogonal w.r.t. make j, distcc, ccache Useful technique to scale continuous integration

23 Summary Continuous integration system Sisyphus distributed architecture selective recompilation Dependency minimization build optimization header restructuring

24 Questions?

25 References Yu et. al., Reducing build time through precompilations for evolving large software, Tech. rep. CSRG 504, Univ. of Toronto, Yu et. al., Removing false code dependencies to speedup software build processes, CASCON, Dayani Fard et. al., Improving the build architecture of legacy C/C++ software systems, submitted to FASE, 2005.

Removing False Code Dependencies to Speedup Software Build Processes

Removing False Code Dependencies to Speedup Software Build Processes Yijun Yu Homy Dayani-Fard John Mylopoulos yijun@cs.toronto.edu homy@cs.toronto.edu jm@cs.toronto.edu University of Toronto IBM Canada