Agile Methods for Analysis

Agile Methods for Analysis Lightweight Concepts for Team-Based Projects Sebastian Neubert CERN PH-LBD Sebastian Neubert Agile Analysis 1/22

Introduction: Data Analysis as a Continuous Improvement Loop What is Agile? Pilot Project: Scrum Analysis Team Sebastian Neubert Agile Analysis 2/22

Introduction: Data Analysis as a Continuous Improvement Loop Sebastian Neubert Agile Analysis 3/22

Core Values of a Physicist Why do we do this job? Curiosity and Creativity Spend my time on interesting problems, learn Make a visible contribution of my own invention Autonomy Work self-determined Flat hierarchies Rigor Attention to detail and scientific argument Continuous strive for improvement and critical scrutiny Sebastian Neubert Agile Analysis 4/22

Core Values of a Physicist Why do we do this job? Curiosity and Creativity Spend my time on interesting problems, learn Make a visible contribution of my own invention Autonomy Work self-determined Flat hierarchies Note clash of interests Interesting problems are hard need to collaborate Attention to detail can get tedious interesting? Continuous improvement might mean I have to question my way of doing things Rigor Attention to detail and scientific argument Continuous strive for improvement and critical scrutiny Sebastian Neubert Agile Analysis 4/22

Complex Problems The need for teams Sebastian Neubert Agile Analysis 5/22

A Short Story of Physics Analysis Typical patterns and pitfalls Sebastian Neubert Agile Analysis 6/22

Physics analysis is a continuous improvement loop Sebastian Neubert Agile Analysis 7/22

What do we need to tackle complex projects? Divide the problem into small but connected pieces (which is going to be non-trivial for a complex project) Inspect and track our progress Make problems visible as soon as possible Be able to adapt our analysis strategy as information becomes available Adapt what we do and how we do things Enable constant improvement and reproducibility Sebastian Neubert Agile Analysis 8/22

Results and Outcomes What do we produce? How can we improve: Repeatability continuous refinement of the analysis requires automated running Reproducibilty make it possible for future analysts to access and repeat what was done Sebastian Neubert Agile Analysis 9/22

Results and Outcomes What do we produce? How can we improve: Repeatability continuous refinement of the analysis requires automated running Reproducibilty make it possible for future analysts to access and repeat what was done Build the software that performs the analysis Enable 3rd parties to produce the plots and numbers Sebastian Neubert Agile Analysis 9/22

What is Agile? Sebastian Neubert Agile Analysis 10/22

How it started: The Agile Manifesto http://agilemanifesto.org/ Individuals and Interactions > Processes and Tools Functional Software > Documentation Collaboration > Negotiation Adaption to Change > Follow the Plan Agile concepts are well aligned with our core values Sebastian Neubert Agile Analysis 11/22

Agile is a Way Of Thinking and a set of values and ideas Sebastian Neubert Agile Analysis 12/22

Agile Practices People practices and Engineering Practices Sebastian Neubert Agile Analysis 13/22

Concrete Fields of Improvements What do we get for our analyses? Reduce double work allow to concentrate on interesting problems Spread knowledge and remove single points of failure the industry factor 1 Achieve sustainable work performance Improved Reproducibility Have more fun learning new things 1 the minimal number of people that have to leave physics for industry for your project to be screwed. Sebastian Neubert Agile Analysis 14/22

Pilot Project: A Scrum Analysis Team How to apply agile concepts in the real HEP world? Sebastian Neubert Agile Analysis 15/22

Scrum Analysis Team Pilot project 3 Summerstudents + 3 CERN Fellows 2 Analysis topics: Search for new decays of Λ b Λ c + (hh)k, B s f 2 (1525)µµ Timeframe: 13 weeks Testing Scrum as a development framework Sebastian Neubert Agile Analysis 16/22

Scrum in a Nutshell A Framework for Agile Software Development 1 A Team commits to delivering software in 30 days or less 2 A time is scheduled to show that 3 The Team creates the software 4 The Team offers their work for inspection and adapt the plan for the next cyle Sebastian Neubert Agile Analysis 17/22

A Tool for Continuous Improvement Transparency and Reflection Sebastian Neubert Agile Analysis 18/22

Artifacts and Roles Nuts and bolts of Scrum Artifacts Project backlog : Prioritized and Estimated ToDo-List Sprint backlog : Items for current sprint, split into tasks Potentially releasable analysis increment Definition of done: when is a task finished? Sebastian Neubert Agile Analysis 19/22

Artifacts and Roles Nuts and bolts of Scrum Artifacts Project backlog : Prioritized and Estimated ToDo-List Sprint backlog : Items for current sprint, split into tasks Potentially releasable analysis increment Definition of done: when is a task finished? Roles Self-organized analysis team Everyone directly working on the analysis Postdocs & Students Analysis owner (Moritz Karbach) Keeps the Backlog up to date (vetoe on backlog changes) Point of contact for working group and reviewers Scrum master (S.N.) Helps the team to do Scrum Vetoe on changes to process Sebastian Neubert Agile Analysis 19/22

What s going to happen? Activities in Scrum Kick-off workshop 2 (half)days workshop Introducing physics topic Setting up computing infrastructure Scrum workshop Sebastian Neubert Agile Analysis 20/22

What s going to happen? Activities in Scrum Kick-off workshop 2 (half)days workshop Introducing physics topic Setting up computing infrastructure Scrum workshop 1-week Sprints: Team working together in a common office Sprint planning workshop Timeboxed: 2h (< 8h) Discuss next steps, create sprint backlog Daily coffee Super-Quick status report < 15min Sprint review 1h (< 4) Demo working analysis software (no slides) Sprint retrospective 1h (< 3): Reflecting on HOW the team worked, discussing improvements Sebastian Neubert Agile Analysis 20/22

Continuous Improvement Requires Courage Need to create a save working climate Sebastian Neubert Agile Analysis 21/22

Easy to Understand, Hard to Master Sebastian Neubert Agile Analysis 22/22

BACKUP Sebastian Neubert Agile Analysis 23/22

Where Physics Projects are Agile How we already work with agility in mind Working in Iterations Weekly meetings with the working group (several project teams together) every 4-8 weeks larger report from the team (depends very much on the state of the project) Handling change Sebastian Neubert Agile Analysis 24/22

Individuals and Interactions > Processes and Tools Process = The Playing Rules used by a team to organize themselves Agile offers techniques to improve the interactions between people Be aware of the process (What are the rules?) Adapt workstyle continously to the needs (What rules help us?) Rules and guidelines can always be changed by the team. (If it doesn t work, we try something else.) What does it mean for physics? We are using formal processes only very sparingly Becoming more aware of processes and implicit rules allows to adapt them to our needs Cultivate an open discussion on how the team works together Sebastian Neubert Agile Analysis 25/22

Functional Software > Documentation Tests as documentation Include users in development, collect feedback Just enough documentation to use, maintain and further develop the software Build software such that it is self-explaining What does it mean for physics? Just enough still means enough! The analysis software is part of the documentation Analysis software can be written such that it can serve this purpose Analysis note more important than software documentation Sebastian Neubert Agile Analysis 26/22

Collaboration > Negotiation Refine and adapt the scope of a project continously Include all stakeholders in this discussion What does it mean for physics? I think we already do that quite well The review process allows a lot of individual flexibility, this should be further cultivated Probably most relevant for discussions between different groups and across institutes Sebastian Neubert Agile Analysis 27/22

Adaption to Change > Follow the Plan What does it mean for physics? There often is little planning now How can we create plans we can believe in? Sebastian Neubert Agile Analysis 28/22

Organisation Sebastian Neubert Agile Analysis 29/22

What is our Product? Things we are supposed to deliver Physics results Measurements, numbers, errors, tables Plots, graphs Slides, presentations Analysis notes, conference notes Papers Sebastian Neubert Agile Analysis 30/22

What is our Product? Things we are supposed to deliver Physics results Measurements, numbers, errors, tables Plots, graphs Slides, presentations Analysis notes, conference notes Papers Education Trained students Sebastian Neubert Agile Analysis 30/22

What is our Product? Things we are supposed to deliver Physics results Measurements, numbers, errors, tables Plots, graphs Slides, presentations Analysis notes, conference notes Papers Education Trained students Software Frameworks, tools (long lifetime, multiple users) Specialized analysis code (few months lifetime, few users) Sebastian Neubert Agile Analysis 30/22

Can something that has been developed for industry be applied to the special situation in physics? Sebastian Neubert Agile Analysis 31/22

(Not so) Special Situation in Physics What are we faced with in a typical HEP collaboration? Learning and Education Education is part of the product Wide range of talent and competence Sebastian Neubert Agile Analysis 32/22

(Not so) Special Situation in Physics What are we faced with in a typical HEP collaboration? Learning and Education Education is part of the product Wide range of talent and competence Unknown outcomes Data always offers surprises This is what research is about! Sebastian Neubert Agile Analysis 32/22

(Not so) Special Situation in Physics What are we faced with in a typical HEP collaboration? Learning and Education Education is part of the product Wide range of talent and competence Unknown outcomes Data always offers surprises This is what research is about! Spreadout workforce Multi-site working groups Sebastian Neubert Agile Analysis 32/22

(Not so) Special Situation in Physics What are we faced with in a typical HEP collaboration? Learning and Education Education is part of the product Wide range of talent and competence Unknown outcomes Data always offers surprises This is what research is about! Spreadout workforce Multi-site working groups External constraints Funding through multiple groups multiple sources of authority (professors, group leaders...) Competing experiments Conference deadlines Sebastian Neubert Agile Analysis 32/22

(Not so) Special Situation in Physics What are we faced with in a typical HEP collaboration? Learning and Education Typical problem in software companies Education is part of the product How does Agile deal with this? Wide range of talent and competence Unknown outcomes Data always offers surprises This is what research is about! Spreadout workforce Multi-site working groups External constraints Funding through multiple groups multiple sources of authority (professors, group leaders...) Competing experiments Conference deadlines Feature of all R&D projects How does Agile embrace change? Difficult problem This is actually not so special How does Agile attack these? Sebastian Neubert Agile Analysis 32/22

Why is it Important to Track Progress? Being stuck indicates problems that need attention We often tend to forget what we already have accomplished Acknowledging progress is essential for motivation Sebastian Neubert Agile Analysis 33/22