Small Data in Big Data July 17, 2013 So6ware Experts Summit Sea>le

Transcription

1 Small Data in Big Data July 17, 2013 So6ware Experts Summit Sea>le Ayse Basar Bener Data Science Lab Mechanical and Industrial Engineering Ryerson University

2 Part 1 ANALYTICS IN SOFTWARE ENGINEERING

3 Data AnalyKcs in So6ware Engineering To make decisions under uncertainty How to assign available resources and budget end- to- end? Where to allocate scarce teskng resources? How much maintenance effort is required? When to stop teskng? How confident are we to release the product?.. Expert judgement is common way to make decisions Bias, availability, limited experience

4 Data AnalyKcs in So6ware Engineering Necessary, but sca>ered Code versioning systems Code metric repositories Issue tracking/ management systems Specialized tools Non- generalizable research outcomes

5 Theory (Kahneman) Human mind works in two modes: Fast Thinking Mode: Default Based on heuriskcs Error prone Slow Thinking Mode: ReacKve: Triggered by Fast Thinking Based on Facts and Logic Clouded by Fast Thinking

6 Where Big Data Techniques fit in SE: Help experts by simulakng human slow thinking mode in a faster mode!

7 The Problem Ø Sca>ered Research Clusters Ø Overlooked Research Clusters Ø Lack of generalizakon Efforts Ø Lack of Theory Ø Privacy Concerns of Industry Turhan, B. & A. Bener, RAISE 2013

8 The Vision Theory Interplay of analykcs techniques and SE to work like human brain Human in the loop models PracKce Use big data and experts to not only predict the future, but cause the future Tool support Caglayan et al., FSE 2012

9 How? Ø Stop validakng, start applying in real sehngs Ø to provide tools that combine individually validated research clusters for enabling applicakons in real sehngs Ø to refocus on overlooked research clusters, i.e. people in 3P (People, Product, Process). Ø to form an academic culture paying a>enkon to underlying theories and assumpkons to avoid academic number crunching exercises. Ø to extend our efforts beyond individual cases to pursue generalizakons. Ø to address the concerns of business side whose data and support are required to realize the above.

10 Puhng the Bricks Together Where we are Where we should be Not only predict the future, but cause the future

11 Part 2 DOES SIZE MATTER?

12 The devil is hidden in the details To uncover hidden pa7erns in big data

13 Big Data versus Data Analysis The sum of small pieces are larger than the whole? Issues Access Storage Processing

14 Size versus Data Any data or meaningful data Centralized or decentralized CollaboraKon or control May be small is beaukful

15 Small Data Data AnalyKcs = Big Data?? Context based Case studies Model assumpkons More data to overcome over fihng? CogniKve biases in determining model parameters

16 Part 3 SMALL DATA EXAMPLES

17 Small data is important in so6ware engineering Sampling: Empirical evidence Under/ micro sampling Dimensionality reduckon T. Menzies, B. Turhan, G. Gay, A. Bener, B. Cukic, H. Jiang PROMISE 08

18 Micro Sampling: Use Even Less... Given N defeckve modules: M = {25, 50, 75,...} <= N Select M defeckve and M defect- free modules. Learn theories on 2M instances Undersampling: M=N 8/12 datasets - > M = 25 1/12 datasets - > M = 75 3/12 datasets - > M = {200, 575, 1025} T. Menzies, B. Turhan, G. Gay, A. Bener, B. Cukic, H. Jiang PROMISE 08

19 QualitaKve Studies in So6ware Engineering ConnecKng the dots Field studies RecommendaKon systems Researcher and PracKKoner work together Be>er understand the overlooked areas

20 Problem Predic'on of defect categories Goal We aimed to increase the informakon content of the output of a defect predickon model by eskmakng the categories of defects in the defect- prone so6ware modules. Challenges Many defect categorizakon methodologies in the literature. No standard categorizakon methodology. Most of the exiskng defect data are not categorized at all.

21 1- Big Data Analysis We had to use the category definikons that were available in mulkple datasets: pre- and post- release defect categories. DefiniKon of pre- release and post- release different among projects. Predictor performance not saksfactory. CategorizaKon not worth the trouble? These categories were not meaningful for the customer.

22 1- Big Data Analysis

23 2- Small Data Analysis We idenkfied the defect categories with the quality assurance team of the company. We idenkfied metrics that were significantly correlated with the categories by analyzing the small data. The model since the categorizakon was tailored for the company needs. We improved defect predickon accuracy with the category- aware defect predictor.

24 2- Small Data Analysis

25 Lessons Learned Analysis of data with the key stakeholders of the organizakon is the key for providing delivering a valuable solukon. Knowledge gained from one customer may not be directly transferrable to another. Caglayan et al., Promise 2010 Tosun et al., WeTSOM 2011

26 Problem Confirma'on biases of so9ware engineers Goal to analyze factors affeckng so6ware engineers confirmakon biases. MoDvaDon due to the confirmatory behavior of so6ware engineers, defects may be introduced during any phase of SDLC IdenKficaKon of the factors affeckng confirmakon bias to circumvent its negakve effects Challenges QuanKficaKon of confirmakon bias

27 1- Big Data Analysis DefiniKon of a methodology to quankfy confirmakon bias levels of so6ware engineers. FormaKon of confirmakon bias metrics set. FormaKon of a single derived metric. ConducKng N- way ANOVA. Some results did not turn out to be meaningful.

28 1- Big Data Analysis

29 2- Small Data Analysis IdenKfied outliers in the data and analyzed them. Interviews with PM s and SE s who are outliers InvesKgate task load distribukon of developers. Outliers had heavy task loads and they were mentally exhausted. Hence, their test results did not reflect their actual confirmakon bias levels. We removed the outliers and repeated the analysis procedure.

30 2- Small Data Analysis

31 Lessons Learned Analysis of data with so6ware engineers and project managers who are involved in the field studies is crucial. Field studies should cover so6ware companies from different domains as much as possible to overcome threats to external validity and to obtain meaningful results. Calikli & Bener, ASE 2013 Calikli, Bener & Arslan, ICSE 2010

32 Part 4 CONCLUSION

33 Raw Data- the process case study Schu>, R. 2012, Data Science Course Blog: h>p:// columbiadatascience.com/blog

34 Small Data Meaningful small data is all you need Theories can be learned from a very small sample of available data We need to understand the underlying concepts Combine with available data and models Combined use of big data techniques and local models Remove errors with small data Access and use enourmous amount of data for analysis- with big data Tosun et al., 2014

35 Thank You