Process Mining An index to the state of the art and an outline of open research challenges at DIAG

Transcription

1 An index to the state of the art and an outline of open research challenges at DIAG Claudio Di Ciccio, Massimo Mecella Seminars in Software and Services for the Information Society

2 Definition [Aalst2011.book], also referred to as Workflow Mining, is the set of techniques that allow the extraction of process descriptions, stemming from a set of recorded real executions (logs). ProM [AalstEtAl2009] is one of the most used plugin based software environment for implementing workflow mining (and more) techniques. The new version 6.0 is available for download at Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 2

3 Definition involves: Process discovery Control flow mining, organizational mining, decision mining; workflow Conformance checking Operational support We will focus on the control flow mining Many control flow mining algorithms proposed α [AalstEtAl2004], α ++ [WenEtAl2007] and α # [WenEtAl2010] Fuzzy [GüntherAalst2007] Heuristic [WeijtersEtAl2001] Genetic [MedeirosEtAl2007] Two-step [AalstEtAl2010] Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 3

4 Further reading The rest of the lesson is based on the following material: Van der Aalst, W. M. P.: Process Discovery: An Introduction Available at process_mining_chapter_05_process_discovery.pdf From the teaching material for [Aalst2011.book] De Medeiros, A. K. A.: : Control-Flow Mining Algorithms Available at lecture3_controlflowminingalgorithms.ppt Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 4

5 A different context for process mining Artful processes and knowledge workers Artful processes [HillEtAl06] informal processes typically carried out by those people whose work is mental rather than physical (managers, professors, researchers, engineers, etc.) knowledge workers [ACTIVE09] Knowledge workers create artful processes on the fly Though artful processes are frequently repeated, they are not exactly reproducible, even by their originators, nor can they be easily shared Loosely structured Highly flexible Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 5

6 A different context (2) conversations In collaborative contexts, knowledge workers share their information and outcomes with other knowledge workers E.g., a software development mgr. Typically, by means of several conversations conversations are actual traces of running processes that knowledge workers adhere to Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 6

7 A different context (4) Some areas of applicability Personal information management (PIM) how to organize one s own activities, contacts, etc. through the usage of software Information warfare in supporting anti-crime intelligence agencies Enterprise engineering for knowledge-heavy industries, where preserving documents making up product data is not enough ehealth for the automatic discovery of medical treatment procedures on top of patient health records Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 7

8 MailOfMine What is MailOfMine? MailOfMine is the approach and the implementation of a collection of techniques, the aim of which is to is to automatically build, on top of a collection of messages, a set of workflow models that represent the artful processes laying behind the knowledge workers activities. [DiCiccioEtAl11] [DiCiccioMecella12] [DiCiccioMecella/TR12] [DiCiccioMecella13] Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 8

9 On the visualization of processes The imperative model Represents the whole process at once The most used notation is based on a subclass of Petri Nets (namely, the Workflow Nets) Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 9

10 On the visualization of processes The declarative model If A is performed, B must be perfomed, no matter before or afterwards (responded existence) Rather than using a procedural language for expressing the allowed sequence of activities, it is based on the description of workflows through the usage of constraints the idea is that every task can be performed, except the ones which do not respect such constraints this technique fits with processes that are highly flexible and subject to changes, such as artful processes The notation here is based on [AalstEtAl06, MaggiEtAl11] (DecSerFlow, Declare) Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) Whenever B is performed, C must be performed afterwards and B can not be repeated until C is done (alternate response) P. 10

11 On the visualization of processes Imperative vs declarative Declarative Imperative Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) Declarative models work better in presence of a partial specification of the process scheme P. 11

12 A real discovered process model Spaghetti process [Aalst2011.book] Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 12

13 Declare constraint templates Existence templates Existence(n, A) Activity A occurs at least n times in the process instance BCAAC BCAAAC BCAC (for n = 2) Absence(A) Activity A does not occur in the process instance BCC BCAC Absence(n+1, A) Activity A occurs at most n+1 times in the process instance BCAAC BCAC BCC (for n = 2) Exactly(n, A) Activity A occurs exactly n times in the process instance BCAAC BCAAAC BCAC (for n = 2) Init(A) Activity A is the first to occur in each process instance BCAAC ACAAAC BCC Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 13

14 Declare constraint templates Relation templates RespondedExistence(A, B) If A occurs in the process instance, then B occurs as well CAC CAACB BCAC BCC Response(A, B) If A occurs in the process instance, then B occurs after A BCAAC CAACB CAC BCC AlternateResponse(A, B) Each time A occurs in the process instance, then B occurs afterwards, before A recurs BCAAC CAACB CACB CABCA BCC CACBBAB ChainResponse(A, B) Each time A occurs in the process instance, then B occurs immediately afterwards BCAAC BCAABC BCABABC Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 14

15 Declare constraint templates Relation templates RespondedExistence(B, A) If B occurs in the process instance, then A occurs as well CAC CAACB BCAC BCC Precedence(A, B) B occurs in the process instance only if preceded by A BCAAC CAACB CAC BCC AlternatePrecedence(A, B) Each time B occurs in the process instance, it is preceded by A and no other B can recur in between BCAAC CAACB CACB CABCA BCC CACBAB ChainPrecedence(A, B) Each time B occurs in the process instance, then B occurs immediately beforehand BCAAC BCAABC CABABCA Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 15

16 Declare constraint templates Relation templates CoExistence(A, B) If B occurs in the process instance, then A occurs, and viceversa CAC CAACB BCAC BCC Succession(A, B) A occurs if and only if it is followed by B in the process instance BCAAC CAACB CAC BCC AlternateSuccession(A, B) A and B occur in the process instance if and only if the latter follows the former, and they alternate each other in the trace BCAAC CAACB CACB CABCA BCC CACBAB ChainSuccession(A, B) A and B occur in the process instance if and only if the latter immediately follows the former BCAAC BCAABC CABABC Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 16

17 Declare constraint templates Negative relation templates NotCoExistence(A, B) A and B never occur together in the process instance CAC CAACB BCAC BCC NotSuccession(A, B) A can never occur before B in the process instance BCAAC CAACB CAC BCC NotChainSuccession(A, B) A and B occur in the process instance if and only if the latter does not immediately follows the former BCAAC BCAABC CBACBA Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 17

18 Relation constraint templates subsumption Constraint templates are not independent of each other Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 18

19 MINERful ++ Our mining algorithm MINERful ++ is the workflow discovery algorithm of MailOfMine Its input is a collection of strings T and an alphabet Σ T Each string t is a trace Each character is an event (enacted task) The collection represents the log Its output is a declarative process model What is a declarative process model? Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 19

20 The declara=ve specifica=on of an process (e.g.) Scenario A project mee+ng is scheduled We suppose that a final agenda will be commi9ed ( confirmagenda ) a>er that requests for a new proposal ( requestagenda ), proposals themselves ( proposeagenda ) and comments ( commentagenda ) have been circulated. Shortcuts for tasks (process alphabet): p ( proposeagenda ) r ( requestagenda ) c ( commentagenda ) n ( confirmagenda ) Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 20

21 The declara=ve specifica=on of an process (e.g.) Constraints on activities Existence constraints 1. Participation(n) 2. Uniqueness(n) 3. End(n) Relation constraints 4. Response(r, p) 5. RespondedExistence(c, p) 6. Succession(p, n) The agenda 1. must be confirmed, 2. only once: 3. it is the last thing to do. During the compilation: 4. the proposal follows a request; 5. if a comment circulates, there has been / will be a proposal; 6. after the proposal, there will be a confirmation, and there can be no confirmation without a preceding proposal. Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 21

22 MINERful ++ Workflow discovery by constraints inference MINERful ++ is a two-step algorithm 1. Construction of a Knowledge Base 2. Constraints inference by means of queries evaluated on the KB This allows the discovery of constraints through a faster procedure on data which are smaller in size than the whole input Returned constraints are weighted with their support the normalized fraction of cases in which the constraint is verified over the set of input traces Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 22

23 MINERful ++ by example Workflow discovery by constraints inference In order to see how it works now we see a run of MINERful ++ over a string, compliant with the previous example: r r p c r p c r c p c n We start with the construction of the algorithm s Knowledge Base Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 23

24 MINERful ++ by example Building the ownplay of p and n p n p occurred 3 +mes in 1 string γ p (3) = 1 For each m 3 γ p (m) = 0 p did not occur as the first nor as the last character g i (p) = 0 g l (p) = 0 γ n (1) = 1 For each m 1, γ n (m) = 0 n occurred as the last character in 1 string g i (n) = 0 g l (n) = 1 r r p c r p c r c p c n r r p c r p c r c p c n Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 24

25 MINERful ++ by example Building the interplay of p and n With respect to the occurrence of p, n occurred i. Never before: 3 times ii. iii. iv. δ p,n (- ) = 3 2 char s after: 1 time δ p,n (2) = 1 6 char s after: 1 time δ p,n (6) = 1 9 char s after: 1 time δ p,n (9) = 1 v. Repetitions in-between: i. onwards: 2 times ii. b p,n = 2 backwards: never b p,n = 0 Looking at the string i. r r p c r p c r c p c n ii. r r p c r p c r c p c n iii. r r p c r p c r c p c n iv. r r p c r p c r c p c n v. i. r r p c r p c r c p c n ii. r r p c r p c r c p c n Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 25

26 MINERful ++ by example Building the interplay of r and p δ r,p b r,p = 1 b r,p = 0 r r p c r p c r c p c n Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 26

27 MINERful ++ Computing the support of constraints Let Γ r and Γ p be the number of times in which r and p respectively appear in the log! x =! x (n)" n we have, e.g., support for Response(r, p) 1!! r, p (+") # r hint: how many times p was not read in the traces after r occurred? In those cases, Response(r, p) does not hold support for Succession(r, p) 1!! r, p (+")+! p,r (!") # r + # p # n>0 how many times p was not read in the traces after r occurred, nor r was read before p occurred? In those cases, Succession(r, p) does not hold Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 27

28 MINERful ++ Computing the support of constraints Constraint Support function Constraint Support function Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 28

29 MINERful ++ by example Computing the support of constraints 1. r r p c r p c r c p c n 2. r c c r p p c p c r c p p n 3. c c c c c r r p c r r p r r p r p p p n 4. c p r p n 5. r p p n 6. r r r c r c p n 7. r p c n 8. c r p r r c c p p n 9. p c c n 10. c r r c c c r r p c p c c p n Support for Response(r, p) 1.0 Succession(r, p) Precedence(c, n) 0.9 The support can be used to prune out those constraints falling under a given threshold E.g., 0.95 A threshold equal to 1.0 selects those constraints which are always valid on the log Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 29

30 Rela=on constraint templates subsump=on Constraint templates are not independent of each other E.g., A trace like a b a b c a b c c sa+sfies (w.r.t. b and a): RespondedExistence(a, b), RespondedExistence(b, a), CoExistence(a, b), CoExistence(b, a), Response(a, b), AlternateResponse(a, b), ChainResponse(a, b), Precedence(a, b), AlternatePrecedence(a, b), ChainPrecedence (a, b), Succession(a, b), AlternateSuccession(a, b), ChainSuccession(a, b) The mining algorithm would have to show the most strict constraint only (ChainSuccession(a, b)) MINERful ++ faces this issue, by pruning the returned constraints on the basis of the subsump=on hierarchy of constraints Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 30

31 Rela=on constraint templates subsump=on Constraint templates are not independent of each other Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 31

32 Rela=on constraint templates subsump=on A hint on the pruning procedure Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 32

33 Rela=on constraint templates subsump=on A hint on the pruning procedure S u p p o r t ?? ?? Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 33

34 Evalua=on Characteristics of MINERful ++ MINERful ++ is Independent on the formalism used for expressing constraints Modular (two- phase) Capable of elimina+ng redundancy in the process model Fast Source Tasks Traces Events processed Total comp. time Engine Synth. log, Setup 1 (avg. ) 00:00: MINERful 00:11: Declare Miner 2011 Synth. log, Setup 2 (avg. ) 00:00: MINERful 00:21: Declare Miner 2011 Financial log (avg. ) 00:00: MINERful [BPIC12] Hospital log [BPIC11] 00:08: Declare Miner 2011 (avg. ) 00:04: MINERful 03:39: Declare Miner 2011 Sony VAIO VGN- FE11H Intel Core Duo T GHz 2 MB L2 cache 2 GB of DDR2 667 Mhz Setup Min. Max. Number Alphabet Total length length of traces size tests Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 34

35 Evalua=on On the complexity of MINERful Linear w.r.t. the number of traces in the log T Quadratic w.r.t. the size of traces in the log t max Quadratic w.r.t. the size of the alphabet Σ T Hence, polynomial in the size of the input O( T t max 2 Σ T 2 ) Total execution time [msec] Number of traces Total execution time [msec] Execution time [msec] Average string length Processed events Alphabet size Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 35

36 Preliminary results MINERful ++ on real data Logs extracted from 2 mailboxes [DiCiccioMecella2012] 5 traces events each on average 139 events read in total Evalua+on of inferred constraints conducted with an expert domain Precision % [ 14 ] % [ 45 ] 0/ % [ 7 ] % [ 173 ] 25 Result evaluation Noticeably right Right Utterly wrong Wrong Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 36

37 Future work Research in progress Integrate MINERful ++ with ProM MINERful ++ is already capable of reading/wri+ng XES logs Study the effects of errors in logs on the inferred workflow Error- injected synthe+c logs can help us conduct an automated analysis on the quality of results Refine the es+ma+on calculi for support Auto- tune the support threshold Depending on the constraint template, which can be more or less robust Enlarge the set of discovered constraint templates to the branching Declare constraints Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 37

38 References Cited articles and resources, in order of appearance [Aalst2011.book] van der Aalst, W.M.P.: : Discovery, Conformance and Enhancement of Business Processes. Springer (2011). [AalstEtAl2009] van der Aalst, W.M.P., van Dongen, B.F., Güther, C.W., Rozinat, A., Verbeek, E., Weijters, T.: Prom: The process mining toolkit. In de Medeiros, A.K.A., Weber, B., eds.: BPM (Demos). Volume 489 of CEUR Workshop Proceedings., CEUR-WS.org (2009) [AalstEtAl2004] van der Aalst, W.M.P., Weijters, T., Maruster, L.: Workflow mining: Discovering process models from event logs. IEEE Trans. Knowl. Data Eng. 16(9) (2004) [WenEtAl2007] Wen, L., van der Aalst, W.M.P., Wang, J., Sun, J.: Mining process models with non-freechoice constructs. Data Min. Knowl. Discov. 15(2) (2007) [WenEtAl2010] Wen, L., Wang, J., van der Aalst, W. M. P., Huang, B., Sun, J.: Mining process models with prime invisible tasks. Data Knowl. Eng., 69 (2010), [GüntherEtAl2007] Günther, C.W., van der Aalst, W.M.P.: Fuzzy Mining - Adaptive Process Simplification Based on Multi-perspective Metrics. Proc. BPM [WeijtersEtAl2001] Weijters, A., van der Aalst, W.: Rediscovering workflow models from event-based data using little thumb. Integrated Computer-Aided Engineering 10 (2001) [MedeirosEtAl2007] Medeiros, A.K., Weijters, A.J., Aalst, W.M.: Genetic process mining: an experimental evaluation. Data Min. Knowl. Discov. 14(2) (2007) [AalstEtAl2010] van der Aalst, W., Rubin, V., Verbeek, H., van Dongen, B., Kindler, E., Gnther, C.: Process mining: a two-step approach to balance between underfitting and overfitting. Software and Systems Modeling 9 (2010) Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 38

39 References Cited articles and resources, in order of appearance [HillEtAl06] Hill, C., Yates, R., Jones, C., Kogan, S.L.: Beyond predictable workflows: Enhancing productivity in artful business processes. IBM Systems Journal 45(4), (2006) [ACTIVE09] Warren, P., Kings, N., et al.: Improving knowledge worker productivity - the ACTIVE integrated approach. BT Technology Journal 26(2), (2009) [DiCiccioEtAl11] Di Ciccio, C., Mecella, M., Catarci, T.: Representing and visualizing mined artful processes in MailOfMine. Proc. USAB 2011:83-94 [DiCiccioMecella12] Di Ciccio, C., Mecella,M.: Mining constraints for artful processes. Proc. BIS 2012 [DiCiccioMecella/TR12] Di Ciccio, C., Mecella, M.: MINERful, a mining algorithm for declarative process constraints in MailOfMine. Technical report, Dipartimento di Ingegneria Informatica, Automatica e Gestionale Antonio Ruberti SAPIENZA, Universita` di Roma (2012). [DiCiccioMecella13] Di Ciccio, C., Mecella, M.: A Two-Step Fast Algorithm for the Automated Discovery of Declarative Workflows. Proc. CIDM 2013 [AalstEtAl06] van der Aalst, W.M.P., Pesic, M.: Decserflow: Towards a truly declarative service flow language. Proc. WS-FM 2006 [MaggiEtAl11] Maggi, F.M., Mooij, A.J., van der Aalst, W.M.P.: User-guided discovery of declar- ative process models. Proc. CIDM 2011 Claudio Di Ciccio (DIAG, SAPIENZA Università di Roma) P. 39