Chapter 4 Getting the Data

Transcription

1 Chapter 4 Getting the Data prof.dr.ir. Wil van der Aalst

2 Overview Chapter 1 Introduction Part I: Preliminaries Chapter 2 Process Modeling and Analysis Chapter 3 Data Mining Part II: From Event Logs to Process Models Chapter 4 Getting the Data Chapter 5 Process Discovery: An Introduction Chapter 6 Advanced Process Discovery Techniques Part III: Beyond Process Discovery Chapter 7 Conformance Checking Chapter 8 Mining Additional Perspectives Chapter 9 Operational Support Part IV: Putting Process Mining to Work Chapter 10 Tool Support Chapter 11 Analyzing Lasagna Processes Chapter 12 Analyzing Spaghetti Processes Part V: Reflection Chapter 13 Cartography and Navigation Chapter 14 Epilogue PAGE 1

3 Goal of process mining What really happened in the past? Why did it happen? What is likely to happen in the future? When and why do organizations and people deviate? How to control a process better? How to redesign a process to improve its performance? PAGE 2

4 Getting the data world business processes people machines components organizations models analyzes supports/ controls specifies configures implements analyzes software system records events, e.g., messages, transactions, etc. (process) model discovery conformance enhancement event logs PAGE 3

5 From heterogeneous data sources to process mining results Extract, Transform, and Load (ELT) optional data source ELT data source data source ELT ELT data warehouse coarse-grained scoping data source data source extract XES, MXML, or similar unfiltered event logs process mining discovery conformance enhancement filter filtered event logs (process) models answers fine-grained scoping PAGE 4

6 Example log A process consists of cases. A case consists of events such that each event relates to precisely one case. Events within a case are ordered. Events can have attributes. Examples of typical attribute names are activity, time, costs, and resource. PAGE 5

7 Another view process cases events attributes activity= register request time = :11.02 resource = Pete costs = activity= reject request time = :14.24 resource = Pete costs = activity= register request time = :11.32 resource = Mike costs = activity= pay compensation time = :12.05 resource = Ellen costs = activity= register request time = :11.32 resource = Pete costs = activity= pay compensation time = :10.45 resource = Ellen costs = PAGE 6

8 Standard transactional life-cycle model schedule assign reassign suspend start resume autoskip manualskip complete withdraw abort_activity abort_case successful termination unsuccessful termination PAGE 7

9 Five activity instances a: schedule assign start complete d: start complete b: c: schedule assign reassign start suspend resume complete start suspend resume suspend abort_activity e: complete schedule assign reassign suspend start resume autoskip manualskip complete withdraw abort_activity abort_case successful termination unsuccessful termination PAGE 8

10 Overlapping activity instances a: start start complete complete start start complete complete a: 6 hours 2 hours 5 hours 9 hours schedule assign reassign suspend start resume autoskip manualskip complete withdraw abort_activity abort_case successful termination unsuccessful termination PAGE 9

11 Using attributes social network showing how work flows from one person to another Pete Sara Sue Mike performance indicators per activity Ellen Sean Activity b Frequency: 456 Waiting time: /- 2.5 hours Service time: 1.2 +/- 0.5 hours Costs: 412 +/- 55 euros start A a register request control flow c1 c2 E b examine thoroughly A c examine casually A d check ticket c3 c4 role M e decide f M c5 reinitiate request A g pay compensation A h reject request Activity g Frequency: 311 Waiting time: /- 2.1 hours Service time: 0.5 +/- 0.2 hours Costs: 198 +/- 35 euros end Activity h Frequency: 407 Waiting time: 7.4 +/- 1.8 hours Service time: 1.1 +/- 0.3 hours Costs: 209 +/- 38 euros PAGE 10

12 XES (extensible Event Stream) See Adopted by the IEEE Task Force on Process Mining. Predecessor: MXML and SA-MXML. The format is supported by tools such as ProM (as of version 6), Nitro, XESame, and OpenXES. ProMimport supports MXML. PAGE 11

13 PAGE 12

14 XES (compatible with MXML) Event log consists of: traces (process instances) events Standard extensions: concept (for naming) lifecycle (for transactional properties) org (for the organizational perspective) time (for timestamps) semantic (for ontology references) PAGE 13

15 extensions loaded every trace has a name every event has a name and a transition start of trace (i.e. process instance) classifier = name + transition name of trace resource timestamp transition name of event (activity name) PAGE 14

16 end of trace (i.e. process instance) start of trace name of trace resource timestamp name of event (activity name) data associated to event PAGE 15

17 Challenges when extracting event logs Correlation: Events in an event log are grouped per case. This simple requirement can be quite challenging as it requires event correlation, i.e., events need to be related to each other. Timestamps: Events need to be ordered per case. Typical problems: only dates, different clocks, delayed logging. Snapshots. Cases may have a lifetime extending beyond the recorded period, e.g., a case was started before the beginning of the event log. Scoping. How to decide which tables to incorporate? Granularity: the events in the event log are at a different level of granularity than the activities relevant for end users. PAGE 16

18 Flattening reality into event logs Orderline Order OrderID : OrderID 1 1..* OrderLineID : OrderLineID OrderID : OrderID Customer : CustID Product : ProdID Amount : Euro NofItems : PosInt Created : DateTime TotalWeight : Weight Paid : DateTime Entered : DateTime Completed : DateTime BackOrdered : DateTime Secured : DateTime DelID : DelID 1..* 0..1 Attempt DelID : DelID 0..* 1 Delivery DelID : DelID When : DateTime DelAddress : Address Successful : Bool Contact : PhoneNo PAGE 17

19 Tables Orderline Order OrderID : OrderID 1 1..* OrderLineID : OrderLineID OrderID : OrderID Customer : CustID Product : ProdID Amount : Euro NofItems : PosInt Created : DateTime TotalWeight : Weight Paid : DateTime Entered : DateTime Completed : DateTime BackOrdered : DateTime Secured : DateTime DelID : DelID 1..* 0..1 Attempt DelID : DelID When : DateTime Successful : Bool 0..* 1 Delivery DelID : DelID DelAddress : Address Contact : PhoneNo PAGE 18

20 Order:91245 Order instance Activity: create order Timestamp: :08.12 Customer: John Amount: 100 Activity: pay order Timestamp: :13.45 Customer: John Amount: 100 Activity: complete order Timestamp: :11.33 Customer: John Amount: 100 OrderLine: Order OrderID : OrderID 1 1..* Orderline OrderLineID : OrderLineID OrderID : OrderID Activity: enter order line Timestamp: :08.13 OrderLineID: Product: iphone 4G NofItems: 1 TotalWeight: Activity: secure order line Timestamp: :08.55 OrderLineID: Product: iphone 4G NofItems: 1 TotalWeight: Customer : CustID Product : ProdID OrderLine: Amount : Euro Created : DateTime Paid : DateTime Completed : DateTime NofItems : PosInt TotalWeight : Weight Entered : DateTime BackOrdered : DateTime Activity: enter order line Timestamp: :08.14 OrderLineID: Product: ipod nano NofItems: 2 TotalWeight: DellID: Activity: create backorder Timestamp: :08.55 OrderLineID: Product: ipod nano NofItems: 2 TotalWeight: DellID: Activity: secure order line Timestamp: :09.06 OrderLineID: Product: ipod nano NofItems: 2 TotalWeight: DellID: Secured : DateTime DelID : DelID 1..* 0..1 OrderLine: Activity: enter order line Timestamp: :08.15 OrderLineID: Product: ipod classic NofItems: 1 TotalWeight: Activity: secure order line Timestamp: :10.06 OrderLineID: Product: ipod classic NofItems: 1 TotalWeight: Attempt DelID : DelID 0..* 1 Delivery DelID : DelID Delivery: When : DateTime DelAddress : Address Successful : Bool Contact : PhoneNo Attempt: Attempt: Attempt: Activity: delivery attempt Timestamp: :08.55 Successful: false DelAddress: 5513VJ-22a Contact: Activity: delivery attempt Timestamp: :09.12 Successful: false DelAddress: 5513VJ-22a Contact: Activity: delivery attempt Timestamp: :08.56 Successful: true DelAddress: 5513VJ-22a Contact: Delivery: Attempt: Activity: delivery attempt Timestamp: :08.43 DellID: Successful: true DelAddress: 5513XG-45 Contact: PAGE 19

21 Order:91245 Activity: create order Timestamp: :08.12 Customer: John Amount: 100 Activity: pay order Timestamp: :13.45 Customer: John Amount: 100 Activity: complete order Timestamp: :11.33 Customer: John Amount: 100 OrderLine: Activity: enter order line Timestamp: :08.13 OrderLineID: Product: iphone 4G NofItems: 1 TotalWeight: Activity: secure order line Timestamp: :08.55 OrderLineID: Product: iphone 4G NofItems: 1 TotalWeight: OrderLine: Activity: enter order line Timestamp: :08.14 OrderLineID: Product: ipod nano NofItems: 2 TotalWeight: DellID: Activity: create backorder Timestamp: :08.55 OrderLineID: Product: ipod nano NofItems: 2 TotalWeight: DellID: Activity: secure order line Timestamp: :09.06 OrderLineID: Product: ipod nano NofItems: 2 TotalWeight: DellID: OrderLine: Activity: enter order line Timestamp: :08.15 OrderLineID: Product: ipod classic NofItems: 1 TotalWeight: Activity: secure order line Timestamp: :10.06 OrderLineID: Product: ipod classic NofItems: 1 TotalWeight: Delivery: Attempt: Attempt: Attempt: Activity: delivery attempt Timestamp: :08.55 Successful: false DelAddress: 5513VJ-22a Contact: Activity: delivery attempt Timestamp: :09.12 Successful: false DelAddress: 5513VJ-22a Contact: Activity: delivery attempt Timestamp: :08.56 Successful: true DelAddress: 5513VJ-22a Contact: Delivery: Attempt: Activity: delivery attempt Timestamp: :08.43 DellID: Successful: true DelAddress: 5513XG-45 Contact: PAGE 20

22 Orderline Orderline instance Order OrderID : OrderID Customer : CustID Amount : Euro 1 1..* OrderLineID : OrderLineID OrderID : OrderID Product : ProdID NofItems : PosInt Created : DateTime TotalWeight : Weight Paid : DateTime Entered : DateTime Completed : DateTime BackOrdered : DateTime Secured : DateTime DelID : DelID 1..* OrderLine: Case id: Activity: enter order line Timestamp: :08.13 OrderLineID: Product: iphone 4G NofItems: 1 TotalWeight: Case id: Activity: secure order line Timestamp: :08.55 OrderLineID: Product: iphone 4G NofItems: 1 TotalWeight: Attempt DelID : DelID When : DateTime Successful : Bool 0..* Delivery DelID : DelID DelAddress : Address Contact : PhoneNo Order:91245 Case id: Activity: create order Timestamp: :08.12 Customer: John Amount: 100 Case id: Activity: pay order Timestamp: :13.45 Customer: John Amount: 100 Case id: Activity: complete order Timestamp: :11.33 Customer: John Amount: 100 Delivery: Attempt: Attempt: Attempt: Case id: Activity: delivery attempt Timestamp: :08.55 Successful: false DelAddress: 5513VJ-22a Contact: Case id: Activity: delivery attempt Timestamp: :09.12 Successful: false DelAddress: 5513VJ-22a Contact: Case id: Activity: delivery attempt Timestamp: :08.56 Successful: true DelAddress: 5513VJ-22a Contact: PAGE 21

23 Other examples The life cycles of reviewers, authors, papers, reviews, PC chairs, etc. The life cycles of job applications and vacancies. X-ray machine logs: machine, machine day, patient, treatment, routine, etc.? Therefore, the selection and scoping of instances is needed. Like making deciding on the elements to be put on map; there may be many maps covering partially overlapping areas. PAGE 22

24 Extracting event logs Not just a syntactical issue. Different views are possible. Important: Selecting the right instance notion. Ordering of events. Selection of events. Proclets: the true fabric of real-life processes. PAGE 23