New Developments in the Automatic Classification of Records. Inge Alberts, André Vellino, Craig Eby, Yves Marleau

Transcription

1 New Developments in the Automatic Classification of Records Inge Alberts, André Vellino, Craig Eby, Yves Marleau ARMA Canada 2014

2 INTRODUCTION

3 OUTLINE 1. Research team 2. Research context / Problem statement 3. Overview of records auto-classification 4. Project objectives 5. Research methodology 6. Qualitative analysis 7. Automatic classification 8. Future work 3

4 RESEARCH TEAM CISRI Center of excellence and a catalyst for collaborative, interdisciplinary research in information science ÉSIS, University of Ottawa Information Studies program grounded in theory, supported by practical work experience, and integrally connected to the trends of the leading knowledge centres in the National Capital Region and beyond 4

5 RESEARCH TEAM Inge Alberts ÉSIS / CISRI André Vellino ÉSIS / Institute for Science, Society and Policy (ISSP) Craig Eby CISRI / Cogniva Yves Marleau CISRI / Cogniva 5

6 BUSINESS PROBLEMS management has always been a problem Terabytes of data in mailboxes and PST files Integration of and ECM systems What is the role of the user? 6

7 GOC EXAMPLE Three new initiatives make the problem even more complex: Transformation Directive on Recordkeeping Open Government 7

8 IMPORTANCE OF BUSINESS CONTEXT Need to identify information of business value Need to better define the concept of business value Need to situate information within its context of use 8

9 DEFINING BUSINESS CONTEXT 9

10 RESEARCH PROJECT File Plan Business Function Function Function Sub- Function Sub- Function Sub- Function Sub- Function 10

11 ISIS Methodology produces models of an organization s business context ISIS Enterprise Software Solution helps organizations implement business context centric classification Classification automation Automated business rules Centralized Taxonomy and Rules Management Goal is to further reduce the requirements on users 11

12 OVERVIEW OF RECORDS AUTO- CLASSIFICATION Current state: Reaching similar classification quality as human users Mix of statistical and rule-based implementations Challenges Challenging to implement Confidence in and acceptance of results Lacking a structured approach to systematically ensure quality of system and interpret results 12

13 RELATED RESEARCH Analysis of - Ph.D. on automatic classification (Inge) - Extrusion protection at Entrust (André) - Business value pilot (André & Inge) Business Modeling - ISIS Methodology (Cogniva ) - Collaborations with LAC (Cogniva ) - Research on faceted classification (CISRI & University of Montreal) Process Discovery & Auto Classification - Auto-classification Research (Cogniva) - IRAP Research (Cogniva & CISRI) - ISIS Software (Cogniva) 13

14 RESEARCH OBJECTIVES 1. Understand how the concept of business value applies to the management of records 2. Develop a model of information experts strategies while appraising value in a work context 3. Propose a set of requirements to automatically classify organizational records 4. Test these requirements on a corpus of s 14

15 RESEARCH METHODOLOGY Phase 1: User Study Qualitative analysis of 8 information experts appraisal strategies Phase 2: Automatic Classification Quantitative analysis of ~900 messages 15

16 Research Focus Phase 1 Criteria to identify the business value of Decision process when appraising the value of Lexical and nonlexical features used when appraising the business value of Requirements needed to automatically classify organizational 16

17 Methodology Phase 1 Semi-structured Interviews Cognitive Inquiries 8 experts ~1h 14 questions on business value & 7 experts ~30 minutes classification exercise sample (n=174) Model Development Manual Classification BV factors feature analysis Classification model of appraisal strategies 2 inboxes (n=1975) 1 corpus classified (n=~800) 17

18 PROFILE OF PARTICIPANTS (N=8) 18

19 RESULTS FROM PHASE Business Value Factors 2. Features Analysis 3. Classification Model of Appraisal Strategies 19

20 BUSINESS VALUE Information resources of business value: Are published and unpublished materials, regardless of medium or form Created or acquired because they enable and document decision-making in support of programs, services and ongoing operations Support departmental reporting, performance and accountability requirements (Directive on Recordkeeping) 20

21 BUSINESS VALUE Process Context Operational = Performance Support actions & decisions Enhance performance Mitigate risks Evidential = Accountability Evidence of transaction Report on results ATIP, litigation Time 21

22 BUSINESS VALUE FACTORS Origin Action Chronology Meaning 22

23 ORIGIN origin is internal (team members, supervisors) or external (clients, professional network) Origin is the main factor affecting the appraisal of business value Appraisal decisions related to origin are based on: Name of the sender Position and organization of the sender Hierarchical relation between the sender & the recipient Active project involving the sender & the recipient 23

24 ACTION action is passive (no engagement from the recipient) or performative (engagement & accountability from the recipient) Action is an important factor affecting the appraisal of business value, both operational & evidential Appraisal decisions related to action are based on: Type of action Level of engagement & accountability of the recipient (high risk or low risk) 24

25 CHRONOLOGY Chronology is operational (during project) or postmortem (after project) For IM consultants, keeping track of action history is a determinant factor to appraise business value, specially for active projects Challenging factor for defining business value Appraisal decisions related to chronology are based on: Project status: active or closed 25

26 MEANING meaning is explicit (rich vocabulary) or latent (based on context) Many solutions available on the market classify based on explicit meaning but appraisal decisions are often based on latent meaning Appraisal decisions related to meaning are based on: EXPLICIT: Keywords, Attachment, Thread, Type of Action IMPLICIT: Origin, Chronology, Level of Engagement 26

27 ANALYSIS OF FEATURES Lexical Features Name & organization of the sender Action verbs: approval, confirmation, request, reminder, negotiation Action objects: SOW, meeting, status report, deadlines, deliverables, decision, reference material Presence of RE or FW in the title Name of attachment Nonlexical Features Message sent or received Hierarchical relation between the sender & the recipient Position of the recipient (TO, CC) Number of recipients (TO, CC) Project status: active or closed Presence of attachment Presence of a thread Presence of high priority symbol 27

28 HUMAN CLASSIFICATION MODEL (1/2) 28

29 HUMAN CLASSIFICATION MODEL (2/2) 29

30 MANUAL CLASSIFICATION CHALLENGES (1/2) More BV=NO than BV=Yes Bilingual messages Two recipients in the TO fields Threads: a message of business value is quickly superseded by a more recent one Sender is accountable for internal messages sent but for both sent and received when external 30

31 MANUAL CLASSIFICATION CHALLENGES (2/2) s of business value and attachments of business value have to be differentiated Perception of value is different between the individuals and the organization Evaluating the importance of some decisions or main revisions of drafts can be challenging Some s of business value during active projects are ephemeral = operational versus evidential 31

32 Objectives Phase 2 Attempt the automation of binary classifications of Business Value No Business Value Compare the human labeling process with the machine learning 32

33 Methodology Phase 2 Corpus Creation Manual Classificatio n Feature Extraction Machine Training Cross- Validation Model Testing 33

34 A Machine Learning toolkit for non-experts For experimenting with text mining technology Based on Weka Data Mining Open Source software Developed by smart Ph.D. students at Carnegie Mellon University Offers Feature extraction Model building Automated analysis and labeling Prediction 34

35 CORPUS FOR TRAINING 2 individual collections (inbox + outbox) 250 s Business Value s No Business Value 172 s Business Value No Business Value Features extracted: Originator and Recipients (To / From / Cc) Content of Subject / Body / Attachments Number of recipients in To and Cc fields Number of attachments Importance flag Forwarded indications Part of Thread indications 35

36 FROM AND TO FIELDS "From sender supervisor colleague client To solerecipient "soleorganizationalrecipient "supervisorisrecipient "oneamongmanyrecipient clientisrecipient" in every other case 36

37 GENERAL RESULTS Business Value SVM compare w/ Spam SVM Accuracy 0.91 Accuracy 0.96 Kappa 0.83 Kappa 0.93 Support Vector Machines (SVMs) are highly accurate predictors of Business Value / Not Business Value SVMs models are very specific to sender / recipient One model does not appear to suffice for organizational automatic classification of business value Training SVMs for greater accuracy makes it more difficult to explain the behaviour of the model 37

38 CONTRIBUTION OF ATTACHMENT CONTENT Without Attachment Content Analysis With Attachment Content Analysis 38

39 No business value Features Company Acronym Company Colleague Colleague Colleague 39

40 business value Features owner Client Acronym Client Client Acronym 40

41 FUTURE WORK Experiment w/ alternative classifiers besides SVMs Grow the corpus of BV and NBV from a wider variety of senders / recipients Add / subtract features Vary text analysis parameters Unigram / Bigram / Trigram PoS tagging Stop words Stemming Punctuation 41

42 ACKNOWLEDGMENTS Special thanks to: The participants for their time and their enthusiasm during this study The organization which granted us permission to analyze data The research assistants for their active contribution This project is supported by a research grant from the University of Ottawa 42

43 THANK YOU 43

44 ACCURACY True-Pos. (229) + True-Neg. (228) (True + False) Pos. + (True + False) Neg (500) = 457 / 500 =

45 Cohen s Kappa The degree to which the machine classifier and the human classifiers agree. κ= Pr(M) Pr(R) / 1-Pr(R) Pr(M) = ( ) / 500 = 0.91 Pr(R) = probability of random agreement = 0.5 κ=