Review & AI Lessons learned while using Artificial Intelligence April 2013

Size: px

Start display at page:

Download "www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence April 2013"

Berniece Stokes
8 years ago
Views:

1 Review & AI Lessons learned while using Artificial Intelligence

2 Why are non-users staying away from PC? source: edj Group s Q Predictive Coding Survey, February 2013, N = 66 Slide 2

3 Introduction Relative costs of producing electronic documents Collection 8% Internal 4% Processing 19% Vendors 26% Review 73% Outside counsel 70% Source: Where the money goes: understanding litigant expenditures for producing electronic discovery / Nicholas M. Pace, Laura Zakaras. Slide 3

Source: Where the money goes: understanding litigant expenditures for

4 Review methodologies Sophistication & efficiency of approach Boolean searching Pattern matching Clustering (un-supervised machine learning) Categorisation (supervised machine learning) Basic keyword search Data volumes Slide 4

(un-supervised machine learning) Categorisation

5 Review methodologies Clustering Slide 5

6 Judicially approved Da Silva Moore v. Publicis Groupe, et al. (February 2012) Don t worry about being the guinea pig Computer-assisted review now can be considered judicially-approved for use in appropriate cases. Work smarter Computer-assisted review appears to be better than the available alternatives, and thus should be used in appropriate cases. Go Faster Computer-assisted review should be seriously considered for use in large-data-volume cases where it may save the producing party (or both parties) significant amounts of legal fees in document review. Slide 6

7 Judicially approved Da Silva Moore v. Publicis Groupe, et al. (February 2012) Don t focus on the black box The idea is not to make [computer-assisted review] perfect, it s not going to be perfect. I may be less interested in the science behind the black box of the vendor s software than in whether it produced responsive documents. Proof of a valid process, including quality control testing, also will be important. Slide 7

8 Computer assisted review Categorisation example Slide 8

9 Computer assisted review Categorisation example Slide 9

10 Computer assisted review Categorisation example Slide 10

11 Computer assisted review Categorisation example Slide 11

12 Computer assisted review Components of assisted review systems Domain expert Analytics engine Statistical validation Slide 12

13 Workflow Document universe Validate results Train the computer Manually review categorised documents Categorise document universe Slide 13

14 Workflow How do we address the following concerns : How are the results? Training of the computer Do we need to continue training? Is the expert training the system well? Are the training docs representative? How many documents will we need for training? Review of documents Which documents should be submitted for review? How do we verify the results? Slide 14

Is the expert training the system well? Are the training docs representative?

15 Computer assisted review Categorisation example False Negative True Positive False Positive True Negative Slide 15

16 Workflow How are the results? Accuracy Recall Precision F-measure Slide 16

17 Workflow How are the results? Accuracy Recall Precision F-measure Review result The truth Relevant Not Relevant Total Categorisation result Relevant Not Relevant Total Slide 17

18 Workflow How are the results? Accuracy = 99% Recall = 0% Precision F-measure Review result The truth Relevant Not Relevant Total Categorisation result Relevant Not Relevant Total Slide 18

19 Workflow How are the results? Accuracy Recall Precision F-measure Review result The truth Relevant Not Relevant Total Categorisation result Relevant Not Relevant Total Slide 19

20 Workflow How are the results? Accuracy = 93% Recall = 20% Precision = 25% F-measure Review result The truth Relevant Not Relevant Total Categorisation result Relevant Not Relevant Total Slide 20

21 Workflow How are the results? Accuracy Recall Precision F-measure: Calculated as the harmonic mean of recall and precision F = 2(P*R)/(P+R) Slide 21

22 Workflow Document universe Validate results Train the computer Manually review categorised documents Categorise document universe Slide 22

23 Workflow How do we address the following concerns : How are the results? Training of the computer Do we need to continue training? Is the expert training the system well? Are the training docs representative? How many documents will we need for training? Review of documents Which documents should be submitted for review? How do we verify the results? Slide 23

24 Workflow Train the computer Do we need to continue training? Intuition? Objective training optimisation criterion Slide 24

25 Workflow Manually review categorised documents Which docs should be reviewed? Slide 25

26 Workflow Manually review categorised documents Which docs should be reviewed? Slide 26

27 Workflow Manually review categorised documents How do we verify the results? Quality assurance provides transparent verification of the generated results and is a key component of the computer assisted review process. Quality assurance: A random sample of the not review docs Size of sample based on level of statistical confidence Review of sample set by attorney Calculation of recall and precision within not review docs Attorney can confirm or modify cut off point Slide 27

28 Workflow Document universe Validate results Train the computer Manually review categorised documents Categorise document universe Slide 28

29 Real life examples Large Second Request Chose not to review 1.3m docs based on 95% confidence level It took only 4 rounds of human review to stabilize at a 95% confidence level Stats were also used to evaluate human review; categorization was found to be 4x s as accurate as the human review team Bankruptcy Case 2m docs categorized after review of 1,500 docs Slide 29

30 Real life examples Civil Litigation 200k docs reviewed and produced in 2 days Corruption Investigation Used on a subset of ~53k docs 97% of the docs were coded after review of the first sample, which was ~3% of the population After the first sample, humans agreed with categorization 87% of the time 91% of the overturned documents were exact dupes or 90% similar Slide 30

31 Creating relationships that create value All rights reserved. Not for further distribution without the permission of. "" refers to the network of member firms of PricewaterhouseCoopers International Limited (IL), or, as the context requires, individual member firms of the network. Each member firm is a separate legal entity and does not act as agent of IL or any other member firm. IL does not provide any services to clients. IL is not responsible or liable for the acts or omissions of any of its member firms nor can it control the exercise of their professional judgment or bind them in any way. No member firm is responsible or liable for the acts or omissions of any other member firm nor can it control the exercise of another member firm's professional judgment or bind another member firm or IL in any way.

ESI and Predictive Coding

Beijing Boston Brussels Chicago Frankfurt Hong Kong ESI and Predictive Coding Houston London Los Angeles Moscow Munich New York Palo Alto Paris São Paulo Charles W. Schwartz Chris Wycliff December 13,