# A Practitioner s Guide to Statistical Sampling in E-Discovery. October 16, 2012

Save this PDF as:

Size: px
Start display at page:

Download "A Practitioner s Guide to Statistical Sampling in E-Discovery. October 16, 2012"

## Transcription

1 A Practitioner s Guide to Statistical Sampling in E-Discovery October 16,

2 Meet the Panelists Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Gordon V. Cormack, Professor at the David R. Cheriton School of Computer Science at the University of Waterloo Jim Wagner, Co-founder and CEO of DiscoverReady Maureen O Neill, SVP, Marketplace Leader at DiscoverReady 2

3 Agenda What is statistical sampling? Why should a practitioner use statistical sampling in e-discovery? Opportunities to use statistical sampling The basics of statistical sampling An example of statistical sampling in the e-discovery context Key decisions when using statistical sampling Take-away recommendations on using statistical sampling 3

4 What is Statistical Sampling? In general, statistical sampling is a method to estimate a characteristic of a large population by examining only a subset of it. In the specific context of e-discovery: Estimate is a reasonably precise mathematical measurement. Characteristic refers to the number (or proportion) of items in the document population having a certain property, such as responsiveness or privilege. Population is a collection of electronic documents. Subset is a small but representative sample of the document population (chosen at random). Note that the size of the subset the sample size determines the precision of the estimate. Sample size is dependent on the acceptable margin of error, the desired confidence level, and to a negligible extent, the size of the population (these variables will be discussed later in the webinar). 4

5 What is Statistical Sampling? Judgmental sampling can be very useful in e-discovery, but it is not what we are discussing in today s webinar Statistical sampling is not the same thing as judgmental sampling Judgmental sampling is not random It involves a selection of items for the subset using some degree of human judgment When sampling is judgmental, inferences cannot be drawn about the population based on an examination of the subset Judgmental sampling is akin to spot-checking 5

6 Why Use Statistical Sampling? Thoughtful use of statistical sampling can improve the quality, efficiency, and defensibility of e-discovery efforts. Quality By counting or measuring the inputs and outputs of an e- discovery process, we can work to improve the process to make it more accurate. For example: If we find that a proposed search term is bringing in too many false positives (i.e., poor precision ) we can try a different search term and test the results using sampling. If statistical sampling confirms that the new term reduces the number of irrelevant documents (better precision), but is not under-inclusive (the recall is good), we have improved the search process. 6

7 Why Use Statistical Sampling? Efficiency Sampling saves time and money by allowing us to count or measure things more efficiently. Statistical sampling is scalable; even with extremely large populations, we can use relatively small samples. For example, if we want to estimate the proportion of relevant documents (the richness or prevalence ) in a collection of one million documents, an appropriate sample size might be 600 documents. If we want to estimate the proportion of relevant documents in a collection of a billion documents, the appropriate sample size would remain the same. 7

8 Why Use Statistical Sampling? Although its use in e-discovery is not yet widespread, we are moving in a direction where it soon will be considered best practice. Defensibility Courts are increasingly requiring the use of sampling as part of a reasonable discovery process. In re Seroquel Prods. Liabil. Litig., 244 F.R.D. 650 (M.D. Fla. 2007) Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251 (D. Md. 2008) William. A. Gross Constr. Assocs. Inc. v. American Mfrs. Mut. Ins. Co., 256 F.R.D. 134 (S.D.N.Y. 2009) Mt. Hawley Ins. Co. v. Felman Prod., Inc., 2010 WL (S.D. W. Va. May 18, 2010) DaSilva Moore v. Publicis Group, No. 11 Civ (S.D.N.Y. Feb. 24, 2012) (Peck, M.J.), aff d (S.D.N.Y. Apr. 26, 2012) (Carter, D.J.) In re: Actos, MDL No. 6:11-md-2299 (W.D. La. Jul. 27, 2012) 8

9 Opportunities to Use Statistical Sampling Incorporate into early case assessment and strategy development Efficiently hone in on the sources and custodians of information likely to be relevant Assess the burdens and costs involved in accessing certain information, such as backup tapes or other offline media Gauge the richness of populations before embarking on review (Later in the program, we will take a step-by-step walk through this use of sampling, as an example of how sampling is performed) Test the culling of a data set to ensure that your cull is neither over-broad nor too restrictive 9

10 Opportunities to Use Statistical Sampling Measure the efficacy of search terms and refine the terms Measure the accuracy of a predictive coding process Test automated methods of screening documents for privilege and confidentiality Sample a document production before it goes out the door to provide additional assurance that privileged content is not inadvertently included 10

11 Opportunities to Use Statistical Sampling Support proportionality arguments Determine whether the cost of reviewing certain types of ESI is reasonable and proportional Is there bang for the buck in reviewing a particular set of documents based on how many responsive documents are estimated to be found? For example: Should we spend money on reviewing this custodian s documents if the collection has very low prevalence? Is it worth the expense to continue reviewing more documents from more custodians if that additional effort is not likely to yield significantly more relevant information? 11

12 Opportunities to Use Statistical Sampling Conduct quality control and quality assurance of human review efforts Measure the error rate on document review decisions for an overall project, or for particular reviewers When done in real time, as the project progresses, error rate measurement can be part of an effective quality control ( QC ) workflow When done at the conclusion of the project, or a phase of the project, the measurement becomes part of the quality assurance ( QA ) and defensibility of process 12

13 Opportunities to Use Statistical Sampling Important caveat about taking statistical measurements using human decisions as a reference point or gold standard Human decisions about documents inevitably involve an element of subjectivity, and even the best decision-makers will make mistakes. (This is true for all decisions, whether in the original review, a QC review, or in a sampling review.) Even the gold standard decisions are not going to be 100% consistent or correct. This element of human error, or legitimate differences of opinion, will always introduce some degree of measurement error into a statistical measurement involving human decisionmaking. Therefore, statistical measurement cannot be more accurate than human judgment permits. 13

14 The Basics of Statistical Sampling: Drawing a Random Sample What is a random sample and how is it generated? A random sample is a subset of documents that is chosen at random from a larger population of interest. Choosing at random means that every document in the collection has an equal chance of being selected in the sample. Random sampling can be achieved a number of different ways: Drawing numbers from a hat Using a computerized random-number generator Choosing documents based on one or more digits from their hash value Many e-discovery tools have built-in random sample generators 14

15 The Basics of Statistical Sampling: Drawing a Random Sample A random sample has been drawn now what? Review the sample and count the number of documents with the characteristic of interest (e.g., responsiveness). The proportion of responsive documents in the sample is calculated by dividing the number of responsive documents in the sample by the total number of documents in the sample. Because the sample is random, we can extrapolate that the proportion of responsive documents in the population is approximately the same as the proportion in the sample. When we say approximately the same, we mean that there is a margin of error in our estimate we ll explain margin of error in a moment. 15

16 The Basics of Statistical Sampling: Drawing a Random Sample Suppose we have a collection of one million documents And we want to estimate how many of them are relevant for the purposes of building a budget and timeline for review and production. Reviewing all one million documents for the purposes of budgeting and project planning is infeasible. So, instead, we take a random sample of 1,000 documents. We review the sample and find that 300 documents are relevant. The proportion of relevant documents in the sample is 300/1,000, or 30%. Therefore, the proportion of relevant documents in the collection is estimated to be approximately 30% (or 300,000 documents). If we repeated this process, we would get a slightly different estimate each time, but in general, each estimate would be close to the actual proportion. 16

17 The Basics of Statistical Sampling: Margin of Error / Confidence Interval The margin of error is a way of expressing a range above and below the estimate that is likely to contain the actual value. In our example: We determined that the proportion of relevant documents in the sample was 30%, and we extrapolated that the proportion of relevant documents in the collection was approximately, but not exactly, 30%. The exact proportion of relevant documents in the collection is unknown, but is likely to fall within a margin of error of +/- 3%. We express this by saying that that the proportion of relevant documents in the collection is estimated to be 30%, plus or minus 3%. 17

18 The Basics of Statistical Sampling: Margin of Error / Confidence Interval An alternative way of stating the estimate is by using a confidence interval instead of margin of error. The confidence interval is the range of values that is likely to contain the actual value. In our previous example, we would state that the proportion of relevant documents is likely to fall within a range of 27% to 33%. As compared to the margin of error, the confidence interval does not have to be exactly symmetrical around the estimate, and can therefore be a more precise way of expressing the uncertainty of the estimate. 18

19 The Basics of Statistical Sampling: Confidence Level What do we mean when we say the confidence interval is likely to contain the actual value? The confidence level is the probability that the confidence interval would contain the actual value if the sampling process were to be repeated a large number of times. For example: If the confidence level is 95%, it means that there is a 95% chance that the actual value is within the confidence interval. (In our previous example, we would say with a 95% confidence, that the proportion of relevant documents falls between 27% and 33%. ) If the confidence level is 99%, it means that there is a 99% chance that the actual value is within the confidence interval. (In our previous example, we would say with a 99% confidence, that the proportion of relevant documents falls between 26% and 34%. ) 19

20 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size The three concepts of confidence level, confidence interval (or margin of error), and sample size are interrelated. Generally speaking, if the confidence level remains constant, as the sample size goes up, the margin of error goes down. Similarly, to increase the confidence level, either the sample size or the margin of error must increase. Finally, if you want to decrease your margin of error, either the confidence level must come down or the sample size must go up. Bear in mind that these relationships are not proportional: Getting a smaller confidence interval, or a higher confidence level, may require drawing a much larger sample. To illustrate this, consider again the previous example, and assume that we find the proportion of relevant documents is 30% in each sample. The following slide depicts the relationship between sample size, confidence level, and margin of error: 20

21 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size Margin of Error Confidence Level Sample Size % 99% 4, % 99% 1, % 99% 500 0% 30% 100% Proportion of Relevant documents 21

22 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size Here s another illustration that also demonstrates how taking relatively small samples can help us understand very large populations. We have a collection of one million documents; we assume the proportion of relevant documents is 50% and we want a 95% confidence level for our estimate. Here are some examples of the (pretty good) margins of error we will achieve with some relatively small sample sizes: Sample Size Margin of Error % % % 22

23 The Basics of Statistical Sampling: The Tradeoffs In general, a higher confidence level is better. But that comes at the price of a larger sample size, and/or A wider confidence interval (i.e., a higher margin of error). Likewise, a smaller margin of error generally is better, because it reflects a more precise estimate. But that requires a lower confidence level (less certainty), and/or A larger sample size (higher cost and/or less efficiency) There are calculators available on the Internet that allow you to plug in the variables and make computations of sample size, margin of error, and confidence level, but be careful when choosing one. Make sure you understand the assumptions each one uses or if you don t, get help from someone who does! 23

24 The Basics of Statistical Sampling: Standards for E-Discovery? Is there a minimum acceptable confidence level and/or margin of error when using statistical sampling in e-discovery? There are no bright line rules regarding confidence level or margin of error 95% confidence level is commonly used in statistical measurement The acceptable margin of error will depend on the consequences of an inaccurate estimate The operative standard is one of reasonableness 24

25 The Basics of Statistical Sampling: Standards for E-Discovery? Is there a minimum acceptable confidence level and/or margin of error when using statistical sampling in e-discovery? Every matter is different, and what is reasonable in one matter may not be in another Factors affecting the reasonableness calculus include: The cost of greater precision in measurement as compared to the amount at stake and the importance of the matter (proportionality) The purpose for which the sampling is being performed The time and resources available for sampling 25

26 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Determine the total number of documents in the collection. We ll call that number N (e.g., N = 1,000,000 documents). Choose the desired confidence level (e.g., 95%), and margin of error (e.g., +/- 2%). Determine the appropriate sample size (we ll call that n ) using an appropriate calculator (e.g., n = 2,395). 26

27 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Select 2,395 of the documents, at random, from the collection. 27

28 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents Not Responsive 1,695 documents Count the number of responsive documents in the sample (say, 700). 28

29 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents The proportion of responsive documents in the sample is calculated by dividing the number of responsive documents in the sample (700) by n (2,395): 700/2,395 = 29.2% 29

30 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents The confidence interval is 29.2% (the proportion of responsive documents) plus or minus 2% (the margin of error): 27.2% to 31.2% 30

31 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents To calculate the number of responsive documents in the collection, multiply the two confidence limits by N (27.2% x 1,000,000 = 272,000 responsive documents; 31.2% x 1,000,000 = 312,000 responsive documents). 31

32 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents In this example, we can state that, with 95% confidence, we estimate there are between 272,000 and 312,000 responsive documents in this collection. 32

33 Examples of Statistical Sampling: Example 2 How many responsive documents are likely to be found in the document collection? 1. Let s tweak Example 1 slightly, and assume that the number of responsive documents in the reviewed sample is only The proportion of responsive documents in this example is 3/2,395 = 0.13%, and with a confidence interval of -1.87% to +2.13% (-18,700 to 21,300 relevant documents). 3. Obviously it is not possible to have a negative number of relevant documents in the collection. This scenario illustrates the problem faced when sampling populations with a low proportion of relevant documents (i.e., low prevalence or low richness ). 4. When sampling such populations, a normal sample size calculation of confidence intervals and margin of error is not precise enough. When dealing with a low prevalence population, a more precise confidence interval can be computed using a binomial confidence interval calculator. 33

34 Examples of Statistical Sampling: Example 2 How many responsive documents are likely to be found in a collection of documents? 5. Here s an example of a binomial confidence interval calculator: 6. If we take our sample of 2,395 documents and re-compute the confidence intervals at the 95% confidence level using the binomial confidence interval calculator, if there are three responsive documents in the sample, the confidence interval is 0.03% to 0.37% (300 to 3,700 documents). 7. This is an example of how the confidence interval may not be symmetrical around the estimate. 34

35 Using Statistical Sampling: Key Decisions a Practitioner Must Make What are you trying to measure? What are trying to accomplish with the measurement? Are you sampling for internal purposes (to improve process) or to defend your process to opposing counsel or the court? Do you need to hire an expert in statistics? What are your acceptable levels of recall, precision, etc.? What is your desired confidence level? How wide a confidence interval will you accept? How large a sample size is feasible under the circumstances of your matter, taking into account cost and timing considerations? How will you document and report your measurements and methodology? What aspects of your statistical sampling efforts are you willing to share with the opposing party? With the court? 35

36 Take-Away Recommendations on Using Statistical Sampling Statistical sampling can be tricky most lawyers do not have sufficient understanding of the concepts (or the math) to go it alone and get it right. Practitioners should use an expert or statistical tool to: Determine whether statistical sampling is an appropriate tool for the matter Determine which statistical measures are best for the situation (precision, recall, elusion, accuracy, etc.) Determine the desired confidence level and margin of error Calculate the proper sample size to achieve this confidence level and margin of error Draw a random sample of the appropriate size Compute the estimate 36

37 Question & Answer Session To contact the panelists: Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Jim Wagner, Co-founder and CEO of DiscoverReady Gordon V. Cormack, Professor at the David R. Cheriton School of Computer Science at the University of Waterloo Maureen O Neill, SVP, Marketplace Leader at DiscoverReady 37

### E-Discovery in Mass Torts:

E-Discovery in Mass Torts: Predictive Coding Friend or Foe? Sherry A. Knutson Sidley Austin One S Dearborn St 32nd Fl Chicago, IL 60603 (312) 853-4710 sknutson@sidley.com Sherry A. Knutson is a partner

### The Evolution, Uses, and Case Studies of Technology Assisted Review

FEBRUARY 4 6, 2014 / THE HILTON NEW YORK The Evolution, Uses, and Case Studies of Technology Assisted Review One Size Does Not Fit All #LTNY Meet Our Panelists The Honorable Dave Waxse U.S. Magistrate

### E-discovery Taking Predictive Coding Out of the Black Box

E-discovery Taking Predictive Coding Out of the Black Box Joseph H. Looby Senior Managing Director FTI TECHNOLOGY IN CASES OF COMMERCIAL LITIGATION, the process of discovery can place a huge burden on

### Predictive Coding Defensibility

Predictive Coding Defensibility Who should read this paper The Veritas ediscovery Platform facilitates a quality control workflow that incorporates statistically sound sampling practices developed in conjunction

### The Benefits of. in E-Discovery. How Smart Sampling Can Help Attorneys Reduce Document Review Costs. A white paper from

The Benefits of Sampling in E-Discovery How Smart Sampling Can Help Attorneys Reduce Document Review Costs A white paper from 615.255.5343 dsi.co 414 Union Street, Suite 1210 Nashville, TN 37219-1771 Table

### Power-Up Your Privilege Review: Protecting Privileged Materials in Ediscovery

Power-Up Your Privilege Review: Protecting Privileged Materials in Ediscovery Jeff Schomig, WilmerHale Stuart Altman, Hogan Lovells Joe White, Kroll Ontrack Sheldon Noel, Kroll Ontrack (moderator) April

### SAMPLING: MAKING ELECTRONIC DISCOVERY MORE COST EFFECTIVE

SAMPLING: MAKING ELECTRONIC DISCOVERY MORE COST EFFECTIVE Milton Luoma Metropolitan State University 700 East Seventh Street St. Paul, Minnesota 55337 651 793-1246 (fax) 651 793-1481 Milt.Luoma@metrostate.edu

### Application of Simple Random Sampling 1 (SRS) in ediscovery

Manuscript submitted to the Organizing Committee of the Fourth DESI Workshop on Setting Standards for Electronically Stored Information in Discovery Proceedings on April 20, 2011. Updated May 18, 2011.

### Technology Assisted Review of Documents

Ashish Prasad, Esq. Noah Miller, Esq. Joshua C. Garbarino, Esq. October 27, 2014 Table of Contents Introduction... 3 What is TAR?... 3 TAR Workflows and Roles... 3 Predictive Coding Workflows... 4 Conclusion...

### Recent Developments in the Law & Technology Relating to Predictive Coding

Recent Developments in the Law & Technology Relating to Predictive Coding Presented by Paul Neale CEO Presented by Gene Klimov VP & Managing Director Presented by Gerard Britton Managing Director 2012

### MANAGING BIG DATA IN LITIGATION

David Han 2015 MANAGING BIG DATA IN LITIGATION DAVID HAN Associate, Morgan Lewis & Bockius, edata Practice Group MANAGING BIG DATA Data volumes always increasing New data sources Mobile Internet of Things

### Predictive Coding: A Primer

MEALEY S TM LITIGATION REPORT Discovery Predictive Coding: A Primer by Amy Jane Longo, Esq. and Usama Kahf, Esq. O Melveny & Myers LLP Los Angeles, California A commentary article reprinted from the March

### Technology Assisted Review: The Disclosure of Training Sets and Related Transparency Issues Whitney Street, Esq. 1

Technology Assisted Review: The Disclosure of Training Sets and Related Transparency Issues Whitney Street, Esq. 1 The potential cost savings and increase in accuracy afforded by technology assisted review

### Pr a c t i c a l Litigator s Br i e f Gu i d e t o Eva l u at i n g Ea r ly Ca s e

Ba k e Offs, De m o s & Kicking t h e Ti r e s: A Pr a c t i c a l Litigator s Br i e f Gu i d e t o Eva l u at i n g Ea r ly Ca s e Assessment So f t wa r e & Search & Review Tools Ronni D. Solomon, King

### The Truth About Predictive Coding: Getting Beyond The Hype

www.encase.com/ceic The Truth About Predictive Coding: Getting Beyond The Hype David R. Cohen Reed Smith LLP Records & E-Discovery Practice Group Leader David leads a group of more than 100 lawyers in

### Judge Peck Provides a Primer on Computer-Assisted Review By John Tredennick

By John Tredennick CEO Catalyst Repository Systems Magistrate Judge Andrew J. Peck issued a landmark decision in Da Silva Moore v. Publicis and MSL Group, filed on Feb. 24, 2012. This decision made headlines

### ESI and Predictive Coding

Beijing Boston Brussels Chicago Frankfurt Hong Kong ESI and Predictive Coding Houston London Los Angeles Moscow Munich New York Palo Alto Paris São Paulo Charles W. Schwartz Chris Wycliff December 13,

### November/December 2010 THE MAGAZINE OF THE AMERICAN INNS OF COURT. rofessionalism. Ethics Issues. and. Today s. Technology. www.innsofcourt.

November/December 2010 THE MAGAZINE OF THE AMERICAN INNS OF COURT rofessionalism and Ethics Issues in Today s Technology www.innsofcourt.org Transparency in E-Discovery: No Longer a Novel Approach By Michael

### Mastering Predictive Coding: The Ultimate Guide

Mastering Predictive Coding: The Ultimate Guide Key considerations and best practices to help you increase ediscovery efficiencies and save money with predictive coding 4.5 Validating the Results and Producing

### The United States Law Week

The United States Law Week Source: U.S. Law Week: News Archive > 2012 > 04/24/2012 > BNA Insights > Under Fire: A Closer Look at Technology- Assisted Document Review E-DISCOVERY Under Fire: A Closer Look

### Legal Arguments & Response Strategies for E-Discovery

Legal Arguments & Response Strategies for E-Discovery The tools to craft strategic discovery requests & mitigate the risks and burdens of production. Discussion Outline Part I Strategies for Requesting

### Quality Control for predictive coding in ediscovery. kpmg.com

Quality Control for predictive coding in ediscovery kpmg.com Advances in technology are changing the way organizations perform ediscovery. Most notably, predictive coding, or technology assisted review,

### The Predictive Coding Soundtrack: Rewind, Play, Fast-Forward

The Predictive Coding Soundtrack: Rewind, Play, Fast-Forward LEGALTECH NEW YORK February 3, 2015 Moderator: Amy Hinzmann Senior Vice President, DiscoverReady DiscoverReady 2014 THE PANELISTS* Marla Bergman

### An Open Look at Keyword Search vs. Predictive Analytics

877.557.4273 catalystsecure.com ARTICLE An Open Look at Keyword Search vs. Can Keyword Search Be As Effective as TAR? John Tredennick, Esq. Founder and CEO, Catalyst Repository Systems 2015 Catalyst Repository

### Measurement in ediscovery

Measurement in ediscovery A Technical White Paper Herbert Roitblat, Ph.D. CTO, Chief Scientist Measurement in ediscovery From an information-science perspective, ediscovery is about separating the responsive

### Predictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review

Predictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review ACEDS Webinar April 23, 2014 Sponsored by Robert Half Legal 1 2014 Robert Half Legal. An Equal Opportunity

### White Paper Technology Assisted Review. Allison Stanfield and Jeff Jarrett 25 February 2015. 1300 136 993 www.elaw.com.au

White Paper Technology Assisted Review Allison Stanfield and Jeff Jarrett 25 February 2015 1300 136 993 www.elaw.com.au Table of Contents 1. INTRODUCTION 3 2. KEYWORD SEARCHING 3 3. KEYWORD SEARCHES: THE

### Document Review Costs

Predictive Coding Gain Earlier Insight and Reduce Document Review Costs Tom Groom Vice President, Discovery Engineering tgroom@d4discovery.com 303.840.3601 D4 LLC Litigation support service provider since

### Predictive Coding Defensibility and the Transparent Predictive Coding Workflow

WHITE PAPER: PREDICTIVE CODING DEFENSIBILITY........................................ Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive

### Cost-Effective and Defensible Technology Assisted Review

WHITE PAPER: SYMANTEC TRANSPARENT PREDICTIVE CODING Symantec Transparent Predictive Coding Cost-Effective and Defensible Technology Assisted Review Who should read this paper Predictive coding is one of

### case 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION

case 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION IN RE: BIOMET M2a MAGNUM HIP IMPLANT PRODUCTS LIABILITY

### Predictive Coding Defensibility and the Transparent Predictive Coding Workflow

Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive coding is one of the most promising technologies to reduce the high cost of review by

### TECHNOLOGY-ASSISTED DOCUMENT REVIEW: IS IT DEFENSIBLE?

TECHNOLOGY-ASSISTED DOCUMENT REVIEW: IS IT DEFENSIBLE? By William W. Belt, Dennis R. Kiker and Daryl E. Shetterly* Cite as: William W. Belt, Dennis R. Kiker & Daryl E. Shetterly, Technology-Assisted Document

### Navigating Information Governance and ediscovery

Navigating Information Governance and ediscovery Implementing Processes & Technology to Reduce Downstream ediscovery Cost and Risk Shannon Smith General Counsel, Globanet March 11 12, 2013 Agenda 1 Overview

### COURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Fall 2014

COURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Professors:Mark Austrian Christopher Racich Fall 2014 Introduction The ubiquitous use of computers, the

### THE PREDICTIVE CODING CASES A CASE LAW REVIEW

THE PREDICTIVE CODING CASES A CASE LAW REVIEW WELCOME Thank you for joining Numerous diverse attendees Please feel free to submit questions Slides, recording and survey coming tomorrow SPEAKERS Matthew

### The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review

ediscovery & Information Management White Paper The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review Scott M. Cohen Elizabeth T. Timkovich John J. Rosenthal February 2014 2014 Winston

### How Good is Your Predictive Coding Poker Face?

How Good is Your Predictive Coding Poker Face? SESSION ID: LAW-W03 Moderator: Panelists: Matthew Nelson ediscovery Counsel Symantec Corporation Hon. Andrew J. Peck US Magistrate Judge Southern District

### The State Of Predictive Coding

MEALEY S TM LITIGATION REPORT Discovery The State Of Predictive Coding by Royce F. Cohen and Derek I.A. Silverman Stroock & Stroock & Lavan LLP New York A commentary article reprinted from the September

### Jason R. Baron Director of Litigation Office of General Counsel National Archives and Records Administration

NARA RACO 2010 Ronald Reagan Bldg., Washington, D.C. May 12, 2010 Jason R. Baron Director of Litigation Office of General Counsel National Archives and Records Administration Email is still the 800 lb.

### 2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500

2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500 5218 South East Street, Suite E-3, Indianapolis, IN 46227 Tel 317.247.4400 Fax 317.247.0044 Presented by Providing

### PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX?

Vol. 46 No. 3 February 6, 2013 PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX? The high costs of e-discovery have led to the development of computerized review technology by which the user may search

### Ethics and ediscovery

Ethics and ediscovery John Mansfield and Devon Newman January 6, 2012 1 2013, MansfieldLaw ediscovery basics We will cover: Preservation and spoliation Searching and producing documents Supervising lawyers

### The case for statistical sampling in e-discovery

Forensic The case for statistical sampling in e-discovery January 2012 kpmg.com 2 The case for statistical sampling in e-discovery The sheer volume and unrelenting production deadlines of today s electronic

### Litigation Solutions insightful interactive culling distributed ediscovery processing powering digital review

Litigation Solutions i n s i g h t f u l i n t e r a c t i ve c u l l i n g d i s t r i b u t e d e d i s cove r y p ro ce s s i n g p owe r i n g d i g i t a l re v i e w Advanced Analytical Review Data

### Comprehending the Challenges of Technology Assisted Document Review

Comprehending the Challenges of Technology Assisted Document Review Predictive Coding in Multi-Language E-Discovery 3 Lagoon Dr., Ste.180, Redwood UBIC North City, America, CA 94065 Inc. +1-650-654-7664

### Predictive Coding Helps Companies Reduce Discovery Costs

Predictive Coding Helps Companies Reduce Discovery Costs Recent Court Decisions Open Door to Wider Use by Businesses to Cut Costs in Document Discovery By John Tredennick As companies struggle to manage

### Predictive Coding in Multi-Language E-Discovery

Comprehending the Challenges of Technology Assisted Document Review Predictive Coding in Multi-Language E-Discovery UBIC North America, Inc. 3 Lagoon Dr., Ste. 180, Redwood City, CA 94065 877-321-8242

### Digital Government Institute. Managing E-Discovery for Government: Integrating Teams and Technology

Digital Government Institute Managing E-Discovery for Government: Integrating Teams and Technology Larry Creech Program Manager Information Catalog Program Corporate Information Security Information Technology

### ediscovery Policies: Planned Protection Saves More than Money Anticipating and Mitigating the Costs of Litigation

Brought to you by: ediscovery Policies: Planned Protection Saves More than Money Anticipating and Mitigating the Costs of Litigation Introduction: Rising costs of litigation The chance of your organization

### Pretrial Practice Course Syllabus Spring, 2014 Meeting -- Tuesdays 1:30-3:20pm Room -- 432(C)

Pretrial Practice Course Syllabus Spring, 2014 Meeting -- Tuesdays 1:30-3:20pm Room -- 432(C) Professor: Rich Kelsey Telephone: (703) 993-8973 Email: rkelsey@gmu.edu Course Materials Material will be assigned

### Presenters: Brett Anders, Esq. Joseph J. Lazzarotti, Esq., CIPP/US. Morristown, NJ

Presenters: Brett Anders, Esq. Joseph J. Lazzarotti, Esq., CIPP/US Morristown, NJ 1 Preservation Privacy & Data Security Search & Review 2 Pre-Litigation Data Map Litigation Hold Procedure Standardized

### How to Manage Costs and Expectations for Successful E-Discovery: Best Practices

How to Manage Costs and Expectations for Successful E-Discovery: Best Practices Mukesh Advani, Esq., Advisory Board Member, UBIC North America, Inc. UBIC North America, Inc. 3 Lagoon Dr., Ste. 180, Redwood

### Software-assisted document review: An ROI your GC can appreciate. kpmg.com

Software-assisted document review: An ROI your GC can appreciate kpmg.com b Section or Brochure name Contents Introduction 4 Approach 6 Metrics to compare quality and effectiveness 7 Results 8 Matter 1

### E-DISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK

E-DISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK If your company is involved in civil litigation, the Federal Rules of Civil Procedure regarding preservation and production of electronic documents

### Top 10 Best Practices in Predictive Coding

Top 10 Best Practices in Predictive Coding Emerging Best Practice Guidelines for the Conduct of a Predictive Coding Project Equivio internal document " design an appropriate process, including use of available

### Technology- Assisted Review 2.0

LITIGATION AND PRACTICE SUPPORT Technology- Assisted Review 2.0 by Ignatius Grande of Hughes Hubbard & Reed LLP and Andrew Paredes of Epiq Systems Legal teams and their outside counsel must deal with an

### Measuring Recall in E-Discovery, Part Two: No Easy Answers

Measuring Recall in E-Discovery, Part Two: No Easy Answers John Tredennick In Part One of this article, I introduced readers to statistical problems inherent in proving the level of recall reached in a

### Industry Leading Solutions: Innovative Technology. Quality Results.

Industry Leading Solutions: Innovative Technology. Quality Results. April 10, 2013 emagsolutions.com Agenda Speaker Introduction A Quick Word about emag Importance of Technology Assisted Review (TAR) Key

### www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence April 2013

www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence Why are non-users staying away from PC? source: edj Group s Q1 2013 Predictive Coding Survey, February 2013, N = 66 Slide 2 Introduction

### Three Methods for ediscovery Document Prioritization:

Three Methods for ediscovery Document Prioritization: Comparing and Contrasting Keyword Search with Concept Based and Support Vector Based "Technology Assisted Review-Predictive Coding" Platforms Tom Groom,

### ESI: Focus on Review and Production Strategy. Meredith Lee, Online Document Review Supervisor, Paralegal

ESI: Focus on Review and Production Strategy Meredith Lee, Online Document Review Supervisor, Paralegal About Us Avansic E-discovery and digital forensics company founded in 2004 by Dr. Gavin W. Manes,

### Minimizing ediscovery risks. What organizations need to know in today s litigious and digital world.

What organizations need to know in today s litigious and digital world. The main objective for a corporation s law department is to mitigate risk throughout the company, while keeping costs under control.

### E-Discovery Tip Sheet

E-Discovery Tip Sheet LegalTech 2015 Some Panels and Briefings Last month I took you on a select tour of the vendor exhibits and products from LegalTech 2015. This month I want to provide a small brief

### THE NEW WORLD OF E-DISCOVERY

THE NEW WORLD OF E-DISCOVERY Ralph Losey: partner and National e-discovery Counsel of Jackson Lewis LLP, a labor & employment firm with 700 lawyers and 46 offices nationwide. JacksonLewis.com author of

### WHITE PAPER: CUSTOMIZE WHITE PAPER: BEST PRACTICES FOR ARCHIVING. Best Practices for Defining and Establishing Effective Archive Retention Policies

WHITE PAPER: CUSTOMIZE WHITE PAPER: BEST PRACTICES FOR ARCHIVING Confidence in a connected world. Best Practices for Defining and Establishing Effective Archive Retention Policies Sponsored by Symantec

### Xact Data Discovery. Xact Data Discovery. Xact Data Discovery. Xact Data Discovery. ediscovery for DUMMIES LAWYERS. MDLA TTS August 23, 2013

MDLA TTS August 23, 2013 ediscovery for DUMMIES LAWYERS Kate Burke Mortensen, Esq. kburke@xactdatadiscovery.com Scott Polus, Director of Forensic Services spolus@xactdatadiscovery.com 1 Where Do I Start??

### A PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE

A PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE Effective February 1, 2010, the Alabama Rules of Civil Procedure were amended to provide for and accommodate

### The Case for Technology Assisted Review and Statistical Sampling in Discovery

The Case for Technology Assisted Review and Statistical Sampling in Discovery Position Paper for DESI VI Workshop, June 8, 2015, ICAIL Conference, San Diego, CA Christopher H Paskach The Claro Group, LLC

### Predictive Coding as a Means to Prioritize Review and Reduce Discovery Costs. White Paper

Predictive Coding as a Means to Prioritize Review and Reduce Discovery Costs White Paper INTRODUCTION Computers and the popularity of digital information have changed the way that the world communicates

### E-Discovery: A Common Sense Approach. In order to know how to handle and address ESI issues, the preliminary and

Jay E. Heidrick Polsinelli jheidrick@polsinelli.com (913) 234-7506 E-Discovery: A Common Sense Approach In order to know how to handle and address ESI issues, the preliminary and obvious question must

### Technology-Assisted Review and Other Discovery Initiatives at the Antitrust Division. Tracy Greer 1 Senior Litigation Counsel E-Discovery

Technology-Assisted Review and Other Discovery Initiatives at the Antitrust Division Tracy Greer 1 Senior Litigation Counsel E-Discovery The Division has moved to implement several discovery initiatives

### Case 1:12-cv-24356-JG Document 404 Entered on FLSD Docket 03/18/2014 Page 1 of 14

Case 1:12-cv-24356-JG Document 404 Entered on FLSD Docket 03/18/2014 Page 1 of 14 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF FLORIDA MIAMI DIVISION CASE NO. 12 24356 CIV GOODMAN PROCAPS S.A., [CONSENT

### for Insurance Claims Professionals

A Practical Guide to Understanding ediscovery for Insurance Claims Professionals ediscovery Defined and its Relationship to an Insurance Claim Simply put, ediscovery (or Electronic Discovery) refers to

### Using Futures Markets to Manage Price Risk for Feeder Cattle (AEC 2013-01) February 2013

Using Futures Markets to Manage Price Risk for Feeder Cattle (AEC 2013-01) February 2013 Kenny Burdine 1 Introduction: Price volatility in feeder cattle markets has greatly increased since 2007. While

### STATISTICAL ANALYSIS AND INTERPRETATION OF DATA COMMONLY USED IN EMPLOYMENT LAW LITIGATION

STATISTICAL ANALYSIS AND INTERPRETATION OF DATA COMMONLY USED IN EMPLOYMENT LAW LITIGATION C. Paul Wazzan Kenneth D. Sulzer ABSTRACT In employment law litigation, statistical analysis of data from surveys,

### E-Discovery in Michigan. Presented by Angela Boufford

E-Discovery in Michigan ESI Presented by Angela Boufford DISCLAIMER: This is by no means a comprehensive examination of E-Discovery issues. You will not be an E-Discovery expert after this presentation.

### E-Discovery in Employment Litigation: Making Practical, Yet Defensible Decisions

E-Discovery in Employment Litigation: Making Practical, Yet Defensible Decisions 11 E-Discovery in Employment Litigation: Making Practical, Yet Defensible Decisions Introduction Much has been said about

### The Random Sampling Road to Reasonableness. Reduce Risk and Cost by Employing a Complete and Integrated Validation Process

The Random Sampling Road to Reasonableness Reduce Risk and Cost by Employing a Complete and Integrated Validation Process By: Michael R. Wade Planet Data Executive Vice President Chief Technology Officer

### E-Discovery for Backup Tapes. How Technology Is Easing the Burden

E-Discovery for Backup Tapes How Technology Is Easing the Burden TABLE OF CONTENTS Introduction...3 The Importance and Challenge of Backup Tapes for Electronic Discovery...3 TAPES AS A SOURCE OF ESI...

### Discovery Data Management

Discovery Data Management in Practice Introductions Reveal Derick Roselli 949-280-3519 droselli@revealdata.com www.revealdata.com Program Outline Pre-Planning Stages Workflow Management Project Management

### Making reviews more consistent and efficient.

Making reviews more consistent and efficient. PREDICTIVE CODING AND ADVANCED ANALYTICS Predictive coding although yet to take hold with the enthusiasm initially anticipated is still considered by many

### Predictability in E-Discovery

Predictability in E-Discovery Presented by: John G. Roman, Jr. National Manager, Practice Group Technology Services Nixon Peabody LLP Tom Barce Assistant Director of Practice Support Fulbright & Jaworski

### Pros And Cons Of Computer-Assisted Review

Portfolio Media. Inc. 860 Broadway, 6th Floor New York, NY 10003 www.law360.com Phone: +1 646 783 7100 Fax: +1 646 783 7161 customerservice@law360.com Pros And Cons Of Computer-Assisted Review Law360,

PRESENTED BY: Sponsored by: Practical Uses of Analytics in E-Discovery - A PRIMER Jenny Le, Esq. Vice President of Discovery Services jle@evolvediscovery.com E-Discovery & Ethics Structured, Conceptual,

### E-Discovery Defensibility ViewS from the Bench page : 1

E-Discovery Defensibility ViewS from the Bench page : 1 E-Discovery Defensibility Views from the Bench A Clearwell White Paper E-Discovery Defensibility ViewS from the Bench page : 2 Table of Contents

### Predictive Coding: Emerging E Discovery Tool Leveraging E Discovery Computer Assisted Review to Reduce Time and Expense of Discovery

Presenting a live 90 minute webinar with interactive Q&A Predictive Coding: Emerging E Discovery Tool Leveraging E Discovery Computer Assisted Review to Reduce Time and Expense of Discovery WEDNESDAY,

### ediscovery Defensibility

Content Introduction... 3 The Duty to Preserve Phillip M. Adams & Assoc., LLC v. Dell, Inc.... 4 Case Facts.... 4 Key Takeaways... 4 Keyword Search Victor Stanley, Inc. v. Creative Pipe, Inc..... 5 Case

### Introduction to Predictive Coding

Introduction to Predictive Coding Herbert L. Roitblat, Ph.D. CTO, Chief Scientist, OrcaTec Predictive coding uses computers and machine learning to reduce the number of documents in large document sets

### EnCase ediscovery. Automatically search, identify, collect, preserve, and process electronically stored information across the network.

TM GUIDANCE SOFTWARE EnCASE ediscovery EnCase ediscovery Automatically search, identify, collect, preserve, and process electronically stored information across the network. GUIDANCE SOFTWARE EnCASE ediscovery

### REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - PRIORITIZATION FOR 100% REVIEW. Bill Tolson Sr. Product Marketing Manager Recommind Inc.

REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - Bill Tolson Sr. Product Marketing Manager Recommind Inc. Introduction... 3 Traditional Linear Review... 3 Advanced Review Strategies: A Typical Predictive

### Electronic Discovery: Litigation Holds, Data Preservation and Production

Electronic Discovery: Litigation Holds, Data Preservation and Production April 27, 2010 Daniel Munsch, Assistant General Counsel John Lerchey, Coordinator for Incident Response 0 E-Discovery Rules Federal

### E-Discovery Getting a Handle on Predictive Coding

E-Discovery Getting a Handle on Predictive Coding John J. Jablonski Goldberg Segalla LLP 665 Main St Ste 400 Buffalo, NY 14203-1425 (716) 566-5400 jjablonski@goldbergsegalla.com Drew Lewis Recommind 7028

### Traditionally, the gold standard for identifying potentially

istockphoto.com/alexandercreative Predictive Coding: It s Here to Stay Predictive coding programs are poised to become a standard practice in e-discovery in the near future. As more courts weigh in on

### THE FEDERAL COURTS LAW REVIEW. Comments on The Implications of Rule 26(g) on the Use of Technology-Assisted Review

THE FEDERAL COURTS LAW REVIEW Volume 7, Issue 1 2014 Comments on The Implications of Rule 26(g) on the Use of Technology-Assisted Review Maura R. Grossman and Gordon V. Cormack ABSTRACT Approaches to technology-assisted

### When E-Discovery Becomes Evidence

Monday, June 11, 2007 When E-Discovery Becomes Evidence Make sure that you can easily authenticate the information that was so costly to produce By Leonard Deutchman Special to the Law Weekly A federal

### Case 1:11-cv-01279-ALC-AJP Document 96 Filed 02/24/12 Page 1 of 49

Case 1:11-cv-01279-ALC-AJP Document 96 Filed 02/24/12 Page 1 of 49 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

### Case 2:11-cv-00678-LRH-PAL Document 174 Filed 07/18/14 Page 1 of 18 UNITED STATES DISTRICT COURT DISTRICT OF NEVADA * * * Plaintiff, Defendants.

Case :-cv-00-lrh-pal Document Filed 0// Page of 0 PROGRESSIVE CASUALTY INSURANCE COMPANY, v. JACKIE K. DELANEY, et al., UNITED STATES DISTRICT COURT DISTRICT OF NEVADA Plaintiff, Defendants. * * * Case