A Practitioner s Guide to Statistical Sampling in EDiscovery. October 16, 2012


 Egbert Mills
 3 years ago
 Views:
Transcription
1 A Practitioner s Guide to Statistical Sampling in EDiscovery October 16,
2 Meet the Panelists Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Gordon V. Cormack, Professor at the David R. Cheriton School of Computer Science at the University of Waterloo Jim Wagner, Cofounder and CEO of DiscoverReady Maureen O Neill, SVP, Marketplace Leader at DiscoverReady 2
3 Agenda What is statistical sampling? Why should a practitioner use statistical sampling in ediscovery? Opportunities to use statistical sampling The basics of statistical sampling An example of statistical sampling in the ediscovery context Key decisions when using statistical sampling Takeaway recommendations on using statistical sampling 3
4 What is Statistical Sampling? In general, statistical sampling is a method to estimate a characteristic of a large population by examining only a subset of it. In the specific context of ediscovery: Estimate is a reasonably precise mathematical measurement. Characteristic refers to the number (or proportion) of items in the document population having a certain property, such as responsiveness or privilege. Population is a collection of electronic documents. Subset is a small but representative sample of the document population (chosen at random). Note that the size of the subset the sample size determines the precision of the estimate. Sample size is dependent on the acceptable margin of error, the desired confidence level, and to a negligible extent, the size of the population (these variables will be discussed later in the webinar). 4
5 What is Statistical Sampling? Judgmental sampling can be very useful in ediscovery, but it is not what we are discussing in today s webinar Statistical sampling is not the same thing as judgmental sampling Judgmental sampling is not random It involves a selection of items for the subset using some degree of human judgment When sampling is judgmental, inferences cannot be drawn about the population based on an examination of the subset Judgmental sampling is akin to spotchecking 5
6 Why Use Statistical Sampling? Thoughtful use of statistical sampling can improve the quality, efficiency, and defensibility of ediscovery efforts. Quality By counting or measuring the inputs and outputs of an e discovery process, we can work to improve the process to make it more accurate. For example: If we find that a proposed search term is bringing in too many false positives (i.e., poor precision ) we can try a different search term and test the results using sampling. If statistical sampling confirms that the new term reduces the number of irrelevant documents (better precision), but is not underinclusive (the recall is good), we have improved the search process. 6
7 Why Use Statistical Sampling? Efficiency Sampling saves time and money by allowing us to count or measure things more efficiently. Statistical sampling is scalable; even with extremely large populations, we can use relatively small samples. For example, if we want to estimate the proportion of relevant documents (the richness or prevalence ) in a collection of one million documents, an appropriate sample size might be 600 documents. If we want to estimate the proportion of relevant documents in a collection of a billion documents, the appropriate sample size would remain the same. 7
8 Why Use Statistical Sampling? Although its use in ediscovery is not yet widespread, we are moving in a direction where it soon will be considered best practice. Defensibility Courts are increasingly requiring the use of sampling as part of a reasonable discovery process. In re Seroquel Prods. Liabil. Litig., 244 F.R.D. 650 (M.D. Fla. 2007) Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251 (D. Md. 2008) William. A. Gross Constr. Assocs. Inc. v. American Mfrs. Mut. Ins. Co., 256 F.R.D. 134 (S.D.N.Y. 2009) Mt. Hawley Ins. Co. v. Felman Prod., Inc., 2010 WL (S.D. W. Va. May 18, 2010) DaSilva Moore v. Publicis Group, No. 11 Civ (S.D.N.Y. Feb. 24, 2012) (Peck, M.J.), aff d (S.D.N.Y. Apr. 26, 2012) (Carter, D.J.) In re: Actos, MDL No. 6:11md2299 (W.D. La. Jul. 27, 2012) 8
9 Opportunities to Use Statistical Sampling Incorporate into early case assessment and strategy development Efficiently hone in on the sources and custodians of information likely to be relevant Assess the burdens and costs involved in accessing certain information, such as backup tapes or other offline media Gauge the richness of populations before embarking on review (Later in the program, we will take a stepbystep walk through this use of sampling, as an example of how sampling is performed) Test the culling of a data set to ensure that your cull is neither overbroad nor too restrictive 9
10 Opportunities to Use Statistical Sampling Measure the efficacy of search terms and refine the terms Measure the accuracy of a predictive coding process Test automated methods of screening documents for privilege and confidentiality Sample a document production before it goes out the door to provide additional assurance that privileged content is not inadvertently included 10
11 Opportunities to Use Statistical Sampling Support proportionality arguments Determine whether the cost of reviewing certain types of ESI is reasonable and proportional Is there bang for the buck in reviewing a particular set of documents based on how many responsive documents are estimated to be found? For example: Should we spend money on reviewing this custodian s documents if the collection has very low prevalence? Is it worth the expense to continue reviewing more documents from more custodians if that additional effort is not likely to yield significantly more relevant information? 11
12 Opportunities to Use Statistical Sampling Conduct quality control and quality assurance of human review efforts Measure the error rate on document review decisions for an overall project, or for particular reviewers When done in real time, as the project progresses, error rate measurement can be part of an effective quality control ( QC ) workflow When done at the conclusion of the project, or a phase of the project, the measurement becomes part of the quality assurance ( QA ) and defensibility of process 12
13 Opportunities to Use Statistical Sampling Important caveat about taking statistical measurements using human decisions as a reference point or gold standard Human decisions about documents inevitably involve an element of subjectivity, and even the best decisionmakers will make mistakes. (This is true for all decisions, whether in the original review, a QC review, or in a sampling review.) Even the gold standard decisions are not going to be 100% consistent or correct. This element of human error, or legitimate differences of opinion, will always introduce some degree of measurement error into a statistical measurement involving human decisionmaking. Therefore, statistical measurement cannot be more accurate than human judgment permits. 13
14 The Basics of Statistical Sampling: Drawing a Random Sample What is a random sample and how is it generated? A random sample is a subset of documents that is chosen at random from a larger population of interest. Choosing at random means that every document in the collection has an equal chance of being selected in the sample. Random sampling can be achieved a number of different ways: Drawing numbers from a hat Using a computerized randomnumber generator Choosing documents based on one or more digits from their hash value Many ediscovery tools have builtin random sample generators 14
15 The Basics of Statistical Sampling: Drawing a Random Sample A random sample has been drawn now what? Review the sample and count the number of documents with the characteristic of interest (e.g., responsiveness). The proportion of responsive documents in the sample is calculated by dividing the number of responsive documents in the sample by the total number of documents in the sample. Because the sample is random, we can extrapolate that the proportion of responsive documents in the population is approximately the same as the proportion in the sample. When we say approximately the same, we mean that there is a margin of error in our estimate we ll explain margin of error in a moment. 15
16 The Basics of Statistical Sampling: Drawing a Random Sample Suppose we have a collection of one million documents And we want to estimate how many of them are relevant for the purposes of building a budget and timeline for review and production. Reviewing all one million documents for the purposes of budgeting and project planning is infeasible. So, instead, we take a random sample of 1,000 documents. We review the sample and find that 300 documents are relevant. The proportion of relevant documents in the sample is 300/1,000, or 30%. Therefore, the proportion of relevant documents in the collection is estimated to be approximately 30% (or 300,000 documents). If we repeated this process, we would get a slightly different estimate each time, but in general, each estimate would be close to the actual proportion. 16
17 The Basics of Statistical Sampling: Margin of Error / Confidence Interval The margin of error is a way of expressing a range above and below the estimate that is likely to contain the actual value. In our example: We determined that the proportion of relevant documents in the sample was 30%, and we extrapolated that the proportion of relevant documents in the collection was approximately, but not exactly, 30%. The exact proportion of relevant documents in the collection is unknown, but is likely to fall within a margin of error of +/ 3%. We express this by saying that that the proportion of relevant documents in the collection is estimated to be 30%, plus or minus 3%. 17
18 The Basics of Statistical Sampling: Margin of Error / Confidence Interval An alternative way of stating the estimate is by using a confidence interval instead of margin of error. The confidence interval is the range of values that is likely to contain the actual value. In our previous example, we would state that the proportion of relevant documents is likely to fall within a range of 27% to 33%. As compared to the margin of error, the confidence interval does not have to be exactly symmetrical around the estimate, and can therefore be a more precise way of expressing the uncertainty of the estimate. 18
19 The Basics of Statistical Sampling: Confidence Level What do we mean when we say the confidence interval is likely to contain the actual value? The confidence level is the probability that the confidence interval would contain the actual value if the sampling process were to be repeated a large number of times. For example: If the confidence level is 95%, it means that there is a 95% chance that the actual value is within the confidence interval. (In our previous example, we would say with a 95% confidence, that the proportion of relevant documents falls between 27% and 33%. ) If the confidence level is 99%, it means that there is a 99% chance that the actual value is within the confidence interval. (In our previous example, we would say with a 99% confidence, that the proportion of relevant documents falls between 26% and 34%. ) 19
20 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size The three concepts of confidence level, confidence interval (or margin of error), and sample size are interrelated. Generally speaking, if the confidence level remains constant, as the sample size goes up, the margin of error goes down. Similarly, to increase the confidence level, either the sample size or the margin of error must increase. Finally, if you want to decrease your margin of error, either the confidence level must come down or the sample size must go up. Bear in mind that these relationships are not proportional: Getting a smaller confidence interval, or a higher confidence level, may require drawing a much larger sample. To illustrate this, consider again the previous example, and assume that we find the proportion of relevant documents is 30% in each sample. The following slide depicts the relationship between sample size, confidence level, and margin of error: 20
21 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size Margin of Error Confidence Level Sample Size % 99% 4, % 99% 1, % 99% 500 0% 30% 100% Proportion of Relevant documents 21
22 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size Here s another illustration that also demonstrates how taking relatively small samples can help us understand very large populations. We have a collection of one million documents; we assume the proportion of relevant documents is 50% and we want a 95% confidence level for our estimate. Here are some examples of the (pretty good) margins of error we will achieve with some relatively small sample sizes: Sample Size Margin of Error % % % 22
23 The Basics of Statistical Sampling: The Tradeoffs In general, a higher confidence level is better. But that comes at the price of a larger sample size, and/or A wider confidence interval (i.e., a higher margin of error). Likewise, a smaller margin of error generally is better, because it reflects a more precise estimate. But that requires a lower confidence level (less certainty), and/or A larger sample size (higher cost and/or less efficiency) There are calculators available on the Internet that allow you to plug in the variables and make computations of sample size, margin of error, and confidence level, but be careful when choosing one. Make sure you understand the assumptions each one uses or if you don t, get help from someone who does! 23
24 The Basics of Statistical Sampling: Standards for EDiscovery? Is there a minimum acceptable confidence level and/or margin of error when using statistical sampling in ediscovery? There are no bright line rules regarding confidence level or margin of error 95% confidence level is commonly used in statistical measurement The acceptable margin of error will depend on the consequences of an inaccurate estimate The operative standard is one of reasonableness 24
25 The Basics of Statistical Sampling: Standards for EDiscovery? Is there a minimum acceptable confidence level and/or margin of error when using statistical sampling in ediscovery? Every matter is different, and what is reasonable in one matter may not be in another Factors affecting the reasonableness calculus include: The cost of greater precision in measurement as compared to the amount at stake and the importance of the matter (proportionality) The purpose for which the sampling is being performed The time and resources available for sampling 25
26 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Determine the total number of documents in the collection. We ll call that number N (e.g., N = 1,000,000 documents). Choose the desired confidence level (e.g., 95%), and margin of error (e.g., +/ 2%). Determine the appropriate sample size (we ll call that n ) using an appropriate calculator (e.g., n = 2,395). 26
27 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Select 2,395 of the documents, at random, from the collection. 27
28 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents Not Responsive 1,695 documents Count the number of responsive documents in the sample (say, 700). 28
29 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents The proportion of responsive documents in the sample is calculated by dividing the number of responsive documents in the sample (700) by n (2,395): 700/2,395 = 29.2% 29
30 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents The confidence interval is 29.2% (the proportion of responsive documents) plus or minus 2% (the margin of error): 27.2% to 31.2% 30
31 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents To calculate the number of responsive documents in the collection, multiply the two confidence limits by N (27.2% x 1,000,000 = 272,000 responsive documents; 31.2% x 1,000,000 = 312,000 responsive documents). 31
32 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents In this example, we can state that, with 95% confidence, we estimate there are between 272,000 and 312,000 responsive documents in this collection. 32
33 Examples of Statistical Sampling: Example 2 How many responsive documents are likely to be found in the document collection? 1. Let s tweak Example 1 slightly, and assume that the number of responsive documents in the reviewed sample is only The proportion of responsive documents in this example is 3/2,395 = 0.13%, and with a confidence interval of 1.87% to +2.13% (18,700 to 21,300 relevant documents). 3. Obviously it is not possible to have a negative number of relevant documents in the collection. This scenario illustrates the problem faced when sampling populations with a low proportion of relevant documents (i.e., low prevalence or low richness ). 4. When sampling such populations, a normal sample size calculation of confidence intervals and margin of error is not precise enough. When dealing with a low prevalence population, a more precise confidence interval can be computed using a binomial confidence interval calculator. 33
34 Examples of Statistical Sampling: Example 2 How many responsive documents are likely to be found in a collection of documents? 5. Here s an example of a binomial confidence interval calculator: 6. If we take our sample of 2,395 documents and recompute the confidence intervals at the 95% confidence level using the binomial confidence interval calculator, if there are three responsive documents in the sample, the confidence interval is 0.03% to 0.37% (300 to 3,700 documents). 7. This is an example of how the confidence interval may not be symmetrical around the estimate. 34
35 Using Statistical Sampling: Key Decisions a Practitioner Must Make What are you trying to measure? What are trying to accomplish with the measurement? Are you sampling for internal purposes (to improve process) or to defend your process to opposing counsel or the court? Do you need to hire an expert in statistics? What are your acceptable levels of recall, precision, etc.? What is your desired confidence level? How wide a confidence interval will you accept? How large a sample size is feasible under the circumstances of your matter, taking into account cost and timing considerations? How will you document and report your measurements and methodology? What aspects of your statistical sampling efforts are you willing to share with the opposing party? With the court? 35
36 TakeAway Recommendations on Using Statistical Sampling Statistical sampling can be tricky most lawyers do not have sufficient understanding of the concepts (or the math) to go it alone and get it right. Practitioners should use an expert or statistical tool to: Determine whether statistical sampling is an appropriate tool for the matter Determine which statistical measures are best for the situation (precision, recall, elusion, accuracy, etc.) Determine the desired confidence level and margin of error Calculate the proper sample size to achieve this confidence level and margin of error Draw a random sample of the appropriate size Compute the estimate 36
37 Question & Answer Session To contact the panelists: Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Jim Wagner, Cofounder and CEO of DiscoverReady Gordon V. Cormack, Professor at the David R. Cheriton School of Computer Science at the University of Waterloo Maureen O Neill, SVP, Marketplace Leader at DiscoverReady 37
EDiscovery in Mass Torts:
EDiscovery in Mass Torts: Predictive Coding Friend or Foe? Sherry A. Knutson Sidley Austin One S Dearborn St 32nd Fl Chicago, IL 60603 (312) 8534710 sknutson@sidley.com Sherry A. Knutson is a partner
More informationThe Evolution, Uses, and Case Studies of Technology Assisted Review
FEBRUARY 4 6, 2014 / THE HILTON NEW YORK The Evolution, Uses, and Case Studies of Technology Assisted Review One Size Does Not Fit All #LTNY Meet Our Panelists The Honorable Dave Waxse U.S. Magistrate
More informationEdiscovery Taking Predictive Coding Out of the Black Box
Ediscovery Taking Predictive Coding Out of the Black Box Joseph H. Looby Senior Managing Director FTI TECHNOLOGY IN CASES OF COMMERCIAL LITIGATION, the process of discovery can place a huge burden on
More informationPredictive Coding Defensibility
Predictive Coding Defensibility Who should read this paper The Veritas ediscovery Platform facilitates a quality control workflow that incorporates statistically sound sampling practices developed in conjunction
More informationThe Benefits of. in EDiscovery. How Smart Sampling Can Help Attorneys Reduce Document Review Costs. A white paper from
The Benefits of Sampling in EDiscovery How Smart Sampling Can Help Attorneys Reduce Document Review Costs A white paper from 615.255.5343 dsi.co 414 Union Street, Suite 1210 Nashville, TN 372191771 Table
More informationPowerUp Your Privilege Review: Protecting Privileged Materials in Ediscovery
PowerUp Your Privilege Review: Protecting Privileged Materials in Ediscovery Jeff Schomig, WilmerHale Stuart Altman, Hogan Lovells Joe White, Kroll Ontrack Sheldon Noel, Kroll Ontrack (moderator) April
More informationSAMPLING: MAKING ELECTRONIC DISCOVERY MORE COST EFFECTIVE
SAMPLING: MAKING ELECTRONIC DISCOVERY MORE COST EFFECTIVE Milton Luoma Metropolitan State University 700 East Seventh Street St. Paul, Minnesota 55337 651 7931246 (fax) 651 7931481 Milt.Luoma@metrostate.edu
More informationApplication of Simple Random Sampling 1 (SRS) in ediscovery
Manuscript submitted to the Organizing Committee of the Fourth DESI Workshop on Setting Standards for Electronically Stored Information in Discovery Proceedings on April 20, 2011. Updated May 18, 2011.
More informationTechnology Assisted Review of Documents
Ashish Prasad, Esq. Noah Miller, Esq. Joshua C. Garbarino, Esq. October 27, 2014 Table of Contents Introduction... 3 What is TAR?... 3 TAR Workflows and Roles... 3 Predictive Coding Workflows... 4 Conclusion...
More informationRecent Developments in the Law & Technology Relating to Predictive Coding
Recent Developments in the Law & Technology Relating to Predictive Coding Presented by Paul Neale CEO Presented by Gene Klimov VP & Managing Director Presented by Gerard Britton Managing Director 2012
More informationMANAGING BIG DATA IN LITIGATION
David Han 2015 MANAGING BIG DATA IN LITIGATION DAVID HAN Associate, Morgan Lewis & Bockius, edata Practice Group MANAGING BIG DATA Data volumes always increasing New data sources Mobile Internet of Things
More informationPredictive Coding: A Primer
MEALEY S TM LITIGATION REPORT Discovery Predictive Coding: A Primer by Amy Jane Longo, Esq. and Usama Kahf, Esq. O Melveny & Myers LLP Los Angeles, California A commentary article reprinted from the March
More informationTechnology Assisted Review: The Disclosure of Training Sets and Related Transparency Issues Whitney Street, Esq. 1
Technology Assisted Review: The Disclosure of Training Sets and Related Transparency Issues Whitney Street, Esq. 1 The potential cost savings and increase in accuracy afforded by technology assisted review
More informationPr a c t i c a l Litigator s Br i e f Gu i d e t o Eva l u at i n g Ea r ly Ca s e
Ba k e Offs, De m o s & Kicking t h e Ti r e s: A Pr a c t i c a l Litigator s Br i e f Gu i d e t o Eva l u at i n g Ea r ly Ca s e Assessment So f t wa r e & Search & Review Tools Ronni D. Solomon, King
More informationThe Truth About Predictive Coding: Getting Beyond The Hype
www.encase.com/ceic The Truth About Predictive Coding: Getting Beyond The Hype David R. Cohen Reed Smith LLP Records & EDiscovery Practice Group Leader David leads a group of more than 100 lawyers in
More informationJudge Peck Provides a Primer on ComputerAssisted Review By John Tredennick
By John Tredennick CEO Catalyst Repository Systems Magistrate Judge Andrew J. Peck issued a landmark decision in Da Silva Moore v. Publicis and MSL Group, filed on Feb. 24, 2012. This decision made headlines
More informationESI and Predictive Coding
Beijing Boston Brussels Chicago Frankfurt Hong Kong ESI and Predictive Coding Houston London Los Angeles Moscow Munich New York Palo Alto Paris São Paulo Charles W. Schwartz Chris Wycliff December 13,
More informationNovember/December 2010 THE MAGAZINE OF THE AMERICAN INNS OF COURT. rofessionalism. Ethics Issues. and. Today s. Technology. www.innsofcourt.
November/December 2010 THE MAGAZINE OF THE AMERICAN INNS OF COURT rofessionalism and Ethics Issues in Today s Technology www.innsofcourt.org Transparency in EDiscovery: No Longer a Novel Approach By Michael
More informationMastering Predictive Coding: The Ultimate Guide
Mastering Predictive Coding: The Ultimate Guide Key considerations and best practices to help you increase ediscovery efficiencies and save money with predictive coding 4.5 Validating the Results and Producing
More informationThe United States Law Week
The United States Law Week Source: U.S. Law Week: News Archive > 2012 > 04/24/2012 > BNA Insights > Under Fire: A Closer Look at Technology Assisted Document Review EDISCOVERY Under Fire: A Closer Look
More informationLegal Arguments & Response Strategies for EDiscovery
Legal Arguments & Response Strategies for EDiscovery The tools to craft strategic discovery requests & mitigate the risks and burdens of production. Discussion Outline Part I Strategies for Requesting
More informationQuality Control for predictive coding in ediscovery. kpmg.com
Quality Control for predictive coding in ediscovery kpmg.com Advances in technology are changing the way organizations perform ediscovery. Most notably, predictive coding, or technology assisted review,
More informationThe Predictive Coding Soundtrack: Rewind, Play, FastForward
The Predictive Coding Soundtrack: Rewind, Play, FastForward LEGALTECH NEW YORK February 3, 2015 Moderator: Amy Hinzmann Senior Vice President, DiscoverReady DiscoverReady 2014 THE PANELISTS* Marla Bergman
More informationAn Open Look at Keyword Search vs. Predictive Analytics
877.557.4273 catalystsecure.com ARTICLE An Open Look at Keyword Search vs. Can Keyword Search Be As Effective as TAR? John Tredennick, Esq. Founder and CEO, Catalyst Repository Systems 2015 Catalyst Repository
More informationMeasurement in ediscovery
Measurement in ediscovery A Technical White Paper Herbert Roitblat, Ph.D. CTO, Chief Scientist Measurement in ediscovery From an informationscience perspective, ediscovery is about separating the responsive
More informationPredictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review
Predictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review ACEDS Webinar April 23, 2014 Sponsored by Robert Half Legal 1 2014 Robert Half Legal. An Equal Opportunity
More informationWhite Paper Technology Assisted Review. Allison Stanfield and Jeff Jarrett 25 February 2015. 1300 136 993 www.elaw.com.au
White Paper Technology Assisted Review Allison Stanfield and Jeff Jarrett 25 February 2015 1300 136 993 www.elaw.com.au Table of Contents 1. INTRODUCTION 3 2. KEYWORD SEARCHING 3 3. KEYWORD SEARCHES: THE
More informationDocument Review Costs
Predictive Coding Gain Earlier Insight and Reduce Document Review Costs Tom Groom Vice President, Discovery Engineering tgroom@d4discovery.com 303.840.3601 D4 LLC Litigation support service provider since
More informationPredictive Coding Defensibility and the Transparent Predictive Coding Workflow
WHITE PAPER: PREDICTIVE CODING DEFENSIBILITY........................................ Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive
More informationCostEffective and Defensible Technology Assisted Review
WHITE PAPER: SYMANTEC TRANSPARENT PREDICTIVE CODING Symantec Transparent Predictive Coding CostEffective and Defensible Technology Assisted Review Who should read this paper Predictive coding is one of
More informationcase 3:12md02391RLMCAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION
case 3:12md02391RLMCAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION IN RE: BIOMET M2a MAGNUM HIP IMPLANT PRODUCTS LIABILITY
More informationPredictive Coding Defensibility and the Transparent Predictive Coding Workflow
Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive coding is one of the most promising technologies to reduce the high cost of review by
More informationTECHNOLOGYASSISTED DOCUMENT REVIEW: IS IT DEFENSIBLE?
TECHNOLOGYASSISTED DOCUMENT REVIEW: IS IT DEFENSIBLE? By William W. Belt, Dennis R. Kiker and Daryl E. Shetterly* Cite as: William W. Belt, Dennis R. Kiker & Daryl E. Shetterly, TechnologyAssisted Document
More informationNavigating Information Governance and ediscovery
Navigating Information Governance and ediscovery Implementing Processes & Technology to Reduce Downstream ediscovery Cost and Risk Shannon Smith General Counsel, Globanet March 11 12, 2013 Agenda 1 Overview
More informationCOURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994001) Fall 2014
COURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994001) Professors:Mark Austrian Christopher Racich Fall 2014 Introduction The ubiquitous use of computers, the
More informationTHE PREDICTIVE CODING CASES A CASE LAW REVIEW
THE PREDICTIVE CODING CASES A CASE LAW REVIEW WELCOME Thank you for joining Numerous diverse attendees Please feel free to submit questions Slides, recording and survey coming tomorrow SPEAKERS Matthew
More informationThe Tested Effectiveness of Equivio>Relevance in Technology Assisted Review
ediscovery & Information Management White Paper The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review Scott M. Cohen Elizabeth T. Timkovich John J. Rosenthal February 2014 2014 Winston
More informationHow Good is Your Predictive Coding Poker Face?
How Good is Your Predictive Coding Poker Face? SESSION ID: LAWW03 Moderator: Panelists: Matthew Nelson ediscovery Counsel Symantec Corporation Hon. Andrew J. Peck US Magistrate Judge Southern District
More informationThe State Of Predictive Coding
MEALEY S TM LITIGATION REPORT Discovery The State Of Predictive Coding by Royce F. Cohen and Derek I.A. Silverman Stroock & Stroock & Lavan LLP New York A commentary article reprinted from the September
More informationJason R. Baron Director of Litigation Office of General Counsel National Archives and Records Administration
NARA RACO 2010 Ronald Reagan Bldg., Washington, D.C. May 12, 2010 Jason R. Baron Director of Litigation Office of General Counsel National Archives and Records Administration Email is still the 800 lb.
More information2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500
2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500 5218 South East Street, Suite E3, Indianapolis, IN 46227 Tel 317.247.4400 Fax 317.247.0044 Presented by Providing
More informationPREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX?
Vol. 46 No. 3 February 6, 2013 PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX? The high costs of ediscovery have led to the development of computerized review technology by which the user may search
More informationEthics and ediscovery
Ethics and ediscovery John Mansfield and Devon Newman January 6, 2012 1 2013, MansfieldLaw ediscovery basics We will cover: Preservation and spoliation Searching and producing documents Supervising lawyers
More informationThe case for statistical sampling in ediscovery
Forensic The case for statistical sampling in ediscovery January 2012 kpmg.com 2 The case for statistical sampling in ediscovery The sheer volume and unrelenting production deadlines of today s electronic
More informationLitigation Solutions insightful interactive culling distributed ediscovery processing powering digital review
Litigation Solutions i n s i g h t f u l i n t e r a c t i ve c u l l i n g d i s t r i b u t e d e d i s cove r y p ro ce s s i n g p owe r i n g d i g i t a l re v i e w Advanced Analytical Review Data
More informationComprehending the Challenges of Technology Assisted Document Review
Comprehending the Challenges of Technology Assisted Document Review Predictive Coding in MultiLanguage EDiscovery 3 Lagoon Dr., Ste.180, Redwood UBIC North City, America, CA 94065 Inc. +16506547664
More informationPredictive Coding Helps Companies Reduce Discovery Costs
Predictive Coding Helps Companies Reduce Discovery Costs Recent Court Decisions Open Door to Wider Use by Businesses to Cut Costs in Document Discovery By John Tredennick As companies struggle to manage
More informationPredictive Coding in MultiLanguage EDiscovery
Comprehending the Challenges of Technology Assisted Document Review Predictive Coding in MultiLanguage EDiscovery UBIC North America, Inc. 3 Lagoon Dr., Ste. 180, Redwood City, CA 94065 8773218242
More informationDigital Government Institute. Managing EDiscovery for Government: Integrating Teams and Technology
Digital Government Institute Managing EDiscovery for Government: Integrating Teams and Technology Larry Creech Program Manager Information Catalog Program Corporate Information Security Information Technology
More informationediscovery Policies: Planned Protection Saves More than Money Anticipating and Mitigating the Costs of Litigation
Brought to you by: ediscovery Policies: Planned Protection Saves More than Money Anticipating and Mitigating the Costs of Litigation Introduction: Rising costs of litigation The chance of your organization
More informationPretrial Practice Course Syllabus Spring, 2014 Meeting  Tuesdays 1:303:20pm Room  432(C)
Pretrial Practice Course Syllabus Spring, 2014 Meeting  Tuesdays 1:303:20pm Room  432(C) Professor: Rich Kelsey Telephone: (703) 9938973 Email: rkelsey@gmu.edu Course Materials Material will be assigned
More informationPresenters: Brett Anders, Esq. Joseph J. Lazzarotti, Esq., CIPP/US. Morristown, NJ
Presenters: Brett Anders, Esq. Joseph J. Lazzarotti, Esq., CIPP/US Morristown, NJ 1 Preservation Privacy & Data Security Search & Review 2 PreLitigation Data Map Litigation Hold Procedure Standardized
More informationHow to Manage Costs and Expectations for Successful EDiscovery: Best Practices
How to Manage Costs and Expectations for Successful EDiscovery: Best Practices Mukesh Advani, Esq., Advisory Board Member, UBIC North America, Inc. UBIC North America, Inc. 3 Lagoon Dr., Ste. 180, Redwood
More informationSoftwareassisted document review: An ROI your GC can appreciate. kpmg.com
Softwareassisted document review: An ROI your GC can appreciate kpmg.com b Section or Brochure name Contents Introduction 4 Approach 6 Metrics to compare quality and effectiveness 7 Results 8 Matter 1
More informationEDISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK
EDISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK If your company is involved in civil litigation, the Federal Rules of Civil Procedure regarding preservation and production of electronic documents
More informationTop 10 Best Practices in Predictive Coding
Top 10 Best Practices in Predictive Coding Emerging Best Practice Guidelines for the Conduct of a Predictive Coding Project Equivio internal document " design an appropriate process, including use of available
More informationTechnology Assisted Review 2.0
LITIGATION AND PRACTICE SUPPORT Technology Assisted Review 2.0 by Ignatius Grande of Hughes Hubbard & Reed LLP and Andrew Paredes of Epiq Systems Legal teams and their outside counsel must deal with an
More informationMeasuring Recall in EDiscovery, Part Two: No Easy Answers
Measuring Recall in EDiscovery, Part Two: No Easy Answers John Tredennick In Part One of this article, I introduced readers to statistical problems inherent in proving the level of recall reached in a
More informationIndustry Leading Solutions: Innovative Technology. Quality Results.
Industry Leading Solutions: Innovative Technology. Quality Results. April 10, 2013 emagsolutions.com Agenda Speaker Introduction A Quick Word about emag Importance of Technology Assisted Review (TAR) Key
More informationwww.pwc.nl Review & AI Lessons learned while using Artificial Intelligence April 2013
www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence Why are nonusers staying away from PC? source: edj Group s Q1 2013 Predictive Coding Survey, February 2013, N = 66 Slide 2 Introduction
More informationThree Methods for ediscovery Document Prioritization:
Three Methods for ediscovery Document Prioritization: Comparing and Contrasting Keyword Search with Concept Based and Support Vector Based "Technology Assisted ReviewPredictive Coding" Platforms Tom Groom,
More informationESI: Focus on Review and Production Strategy. Meredith Lee, Online Document Review Supervisor, Paralegal
ESI: Focus on Review and Production Strategy Meredith Lee, Online Document Review Supervisor, Paralegal About Us Avansic Ediscovery and digital forensics company founded in 2004 by Dr. Gavin W. Manes,
More informationMinimizing ediscovery risks. What organizations need to know in today s litigious and digital world.
What organizations need to know in today s litigious and digital world. The main objective for a corporation s law department is to mitigate risk throughout the company, while keeping costs under control.
More informationEDiscovery Tip Sheet
EDiscovery Tip Sheet LegalTech 2015 Some Panels and Briefings Last month I took you on a select tour of the vendor exhibits and products from LegalTech 2015. This month I want to provide a small brief
More informationTHE NEW WORLD OF EDISCOVERY
THE NEW WORLD OF EDISCOVERY Ralph Losey: partner and National ediscovery Counsel of Jackson Lewis LLP, a labor & employment firm with 700 lawyers and 46 offices nationwide. JacksonLewis.com author of
More informationWHITE PAPER: CUSTOMIZE WHITE PAPER: BEST PRACTICES FOR ARCHIVING. Best Practices for Defining and Establishing Effective Archive Retention Policies
WHITE PAPER: CUSTOMIZE WHITE PAPER: BEST PRACTICES FOR ARCHIVING Confidence in a connected world. Best Practices for Defining and Establishing Effective Archive Retention Policies Sponsored by Symantec
More informationXact Data Discovery. Xact Data Discovery. Xact Data Discovery. Xact Data Discovery. ediscovery for DUMMIES LAWYERS. MDLA TTS August 23, 2013
MDLA TTS August 23, 2013 ediscovery for DUMMIES LAWYERS Kate Burke Mortensen, Esq. kburke@xactdatadiscovery.com Scott Polus, Director of Forensic Services spolus@xactdatadiscovery.com 1 Where Do I Start??
More informationA PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE
A PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE Effective February 1, 2010, the Alabama Rules of Civil Procedure were amended to provide for and accommodate
More informationThe Case for Technology Assisted Review and Statistical Sampling in Discovery
The Case for Technology Assisted Review and Statistical Sampling in Discovery Position Paper for DESI VI Workshop, June 8, 2015, ICAIL Conference, San Diego, CA Christopher H Paskach The Claro Group, LLC
More informationPredictive Coding as a Means to Prioritize Review and Reduce Discovery Costs. White Paper
Predictive Coding as a Means to Prioritize Review and Reduce Discovery Costs White Paper INTRODUCTION Computers and the popularity of digital information have changed the way that the world communicates
More informationEDiscovery: A Common Sense Approach. In order to know how to handle and address ESI issues, the preliminary and
Jay E. Heidrick Polsinelli jheidrick@polsinelli.com (913) 2347506 EDiscovery: A Common Sense Approach In order to know how to handle and address ESI issues, the preliminary and obvious question must
More informationTechnologyAssisted Review and Other Discovery Initiatives at the Antitrust Division. Tracy Greer 1 Senior Litigation Counsel EDiscovery
TechnologyAssisted Review and Other Discovery Initiatives at the Antitrust Division Tracy Greer 1 Senior Litigation Counsel EDiscovery The Division has moved to implement several discovery initiatives
More informationCase 1:12cv24356JG Document 404 Entered on FLSD Docket 03/18/2014 Page 1 of 14
Case 1:12cv24356JG Document 404 Entered on FLSD Docket 03/18/2014 Page 1 of 14 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF FLORIDA MIAMI DIVISION CASE NO. 12 24356 CIV GOODMAN PROCAPS S.A., [CONSENT
More informationfor Insurance Claims Professionals
A Practical Guide to Understanding ediscovery for Insurance Claims Professionals ediscovery Defined and its Relationship to an Insurance Claim Simply put, ediscovery (or Electronic Discovery) refers to
More informationUsing Futures Markets to Manage Price Risk for Feeder Cattle (AEC 201301) February 2013
Using Futures Markets to Manage Price Risk for Feeder Cattle (AEC 201301) February 2013 Kenny Burdine 1 Introduction: Price volatility in feeder cattle markets has greatly increased since 2007. While
More informationSTATISTICAL ANALYSIS AND INTERPRETATION OF DATA COMMONLY USED IN EMPLOYMENT LAW LITIGATION
STATISTICAL ANALYSIS AND INTERPRETATION OF DATA COMMONLY USED IN EMPLOYMENT LAW LITIGATION C. Paul Wazzan Kenneth D. Sulzer ABSTRACT In employment law litigation, statistical analysis of data from surveys,
More informationEDiscovery in Michigan. Presented by Angela Boufford
EDiscovery in Michigan ESI Presented by Angela Boufford DISCLAIMER: This is by no means a comprehensive examination of EDiscovery issues. You will not be an EDiscovery expert after this presentation.
More informationEDiscovery in Employment Litigation: Making Practical, Yet Defensible Decisions
EDiscovery in Employment Litigation: Making Practical, Yet Defensible Decisions 11 EDiscovery in Employment Litigation: Making Practical, Yet Defensible Decisions Introduction Much has been said about
More informationThe Random Sampling Road to Reasonableness. Reduce Risk and Cost by Employing a Complete and Integrated Validation Process
The Random Sampling Road to Reasonableness Reduce Risk and Cost by Employing a Complete and Integrated Validation Process By: Michael R. Wade Planet Data Executive Vice President Chief Technology Officer
More informationEDiscovery for Backup Tapes. How Technology Is Easing the Burden
EDiscovery for Backup Tapes How Technology Is Easing the Burden TABLE OF CONTENTS Introduction...3 The Importance and Challenge of Backup Tapes for Electronic Discovery...3 TAPES AS A SOURCE OF ESI...
More informationDiscovery Data Management
Discovery Data Management in Practice Introductions Reveal Derick Roselli 9492803519 droselli@revealdata.com www.revealdata.com Program Outline PrePlanning Stages Workflow Management Project Management
More informationMaking reviews more consistent and efficient.
Making reviews more consistent and efficient. PREDICTIVE CODING AND ADVANCED ANALYTICS Predictive coding although yet to take hold with the enthusiasm initially anticipated is still considered by many
More informationPredictability in EDiscovery
Predictability in EDiscovery Presented by: John G. Roman, Jr. National Manager, Practice Group Technology Services Nixon Peabody LLP Tom Barce Assistant Director of Practice Support Fulbright & Jaworski
More informationPros And Cons Of ComputerAssisted Review
Portfolio Media. Inc. 860 Broadway, 6th Floor New York, NY 10003 www.law360.com Phone: +1 646 783 7100 Fax: +1 646 783 7161 customerservice@law360.com Pros And Cons Of ComputerAssisted Review Law360,
More informationPRESENTED BY: Sponsored by:
PRESENTED BY: Sponsored by: Practical Uses of Analytics in EDiscovery  A PRIMER Jenny Le, Esq. Vice President of Discovery Services jle@evolvediscovery.com EDiscovery & Ethics Structured, Conceptual,
More informationEDiscovery Defensibility ViewS from the Bench page : 1
EDiscovery Defensibility ViewS from the Bench page : 1 EDiscovery Defensibility Views from the Bench A Clearwell White Paper EDiscovery Defensibility ViewS from the Bench page : 2 Table of Contents
More informationPredictive Coding: Emerging E Discovery Tool Leveraging E Discovery Computer Assisted Review to Reduce Time and Expense of Discovery
Presenting a live 90 minute webinar with interactive Q&A Predictive Coding: Emerging E Discovery Tool Leveraging E Discovery Computer Assisted Review to Reduce Time and Expense of Discovery WEDNESDAY,
More informationediscovery Defensibility
Content Introduction... 3 The Duty to Preserve Phillip M. Adams & Assoc., LLC v. Dell, Inc.... 4 Case Facts.... 4 Key Takeaways... 4 Keyword Search Victor Stanley, Inc. v. Creative Pipe, Inc..... 5 Case
More informationIntroduction to Predictive Coding
Introduction to Predictive Coding Herbert L. Roitblat, Ph.D. CTO, Chief Scientist, OrcaTec Predictive coding uses computers and machine learning to reduce the number of documents in large document sets
More informationEnCase ediscovery. Automatically search, identify, collect, preserve, and process electronically stored information across the network.
TM GUIDANCE SOFTWARE EnCASE ediscovery EnCase ediscovery Automatically search, identify, collect, preserve, and process electronically stored information across the network. GUIDANCE SOFTWARE EnCASE ediscovery
More informationREDUCING COSTS WITH ADVANCED REVIEW STRATEGIES  PRIORITIZATION FOR 100% REVIEW. Bill Tolson Sr. Product Marketing Manager Recommind Inc.
REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES  Bill Tolson Sr. Product Marketing Manager Recommind Inc. Introduction... 3 Traditional Linear Review... 3 Advanced Review Strategies: A Typical Predictive
More informationElectronic Discovery: Litigation Holds, Data Preservation and Production
Electronic Discovery: Litigation Holds, Data Preservation and Production April 27, 2010 Daniel Munsch, Assistant General Counsel John Lerchey, Coordinator for Incident Response 0 EDiscovery Rules Federal
More informationEDiscovery Getting a Handle on Predictive Coding
EDiscovery Getting a Handle on Predictive Coding John J. Jablonski Goldberg Segalla LLP 665 Main St Ste 400 Buffalo, NY 142031425 (716) 5665400 jjablonski@goldbergsegalla.com Drew Lewis Recommind 7028
More informationTraditionally, the gold standard for identifying potentially
istockphoto.com/alexandercreative Predictive Coding: It s Here to Stay Predictive coding programs are poised to become a standard practice in ediscovery in the near future. As more courts weigh in on
More informationTHE FEDERAL COURTS LAW REVIEW. Comments on The Implications of Rule 26(g) on the Use of TechnologyAssisted Review
THE FEDERAL COURTS LAW REVIEW Volume 7, Issue 1 2014 Comments on The Implications of Rule 26(g) on the Use of TechnologyAssisted Review Maura R. Grossman and Gordon V. Cormack ABSTRACT Approaches to technologyassisted
More informationWhen EDiscovery Becomes Evidence
Monday, June 11, 2007 When EDiscovery Becomes Evidence Make sure that you can easily authenticate the information that was so costly to produce By Leonard Deutchman Special to the Law Weekly A federal
More informationCase 1:11cv01279ALCAJP Document 96 Filed 02/24/12 Page 1 of 49
Case 1:11cv01279ALCAJP Document 96 Filed 02/24/12 Page 1 of 49 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK                                      
More informationCase 2:11cv00678LRHPAL Document 174 Filed 07/18/14 Page 1 of 18 UNITED STATES DISTRICT COURT DISTRICT OF NEVADA * * * Plaintiff, Defendants.
Case :cv00lrhpal Document Filed 0// Page of 0 PROGRESSIVE CASUALTY INSURANCE COMPANY, v. JACKIE K. DELANEY, et al., UNITED STATES DISTRICT COURT DISTRICT OF NEVADA Plaintiff, Defendants. * * * Case
More informationProactively Using Information Governance and Advance Planning to Reduce the Burden and Expense of EDiscovery
KNOW THYSELF: Proactively Using Information Governance and Advance Planning to Reduce the Burden and Expense of EDiscovery Jonathan D. Rudolph, General Counsel of Accumen Data Services Jeffrey D. Bukowski,
More informationPredictive Coding in UK Civil Litigation. White Paper by Chris Dale of the edisclosure Information Project
Predictive Coding in UK Civil Litigation White Paper by Chris Dale of the edisclosure Information Project THE PURPOSE OF THIS PAPER This paper considers the use of technology known as Predictive Coding
More information