A Practitioner s Guide to Statistical Sampling in E-Discovery. October 16, 2012
|
|
- Egbert Mills
- 8 years ago
- Views:
Transcription
1 A Practitioner s Guide to Statistical Sampling in E-Discovery October 16,
2 Meet the Panelists Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Gordon V. Cormack, Professor at the David R. Cheriton School of Computer Science at the University of Waterloo Jim Wagner, Co-founder and CEO of DiscoverReady Maureen O Neill, SVP, Marketplace Leader at DiscoverReady 2
3 Agenda What is statistical sampling? Why should a practitioner use statistical sampling in e-discovery? Opportunities to use statistical sampling The basics of statistical sampling An example of statistical sampling in the e-discovery context Key decisions when using statistical sampling Take-away recommendations on using statistical sampling 3
4 What is Statistical Sampling? In general, statistical sampling is a method to estimate a characteristic of a large population by examining only a subset of it. In the specific context of e-discovery: Estimate is a reasonably precise mathematical measurement. Characteristic refers to the number (or proportion) of items in the document population having a certain property, such as responsiveness or privilege. Population is a collection of electronic documents. Subset is a small but representative sample of the document population (chosen at random). Note that the size of the subset the sample size determines the precision of the estimate. Sample size is dependent on the acceptable margin of error, the desired confidence level, and to a negligible extent, the size of the population (these variables will be discussed later in the webinar). 4
5 What is Statistical Sampling? Judgmental sampling can be very useful in e-discovery, but it is not what we are discussing in today s webinar Statistical sampling is not the same thing as judgmental sampling Judgmental sampling is not random It involves a selection of items for the subset using some degree of human judgment When sampling is judgmental, inferences cannot be drawn about the population based on an examination of the subset Judgmental sampling is akin to spot-checking 5
6 Why Use Statistical Sampling? Thoughtful use of statistical sampling can improve the quality, efficiency, and defensibility of e-discovery efforts. Quality By counting or measuring the inputs and outputs of an e- discovery process, we can work to improve the process to make it more accurate. For example: If we find that a proposed search term is bringing in too many false positives (i.e., poor precision ) we can try a different search term and test the results using sampling. If statistical sampling confirms that the new term reduces the number of irrelevant documents (better precision), but is not under-inclusive (the recall is good), we have improved the search process. 6
7 Why Use Statistical Sampling? Efficiency Sampling saves time and money by allowing us to count or measure things more efficiently. Statistical sampling is scalable; even with extremely large populations, we can use relatively small samples. For example, if we want to estimate the proportion of relevant documents (the richness or prevalence ) in a collection of one million documents, an appropriate sample size might be 600 documents. If we want to estimate the proportion of relevant documents in a collection of a billion documents, the appropriate sample size would remain the same. 7
8 Why Use Statistical Sampling? Although its use in e-discovery is not yet widespread, we are moving in a direction where it soon will be considered best practice. Defensibility Courts are increasingly requiring the use of sampling as part of a reasonable discovery process. In re Seroquel Prods. Liabil. Litig., 244 F.R.D. 650 (M.D. Fla. 2007) Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251 (D. Md. 2008) William. A. Gross Constr. Assocs. Inc. v. American Mfrs. Mut. Ins. Co., 256 F.R.D. 134 (S.D.N.Y. 2009) Mt. Hawley Ins. Co. v. Felman Prod., Inc., 2010 WL (S.D. W. Va. May 18, 2010) DaSilva Moore v. Publicis Group, No. 11 Civ (S.D.N.Y. Feb. 24, 2012) (Peck, M.J.), aff d (S.D.N.Y. Apr. 26, 2012) (Carter, D.J.) In re: Actos, MDL No. 6:11-md-2299 (W.D. La. Jul. 27, 2012) 8
9 Opportunities to Use Statistical Sampling Incorporate into early case assessment and strategy development Efficiently hone in on the sources and custodians of information likely to be relevant Assess the burdens and costs involved in accessing certain information, such as backup tapes or other offline media Gauge the richness of populations before embarking on review (Later in the program, we will take a step-by-step walk through this use of sampling, as an example of how sampling is performed) Test the culling of a data set to ensure that your cull is neither over-broad nor too restrictive 9
10 Opportunities to Use Statistical Sampling Measure the efficacy of search terms and refine the terms Measure the accuracy of a predictive coding process Test automated methods of screening documents for privilege and confidentiality Sample a document production before it goes out the door to provide additional assurance that privileged content is not inadvertently included 10
11 Opportunities to Use Statistical Sampling Support proportionality arguments Determine whether the cost of reviewing certain types of ESI is reasonable and proportional Is there bang for the buck in reviewing a particular set of documents based on how many responsive documents are estimated to be found? For example: Should we spend money on reviewing this custodian s documents if the collection has very low prevalence? Is it worth the expense to continue reviewing more documents from more custodians if that additional effort is not likely to yield significantly more relevant information? 11
12 Opportunities to Use Statistical Sampling Conduct quality control and quality assurance of human review efforts Measure the error rate on document review decisions for an overall project, or for particular reviewers When done in real time, as the project progresses, error rate measurement can be part of an effective quality control ( QC ) workflow When done at the conclusion of the project, or a phase of the project, the measurement becomes part of the quality assurance ( QA ) and defensibility of process 12
13 Opportunities to Use Statistical Sampling Important caveat about taking statistical measurements using human decisions as a reference point or gold standard Human decisions about documents inevitably involve an element of subjectivity, and even the best decision-makers will make mistakes. (This is true for all decisions, whether in the original review, a QC review, or in a sampling review.) Even the gold standard decisions are not going to be 100% consistent or correct. This element of human error, or legitimate differences of opinion, will always introduce some degree of measurement error into a statistical measurement involving human decisionmaking. Therefore, statistical measurement cannot be more accurate than human judgment permits. 13
14 The Basics of Statistical Sampling: Drawing a Random Sample What is a random sample and how is it generated? A random sample is a subset of documents that is chosen at random from a larger population of interest. Choosing at random means that every document in the collection has an equal chance of being selected in the sample. Random sampling can be achieved a number of different ways: Drawing numbers from a hat Using a computerized random-number generator Choosing documents based on one or more digits from their hash value Many e-discovery tools have built-in random sample generators 14
15 The Basics of Statistical Sampling: Drawing a Random Sample A random sample has been drawn now what? Review the sample and count the number of documents with the characteristic of interest (e.g., responsiveness). The proportion of responsive documents in the sample is calculated by dividing the number of responsive documents in the sample by the total number of documents in the sample. Because the sample is random, we can extrapolate that the proportion of responsive documents in the population is approximately the same as the proportion in the sample. When we say approximately the same, we mean that there is a margin of error in our estimate we ll explain margin of error in a moment. 15
16 The Basics of Statistical Sampling: Drawing a Random Sample Suppose we have a collection of one million documents And we want to estimate how many of them are relevant for the purposes of building a budget and timeline for review and production. Reviewing all one million documents for the purposes of budgeting and project planning is infeasible. So, instead, we take a random sample of 1,000 documents. We review the sample and find that 300 documents are relevant. The proportion of relevant documents in the sample is 300/1,000, or 30%. Therefore, the proportion of relevant documents in the collection is estimated to be approximately 30% (or 300,000 documents). If we repeated this process, we would get a slightly different estimate each time, but in general, each estimate would be close to the actual proportion. 16
17 The Basics of Statistical Sampling: Margin of Error / Confidence Interval The margin of error is a way of expressing a range above and below the estimate that is likely to contain the actual value. In our example: We determined that the proportion of relevant documents in the sample was 30%, and we extrapolated that the proportion of relevant documents in the collection was approximately, but not exactly, 30%. The exact proportion of relevant documents in the collection is unknown, but is likely to fall within a margin of error of +/- 3%. We express this by saying that that the proportion of relevant documents in the collection is estimated to be 30%, plus or minus 3%. 17
18 The Basics of Statistical Sampling: Margin of Error / Confidence Interval An alternative way of stating the estimate is by using a confidence interval instead of margin of error. The confidence interval is the range of values that is likely to contain the actual value. In our previous example, we would state that the proportion of relevant documents is likely to fall within a range of 27% to 33%. As compared to the margin of error, the confidence interval does not have to be exactly symmetrical around the estimate, and can therefore be a more precise way of expressing the uncertainty of the estimate. 18
19 The Basics of Statistical Sampling: Confidence Level What do we mean when we say the confidence interval is likely to contain the actual value? The confidence level is the probability that the confidence interval would contain the actual value if the sampling process were to be repeated a large number of times. For example: If the confidence level is 95%, it means that there is a 95% chance that the actual value is within the confidence interval. (In our previous example, we would say with a 95% confidence, that the proportion of relevant documents falls between 27% and 33%. ) If the confidence level is 99%, it means that there is a 99% chance that the actual value is within the confidence interval. (In our previous example, we would say with a 99% confidence, that the proportion of relevant documents falls between 26% and 34%. ) 19
20 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size The three concepts of confidence level, confidence interval (or margin of error), and sample size are interrelated. Generally speaking, if the confidence level remains constant, as the sample size goes up, the margin of error goes down. Similarly, to increase the confidence level, either the sample size or the margin of error must increase. Finally, if you want to decrease your margin of error, either the confidence level must come down or the sample size must go up. Bear in mind that these relationships are not proportional: Getting a smaller confidence interval, or a higher confidence level, may require drawing a much larger sample. To illustrate this, consider again the previous example, and assume that we find the proportion of relevant documents is 30% in each sample. The following slide depicts the relationship between sample size, confidence level, and margin of error: 20
21 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size Margin of Error Confidence Level Sample Size % 99% 4, % 99% 1, % 99% 500 0% 30% 100% Proportion of Relevant documents 21
22 The Basics of Statistical Sampling: The Relationship Between Confidence Level, Margin of Error, and Sample Size Here s another illustration that also demonstrates how taking relatively small samples can help us understand very large populations. We have a collection of one million documents; we assume the proportion of relevant documents is 50% and we want a 95% confidence level for our estimate. Here are some examples of the (pretty good) margins of error we will achieve with some relatively small sample sizes: Sample Size Margin of Error % % % 22
23 The Basics of Statistical Sampling: The Tradeoffs In general, a higher confidence level is better. But that comes at the price of a larger sample size, and/or A wider confidence interval (i.e., a higher margin of error). Likewise, a smaller margin of error generally is better, because it reflects a more precise estimate. But that requires a lower confidence level (less certainty), and/or A larger sample size (higher cost and/or less efficiency) There are calculators available on the Internet that allow you to plug in the variables and make computations of sample size, margin of error, and confidence level, but be careful when choosing one. Make sure you understand the assumptions each one uses or if you don t, get help from someone who does! 23
24 The Basics of Statistical Sampling: Standards for E-Discovery? Is there a minimum acceptable confidence level and/or margin of error when using statistical sampling in e-discovery? There are no bright line rules regarding confidence level or margin of error 95% confidence level is commonly used in statistical measurement The acceptable margin of error will depend on the consequences of an inaccurate estimate The operative standard is one of reasonableness 24
25 The Basics of Statistical Sampling: Standards for E-Discovery? Is there a minimum acceptable confidence level and/or margin of error when using statistical sampling in e-discovery? Every matter is different, and what is reasonable in one matter may not be in another Factors affecting the reasonableness calculus include: The cost of greater precision in measurement as compared to the amount at stake and the importance of the matter (proportionality) The purpose for which the sampling is being performed The time and resources available for sampling 25
26 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Determine the total number of documents in the collection. We ll call that number N (e.g., N = 1,000,000 documents). Choose the desired confidence level (e.g., 95%), and margin of error (e.g., +/- 2%). Determine the appropriate sample size (we ll call that n ) using an appropriate calculator (e.g., n = 2,395). 26
27 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Select 2,395 of the documents, at random, from the collection. 27
28 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents Not Responsive 1,695 documents Count the number of responsive documents in the sample (say, 700). 28
29 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents The proportion of responsive documents in the sample is calculated by dividing the number of responsive documents in the sample (700) by n (2,395): 700/2,395 = 29.2% 29
30 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents The confidence interval is 29.2% (the proportion of responsive documents) plus or minus 2% (the margin of error): 27.2% to 31.2% 30
31 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents To calculate the number of responsive documents in the collection, multiply the two confidence limits by N (27.2% x 1,000,000 = 272,000 responsive documents; 31.2% x 1,000,000 = 312,000 responsive documents). 31
32 Examples of Statistical Sampling: Example 1 Collection 1,000,000 documents How many responsive documents are likely to be found in the document collection? Select documents at random Sample 2,395 documents Review Documents Responsive 700 documents In this example, we can state that, with 95% confidence, we estimate there are between 272,000 and 312,000 responsive documents in this collection. 32
33 Examples of Statistical Sampling: Example 2 How many responsive documents are likely to be found in the document collection? 1. Let s tweak Example 1 slightly, and assume that the number of responsive documents in the reviewed sample is only The proportion of responsive documents in this example is 3/2,395 = 0.13%, and with a confidence interval of -1.87% to +2.13% (-18,700 to 21,300 relevant documents). 3. Obviously it is not possible to have a negative number of relevant documents in the collection. This scenario illustrates the problem faced when sampling populations with a low proportion of relevant documents (i.e., low prevalence or low richness ). 4. When sampling such populations, a normal sample size calculation of confidence intervals and margin of error is not precise enough. When dealing with a low prevalence population, a more precise confidence interval can be computed using a binomial confidence interval calculator. 33
34 Examples of Statistical Sampling: Example 2 How many responsive documents are likely to be found in a collection of documents? 5. Here s an example of a binomial confidence interval calculator: 6. If we take our sample of 2,395 documents and re-compute the confidence intervals at the 95% confidence level using the binomial confidence interval calculator, if there are three responsive documents in the sample, the confidence interval is 0.03% to 0.37% (300 to 3,700 documents). 7. This is an example of how the confidence interval may not be symmetrical around the estimate. 34
35 Using Statistical Sampling: Key Decisions a Practitioner Must Make What are you trying to measure? What are trying to accomplish with the measurement? Are you sampling for internal purposes (to improve process) or to defend your process to opposing counsel or the court? Do you need to hire an expert in statistics? What are your acceptable levels of recall, precision, etc.? What is your desired confidence level? How wide a confidence interval will you accept? How large a sample size is feasible under the circumstances of your matter, taking into account cost and timing considerations? How will you document and report your measurements and methodology? What aspects of your statistical sampling efforts are you willing to share with the opposing party? With the court? 35
36 Take-Away Recommendations on Using Statistical Sampling Statistical sampling can be tricky most lawyers do not have sufficient understanding of the concepts (or the math) to go it alone and get it right. Practitioners should use an expert or statistical tool to: Determine whether statistical sampling is an appropriate tool for the matter Determine which statistical measures are best for the situation (precision, recall, elusion, accuracy, etc.) Determine the desired confidence level and margin of error Calculate the proper sample size to achieve this confidence level and margin of error Draw a random sample of the appropriate size Compute the estimate 36
37 Question & Answer Session To contact the panelists: Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Jim Wagner, Co-founder and CEO of DiscoverReady Gordon V. Cormack, Professor at the David R. Cheriton School of Computer Science at the University of Waterloo Maureen O Neill, SVP, Marketplace Leader at DiscoverReady maureen.oneill@discoverready.com 37
E-Discovery in Mass Torts:
E-Discovery in Mass Torts: Predictive Coding Friend or Foe? Sherry A. Knutson Sidley Austin One S Dearborn St 32nd Fl Chicago, IL 60603 (312) 853-4710 sknutson@sidley.com Sherry A. Knutson is a partner
More informationThe Evolution, Uses, and Case Studies of Technology Assisted Review
FEBRUARY 4 6, 2014 / THE HILTON NEW YORK The Evolution, Uses, and Case Studies of Technology Assisted Review One Size Does Not Fit All #LTNY Meet Our Panelists The Honorable Dave Waxse U.S. Magistrate
More informationThe Benefits of. in E-Discovery. How Smart Sampling Can Help Attorneys Reduce Document Review Costs. A white paper from
The Benefits of Sampling in E-Discovery How Smart Sampling Can Help Attorneys Reduce Document Review Costs A white paper from 615.255.5343 dsi.co 414 Union Street, Suite 1210 Nashville, TN 37219-1771 Table
More informationPredictive Coding Defensibility
Predictive Coding Defensibility Who should read this paper The Veritas ediscovery Platform facilitates a quality control workflow that incorporates statistically sound sampling practices developed in conjunction
More informationE-discovery Taking Predictive Coding Out of the Black Box
E-discovery Taking Predictive Coding Out of the Black Box Joseph H. Looby Senior Managing Director FTI TECHNOLOGY IN CASES OF COMMERCIAL LITIGATION, the process of discovery can place a huge burden on
More informationPower-Up Your Privilege Review: Protecting Privileged Materials in Ediscovery
Power-Up Your Privilege Review: Protecting Privileged Materials in Ediscovery Jeff Schomig, WilmerHale Stuart Altman, Hogan Lovells Joe White, Kroll Ontrack Sheldon Noel, Kroll Ontrack (moderator) April
More informationTechnology Assisted Review of Documents
Ashish Prasad, Esq. Noah Miller, Esq. Joshua C. Garbarino, Esq. October 27, 2014 Table of Contents Introduction... 3 What is TAR?... 3 TAR Workflows and Roles... 3 Predictive Coding Workflows... 4 Conclusion...
More informationApplication of Simple Random Sampling 1 (SRS) in ediscovery
Manuscript submitted to the Organizing Committee of the Fourth DESI Workshop on Setting Standards for Electronically Stored Information in Discovery Proceedings on April 20, 2011. Updated May 18, 2011.
More informationSAMPLING: MAKING ELECTRONIC DISCOVERY MORE COST EFFECTIVE
SAMPLING: MAKING ELECTRONIC DISCOVERY MORE COST EFFECTIVE Milton Luoma Metropolitan State University 700 East Seventh Street St. Paul, Minnesota 55337 651 793-1246 (fax) 651 793-1481 Milt.Luoma@metrostate.edu
More informationRecent Developments in the Law & Technology Relating to Predictive Coding
Recent Developments in the Law & Technology Relating to Predictive Coding Presented by Paul Neale CEO Presented by Gene Klimov VP & Managing Director Presented by Gerard Britton Managing Director 2012
More informationThe Truth About Predictive Coding: Getting Beyond The Hype
www.encase.com/ceic The Truth About Predictive Coding: Getting Beyond The Hype David R. Cohen Reed Smith LLP Records & E-Discovery Practice Group Leader David leads a group of more than 100 lawyers in
More informationPredictive Coding: A Primer
MEALEY S TM LITIGATION REPORT Discovery Predictive Coding: A Primer by Amy Jane Longo, Esq. and Usama Kahf, Esq. O Melveny & Myers LLP Los Angeles, California A commentary article reprinted from the March
More informationMANAGING BIG DATA IN LITIGATION
David Han 2015 MANAGING BIG DATA IN LITIGATION DAVID HAN Associate, Morgan Lewis & Bockius, edata Practice Group MANAGING BIG DATA Data volumes always increasing New data sources Mobile Internet of Things
More informationESI and Predictive Coding
Beijing Boston Brussels Chicago Frankfurt Hong Kong ESI and Predictive Coding Houston London Los Angeles Moscow Munich New York Palo Alto Paris São Paulo Charles W. Schwartz Chris Wycliff December 13,
More informationNovember/December 2010 THE MAGAZINE OF THE AMERICAN INNS OF COURT. rofessionalism. Ethics Issues. and. Today s. Technology. www.innsofcourt.
November/December 2010 THE MAGAZINE OF THE AMERICAN INNS OF COURT rofessionalism and Ethics Issues in Today s Technology www.innsofcourt.org Transparency in E-Discovery: No Longer a Novel Approach By Michael
More informationTechnology Assisted Review: The Disclosure of Training Sets and Related Transparency Issues Whitney Street, Esq. 1
Technology Assisted Review: The Disclosure of Training Sets and Related Transparency Issues Whitney Street, Esq. 1 The potential cost savings and increase in accuracy afforded by technology assisted review
More informationThe United States Law Week
The United States Law Week Source: U.S. Law Week: News Archive > 2012 > 04/24/2012 > BNA Insights > Under Fire: A Closer Look at Technology- Assisted Document Review E-DISCOVERY Under Fire: A Closer Look
More informationQuality Control for predictive coding in ediscovery. kpmg.com
Quality Control for predictive coding in ediscovery kpmg.com Advances in technology are changing the way organizations perform ediscovery. Most notably, predictive coding, or technology assisted review,
More informationJudge Peck Provides a Primer on Computer-Assisted Review By John Tredennick
By John Tredennick CEO Catalyst Repository Systems Magistrate Judge Andrew J. Peck issued a landmark decision in Da Silva Moore v. Publicis and MSL Group, filed on Feb. 24, 2012. This decision made headlines
More informationMastering Predictive Coding: The Ultimate Guide
Mastering Predictive Coding: The Ultimate Guide Key considerations and best practices to help you increase ediscovery efficiencies and save money with predictive coding 4.5 Validating the Results and Producing
More informationLegal Arguments & Response Strategies for E-Discovery
Legal Arguments & Response Strategies for E-Discovery The tools to craft strategic discovery requests & mitigate the risks and burdens of production. Discussion Outline Part I Strategies for Requesting
More informationDocument Review Costs
Predictive Coding Gain Earlier Insight and Reduce Document Review Costs Tom Groom Vice President, Discovery Engineering tgroom@d4discovery.com 303.840.3601 D4 LLC Litigation support service provider since
More informationPredictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review
Predictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review ACEDS Webinar April 23, 2014 Sponsored by Robert Half Legal 1 2014 Robert Half Legal. An Equal Opportunity
More informationTHE PREDICTIVE CODING CASES A CASE LAW REVIEW
THE PREDICTIVE CODING CASES A CASE LAW REVIEW WELCOME Thank you for joining Numerous diverse attendees Please feel free to submit questions Slides, recording and survey coming tomorrow SPEAKERS Matthew
More informationThe Predictive Coding Soundtrack: Rewind, Play, Fast-Forward
The Predictive Coding Soundtrack: Rewind, Play, Fast-Forward LEGALTECH NEW YORK February 3, 2015 Moderator: Amy Hinzmann Senior Vice President, DiscoverReady DiscoverReady 2014 THE PANELISTS* Marla Bergman
More informationCOURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Fall 2014
COURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Professors:Mark Austrian Christopher Racich Fall 2014 Introduction The ubiquitous use of computers, the
More informationTECHNOLOGY-ASSISTED DOCUMENT REVIEW: IS IT DEFENSIBLE?
TECHNOLOGY-ASSISTED DOCUMENT REVIEW: IS IT DEFENSIBLE? By William W. Belt, Dennis R. Kiker and Daryl E. Shetterly* Cite as: William W. Belt, Dennis R. Kiker & Daryl E. Shetterly, Technology-Assisted Document
More informationPr a c t i c a l Litigator s Br i e f Gu i d e t o Eva l u at i n g Ea r ly Ca s e
Ba k e Offs, De m o s & Kicking t h e Ti r e s: A Pr a c t i c a l Litigator s Br i e f Gu i d e t o Eva l u at i n g Ea r ly Ca s e Assessment So f t wa r e & Search & Review Tools Ronni D. Solomon, King
More informationCost-Effective and Defensible Technology Assisted Review
WHITE PAPER: SYMANTEC TRANSPARENT PREDICTIVE CODING Symantec Transparent Predictive Coding Cost-Effective and Defensible Technology Assisted Review Who should read this paper Predictive coding is one of
More informationThe State Of Predictive Coding
MEALEY S TM LITIGATION REPORT Discovery The State Of Predictive Coding by Royce F. Cohen and Derek I.A. Silverman Stroock & Stroock & Lavan LLP New York A commentary article reprinted from the September
More informationWhite Paper Technology Assisted Review. Allison Stanfield and Jeff Jarrett 25 February 2015. 1300 136 993 www.elaw.com.au
White Paper Technology Assisted Review Allison Stanfield and Jeff Jarrett 25 February 2015 1300 136 993 www.elaw.com.au Table of Contents 1. INTRODUCTION 3 2. KEYWORD SEARCHING 3 3. KEYWORD SEARCHES: THE
More informationNavigating Information Governance and ediscovery
Navigating Information Governance and ediscovery Implementing Processes & Technology to Reduce Downstream ediscovery Cost and Risk Shannon Smith General Counsel, Globanet March 11 12, 2013 Agenda 1 Overview
More informationPredictive Coding Defensibility and the Transparent Predictive Coding Workflow
WHITE PAPER: PREDICTIVE CODING DEFENSIBILITY........................................ Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive
More informationEthics and ediscovery
Ethics and ediscovery John Mansfield and Devon Newman January 6, 2012 1 2013, MansfieldLaw ediscovery basics We will cover: Preservation and spoliation Searching and producing documents Supervising lawyers
More informationJason R. Baron Director of Litigation Office of General Counsel National Archives and Records Administration
NARA RACO 2010 Ronald Reagan Bldg., Washington, D.C. May 12, 2010 Jason R. Baron Director of Litigation Office of General Counsel National Archives and Records Administration Email is still the 800 lb.
More informationPredictive Coding Defensibility and the Transparent Predictive Coding Workflow
Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive coding is one of the most promising technologies to reduce the high cost of review by
More informationE-DISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK
E-DISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK If your company is involved in civil litigation, the Federal Rules of Civil Procedure regarding preservation and production of electronic documents
More informationMeasurement in ediscovery
Measurement in ediscovery A Technical White Paper Herbert Roitblat, Ph.D. CTO, Chief Scientist Measurement in ediscovery From an information-science perspective, ediscovery is about separating the responsive
More information2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500
2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500 5218 South East Street, Suite E-3, Indianapolis, IN 46227 Tel 317.247.4400 Fax 317.247.0044 Presented by Providing
More informationAn Open Look at Keyword Search vs. Predictive Analytics
877.557.4273 catalystsecure.com ARTICLE An Open Look at Keyword Search vs. Can Keyword Search Be As Effective as TAR? John Tredennick, Esq. Founder and CEO, Catalyst Repository Systems 2015 Catalyst Repository
More informationPretrial Practice Course Syllabus Spring, 2014 Meeting -- Tuesdays 1:30-3:20pm Room -- 432(C)
Pretrial Practice Course Syllabus Spring, 2014 Meeting -- Tuesdays 1:30-3:20pm Room -- 432(C) Professor: Rich Kelsey Telephone: (703) 993-8973 Email: rkelsey@gmu.edu Course Materials Material will be assigned
More informationPresenters: Brett Anders, Esq. Joseph J. Lazzarotti, Esq., CIPP/US. Morristown, NJ
Presenters: Brett Anders, Esq. Joseph J. Lazzarotti, Esq., CIPP/US Morristown, NJ 1 Preservation Privacy & Data Security Search & Review 2 Pre-Litigation Data Map Litigation Hold Procedure Standardized
More informationLitigation Solutions insightful interactive culling distributed ediscovery processing powering digital review
Litigation Solutions i n s i g h t f u l i n t e r a c t i ve c u l l i n g d i s t r i b u t e d e d i s cove r y p ro ce s s i n g p owe r i n g d i g i t a l re v i e w Advanced Analytical Review Data
More informationHow to Manage Costs and Expectations for Successful E-Discovery: Best Practices
How to Manage Costs and Expectations for Successful E-Discovery: Best Practices Mukesh Advani, Esq., Advisory Board Member, UBIC North America, Inc. UBIC North America, Inc. 3 Lagoon Dr., Ste. 180, Redwood
More informationPREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX?
Vol. 46 No. 3 February 6, 2013 PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX? The high costs of e-discovery have led to the development of computerized review technology by which the user may search
More informationThe Tested Effectiveness of Equivio>Relevance in Technology Assisted Review
ediscovery & Information Management White Paper The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review Scott M. Cohen Elizabeth T. Timkovich John J. Rosenthal February 2014 2014 Winston
More informationHow To Write A Document Review
Comprehending the Challenges of Technology Assisted Document Review Predictive Coding in Multi-Language E-Discovery 3 Lagoon Dr., Ste.180, Redwood UBIC North City, America, CA 94065 Inc. +1-650-654-7664
More informationPredictive Coding in Multi-Language E-Discovery
Comprehending the Challenges of Technology Assisted Document Review Predictive Coding in Multi-Language E-Discovery UBIC North America, Inc. 3 Lagoon Dr., Ste. 180, Redwood City, CA 94065 877-321-8242
More informationSoftware-assisted document review: An ROI your GC can appreciate. kpmg.com
Software-assisted document review: An ROI your GC can appreciate kpmg.com b Section or Brochure name Contents Introduction 4 Approach 6 Metrics to compare quality and effectiveness 7 Results 8 Matter 1
More informationHow Good is Your Predictive Coding Poker Face?
How Good is Your Predictive Coding Poker Face? SESSION ID: LAW-W03 Moderator: Panelists: Matthew Nelson ediscovery Counsel Symantec Corporation Hon. Andrew J. Peck US Magistrate Judge Southern District
More informationcase 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION
case 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION IN RE: BIOMET M2a MAGNUM HIP IMPLANT PRODUCTS LIABILITY
More informationPredictive Coding Helps Companies Reduce Discovery Costs
Predictive Coding Helps Companies Reduce Discovery Costs Recent Court Decisions Open Door to Wider Use by Businesses to Cut Costs in Document Discovery By John Tredennick As companies struggle to manage
More informationTechnology- Assisted Review 2.0
LITIGATION AND PRACTICE SUPPORT Technology- Assisted Review 2.0 by Ignatius Grande of Hughes Hubbard & Reed LLP and Andrew Paredes of Epiq Systems Legal teams and their outside counsel must deal with an
More informationThree Methods for ediscovery Document Prioritization:
Three Methods for ediscovery Document Prioritization: Comparing and Contrasting Keyword Search with Concept Based and Support Vector Based "Technology Assisted Review-Predictive Coding" Platforms Tom Groom,
More informationThe case for statistical sampling in e-discovery
Forensic The case for statistical sampling in e-discovery January 2012 kpmg.com 2 The case for statistical sampling in e-discovery The sheer volume and unrelenting production deadlines of today s electronic
More informationDigital Government Institute. Managing E-Discovery for Government: Integrating Teams and Technology
Digital Government Institute Managing E-Discovery for Government: Integrating Teams and Technology Larry Creech Program Manager Information Catalog Program Corporate Information Security Information Technology
More informationTHE NEW WORLD OF E-DISCOVERY
THE NEW WORLD OF E-DISCOVERY Ralph Losey: partner and National e-discovery Counsel of Jackson Lewis LLP, a labor & employment firm with 700 lawyers and 46 offices nationwide. JacksonLewis.com author of
More informationA PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE
A PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE Effective February 1, 2010, the Alabama Rules of Civil Procedure were amended to provide for and accommodate
More informationPRESENTED BY: Sponsored by:
PRESENTED BY: Sponsored by: Practical Uses of Analytics in E-Discovery - A PRIMER Jenny Le, Esq. Vice President of Discovery Services jle@evolvediscovery.com E-Discovery & Ethics Structured, Conceptual,
More informationMeasuring Recall in E-Discovery, Part Two: No Easy Answers
Measuring Recall in E-Discovery, Part Two: No Easy Answers John Tredennick In Part One of this article, I introduced readers to statistical problems inherent in proving the level of recall reached in a
More informationE-Discovery: A Common Sense Approach. In order to know how to handle and address ESI issues, the preliminary and
Jay E. Heidrick Polsinelli jheidrick@polsinelli.com (913) 234-7506 E-Discovery: A Common Sense Approach In order to know how to handle and address ESI issues, the preliminary and obvious question must
More informationE-Discovery in Employment Litigation: Making Practical, Yet Defensible Decisions
E-Discovery in Employment Litigation: Making Practical, Yet Defensible Decisions 11 E-Discovery in Employment Litigation: Making Practical, Yet Defensible Decisions Introduction Much has been said about
More informationCase 1:12-cv-24356-JG Document 404 Entered on FLSD Docket 03/18/2014 Page 1 of 14
Case 1:12-cv-24356-JG Document 404 Entered on FLSD Docket 03/18/2014 Page 1 of 14 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF FLORIDA MIAMI DIVISION CASE NO. 12 24356 CIV GOODMAN PROCAPS S.A., [CONSENT
More informationE-Discovery Tip Sheet
E-Discovery Tip Sheet LegalTech 2015 Some Panels and Briefings Last month I took you on a select tour of the vendor exhibits and products from LegalTech 2015. This month I want to provide a small brief
More informationThe Legal Advantages of Retaining Information
WHITE PAPER: CUSTOMIZE WHITE PAPER: BEST PRACTICES FOR ARCHIVING Confidence in a connected world. Best Practices for Defining and Establishing Effective Archive Retention Policies Sponsored by Symantec
More informationPredictive Coding: Emerging E Discovery Tool Leveraging E Discovery Computer Assisted Review to Reduce Time and Expense of Discovery
Presenting a live 90 minute webinar with interactive Q&A Predictive Coding: Emerging E Discovery Tool Leveraging E Discovery Computer Assisted Review to Reduce Time and Expense of Discovery WEDNESDAY,
More informationXact Data Discovery. Xact Data Discovery. Xact Data Discovery. Xact Data Discovery. ediscovery for DUMMIES LAWYERS. MDLA TTS August 23, 2013
MDLA TTS August 23, 2013 ediscovery for DUMMIES LAWYERS Kate Burke Mortensen, Esq. kburke@xactdatadiscovery.com Scott Polus, Director of Forensic Services spolus@xactdatadiscovery.com 1 Where Do I Start??
More informationESI: Focus on Review and Production Strategy. Meredith Lee, Online Document Review Supervisor, Paralegal
ESI: Focus on Review and Production Strategy Meredith Lee, Online Document Review Supervisor, Paralegal About Us Avansic E-discovery and digital forensics company founded in 2004 by Dr. Gavin W. Manes,
More informationTraditionally, the gold standard for identifying potentially
istockphoto.com/alexandercreative Predictive Coding: It s Here to Stay Predictive coding programs are poised to become a standard practice in e-discovery in the near future. As more courts weigh in on
More informationPredictability in E-Discovery
Predictability in E-Discovery Presented by: John G. Roman, Jr. National Manager, Practice Group Technology Services Nixon Peabody LLP Tom Barce Assistant Director of Practice Support Fulbright & Jaworski
More informationediscovery Policies: Planned Protection Saves More than Money Anticipating and Mitigating the Costs of Litigation
Brought to you by: ediscovery Policies: Planned Protection Saves More than Money Anticipating and Mitigating the Costs of Litigation Introduction: Rising costs of litigation The chance of your organization
More informationTechnology-Assisted Review and Other Discovery Initiatives at the Antitrust Division. Tracy Greer 1 Senior Litigation Counsel E-Discovery
Technology-Assisted Review and Other Discovery Initiatives at the Antitrust Division Tracy Greer 1 Senior Litigation Counsel E-Discovery The Division has moved to implement several discovery initiatives
More informationE-Discovery Defensibility ViewS from the Bench page : 1
E-Discovery Defensibility ViewS from the Bench page : 1 E-Discovery Defensibility Views from the Bench A Clearwell White Paper E-Discovery Defensibility ViewS from the Bench page : 2 Table of Contents
More informationDiscovery Data Management
Discovery Data Management in Practice Introductions Reveal Derick Roselli 949-280-3519 droselli@revealdata.com www.revealdata.com Program Outline Pre-Planning Stages Workflow Management Project Management
More informationREDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - PRIORITIZATION FOR 100% REVIEW. Bill Tolson Sr. Product Marketing Manager Recommind Inc.
REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - Bill Tolson Sr. Product Marketing Manager Recommind Inc. Introduction... 3 Traditional Linear Review... 3 Advanced Review Strategies: A Typical Predictive
More informationediscovery Defensibility
Content Introduction... 3 The Duty to Preserve Phillip M. Adams & Assoc., LLC v. Dell, Inc.... 4 Case Facts.... 4 Key Takeaways... 4 Keyword Search Victor Stanley, Inc. v. Creative Pipe, Inc..... 5 Case
More informationE-Discovery in Michigan. Presented by Angela Boufford
E-Discovery in Michigan ESI Presented by Angela Boufford DISCLAIMER: This is by no means a comprehensive examination of E-Discovery issues. You will not be an E-Discovery expert after this presentation.
More informationThe Case for Technology Assisted Review and Statistical Sampling in Discovery
The Case for Technology Assisted Review and Statistical Sampling in Discovery Position Paper for DESI VI Workshop, June 8, 2015, ICAIL Conference, San Diego, CA Christopher H Paskach The Claro Group, LLC
More informationfor Insurance Claims Professionals
A Practical Guide to Understanding ediscovery for Insurance Claims Professionals ediscovery Defined and its Relationship to an Insurance Claim Simply put, ediscovery (or Electronic Discovery) refers to
More informationTHE FEDERAL COURTS LAW REVIEW. Comments on The Implications of Rule 26(g) on the Use of Technology-Assisted Review
THE FEDERAL COURTS LAW REVIEW Volume 7, Issue 1 2014 Comments on The Implications of Rule 26(g) on the Use of Technology-Assisted Review Maura R. Grossman and Gordon V. Cormack ABSTRACT Approaches to technology-assisted
More informationIndustry Leading Solutions: Innovative Technology. Quality Results.
Industry Leading Solutions: Innovative Technology. Quality Results. April 10, 2013 emagsolutions.com Agenda Speaker Introduction A Quick Word about emag Importance of Technology Assisted Review (TAR) Key
More informationWhen E-Discovery Becomes Evidence
Monday, June 11, 2007 When E-Discovery Becomes Evidence Make sure that you can easily authenticate the information that was so costly to produce By Leonard Deutchman Special to the Law Weekly A federal
More informationCase 2:11-cv-00678-LRH-PAL Document 174 Filed 07/18/14 Page 1 of 18 UNITED STATES DISTRICT COURT DISTRICT OF NEVADA * * * Plaintiff, Defendants.
Case :-cv-00-lrh-pal Document Filed 0// Page of 0 PROGRESSIVE CASUALTY INSURANCE COMPANY, v. JACKIE K. DELANEY, et al., UNITED STATES DISTRICT COURT DISTRICT OF NEVADA Plaintiff, Defendants. * * * Case
More informationTop 10 Best Practices in Predictive Coding
Top 10 Best Practices in Predictive Coding Emerging Best Practice Guidelines for the Conduct of a Predictive Coding Project Equivio internal document " design an appropriate process, including use of available
More informationPredictive Coding and The Return on Investment (ROI) of Advanced Review Strategies in ediscovery
Predictive Coding and The Return on Investment (ROI) of Advanced Review Strategies in ediscovery Drew Lewis ediscovery Counsel AGENDA A Predictive Coding Primer Predictive Coding and Market Trends Predictive
More informationPredictive Coding as a Means to Prioritize Review and Reduce Discovery Costs. White Paper
Predictive Coding as a Means to Prioritize Review and Reduce Discovery Costs White Paper INTRODUCTION Computers and the popularity of digital information have changed the way that the world communicates
More informationSocial Media & ediscovery: Untangling the Tweets for the Trials
Social Media & ediscovery: Untangling the Tweets for the Trials Jack Halprin, Esq. Vice President, ediscovery & Compliance POWER PROTECT PROMOTE Agenda Social Media Sites: What, Who, Where & How Discoverability
More informationSTATISTICAL ANALYSIS AND INTERPRETATION OF DATA COMMONLY USED IN EMPLOYMENT LAW LITIGATION
STATISTICAL ANALYSIS AND INTERPRETATION OF DATA COMMONLY USED IN EMPLOYMENT LAW LITIGATION C. Paul Wazzan Kenneth D. Sulzer ABSTRACT In employment law litigation, statistical analysis of data from surveys,
More informationCase 1:11-cv-01279-ALC-AJP Document 96 Filed 02/24/12 Page 1 of 49
Case 1:11-cv-01279-ALC-AJP Document 96 Filed 02/24/12 Page 1 of 49 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
More informationMinimizing ediscovery risks. What organizations need to know in today s litigious and digital world.
What organizations need to know in today s litigious and digital world. The main objective for a corporation s law department is to mitigate risk throughout the company, while keeping costs under control.
More informationAny and all documents Meets Electronically Stored Information: Discovery in the Electronic Age
Any and all documents Meets Electronically Stored Information: Discovery in the Electronic Age Panel Members Judge Ronald L. Buch, Moderator Panelists The Honorable Paul W. Grimm U.S. District Court for
More informationE-Discovery for Backup Tapes. How Technology Is Easing the Burden
E-Discovery for Backup Tapes How Technology Is Easing the Burden TABLE OF CONTENTS Introduction...3 The Importance and Challenge of Backup Tapes for Electronic Discovery...3 TAPES AS A SOURCE OF ESI...
More informationHow To Protect A Company Or Law Firm From Being Sanctioned For Violating A Discovery Order
A SCARAB CONSULTING WHITEPAPER January, 2012 Third-party ediscovery Providers = Insulation Against Cold, Harsh Sanctions? Synopsis The risk of sanctions resulting from 1) corporations doing selfcollections
More informationBuilt by the clients, for the clients. Utilizing Contract Attorneys for Document Review
Built by the clients, for the clients Utilizing Contract Attorneys for Document Review Ashish Prasad, Esq. October 27, 2014 Table of Contents Introduction... 3 Ethical Issues... 3 Staffing of Reviewers...
More informationThe Ethics of E-Discovery. computer technologies in civil litigation, courts are faced with a myriad of issues
The Ethics of E-Discovery By John M. Barkett Chicago, Illinois, 2009; ISBN 978-1-60442-256-6 Price $69.95, pp. 125 Reviewed by Tracy Flynn Journal of High Technology Law Suffolk University Law School E-discovery
More informationEmerging Topics for E-Discovery. October 22, 2014
Emerging Topics for E-Discovery October 22, 2014 ACEDS Membership Benefits Training, Resources and Networking for the E-Discovery Community! Exclusive News and Analysis! Weekly Web Seminars! Podcasts!
More informationPros And Cons Of Computer-Assisted Review
Portfolio Media. Inc. 860 Broadway, 6th Floor New York, NY 10003 www.law360.com Phone: +1 646 783 7100 Fax: +1 646 783 7161 customerservice@law360.com Pros And Cons Of Computer-Assisted Review Law360,
More informationCOALSP 2013 E-Discovery Case Law Update. Drew Unthank Partner Wheeler Trigg O Donnell LLP
COALSP 2013 E-Discovery Case Law Update Drew Unthank Partner Wheeler Trigg O Donnell LLP Agenda Where have we come from? Where are we now? Where are we going? Antacids Where Have We Come From? Litigation
More informationThe Five Pillars of E-Discovery
The Five Pillars of E-Discovery David M. Fuqua Whitney L. Foster I. Introduction Without discovery, modern litigation would be impossible. In just a few years, discovery has migrated from paper to electronic
More information