1 Forensic The case for statistical sampling in e-discovery January 2012 kpmg.com
2 2 The case for statistical sampling in e-discovery The sheer volume and unrelenting production deadlines of today s electronic discovery efforts demand ever-increasing review capacity and throughput. Meanwhile, a growing chorus within the judiciary is urging adoption of technology-assisted document review. Attribute sampling in e-discovery can help litigators meet search-efficiency and qualityassurance goals, increase productivity, and reduce costs. A robust statistical sampling process can confirm whether technology-assisted results are as effective as human review or more so. Technology and practice considerations The increasing volume and complexity of electronically stored information and the cost of its review in connection with litigation and regulatory proceedings continue to drive the need for technology-assisted document review and the development of predictive-coding software. Because the majority of sophisticated high-speed processing, indexing, and categorization tools rely on sophisticated proprietary algorithms often referred to as black-box technologies, there has been some reluctance to use them. The fear is that the results they produce may be challenged and rejected as not meeting the required standard of reasonableness under the Federal Rules of Civil Procedure and other applicable rules. 1 Note: This observation relates solely to process control sampling in the performance of document review in connection with electronic discovery it is NOT a commentary on sampling techniques relative to financial statement auditing. 2 Confidence level: The probability, expressed as a percentage, that the true value of an attribute being tested through sampling lies within a range called the confidence interval. Confidence interval: An estimated range of values that likely includes the attribute value that is being tested through sampling. Example: A review of sample documents shows that 18 percent of the documents have call tag A. If we are using a confidence interval of plus or minus 2 percent and a confidence level of 95 percent, then we can say that we are 95 percent confident that the actual percentage of call tag A in the entire population is between 16 and 20 percent. Effective use of statistical sampling 1 can help overcome this concern by quantifying the reliability of a system s technology assisted predictive-coding calls through testing and comparison of both human-reviewed and machine-assisted coding. Statistical sampling also can determine with a stated level of confidence 2 whether the system has produced results at least as consistent and reliable as those obtained by having attorneys review the documents manually. Today, deciding whether an electronic discovery review process meets the reasonableness standard generally hinges on whether enough personnel have performed the review to ensure reliable results. Human (manual) review traditionally has been considered the most desirable form or gold standard for assessing the adequacy of a document review process.
3 The case for statistical sampling in e-discovery 3 Results produced by manual review, however, have rarely been properly measured, questioned, or tested for consistency and reliability, and sophisticated sampling and testing of physical review processes are equally rare. Nevertheless, such review remains the dominant method despite academic and anecdotal evidence to the contrary: The idea that exhaustive manual review is the most effective and therefore the most defensible approach to document review is strongly refuted. Technology-assisted review can (and does) yield more accurate results than exhaustive manual review, with much lower effort. 3 In the world of electronic discovery, the myth of the superiority of human review persists, as does the fear of using statistical sampling to defend other methods of review. Rigorous statistical sampling, which is widely accepted as a basic quality-control methodology to manage and quantify quality risk in many product and service industries, has not been widely adopted in electronic discovery review projects. Automotive operations, electronics manufacturing, and financial services, for instance, rely on sampling for quality control and setting warranty reserves, among other things; ironically, their legal counsel may eschew the same statistical processes that underpin the analysis of critical elements of their client s primary business. Scaling up traditional processes by adding more reviewers is impractical given the cost- and quality-control initiatives that are of utmost importance to corporations and the counsel on whom they rely. Today s conditions demand new technologies Most practitioners still review document collections, culled in some form (typically by keywords), in a linear manner. While they may organize by date, custodian, or some other parameter, every document gathered essentially starts in one pile and ends up in another many months and dollars later. Accordingly, technology development efforts have largely focused on helping review teams manage the efficiency and cost of large scale reviews. Initially, tools categorized and clustered documents based on content or concepts or degree of similarity, and grouped near duplicate documents for review. These tools prioritized to improve linear review by helping reviewers speed through nonresponsive documents or similar sets of responsive documents, bulk-tagging them to more quickly isolate material requiring further review. Such prioritization tools also enable different documents to be reviewed at different cost levels. For instance, reviewers working on contract to law firms can weed out nonresponsive documents, allowing the firms full-time lawyers (typically, associates) to make privilege calls and address more complex issues. The latest technologies go beyond prioritization and clustering, and actually automate the review process by minimizing the need for human reviewers to look at every document. Senior reviewers or subject matter experts (SMEs) train the software to make document coding calls predictions based on their coding of training sets of documents. Properly configured, such software 3 Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Richmond J. of Law & Tech., Vol. XVII, Issue 3, 1-48 (2011).
4 4 The case for statistical sampling in e-discovery review is quicker and more consistent than manual review. The results of initial training iterations can then be used to structure further review strategies. Regardless of where an organization falls on the linear/ prioritized/automated review continuum, achieving a faster, more consistent, more predictable, and less costly review takes more than just enhanced efficiency. A valid, supportable document review project requires proven quality control (QC) processes to demonstrate its effectiveness (see sidebar). Cost, risk, and time: Technology vs. human review While the cost-effectiveness of technology-based processes has been largely established, their reliability remains a topic of ongoing discussion. Over a recent three-year period, KPMG conducted four test projects 7 that compared the results of a human review process with results obtained by reprocessing the same set of documents using a predictive coding software tool. The tool used in these tests is trained by an SME. The training used an initial sample of 400 documents augmented by a subsequent series of randomly selected sample batches of 40 documents each. These are reviewed consecutively by the SME until the software can reliably predict the right calls (those that the SME would have made). Based on the SME s decisions about document relevance in the training samples, the trained software calculated the relevance of the remaining documents: In all three matters, the software-assisted review had better recall than human review; i.e., it was able to identify documents that were actually relevant that had been missed in the traditional human review workflow. In addition, the software assisted workflow identified the subset of the total population that was likely to contain relevant documents this subset would have to be reviewed as the next step of the workflow. In the matters we studied, the software system estimated that percent of the relevant documents were in subsets that were only percent of the total population. 8 In other words, for large-scale reviews, software-assisted review consistently finds more relevant documents without the need for human reviewers to look at all of the documents. The completeness of the recall can be confirmed to a stated confidence level 9 by statistical sampling. 7 Three of the test projects are described in detail in KPMG s August 2011 white paper Softwareassisted document review: An ROI your GC can appreciate. Similar results for a fourth project conducted in 2011 were not published. 8 Ibid. 9 Ibid. (footnote 2). The need for proven quality control (QC) processes In a keynote address at the July 2011 Carmel Valley e-discovery Retreat and subsequent article 4 on the issue, United States Magistrate Judge Andrew J. Peck, United States District Court, Southern District of New York, provided support for expanded use of statistical sampling of electronic discovery review results. Statistical sampling can determine the recall and precision the efficacy of the predictive coding software on both a stand-alone and relative-to-human-review basis. He said: If the use of predictive coding is challenged in a case before me, I will want to know what was done and why that produced defensible results. I may be less interested in the science behind the black box of the vendor s software than in whether it produced responsive documents with reasonably high recall and high precision. In Peck s view, the validation of review quality is too often subordinate to the review itself, which is expressed in informal or casual observations that lack the scientific rigor and quantifiable measures necessary to defend the quality process. Judge Peck alludes to more sophisticated quality assurance methods such as those relying on statistical sampling to validate and definitively quantify technologyderived results. Blogger chrisdale.wordpress.com also noted Judge Peck s keynote endorsement of sampling and relevance ranking 5 : If 2010 was the year of proportionality, Judge Peck said, it seems that 2011 is the year of predictive coding. He explained this as meaning (and it is his meaning which counts here, not any one provider s definition) that a senior lawyer, using a random sample or pre-filtered set of documents decides whether they are responsive or not responsive; one goes through the process several times with senior people until the computer is sufficiently trained to apply those conclusions across a wider set or the whole. Some systems, he said, merely discriminate between relevant and non-relevant documents, while others prioritize them for review on a scale from 0 to 100. Such a system might be used to find keywords or for quality testing as well as for making primary selections. The idea is that lawyers do not spend their clients money reviewing irrelevant or low-rated documents. Similarly, in Victor Stanley v. Creative Pipe, Judge Grimm said, [t]he only prudent way to test the reliability of the keyword search is to perform some appropriate sampling of the documents determined to be privileged and those determined not to be in order to arrive at a comfort level that the categories are neither over-inclusive nor under-inclusive. 6 Such quality assurance methods as sampling need to demonstrate that technology-derived results are at least as accurate and consistent as those obtained by unaided human review. 4 Peck, Andrew, Search, Forward, Law Technology News (October 1, 2011) (http://www.law.com/jsp/lawtechnologynews/pubarticleltn. jsp?id= &search_forward&slreturn=1) Victor Stanley v. Creative Pipe, Inc WL (D. Md. May 29, 2008).
5 The case for statistical sampling in e-discovery 5 Measuring and assuring process quality The ability to carefully select a sample and infer from it the condition of the larger population with a high degree of confidence in the reliability of the inference has tremendous utility in electronic discovery. In KPMG s test projects, the software consistently recalled a greater number of relevant documents than the human reviewers did. By retrieving more of the relevant documents than a human first review and significantly reducing how much of the corpus required review, considerable savings may be achieved. The proprietary algorithm used was not disclosed, but in practice this should be increasingly irrelevant. In Judge Peck s words: I may be less interested in the science behind the black box of the vendor s software than in whether it produced responsive documents with reasonably high recall and high precision. Demonstrating and assuring process capability to defend a technology-assisted review is a matter of (1) sound design, (2) transparency, and (3) quantifiable results. QC by process sampling is a scientifically rigorous method that delivers all three. Clearly, statistical sampling can provide benefits to QC in electronic discovery review so why isn t it more widely used? Case study 1: Confidence interval sampling (Process Validation) Statistical sampling is being successfully used in e-discovery to quantify process quality by clients who have come to understand its value in helping them meet tight time lines. One KPMG client who wanted a simple method of assessing the effectiveness of keywords requested testing on nonresponsive documents. KPMG provided the client with alternative statistical sampling methods: (1) select a sample of 1,068 nonresponsive documents and be 95 percent confident that the sample results represent the entire nonresponsive document set; or (2) select and test 4,147 nonresponsive documents and be 99 percent confident that the results represent the entire nonresponsive document set. The entire set of nonresponsive documents was several hundred thousand. Although these two alternatives met the client s requirements, many other sample size/confidence levels are possible, depending on the specific needs of the client. Sound design Sound design involves following an accepted QC sampling methodology, determining the appropriate confidence level, calculating the appropriate sample size, quantifying any process risks, and determining acceptance and rejection criteria. Transparency Transparency involves documenting every step of the process as well as its underlying rationale, and exposing the sample and the acceptance/rejection determinations for the documents in the sample. To quote Judge Peck once more: That may mean allowing the requesting party to see the documents that were used to train the computer-assisted coding system. (Counsel would not be required to explain why documents were coded as responsive or non-responsive, just what the coding was.) Proof of a valid process, including quality control testing, also will be important. Quantifiable results Quantifiable results involves a meaningful discussion of the QC methodology.
6 6 The case for statistical sampling in e-discovery Case study 2: Lot acceptance sampling (Process Control) With a review under way, our client wanted to evaluate contract reviewer performance near real time. After analyzing the average number of documents the reviewers were completing each day, and relying on well established batch or lot sampling methods using a 95 percent confidence level, we provided the following sampling approach: (1) Reviewers completing 1,200 documents or fewer per day would have 80 documents randomly sampled and checked. If three or more documents were improperly coded, the entire batch, up to 1,200 documents, would be returned for rereview. (2) Reviewers completing between 1,201 and 3,200 documents per day would have 125 documents checked and, if four or more documents were improperly coded, the entire batch would be returned for rereview. By using this batch acceptance sampling approach, the lead reviewer was able to provide timely feedback and corrective instruction to the reviewers and thus be more confident that ongoing review quality standards were being met. Ideas for overcoming resistance While using statistical sampling may satisfy an opposing attorney, a regulator, or the magistrate or judge presiding over the discovery proceedings, even more fundamentally, it can provide the electronic discovery practitioner with greater comfort about the accuracy of the technology used in review. So with this degree of reliability, why is there still so much reluctance to use statistical sampling? Possible reasons include lack of familiarity with statistical (generally mathematical) methods, or concerns about mastering its perceived complexity. Another concern may be that a small error found in sampling could potentially render the entire review results unacceptable. Additionally, there seems to be broad concern that, in the discovery review process, there is a lack of legal precedent confirming the acceptability of statistical sampling methods. The following suggestions may help move reluctant electronic discovery practitioners to overcome their bias against statistical sampling: Note that statistical sampling is already part of your life. Practitioners who balk at statistical sampling in electronic discovery may vigorously debate colleagues about the latest presidential poll numbers, make purchasing or investment decisions based on the latest inflation rate, or think nothing of consuming pharmaceuticals that have gone through FDA testing. It defies logic to accept statistical sampling as an integral verification tool of politics, finance, or pharmaceutical manufacturing, for instance, but reject those same methods in professional practice. The Sedona Conference has remarked on statistical sampling in political polling, 10 saying: [Even though statistical sampling is widely used in political polling] mention statistical sampling in most other contexts and there is such apprehension about the process and the math involved, such that, at least to date, statistical sampling is rarely used or cited in assessing the quality and consistency of the electronic document review process. [But] the document review process is very well suited to the application of statistical sampling to improve quality and reduce costs. Use experts. The legal profession consults experts to overcome lack of familiarity with scientific phenomena, technologies used for alleged patent infringement, and the psychology of jurors. Your company can similarly address lack of familiarity with sampling through the use of experts who understand QC sampling. Ultimately, the experts may have to explain the sampling process to an uninitiated magistrate or an overworked regulator, so why not start with your own case team? Sampling experts also can assist with data stratification, determining sample sizes, and calculating confidence levels for statistical inferences. They can point to the generally accepted formulas on which they rely (which are widely available e.g., in college statistics textbooks). As with all experts, cost considerations can be evaluated using a cost benefit calculation that compares the cost of the experts to the avoided costs of more extensive testing using nonstatistical approaches. 10 See Sampling 101 for the E-Discovery Lawyer, an appendix to the May 2009 Sedona Conference Commentary on Achieving Quality in the E-Discovery Process.
7 The case for statistical sampling in e-discovery 7 Take incremental steps. Another approach to adopting the use of statistical sampling would be to programmatically integrate generally accepted QC sampling methods into widely used electronic discovery applications. One does not have to use predictive coding to benefit from sampling in electronic discovery. Keyword-searching, which has become mainstream, relies on somewhat-arcane search techniques and could benefit from statistical sampling and validation. Numerous examples of statistical sampling in electronic discovery (other than predictive coding) exist. Applying a new sampling methodology could include periodic batch sampling throughout the review process with a mechanism designed to address reviewer error as it is detected, thus reducing the need to repeat a significant portion of the review process. Others include testing potential keywords (inferring responsiveness rates based on the existence of a particular keyword), comparing responsiveness rates for different types of documents ( vs. loose files), or other examples described in The Sedona Conference Commentary. 11 Carefully designed user interfaces for selecting samples, testing them, and reporting the results can guide users through the sampling process, minimizing if not eliminating most common sampling mistakes. This would increase consistency, repeatability, and reproducibility in the QC process, and would help increase familiarity with statistical methods and their defensibility in electronic discovery. Sample your sampling. Sampling error caused by errors in evaluating the sample items can lead to invalid interpretation of the sample results. This can cause extra work to rereview documents or significantly expand the sample size. The risk of sampling error can be addressed with a sample evaluation process in which senior SMEs check the sample results before they are finalized. Conclusion: Hurry up and wait (but not for long) The lack of a clear legal precedent is likely to be addressed in the near future by the courts, which are becoming increasingly aware of the benefits of new technology and statistical sampling in dealing with the challenges posed by very large populations of documents. Even without a definitive Federal District Court opinion on the subject, members of the bench are trying to speed this process. [U]ntil there is a judicial opinion approving (or even critiquing) the use of predictive coding, counsel will just have to rely on this article as a sign of judicial approval, Judge Peck has said. Without clear legal precedent, there may be some additional risk to applying new technologies and relying on statistical sampling to demonstrate their efficacy. Companies will have to decide whether the benefits in terms of quality and cost of QC sampling that result from these new technologies can more than offset such risks until the legal precedents supporting their use are clearly established. Until then, even the most reluctant electronic discovery practitioner can begin to familiarize him- or herself with statistical methods to get more comfortable with incorporating them into their existing electronic discovery processes. About the authors Chris Paskach is the national leader for KPMG s 220-member Forensic Technology Services practice. He coordinates the full range of KPMG s ediscovery services, from records and information management, digital evidence recovery, and evidence and discovery management services, including KPMG s secure ediscovery hosted review repository services. As a member of The Sedona Conference Working Group 1, he participated in the drafting of The Sedona Conference Commentary on Achieving Quality in the E-Discovery Process (May 2009) and The Sedona Conference Database Principles (March 2011). Michael Carter is a director in KPMG s ediscovery practice, leading its process improvement deployment. His responsibilities include the technical integration of statistical sampling into KPMG s ediscovery processes and technology. Michael is a certified Six Sigma Master Black Belt and member of the American Society of Quality (ASQ). Phil Strauss is a director in the firm s San Francisco office, coordinating KPMG s ediscovery and other services for Northern California and the Pacific Northwest. He has experience in discovery and litigation matters from private practice with AmLaw firms, as general counsel of two public software companies, and as an outside consultant. He is on the adjunct faculty of the University of California, Berkeley s Haas School of Business, where he teaches business law. 11 Ibid.
8 For more information, please contact: Christopher H. Paskach, CPA Partner T: E: Michael J. Carter Director T: E: Philip R. Strauss, JD Director T: E: kpmg.com This article represents the views of the authors only, and does not necessarily represent the views or professional advice of KPMG LLP. The information contained herein is of a general nature and is not intended to address the circumstances of any particular individual or entity. Although we endeavor to provide accurate and timely information, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future. No one should act upon such information without appropriate professional advice after a thorough examination of the particular situation KPMG LLP, a Delaware limited liability partnership and the U.S. member firm of the KPMG network of independent member firms affiliated with KPMG International Cooperative ( KPMG International ), a Swiss entity. All rights reserved. Printed in the U.S.A NSS The KPMG name, logo, and cutting through complexity are registered trademarks or trademarks of KPMG International.
Software-assisted document review: An ROI your GC can appreciate kpmg.com b Section or Brochure name Contents Introduction 4 Approach 6 Metrics to compare quality and effectiveness 7 Results 8 Matter 1
Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive coding is one of the most promising technologies to reduce the high cost of review by
WHITE PAPER: PREDICTIVE CODING DEFENSIBILITY........................................ Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive
By John Tredennick CEO Catalyst Repository Systems Magistrate Judge Andrew J. Peck issued a landmark decision in Da Silva Moore v. Publicis and MSL Group, filed on Feb. 24, 2012. This decision made headlines
The United States Law Week Source: U.S. Law Week: News Archive > 2012 > 04/24/2012 > BNA Insights > Under Fire: A Closer Look at Technology- Assisted Document Review E-DISCOVERY Under Fire: A Closer Look
Quality Control for predictive coding in ediscovery kpmg.com Advances in technology are changing the way organizations perform ediscovery. Most notably, predictive coding, or technology assisted review,
Predictive Coding Defensibility Who should read this paper The Veritas ediscovery Platform facilitates a quality control workflow that incorporates statistically sound sampling practices developed in conjunction
Predictive Coding Helps Companies Reduce Discovery Costs Recent Court Decisions Open Door to Wider Use by Businesses to Cut Costs in Document Discovery By John Tredennick As companies struggle to manage
WHITE PAPER: SYMANTEC TRANSPARENT PREDICTIVE CODING Symantec Transparent Predictive Coding Cost-Effective and Defensible Technology Assisted Review Who should read this paper Predictive coding is one of
Vol. 46 No. 3 February 6, 2013 PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX? The high costs of e-discovery have led to the development of computerized review technology by which the user may search
Ashish Prasad, Esq. Noah Miller, Esq. Joshua C. Garbarino, Esq. October 27, 2014 Table of Contents Introduction... 3 What is TAR?... 3 TAR Workflows and Roles... 3 Predictive Coding Workflows... 4 Conclusion...
Manuscript submitted to the Organizing Committee of the Fourth DESI Workshop on Setting Standards for Electronically Stored Information in Discovery Proceedings on April 20, 2011. Updated May 18, 2011.
Ba k e Offs, De m o s & Kicking t h e Ti r e s: A Pr a c t i c a l Litigator s Br i e f Gu i d e t o Eva l u at i n g Ea r ly Ca s e Assessment So f t wa r e & Search & Review Tools Ronni D. Solomon, King
SAMPLING: MAKING ELECTRONIC DISCOVERY MORE COST EFFECTIVE Milton Luoma Metropolitan State University 700 East Seventh Street St. Paul, Minnesota 55337 651 793-1246 (fax) 651 793-1481 Milt.Luoma@metrostate.edu
A Practitioner s Guide to Statistical Sampling in E-Discovery October 16, 2012 1 Meet the Panelists Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Gordon V. Cormack, Professor at the David
E-Discovery in Mass Torts: Predictive Coding Friend or Foe? Sherry A. Knutson Sidley Austin One S Dearborn St 32nd Fl Chicago, IL 60603 (312) 853-4710 email@example.com Sherry A. Knutson is a partner
The Benefits of Sampling in E-Discovery How Smart Sampling Can Help Attorneys Reduce Document Review Costs A white paper from 615.255.5343 dsi.co 414 Union Street, Suite 1210 Nashville, TN 37219-1771 Table
REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - Bill Tolson Sr. Product Marketing Manager Recommind Inc. Introduction... 3 Traditional Linear Review... 3 Advanced Review Strategies: A Typical Predictive
ediscovery & Information Management White Paper The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review Scott M. Cohen Elizabeth T. Timkovich John J. Rosenthal February 2014 2014 Winston
MEALEY S TM LITIGATION REPORT Discovery Predictive Coding: A Primer by Amy Jane Longo, Esq. and Usama Kahf, Esq. O Melveny & Myers LLP Los Angeles, California A commentary article reprinted from the March
LITIGATION AND PRACTICE SUPPORT Technology- Assisted Review 2.0 by Ignatius Grande of Hughes Hubbard & Reed LLP and Andrew Paredes of Epiq Systems Legal teams and their outside counsel must deal with an
Amendments to the Rules to Civil Procedure: Yours to E-Discover Prepared by Christopher M. Bartlett Cassels Brock & Blackwell LLP September 25, 2009 Amendments to the Rules of Civil Procedure: Yours to
Top 10 Best Practices in Predictive Coding Emerging Best Practice Guidelines for the Conduct of a Predictive Coding Project Equivio internal document " design an appropriate process, including use of available
Three Methods for ediscovery Document Prioritization: Comparing and Contrasting Keyword Search with Concept Based and Support Vector Based "Technology Assisted Review-Predictive Coding" Platforms Tom Groom,
e-discovery Advisory Forensic Services kpmg.ch e-discovery You or your client are involved in a dispute, investigation, regulatory or internal review. You need to review evidence and may need to disclose
WHITE PAPER: THE ediscovery BALANCING ACT The ediscovery Balancing Act Striking the Right Mix of In-House and Outsourced Expertise The ediscovery Balancing Act Contents Introduction...........................................
Litigation Solutions i n s i g h t f u l i n t e r a c t i ve c u l l i n g d i s t r i b u t e d e d i s cove r y p ro ce s s i n g p owe r i n g d i g i t a l re v i e w Advanced Analytical Review Data
White Paper Technology Assisted Review Allison Stanfield and Jeff Jarrett 25 February 2015 1300 136 993 www.elaw.com.au Table of Contents 1. INTRODUCTION 3 2. KEYWORD SEARCHING 3 3. KEYWORD SEARCHES: THE
Power-Up Your Privilege Review: Protecting Privileged Materials in Ediscovery Jeff Schomig, WilmerHale Stuart Altman, Hogan Lovells Joe White, Kroll Ontrack Sheldon Noel, Kroll Ontrack (moderator) April
Mastering Predictive Coding: The Ultimate Guide Key considerations and best practices to help you increase ediscovery efficiencies and save money with predictive coding 4.5 Validating the Results and Producing
ABA SECTION OF LITIGATION 2012 SECTION ANNUAL CONFERENCE APRIL 18-20, 2012: PREDICTIVE CODING Predictive Coding SUBMITTED IN SUPPORT OF THE PANEL DISCUSSION INTRODUCTION Technology has created a problem.
The Case for Technology Assisted Review and Statistical Sampling in Discovery Position Paper for DESI VI Workshop, June 8, 2015, ICAIL Conference, San Diego, CA Christopher H Paskach The Claro Group, LLC
2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500 5218 South East Street, Suite E-3, Indianapolis, IN 46227 Tel 317.247.4400 Fax 317.247.0044 Presented by Providing
Artificial Intelligence and Transactional Law: Automated M&A Due Diligence By Ben Klaber Introduction Largely due to the pervasiveness of electronically stored information (ESI) and search and retrieval
Continuous Active Learning for Technology Assisted Review How It Works and Why It Matters for E-Discovery John Tredennick, Esq. Founder and CEO, Catalyst Repository Systems Peer-Reviewed Study Compares
1 of 6 10/16/2014 2:36 PM New York Law Journal (Online) May 25, 2012 Friday Copyright 2012 ALM Media Properties, LLC All Rights Reserved Further duplication without permission is prohibited Length: 2327
PICTERA SOLUTIONS An What Is Intell1gent One? Created by the clients, for the clients This white paper discusses: Understanding How Intell1gent One Saves Time and Money Using Intell1gent One to Save Money
Compliance Cost Associated with the Storage of Unstructured Information Sponsored by Novell Independently conducted by Ponemon Institute LLC Publication Date: May 2011 Ponemon Institute Research Report
Multi language e Discovery Three Critical Steps for Litigating in a Global Economy 2 3 5 6 7 Introduction e Discovery has become a pressure point in many boardrooms. Companies with international operations
ediscovery and Information Governance Practice Overview ediscovery and Information Governance Electronic discovery, or ediscovery, is increasingly changing from the exception to the norm in modern litigation.
Industry Leading Solutions: Innovative Technology. Quality Results. April 10, 2013 emagsolutions.com Agenda Speaker Introduction A Quick Word about emag Importance of Technology Assisted Review (TAR) Key
E-Discovery Getting a Handle on Predictive Coding John J. Jablonski Goldberg Segalla LLP 665 Main St Ste 400 Buffalo, NY 14203-1425 (716) 566-5400 firstname.lastname@example.org Drew Lewis Recommind 7028
Predictive Coding in UK Civil Litigation White Paper by Chris Dale of the e-disclosure Information Project THE PURPOSE OF THIS PAPER This paper considers the use of technology known as Predictive Coding
Xerox Legal Services Viewpoint ediscovery Platform Technical Brief Viewpoint ediscovery Services Viewpoint by Xerox delivers a flexible approach to ediscovery designed to help you manage your litigation,
COVER STORY ] THINKING SMARTER Jason R. Baron Revolutionizing how the world handles information It is common knowledge that we are living in what has been termed The Information Age. With the advent of
Managed Services: Maximizing Transparency and Minimizing Expense and Risk in ediscovery and Information Governance January 18, 2013 Andrew Bayer, Director of Business Development Adam Wells, VP, Business
AccessData Group CASE STUDY: Top 5 Communications Company Evaluates Leading ediscovery Solutions White Paper "#$%&'()'*+&*+, Overview... 1 Top 5 Communications Company: Background and Its ediscovery Requirements...
Page 1 LMG GROUP vs. THE BIG DATA TIDAL WAVE Recognizing that corporations, law firms and government entities are faced with tough questions in today s business climate, LMG Group LLC ( LMG Group ) has
PRESENTED BY: Sponsored by: Practical Uses of Analytics in E-Discovery - A PRIMER Jenny Le, Esq. Vice President of Discovery Services email@example.com E-Discovery & Ethics Structured, Conceptual,
ediscovery SERVICES Understanding How Service Providers Charge for ediscovery Services The objective of this document is to briefly define the prominent phases of the ediscovery lifecycle, the fees associated
www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence Why are non-users staying away from PC? source: edj Group s Q1 2013 Predictive Coding Survey, February 2013, N = 66 Slide 2 Introduction
MATERIALS / MATÉRIAUX 2012 Competition Law Fall Conference Conférence annuelle d'automne 2012 en droit de la concurrence E-discovery The Basics: From Proportionality to Technology Assisted Review Susan
CT TyMetrix For Claims Departments, Convergence of Matter Management and E-Billing Creates Opportunity A single litigation management solution yields numerous benefits for insurers CT Representation Services
ediscovery Document Review: Understanding the Key Differences Between Conceptual Searching and Near Duplicate Grouping White Paper by Kimberlee L. Gunning, Esq 1 INTRODUCTION As anyone who has spent time
November/December 2010 THE MAGAZINE OF THE AMERICAN INNS OF COURT rofessionalism and Ethics Issues in Today s Technology www.innsofcourt.org Transparency in E-Discovery: No Longer a Novel Approach By Michael
David Han 2015 MANAGING BIG DATA IN LITIGATION DAVID HAN Associate, Morgan Lewis & Bockius, edata Practice Group MANAGING BIG DATA Data volumes always increasing New data sources Mobile Internet of Things
Making Sense of E-Discovery: 10 Plain Steps for Producing ESI The following article provides a practical guide to producing electronically stored information (ESI) that lawyers can apply immediately in
E-DISCOVERY AND KEYWORDS: NOT SO KEY AFTER ALL FACE 2 FACE A CONFERENCE FOR LITIGATION SUPPORT CBA - NS FRIDAY, DECEMBER 7, 2012 HALIFAX, NOVA SCOTIA HALIFAX MARRIOTT HARBOURFRONT PRESENTATION BY: DANIELA
A BNA, INC. DIGITAL DISCOVERY & E-EVIDENCE! VOL. 7, NO. 11 232-235 REPORT NOVEMBER 1, 2007 Reproduced with permission from Digital Discovery & e-evidence, Vol. 7, No. 11, 11/01/2007, pp. 232-235. Copyright
How Good is Your Predictive Coding Poker Face? SESSION ID: LAW-W03 Moderator: Panelists: Matthew Nelson ediscovery Counsel Symantec Corporation Hon. Andrew J. Peck US Magistrate Judge Southern District
Take an Enterprise Approach to E-Discovery Streamline Discovery and Control Review Cost Using a Central, Secure E-Discovery Cloud Platform A Smarter Approach Catalyst s e-discovery cloud platform provides
Today s elunch Presenters John Rosenthal Litigation Washington, D.C. JRosenthal@winston.com Scott Cohen Director of E Discovery Support Services New York SCohen@winston.com 2 What Was Advertised Effective
SUSTAINABILITY Sustainability reporting What you should know kpmg.com b Sustainability reporting What you should know KPMG LLP (KPMG) defines corporate sustainability as adopting business strategies that
Understanding Search and Retrieval for Effective E- Discovery Results Digital Government Institute ediscovery, Records & Information Management March 5, 2009 Introductions Panellists Harley A.J. (Bucky)
MEALEY S TM LITIGATION REPORT Discovery The State Of Predictive Coding by Royce F. Cohen and Derek I.A. Silverman Stroock & Stroock & Lavan LLP New York A commentary article reprinted from the September
Managing Workflow Tracking in Discovery Efforts How a proactive solution can help you As used in this document, Deloitte means Deloitte Financial Advisory Services LLP, which provides forensic, dispute,
ESI: Focus on Review and Production Strategy Meredith Lee, Online Document Review Supervisor, Paralegal About Us Avansic E-discovery and digital forensics company founded in 2004 by Dr. Gavin W. Manes,
TM GUIDANCE SOFTWARE EnCASE ediscovery EnCase ediscovery Automatically search, identify, collect, preserve, and process electronically stored information across the network. GUIDANCE SOFTWARE EnCASE ediscovery
Navigating Information Governance and ediscovery Implementing Processes & Technology to Reduce Downstream ediscovery Cost and Risk Shannon Smith General Counsel, Globanet March 11 12, 2013 Agenda 1 Overview
The State Bar of Michigan Paralegal Section Annual Day of Education at Davenport University 1 May Grand Rapids, MI USA The Wave of the Future: New Tools and Resources in an E-discovery World 1 May 2009
Relativity and Beyond e-discovery Review Services As a Relativity Premium Hosting Partner, we bring nearly three decades of litigation support experience to one of the most popular, respected and feature-rich
Relativity and Beyond e-discovery Review Services As a Relativity Premium Hosting Partner, we bring nearly three decades of litigation support experience to one of the most popular, respected and feature-rich
KPMG Forensic Technology Services Managing Costs in e-discoverye October 14, 2010 1 Agenda: Strategies to Manage Costs in e-discovery Pre-collection Strategies Filtering Strategies Review and Production
StoredIQ s Flagship ediscovery for SharePoint February 2011 Many ediscovery companies announced integration with Microsoft SharePoint last year. Frankly, our usual response was a resounding yawn. These
Simplifying Cost Savings in E-Discovery PROVEN, EFFECTIVE STRATEGIES FOR RESOURCE ALLOCATION IN DOCUMENT REVIEW Simplifying Cost Savings in E-Discovery PROVEN, EFFECTIVE STRATEGIES FOR RESOURCE ALLOCATION
Investment Management Group Legal Update: SEC Adopts Rules on Compliance Programs for Funds & Advisers If you have questions or would like additional information on the material presented herein, please
EXPERT EVIDENCE PAPER 2.1 Working with an Expert These materials were prepared by Leslie J. Mackoff of Mackoff & Company, Vancouver, BC for the Continuing Legal Education Society of British Columbia, October