Quality Control for predictive coding in ediscovery. kpmg.com
|
|
- Zoe Dixon
- 8 years ago
- Views:
Transcription
1 Quality Control for predictive coding in ediscovery kpmg.com
2 Advances in technology are changing the way organizations perform ediscovery. Most notably, predictive coding, or technology assisted review, is becoming more widely accepted as part of the document review process. While it promises to be a powerful tool to reduce ediscovery costs, the strategies, implications, and leading practices for predictive coding are still evolving. More and more courts are taking up the question of acceptability of the use of predictive coding under the rules of civil procedure. The issue is whether the use of technology to replace human review is sufficient to discharge the parties discovery obligations. Predictive coding has been the subject of recent court decisions, but there has not been a definitive endorsement by a court in a case in which one party objected to the use of predictive coding by the other. 1 It is clear, however, that high standards of quality control during predictive coding will help lower the risk of a dispute over an ediscovery tool. While the courts view will continue to evolve over time, predictive coding appears likely to become a standard tool in ediscovery, and litigants should change their approach to quality control in ediscovery as a result. This paper will discuss several strategies that companies can use now for improving quality control while using predictive coding for document review. Training Phase Select Sample Code Train Measure Improve Application Phase Cull Review Validate Produce 1 See Da Silva Moore v. Publicis Group & MSL Group, No. 11 Civ (ALC) (AJP), 2012 U.S. Dist. LEXIS (S.D.N.Y. Feb. 24, 2012) (endorsing predictive coding where both parties agreed to use it but argued over methodology); Kleen Products v. Packaging Corp of America, No. 10 C 5711, 2012 U.S. Dist. LEXIS (N.D. Ill. Sept. 28, 2012) (Parties ultimately agreed to use Boolean search terms instead of predictive coding); EORHB, Inc. et al v. HOA Holdings, LLC, C.A. No VCL (Del. Ch. Oct. 15, 2012) (court sua sponte ordered parties to use predictive coding)
3 Quality Control for predictive coding in ediscovery 2 How technology has changed the ediscovery process In addition to increased efficiency and reduced costs, technology has made the ediscovery process more complex, and this complexity increases risk. Statistical sampling has become a vital tool for quality control under predictive coding, and the complexity of the predictive coding work flow has highlighted the need for project management strategies to mitigate risk. Under the traditional model of document review, documents were examined, coded, and organized by hand. The first electronic tools for document review were designed to recreate this process, as electronic documents were divided into assignments in the order in which they were loaded onto the software, and each document was reviewed linearly. Using an electronic document depository with the ability to extract metadata from each document, however, has enabled work flow improvements and allowed for stratification, prioritization, and contextualization. Stratification means that file types like spreadsheets or image files can be handled separately, perhaps even by specialized review teams. Prioritization based on search terms, date ranges, or other meta data can be used to similar effect. Finally, technology permits documents to be contextualized by extracting concepts and presenting the reviewer with clusters of similar documents. These strategies have improved review efficiency and quality.
4 3 Quality Control for predictive coding in ediscovery The need for improved quality control While technical tools have led to improvements, the increased complexity only emphasizes the importance of quality control during document review, and statistical sampling is an important tool in this regard. Used early in the process, the insight gained from statistical sampling can be used to refine the training of reviewers, adjust the work flow of secondlevel review, or improve search-term lists and other strategies used to guide reviewers. When predictive coding technology is used, statistical sampling is indispensable. Predictive coding projects proceed in two distinct phases: the training phase and the application phase. The training phase involves the review of documents in order to train the classifier. The application phase involves the classifier making decisions about the documents not reviewed during the training phase. Each phase requires a different approach to quality control. In addition, underlying both the training and application phases, is the issue of work flow complexity, which deals with incremental data loads and with document types that either cannot be classified or that require special treatment. Training: Predictive coding software must be trained to recognize the same distinctions between responsive and nonresponsive documents as a person reviewing the documents would recognize. The benefit of predictive coding lies in the fact that human decisions are limited to the training set, and are then leveraged across the entire body of documents. Thus, each decision on a training document potentially disposes of many documents. The primary quality-control goal during the training phase is to achieve consistent and accurate coding of the training documents, since consistency and accuracy in the training documents will determine the success of the predictive coding project. Human reviewers tend to be inconsistent, including during the training phase. Their views on the classification of documents evolve, and mistakes happen. With a large set of training documents, the predictive-coding software has the capacity, within limits, to correct mistakes made to a small portion of the documents during training, and the software will still learn the correct coding for the type of document. The impact of inconsistent coding during the training phase thus depends on the absolute number of training documents that are relevant to the specific issue. For this reason, it is important to monitor the prevalence, also known as richness, of responsive documents among nonresponsive documents during the training phase. Particular care should be taken to conduct effective quality control during the training phase. One approach is a double blind review of the training set in which the training documents are reviewed by two independent reviewers or review teams. Documents for which there is disagreement between the reviewers are then reviewed by a subject-matter authority to resolve disagreements about the training documents before they become the basis for training the classifier. The same effect can be achieved by using the predictive-coding classifier to review the training documents. Predictive coding classifiers abstract from specific training documents to discover patterns and similarities, and the classifier will often suggest that training documents be coded differently. Those documents should be reviewed again, ideally by a subject matter authority, to resolve the issue. The classifier can then be retrained to avoid the problem in the future. Notably, some predictive coding solutions feature built-in consistency checks designed to eliminate disagreements between the authority and the software. Application: There are several options for how predictive coding can be applied. In a traditional coding work flow, where each document is reviewed, predictive coding can be used to reveal inconsistencies and function as a powerful quality-control tool. Predictive coding can also be used as the basis of production decisions by separating documents that were not reviewed but were classified as responsive by the predictive coding technology. In the most popular approach, predictive coding is used to eliminate from further review documents that were classified as nonresponsive, while documents classified as responsive are then reviewed. Because the majority of documents are typically classified as nonresponsive, this last approach improves efficiency while eliminating the risks of producing documents that were not reviewed by an attorney.
5 Quality Control for predictive coding in ediscovery 4 Running Cost $2,100,000 $1,800,000 $1,500,000 $1,200,000 $900,000 $600,000 $300,000 $0 1/30/12 2/6/12 2/13/12 2/20/12 2/27/12 3/5/12 3/12/12 3/19/12 3/26/12 4/2/12 4/9/12 4/16/12 4/23/12 4/30/12 5/7/12 5/14/12 Time line Wave 1 First Pass (250,000 Docs) Wave 3 First Pass (250,000 Docs) Linear Review Review ends 5/14/2012 after 106 days at a cost of $1,968,000 Wave 1 Second Level (62,500 Docs) Wave 3 Second Level (62,900 Docs) Wave 2 First Pass (250,000 Docs) Wave 4 First Pass (250,000 Docs) Wave 2 Second Level (62,500 Docs) Wave 4 Second Level (62,900 Docs) Using predictive-coding technology to limit review by attorneys to the documents most likely to be relevant, can reduce time and overall ediscovery costs significantly. In this example, document review time was reduced from 106 days to about 20 days, and cost was reduced from nearly $2 million to about $400,000 Linear Relevance Review Review ends 2/17/2012 after 18 days at a cost of $403,500 $400,000 Running Cost $300,000 $200,000 $100,000 $0 1/30/12 1/31/12 2/1/12 2/2/12 2/3/12 2/6/12 2/7/12 2/8/12 2/9/12 2/10/12 2/13/12 2/14/12 2/15/12 2/16/12 2/17/12 Time line Relevance Loading (1,000,000 Docs) Relevance Training (2,100 Docs) Relevance Range (57,390 Docs) Relevance Range (1,569 Docs) Relevance Range (1,327 Docs) Relevance Range (0 Docs) Low Relevance (334-0) 95% Confident Sample of 996,917 Resolution Assessment (656 Docs) $400,000 Relevance Sampling Review Review ends 2/20/2012 after 21 days at a cost of $371,200 Running Cost $300,000 $200,000 $100,000 $0 1/30/12 1/31/12 2/1/12 2/2/12 2/3/12 2/6/12 2/7/12 2/8/12 2/9/12 2/10/12 2/13/12 2/14/12 2/15/12 2/16/12 2/17/12 2/20/12 Time line Relevance Loading (1,000,000 Docs) High Relevance (100-70) Second Level ( Docs) Low Relevance (34-0) 95% Confident Sample of 996, 917 Population Assessment (656 Docs) Relevance Training (2,100 Docs) Less Relevance (69-35) 95% Confident Sample of 1,749 Population Assessment (696 Docs) Using statistical sampling for quality control Regardless of the work flow used, quality control over a large number of documents classified through predictive coding remains a challenge. It is similar to quality control for large document reviews performed by humans. There are three basic options for quality control in the application phase: a second review of a subset of the documents, judgmental sampling by nonstatistical methods, or statistical sampling. Both a second-level review and judgmental sampling are important parts of a well-rounded quality control program. Statistical sampling, however, is a much more powerful way to provide insight into the overall population of documents and the quality of coding. Statistical sampling solutions should be built into any ediscovery software platform. This requirement is especially important for predictive coding applications. KPMG s proprietary enterprise-level ediscovery software, DiscoveryRadar, provides one example. There are three basic rules to remember about using statistical sampling to validate 2 the results of predictive coding: the sampling population must be defined, the sample size must be calculated correctly, and the samples must be drawn randomly. 2 Validate is used here in a meaning specific to quality control in ediscovery; it does not refer to AICPA standards.
6 5 Quality Control for predictive coding in ediscovery Define the sampling population: Statistical sampling is used to draw an inference about the population from the sample. This process first requires the population to be defined correctly in order to interpret the results correctly. There are several approaches to defining the population. The first approach is to define the population as all documents that were not part of the training set. This comprehensive approach will yield an inference about the entire document population and will test the overall quality of the predictivecoding work flow. A second approach is to sample only the set of the documents that were coded nonresponsive in order to gauge whether or how many responsive documents were missed. Calculate sample size correctly: The most common calculation of sample size is a straightforward binomial formula. The most important factors determining the sample size are the desired confidence level, error rate (confidence interval), and prevalence. The confidence level reflects the likelihood that the sample is a true representation of the overall population. For example, a 95 percent confidence level means that if 100 independent samples were randomly selected, 95 of them would accurately represent the population (within the error rate). The error rate expresses the range of expected results. For example, with an error rate of +/-5 percent, if the sample shows that 10 percent of the documents were classified incorrectly in the validation sample, the actual number will be between 5 percent and 15 percent. Increasing the sample size may lower the error rate. Finally, prevalence represents the expected percentage of responsive documents in the population. As a note of caution, correctly interpreting prevalence and adjusting the sample size accordingly requires an advanced-level understanding of statistics. Draw samples randomly: Randomizing software can help any user draw sufficiently random results easily. Randomization becomes a challenge, however, when there are changes in the population, such as the addition of new documents. In practice, a validation sample against half the documents at the midpoint of the project cannot be updated simply by sampling the second half of the documents at the completion of the project. Both validation samples may be useful, but neither will permit a statistically-valid statement about the entire population because each sample will not have been randomly selected from the total population. Getting sampling right is an important part of making the work flow defensible and ensuring quality. The KPMG white paper The case for statistical sampling in e-discovery, provides an excellent resource on statistical sampling and statistical process control in document review. Quality control for documents that require individual review The training and application phases represent the core of the predictive coding work flow. By necessity, predictive coding work flows are complex, as they require the identification and tracking of different categories of documents. In addition to the bulk of the documents for which predictions are generated, there are five categories of documents that will require individual review and should be a special focus of quality control: training documents, validation samples, ambiguous documents, nontext files, and potentially privileged documents. Training documents These documents enable the classifier to learn how a reviewer would handle a specific document. They must be reviewed to give the technology the required input and, as discussed, should be a special focus of quality control. Validation samples These randomly selected documents must be reviewed by an attorney in order to assess the performance of the predictive-coding classifier. Validation samples are statistical samples, and the rules discussed above apply. Ambiguous documents Given the variation in documents, case strategy used, and the complexity of the subject matter, predictive coding technology may not achieve sufficiently clear results for all documents, leaving these documents to be reviewed by an attorney. Depending on the software used, ambiguous documents may not be explicitly identified. Nontext documents Since predictive-coding technology is based on the content of text documents, nontext documents such as image files or poor-quality scans must be reviewed. Potentially privileged documents These documents need to be reviewed by an attorney to produce a privilege log and confirm that the information is subject to privilege.
7 Quality Control for predictive coding in ediscovery 6 Tracking, checklists, and an enterprise-level approach have emerged as the primary tools and strategies for quality control. Tracking: Quality control is essential to the integrity of the ediscovery process, and the foundation of quality control rests on the tracking of all data and activity. Ideally, a tracking system should connect all relevant information in an accessible manner, linking documents to the electronic media on which they were collected as well as to specific work flows and to the final disposition of the documents. Tracking technology should produce a record of data collection, processing, review, and production. The goal is not only to provide chain-of-custody documentation, but also to associate each document with all its relevant process-related information. Using such technology as KPMG s Global Evidence Tracking System (GETS) can help to ensure quality control and help minimize the errors that often result from manual data entry. Checklists: The second principle of quality control in ediscovery is the use of checklists. ediscovery projects are extremely complex, and the use of predictive coding only adds to the complexity. Simple checklists that are followed consistently can help mitigate the risks of error. 3 Checklists make project delivery documentable and auditable by preserving a record of the tasks performed. Checklists can also be customized for large projects and enterprise solutions. In order to unlock cost savings while minimizing risk, checklists should be living documents that are amended as optimal work flows for a particular client are developed and information is shared. An enterprise-level approach: An enterprise-level approach can reduce costs and increase efficiency by allowing the ediscovery provider to become familiar with the data sources, share information across different cases, and to avoid an ad hoc approach for each stage of the process. The enterprise-level approach can also allow for continual improvement and consistency in work flow. For example, the protection of privileged information is one of the core concerns of ediscovery, as privilege considerations are often subject to interpretation and different approaches. Corporations also tend to work with several law firms, and often with various teams within each firm. Consistency in maintaining claims of privilege among matters and over time is important, as any variation in how that information is handled increases risk. Documents that were produced in one matter may no longer be subject to privilege in other matters. 3 Effective checklists require some thought and testing. For example, checklists should use natural breaks in the work flow, be simple and logical, fit on one page, and have a clear objective. See Gawande, Atul, The Checklist Manifesto: How to Get Things Right, Conclusion Predictive coding technology demonstrates the general principle that more sophisticated technical tools lead to work flow complexity. While defensibility and disclosure requirements may be top of mind for outside counsel, successfully navigating the many moving parts in predictive coding technology should be the foremost project-management concern. Predictive coding is a powerful document review tool. Nonetheless, the increased use of technology has also increased the complexity of the ediscovery process, which can result in increased risks. Companies should consider the strategies discussed above for improving quality control during complex ediscovery work flows, particularly for predictive coding. About the author Manfred Gabriel is a principal in KPMG s Forensic Technology Services practice, where he focuses on ediscovery. He provides clients with a wide range of services from enterpriselevel ediscovery management to delivery on large, complex ediscovery projects. As a former practicing antitrust attorney, Manfred has successfully assisted clients in responding to large, fast-paced regulatory requests and in litigations, both domestic and international.
8 Contact us Kelli Brooks U.S. Forensic Technology Network Co-Leader T: E: Ed Goings U.S. Forensic Technology Network Co-Leader T: E: Manfred Gabriel Principal T: E: kpmg.com 2013 KPMG LLP, a Delaware limited liability partnership and the U.S. member firm of the KPMG network of independent member firms affiliated with KPMG International Cooperative ( KPMG International ), a Swiss entity. All rights reserved. Printed in the U.S.A. The KPMG name, logo and cutting through complexity are registered trademarks or trademarks of KPMG International. NDPPS
ESI and Predictive Coding
Beijing Boston Brussels Chicago Frankfurt Hong Kong ESI and Predictive Coding Houston London Los Angeles Moscow Munich New York Palo Alto Paris São Paulo Charles W. Schwartz Chris Wycliff December 13,
More informationPREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX?
Vol. 46 No. 3 February 6, 2013 PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX? The high costs of e-discovery have led to the development of computerized review technology by which the user may search
More informationMaking reviews more consistent and efficient.
Making reviews more consistent and efficient. PREDICTIVE CODING AND ADVANCED ANALYTICS Predictive coding although yet to take hold with the enthusiasm initially anticipated is still considered by many
More informationPredictive Coding and The Return on Investment (ROI) of Advanced Review Strategies in ediscovery
Predictive Coding and The Return on Investment (ROI) of Advanced Review Strategies in ediscovery Drew Lewis ediscovery Counsel AGENDA A Predictive Coding Primer Predictive Coding and Market Trends Predictive
More informationPredictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review
Predictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review ACEDS Webinar April 23, 2014 Sponsored by Robert Half Legal 1 2014 Robert Half Legal. An Equal Opportunity
More informationTop 10 Best Practices in Predictive Coding
Top 10 Best Practices in Predictive Coding Emerging Best Practice Guidelines for the Conduct of a Predictive Coding Project Equivio internal document " design an appropriate process, including use of available
More informationPresenters: Brett Anders, Esq. Joseph J. Lazzarotti, Esq., CIPP/US. Morristown, NJ
Presenters: Brett Anders, Esq. Joseph J. Lazzarotti, Esq., CIPP/US Morristown, NJ 1 Preservation Privacy & Data Security Search & Review 2 Pre-Litigation Data Map Litigation Hold Procedure Standardized
More informationPredictive Coding Defensibility
Predictive Coding Defensibility Who should read this paper The Veritas ediscovery Platform facilitates a quality control workflow that incorporates statistically sound sampling practices developed in conjunction
More information2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500
2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500 5218 South East Street, Suite E-3, Indianapolis, IN 46227 Tel 317.247.4400 Fax 317.247.0044 Presented by Providing
More informationLitigation Solutions insightful interactive culling distributed ediscovery processing powering digital review
Litigation Solutions i n s i g h t f u l i n t e r a c t i ve c u l l i n g d i s t r i b u t e d e d i s cove r y p ro ce s s i n g p owe r i n g d i g i t a l re v i e w Advanced Analytical Review Data
More informationPredictive Coding: Understanding the Wows & Weaknesses
Predictive Coding: Understanding the Wows & Weaknesses Bryan Callahan, CPA, CFF, CFE, CVA Managing Consultant Forensics & Valuation Services bcallahan@bkd.com Lanny Morrow, EnCE Supervising Consultant
More informationPredictive Coding Defensibility and the Transparent Predictive Coding Workflow
Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive coding is one of the most promising technologies to reduce the high cost of review by
More informationPredictive Coding: A Primer
MEALEY S TM LITIGATION REPORT Discovery Predictive Coding: A Primer by Amy Jane Longo, Esq. and Usama Kahf, Esq. O Melveny & Myers LLP Los Angeles, California A commentary article reprinted from the March
More informationPredictive Coding Defensibility and the Transparent Predictive Coding Workflow
WHITE PAPER: PREDICTIVE CODING DEFENSIBILITY........................................ Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive
More informationSoftware-assisted document review: An ROI your GC can appreciate. kpmg.com
Software-assisted document review: An ROI your GC can appreciate kpmg.com b Section or Brochure name Contents Introduction 4 Approach 6 Metrics to compare quality and effectiveness 7 Results 8 Matter 1
More informationJudge Peck Provides a Primer on Computer-Assisted Review By John Tredennick
By John Tredennick CEO Catalyst Repository Systems Magistrate Judge Andrew J. Peck issued a landmark decision in Da Silva Moore v. Publicis and MSL Group, filed on Feb. 24, 2012. This decision made headlines
More informationRecent Developments in the Law & Technology Relating to Predictive Coding
Recent Developments in the Law & Technology Relating to Predictive Coding Presented by Paul Neale CEO Presented by Gene Klimov VP & Managing Director Presented by Gerard Britton Managing Director 2012
More informationMANAGING BIG DATA IN LITIGATION
David Han 2015 MANAGING BIG DATA IN LITIGATION DAVID HAN Associate, Morgan Lewis & Bockius, edata Practice Group MANAGING BIG DATA Data volumes always increasing New data sources Mobile Internet of Things
More informationPredictive Coding, TAR, CAR NOT Just for Litigation
Predictive Coding, TAR, CAR NOT Just for Litigation February 26, 2015 Olivia Gerroll VP Professional Services, D4 Agenda Drivers The Evolution of Discovery Technology Definitions & Benefits How Predictive
More informationTechnology Assisted Review of Documents
Ashish Prasad, Esq. Noah Miller, Esq. Joshua C. Garbarino, Esq. October 27, 2014 Table of Contents Introduction... 3 What is TAR?... 3 TAR Workflows and Roles... 3 Predictive Coding Workflows... 4 Conclusion...
More informationTraditionally, the gold standard for identifying potentially
istockphoto.com/alexandercreative Predictive Coding: It s Here to Stay Predictive coding programs are poised to become a standard practice in e-discovery in the near future. As more courts weigh in on
More informationA Practitioner s Guide to Statistical Sampling in E-Discovery. October 16, 2012
A Practitioner s Guide to Statistical Sampling in E-Discovery October 16, 2012 1 Meet the Panelists Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Gordon V. Cormack, Professor at the David
More informationMaking The Most Of Document Analytics
Portfolio Media. Inc. 860 Broadway, 6th Floor New York, NY 10003 www.law360.com Phone: +1 646 783 7100 Fax: +1 646 783 7161 customerservice@law360.com Making The Most Of Document Analytics Law360, New
More informationTax Dispute Resolution Services kpmg.com
TAX Tax Dispute Resolution Services kpmg.com 1 Tax Dispute Resolution Services Are you prepared for a federal, state, or local income tax examination? Facing potential double taxation from an adjustment
More informationPredictive Coding as a Means to Prioritize Review and Reduce Discovery Costs. White Paper
Predictive Coding as a Means to Prioritize Review and Reduce Discovery Costs White Paper INTRODUCTION Computers and the popularity of digital information have changed the way that the world communicates
More informationTHE CORPORATE COUNSELOR
THE CORPORATE COUNSELOR NOVEMBER 2013 Third-Party Litigation Investing and Attorney-Client Privilege By David A. Prange Civil litigation is potentially expensive, and achieving lucrative outcomes is not
More informationTechnology- Assisted Review 2.0
LITIGATION AND PRACTICE SUPPORT Technology- Assisted Review 2.0 by Ignatius Grande of Hughes Hubbard & Reed LLP and Andrew Paredes of Epiq Systems Legal teams and their outside counsel must deal with an
More informationE-Discovery Guidance for Federal Government Professionals Summer 2014
E-Discovery Guidance for Federal Government Professionals Summer 2014 Allison Stanton Director, E-Discovery, FOIA, & Records Civil Division, Department of Justice Adam Bain Senior Trial Counsel Civil Division,
More informationE-Discovery in Mass Torts:
E-Discovery in Mass Torts: Predictive Coding Friend or Foe? Sherry A. Knutson Sidley Austin One S Dearborn St 32nd Fl Chicago, IL 60603 (312) 853-4710 sknutson@sidley.com Sherry A. Knutson is a partner
More informationFlorida E-Discovery 2013
Florida E-Discovery 2013 Christopher.Hopkins @Akerman.com Palm Beach Bar Association Employment Law Committee Florida E-Discovery 2013 Download This PPT: InternetLawCommentary.com Palm Beach Bar Association
More information13 Reasons why document management systems fail. kpmg.com
13 Reasons why document management systems fail kpmg.com Document management systems have transformed the way businesses operate. 13 Reasons why document management systems fail 1 Whether solving a paper
More informationE-discovery Taking Predictive Coding Out of the Black Box
E-discovery Taking Predictive Coding Out of the Black Box Joseph H. Looby Senior Managing Director FTI TECHNOLOGY IN CASES OF COMMERCIAL LITIGATION, the process of discovery can place a huge burden on
More informationNavigating Information Governance and ediscovery
Navigating Information Governance and ediscovery Implementing Processes & Technology to Reduce Downstream ediscovery Cost and Risk Shannon Smith General Counsel, Globanet March 11 12, 2013 Agenda 1 Overview
More informationReduce Cost and Risk during Discovery E-DISCOVERY GLOSSARY
2016 CLM Annual Conference April 6-8, 2016 Orlando, FL Reduce Cost and Risk during Discovery E-DISCOVERY GLOSSARY Understanding e-discovery definitions and concepts is critical to working with vendors,
More informationThe Case for Technology Assisted Review and Statistical Sampling in Discovery
The Case for Technology Assisted Review and Statistical Sampling in Discovery Position Paper for DESI VI Workshop, June 8, 2015, ICAIL Conference, San Diego, CA Christopher H Paskach The Claro Group, LLC
More informationViewpoint ediscovery Services
Xerox Legal Services Viewpoint ediscovery Platform Technical Brief Viewpoint ediscovery Services Viewpoint by Xerox delivers a flexible approach to ediscovery designed to help you manage your litigation,
More informationManaged Services: Maximizing Transparency and Minimizing Expense and Risk in ediscovery and Information Governance
Managed Services: Maximizing Transparency and Minimizing Expense and Risk in ediscovery and Information Governance January 18, 2013 Andrew Bayer, Director of Business Development Adam Wells, VP, Business
More informationPredictive Coding Helps Companies Reduce Discovery Costs
Predictive Coding Helps Companies Reduce Discovery Costs Recent Court Decisions Open Door to Wider Use by Businesses to Cut Costs in Document Discovery By John Tredennick As companies struggle to manage
More informationCost-Effective and Defensible Technology Assisted Review
WHITE PAPER: SYMANTEC TRANSPARENT PREDICTIVE CODING Symantec Transparent Predictive Coding Cost-Effective and Defensible Technology Assisted Review Who should read this paper Predictive coding is one of
More informationThe State Of Predictive Coding
MEALEY S TM LITIGATION REPORT Discovery The State Of Predictive Coding by Royce F. Cohen and Derek I.A. Silverman Stroock & Stroock & Lavan LLP New York A commentary article reprinted from the September
More informationREDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - PRIORITIZATION FOR 100% REVIEW. Bill Tolson Sr. Product Marketing Manager Recommind Inc.
REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - Bill Tolson Sr. Product Marketing Manager Recommind Inc. Introduction... 3 Traditional Linear Review... 3 Advanced Review Strategies: A Typical Predictive
More informationThree Methods for ediscovery Document Prioritization:
Three Methods for ediscovery Document Prioritization: Comparing and Contrasting Keyword Search with Concept Based and Support Vector Based "Technology Assisted Review-Predictive Coding" Platforms Tom Groom,
More informationThe case for statistical sampling in e-discovery
Forensic The case for statistical sampling in e-discovery January 2012 kpmg.com 2 The case for statistical sampling in e-discovery The sheer volume and unrelenting production deadlines of today s electronic
More informationMassachusetts Bay Transportation Authority
Status Report to the Fiscal Management and Control Board Massachusetts Bay Transportation Authority Year ended June 30, 2015 January 2016 kpmg.com KPMG Audit Objectives Audit Objectives Conduct an independent
More informationeops 2010: Electronic Discovery Operational Parameters Survey Executive Summary April, 2010
eops 2010: Electronic Discovery Operational Parameters Survey Executive Summary April, 2010 Better information will make E-Discovery more efficient. The multi-billion dollar electronic discovery market
More informationNew York Law Journal (Online) May 25, 2012 Friday
1 of 6 10/16/2014 2:36 PM New York Law Journal (Online) May 25, 2012 Friday Copyright 2012 ALM Media Properties, LLC All Rights Reserved Further duplication without permission is prohibited Length: 2327
More informationElectronically Stored Information in Litigation
Electronically Stored Information in Litigation Volume 69, November 2013 By Timothy J. Chorvat and Laura E. Pelanek* I. Introduction Recent developments in the use of electronically stored information
More informationE- Discovery in Criminal Law
E- Discovery in Criminal Law ! An e-discovery Solution for the Criminal Context Criminal lawyers often lack formal procedures to guide them through preservation, collection and analysis of electronically
More informationIBM ediscovery Identification and Collection
IBM ediscovery Identification and Collection Turning unstructured data into relevant data for intelligent ediscovery Highlights Analyze data in-place with detailed data explorers to gain insight into data
More informationDISCOVERY OF ELECTRONICALLY-STORED INFORMATION IN STATE COURT: WHAT TO DO WHEN YOUR COURT S RULES DON T HELP
DISCOVERY OF ELECTRONICALLY-STORED INFORMATION IN STATE COURT: WHAT TO DO WHEN YOUR COURT S RULES DON T HELP Presented by Frank H. Gassler, Esq. Written by Jeffrey M. James, Esq. Over the last few years,
More informationFEDERAL PRACTICE. In some jurisdictions, understanding the December 1, 2006 Amendments to the Federal Rules of Civil Procedure is only the first step.
A BNA, INC. DIGITAL DISCOVERY & E-EVIDENCE! VOL. 7, NO. 11 232-235 REPORT NOVEMBER 1, 2007 Reproduced with permission from Digital Discovery & e-evidence, Vol. 7, No. 11, 11/01/2007, pp. 232-235. Copyright
More informationThe Truth About Predictive Coding: Getting Beyond The Hype
www.encase.com/ceic The Truth About Predictive Coding: Getting Beyond The Hype David R. Cohen Reed Smith LLP Records & E-Discovery Practice Group Leader David leads a group of more than 100 lawyers in
More informationThe United States Law Week
The United States Law Week Source: U.S. Law Week: News Archive > 2012 > 04/24/2012 > BNA Insights > Under Fire: A Closer Look at Technology- Assisted Document Review E-DISCOVERY Under Fire: A Closer Look
More informationTransforming Internal Audit: A Maturity Model from Data Analytics to Continuous Assurance
ADVISORY SERVICES Transforming Internal Audit: A Model from Data Analytics to Assurance kpmg.com Contents Executive summary 1 Making the journey 2 The value of identifying maturity levels 4 Internal audit
More informationElectronic Discovery: Litigation Holds, Data Preservation and Production
Electronic Discovery: Litigation Holds, Data Preservation and Production April 27, 2010 Daniel Munsch, Assistant General Counsel John Lerchey, Coordinator for Incident Response 0 E-Discovery Rules Federal
More informationTechnology-Assisted Review and Other Discovery Initiatives at the Antitrust Division. Tracy Greer 1 Senior Litigation Counsel E-Discovery
Technology-Assisted Review and Other Discovery Initiatives at the Antitrust Division Tracy Greer 1 Senior Litigation Counsel E-Discovery The Division has moved to implement several discovery initiatives
More informationIntermountain ediscovery Conference 2012
Intermountain ediscovery Conference 2012 From Technology Assisted Review to Twi6er: What Clients, Law Firms, and Vendors Need to Know David Horrigan, 451 Research 451 Research Global research analyst firm
More information2011 Winston & Strawn LLP
Today s elunch Presenters John Rosenthal Litigation Washington, D.C. JRosenthal@winston.com Scott Cohen Director of E Discovery Support Services New York SCohen@winston.com 2 What Was Advertised Effective
More informationLowering E-Discovery Costs Through Enterprise Records and Retention Management. An Oracle White Paper March 2007
Lowering E-Discovery Costs Through Enterprise Records and Retention Management An Oracle White Paper March 2007 Lowering E-Discovery Costs Through Enterprise Records and Retention Management Exponential
More informationArtificial Intelligence and Transactional Law: Automated M&A Due Diligence. By Ben Klaber
Artificial Intelligence and Transactional Law: Automated M&A Due Diligence By Ben Klaber Introduction Largely due to the pervasiveness of electronically stored information (ESI) and search and retrieval
More informationLeveraging Data Analytics and Continuous Auditing. Internal Audit. January 9, 2014
Leveraging Data Analytics and Continuous Auditing to Transform Internal Audit January 9, 2014 Presenter Introductions John Isenberg, Director KPMG Risk Consulting Dallas Cortnye King, Manager KPMG Risk
More informationMastering Predictive Coding: The Ultimate Guide
Mastering Predictive Coding: The Ultimate Guide Key considerations and best practices to help you increase ediscovery efficiencies and save money with predictive coding 4.5 Validating the Results and Producing
More informationE-Discovery Tip Sheet
E-Discovery Tip Sheet Random Sampling In days past, one could look at a body of discovery and pretty well calculate how many pairs of eyeballs would be required to examine and code every document within
More informationSet out below are our comments, which are quite minor, on each of the specific guidelines.
Vincent T. Chang, Chair Federal Courts Committee New York County Lawyers Association 14 Vesey Street New York, NY 10007 March 20, 2013 COMMENTS OF THE NEW YORK COUNTY LAWYERS ASSOCIATION FEDERAL COURTS
More informationADVISORY SERVICES. Risk management in an evolving world. Making the case for social media governance. kpmg.com
ADVISORY SERVICES Risk management in an evolving world Making the case for social media governance kpmg.com Risk management in an evolving world 3 Why good governance should be the foundation of your social
More informationCOURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Fall 2014
COURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Professors:Mark Austrian Christopher Racich Fall 2014 Introduction The ubiquitous use of computers, the
More informationApril Edition of Notable Cases and Events in E-Discovery
APRIL 16, 2015 E-DISCOVERY UPDATE April Edition of Notable Cases and Events in E-Discovery This update addresses the following recent developments and court decisions involving e-discovery issues: 1. A
More informationDOCSVAULT WhitePaper. Concise Guide to E-discovery. Contents
WhitePaper Concise Guide to E-discovery Contents i. Overview ii. Importance of e-discovery iii. How to prepare for e-discovery? iv. Key processes & issues v. The next step vi. Conclusion Overview E-discovery
More informationCOST-EFFECTIVE SOLUTIONS TO MITIGATE RISK AND CONTROL LITIGATION COSTS
Legal White Paper Series: The Real Costs & Risks of Email, Part2 COST-EFFECTIVE SOLUTIONS TO MITIGATE RISK AND CONTROL LITIGATION COSTS How in-house counsel can leverage Mid-Market Information Management
More informationElements of a Good Document Retention Policy. Discovery Services WHITE PAPER
Elements of a Good Document Retention Policy Discovery Services WHITE PAPER Document retention especially the retention of electronic data has become a hot topic in the legal industry. In the wake of several
More informationE-discovery: Federal Rules of Civil Procedure and their Implications for Public Sector Corrections Departments
E-discovery: Federal Rules of Civil Procedure and their Implications for Public Sector Corrections Departments Andres De Aguero, Senior Lead, Deloitte Consulting LLP David F. Axelrod, Director, Deloitte
More informationClearwell Legal ediscovery Solution
SOLUTION BRIEF: CLEARWELL LEGAL ediscovery SOLUTION Solution Brief Clearwell Legal ediscovery Solution The Challenge: Months Delay in Ascertaining Case Facts and Determining Case Strategy, High Cost of
More informationZL UNIFIED ARCHIVE A Project Manager s Guide to E-Discovery. ZL TECHNOLOGIES White Paper
ZL UNIFIED ARCHIVE A Project Manager s Guide to E-Discovery ZL TECHNOLOGIES White Paper PAGE 1 A project manager s guide to e-discovery In civil litigation, the parties in a dispute are required to provide
More information(Previously published in The Legal Intelligencer, November 8, 2011) New Cost Guidelines for E-Discovery by Peter Vaira
(Previously published in The Legal Intelligencer, November 8, 2011) New Cost Guidelines for E-Discovery by Peter Vaira In a recent case in the Eastern District, Judge Legrome Davis upheld court costs of
More informationElectronic Discovery and the New Amendments to the Federal Rules of Civil Procedure: A Guide For In-House Counsel and Attorneys
Electronic Discovery and the New Amendments to the Federal Rules of Civil Procedure: A Guide For In-House Counsel and Attorneys By Ronald S. Allen, Esq. As technology has evolved, the federal courts have
More informationE-Discovery Best Practices
José Ramón González-Magaz jrgonzalez@steptoe.com E-Discovery Best Practices www.steptoe.com November 10, 2010 Importance of E-Discovery 92% of all data is ESI. Source: Berkeley Study. 97 billion e-mails
More informationReducing the Cost of ediscovery in Today s Economic Climate. risk of ediscovery
Reducing the Cost of ediscovery in Today s Economic Climate Strategies for reducing the cost and Strategies for reducing the cost and risk of ediscovery The 2009 Corporate ediscovery Challenge Corporate
More informationSymantec Enterprise Vault E-Discovery Connectors
Symantec Enterprise Vault E-Discovery Connectors Extending chain of custody beyond Enterprise Vault Discovery Accelerator Overview Symantec Enterprise Vault Discovery Accelerator extends the basic search
More informationE-Discovery Basics For the RIM Professional. Learning Objectives 5/18/2015. What is Electronic Discovery?
E-Discovery Basics For the RIM Professional By: Andy Sokol, CEDS, CSDS Adding A New Service Offering For Your Legal & Corporate Clients Learning Objectives What is Electronic Discovery? How Does E-Discovery
More informationThe Tested Effectiveness of Equivio>Relevance in Technology Assisted Review
ediscovery & Information Management White Paper The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review Scott M. Cohen Elizabeth T. Timkovich John J. Rosenthal February 2014 2014 Winston
More informationPros And Cons Of Computer-Assisted Review
Portfolio Media. Inc. 860 Broadway, 6th Floor New York, NY 10003 www.law360.com Phone: +1 646 783 7100 Fax: +1 646 783 7161 customerservice@law360.com Pros And Cons Of Computer-Assisted Review Law360,
More informationThe Predictive Coding Soundtrack: Rewind, Play, Fast-Forward
The Predictive Coding Soundtrack: Rewind, Play, Fast-Forward LEGALTECH NEW YORK February 3, 2015 Moderator: Amy Hinzmann Senior Vice President, DiscoverReady DiscoverReady 2014 THE PANELISTS* Marla Bergman
More informationTHE PREDICTIVE CODING CASES A CASE LAW REVIEW
THE PREDICTIVE CODING CASES A CASE LAW REVIEW WELCOME Thank you for joining Numerous diverse attendees Please feel free to submit questions Slides, recording and survey coming tomorrow SPEAKERS Matthew
More informationOrganizations today are faced with an CORPORATE COUNSEL PROVIDE AN INSIDE LOOK AT THE STATE OF E-DISCOVERY EXECUTIVE REPORT
2014 www.bdo.com EXECUTIVE REPORT CORPORATE COUNSEL PROVIDE AN INSIDE LOOK AT THE STATE OF E-DISCOVERY Survey Finds E-Discovery Effectiveness Requires Significant Improvement, While Scope and Total Cost
More informationKPMG Forensic Technology Services
KPMG Forensic Technology Services Managing Costs in e-discoverye October 14, 2010 1 Agenda: Strategies to Manage Costs in e-discovery Pre-collection Strategies Filtering Strategies Review and Production
More informationSimplify the e-discovery process by learning which tools to use and when to use them. CHAPTER 7. Proactive. Review tools. litigation hold tools.
THE WINDOWS MANAGER S GUIDE TO INSIDE: Reactive litigation hold tools Proactive litigation hold tools Review tools Enterprise search tools Archive systems CHAPTER Exploring e-discovery tools Simplify the
More informationediscovery and Information Governance Practice Overview
ediscovery and Information Governance Practice Overview ediscovery and Information Governance Electronic discovery, or ediscovery, is increasingly changing from the exception to the norm in modern litigation.
More informationwww.pwc.nl Review & AI Lessons learned while using Artificial Intelligence April 2013
www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence Why are non-users staying away from PC? source: edj Group s Q1 2013 Predictive Coding Survey, February 2013, N = 66 Slide 2 Introduction
More informationMeasurement in ediscovery
Measurement in ediscovery A Technical White Paper Herbert Roitblat, Ph.D. CTO, Chief Scientist Measurement in ediscovery From an information-science perspective, ediscovery is about separating the responsive
More informationIBM Unstructured Data Identification and Management
IBM Unstructured Data Identification and Management Discover, recognize, and act on unstructured data in-place Highlights Identify data in place that is relevant for legal collections or regulatory retention.
More informationPretrial Practice Course Syllabus Spring, 2014 Meeting -- Tuesdays 1:30-3:20pm Room -- 432(C)
Pretrial Practice Course Syllabus Spring, 2014 Meeting -- Tuesdays 1:30-3:20pm Room -- 432(C) Professor: Rich Kelsey Telephone: (703) 993-8973 Email: rkelsey@gmu.edu Course Materials Material will be assigned
More informationOnly 1% of that data has preservation requirements Only 5% has regulatory requirements Only 34% is active and useful
Page 1 LMG GROUP vs. THE BIG DATA TIDAL WAVE Recognizing that corporations, law firms and government entities are faced with tough questions in today s business climate, LMG Group LLC ( LMG Group ) has
More informationProactive Data Management for ediscovery
Proactive Data Management for ediscovery Simon Taylor Snr. Director Information Management CommVault Systems Inc. Why ediscovery sucks for IT The US Federal Rules of Civil Procedure Rule 34(a), (b) Definition
More informationE-Discovery Getting a Handle on Predictive Coding
E-Discovery Getting a Handle on Predictive Coding John J. Jablonski Goldberg Segalla LLP 665 Main St Ste 400 Buffalo, NY 14203-1425 (716) 566-5400 jjablonski@goldbergsegalla.com Drew Lewis Recommind 7028
More informationThis Webcast Will Begin Shortly
This Webcast Will Begin Shortly If you have any technical problems with the Webcast or the streaming audio, please contact us via email at: accwebcast@commpartners.com Thank You! Welcome! Electronic Data
More informationProactively Using Information Governance and Advance Planning to Reduce the Burden and Expense of E-Discovery
KNOW THYSELF: Proactively Using Information Governance and Advance Planning to Reduce the Burden and Expense of E-Discovery Jonathan D. Rudolph, General Counsel of Accumen Data Services Jeffrey D. Bukowski,
More informationcase 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION
case 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION IN RE: BIOMET M2a MAGNUM HIP IMPLANT PRODUCTS LIABILITY
More informationEnhancing Document Review Efficiency with OmniX
Xerox Litigation Services OmniX Platform Review Technical Brief Enhancing Document Review Efficiency with OmniX Xerox Litigation Services delivers a flexible suite of end-to-end technology-driven services,
More informationFrom Chaos to Clarity.
LITIGATION READINESS 3 PRESERVATION & COLLECTION 3 PROCESSING 3 DATA ANALYTICS 3 DOCUMENT REVIEW 3 PRODUCTION 3 POST PRODUCTION From Chaos to Clarity. The AlixPartners Difference Experienced. AlixPartners
More information