Predictive Coding Defensibility

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Predictive Coding Defensibility"

Transcription

1 Predictive Coding Defensibility Who should read this paper The Veritas ediscovery Platform facilitates a quality control workflow that incorporates statistically sound sampling practices developed in conjunction with expert statisticians, leading enterprises, law firms, and legal service providers. As legal professionals approach a case using predictive coding, linear review, or a combination, this patent-pending sampling technology and workflow from Veritas makes it easy for reviewers to ensure the defensibility of the review process.

2 Content Introduction Measuring review accuracy Challenges with existing approaches The Veritas Solution Conclusion

3 Introduction Numerous technologies and practices have been developed over the last decade to address the challenges of electronic discovery. Among them are more targeted collection tools, intelligent data culling, and early case assessments which have made this process more manageable and cost-effective. However, document review remains expensive whether organizations perform it in-house, through an outside law firm, or a service provider. Predictive coding is an exciting technological development in electronic discovery because it has the potential to reduce review time and cost while simultaneously improving review accuracy. Despite the widespread misconception that linear review is the electronic discovery process gold standard, exhaustive manual review is surprisingly inaccurate, considering its high cost. Academic research on legal review as part of the TREC Legal Track has shown linear review is often only percent accurate. 1 Predictive coding technology involves an iterative process that senior attorneys follow to train software on review criteria, creating a mathematical model that predictive coding software uses to generate predictions of how the remaining documents would otherwise be tagged if reviewed by an experienced attorney. Studies show that predictive coding can achieve much higher levels of accuracy at a fraction of the time and cost. 2 While many believe recent court cases such as Da Silva Moore v. Publicis Groupe provide initial judicial acceptance of for using predictive coding more widely, this and other recent cases also provide some cautionary lessons. Perhaps most importantly, since the accuracy of predictive coding can depend on a number of factors such as the number and quality of training iterations it is not sufficient to use academic studies as evidence that predictive coding is accurate. Parties using predictive coding should be prepared to measure review accuracy in a statistically sound manner to demonstrate their results for every case. Given the disagreements over the use of predictive coding in these early cases, the key question for many electronic discovery practitioners is how to measure review accuracy in a cost-effective and defensible way. Measuring review accuracy Academic studies that measure review accuracy rely on statistical random sampling, a well-accepted method for estimating the characteristics of a large population. Sampling is routinely used in many domains including opinion polling and product quality control. The underlying principle is that by measuring the characteristics of a small random sample, one can project those same characteristics across a much larger population with a known degree of error. For instance, in an election 46 percent of voters plan to vote for Candidate A with a margin of error of +/- three percent. While manufacturers commonly use sampling to ensure the quality of products coming off an assembly line, until now it was rarely used in electronic discovery for measuring the quality of document review, or even the results of predictive coding. However, as predictive coding becomes more widely adopted for electronic discovery, it will be increasingly important to demonstrate the results of the review process in a scientific way. According to Judge Peck of the Southern District of New York, if predictive coding is challenged, parties should be prepared to discuss whether it produced responsive documents with reasonably high Recall and high Precision Overview of the TREC 2010 Legal Track (2012) 2. Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review 3. Andrew Peck, Search, Forward, October 2011 issue of Law Technology News 3

4 Before diving into sampling, parties should first understand the metrics used to measure accuracy in electronic discovery: Recall, Precision, and F-measure. Using the right metrics For most people, the most intuitive method for measuring predictive coding accuracy may be simply calculating the percentage of documents predicted correctly. If 80 out of 100 documents are correctly predicted, the accuracy would be 80 percent. One of the standard methods of calculating a test score is to calculate the number of questions answered correctly, divide it by total number of questions on the test and multiply the resulting number by 100 to get a percentage value. This is not an ideal way to measure accuracy in electronic discovery, which is an exercise in finding as many responsive documents, not simply categorizing all documents correctly. For example, let s assume there are 50,000 documents in a case and each has been reviewed by attorneys and predictive coding software, resulting in the human-software comparison chart shown below. Number of documents Human decision Software prediction Description 2,000 Responsive Responsive Agreements: Human and software agree on responsiveness 6,000 Responsive Not-Responsive Disagreements: Human and software disagree on responsiveness 40,000 Not-Responsive Not-Responsive Agreements: Human and software agree on non-responsiveness 2,000 Not-Responsive Responsive Disagreements: Human and software disagree on non-responsiveness Table 1: Comparison of human review decisions with software predictions of responsiveness. Based on this chart, one could calculate that out of 50,000 total documents, the software predicted 42,000 documents (sum of row #1 and #3) correctly and therefore its accuracy is 84 percent (42,000/50,000). However, analyzing the chart closely reveals a very different picture. The results of human review shows there are 8,000 total responsive documents (sum of row #1 and #2) but the software found only 2,000 of those (row #1), meaning software was able to find only 25 percent of truly responsive documents. This measure is called Recall. 4 Also, of 4,000 documents the software predicted to be responsive (sum of row #1 and #4), only 2,000 are actually responsive (row #1), meaning software is right only 50 percent of the times when it predicts a document responsive. This measure is called Precision. 5 Why is Recall only 25 percent and Precision only 50 percent when the software s predictions are right 84 percent of the time? That s because the software did very well at predicting non-responsive documents. Based on the human review, there are 42,000 non-responsive documents (sum of row #3 and #4), of which the software found 40,000 correctly, meaning it is right 95 percent (40,000/42,000) of time when it predicts a document to be non-responsive. While the software is right only 50 percent of the time when predicting a document responsive, it is right 95 percent of the time when predicting a document non-responsive, driving up the percentage of correct predictions across all documents to 84 percent. 4. Recall is the number or percentage of truly responsive documents identified within a defined document population that is identified as responsive. In other words, Recall is a measure of completeness. 5. Refers to the number or percentage of documents identified within a defined document population that are truly responsive. In other words, Precision is a measure of exactness. 4

5 During electronic discovery, the objective is to make a reasonable effort to identify as many responsive documents, while at the same time minimizing the number of non-responsive documents which are produced. The example above illustrates that the percentage of correct predictions across all documents metric may paint an inaccurate view of the number of responsive documents found or missed by the software. This is especially true when most of the documents in a case are non-responsive which happen to be the most common scenario in electronic discovery. Therefore, most academics use Recall and Precision to measure the accuracy of review since the electronic discovery process is an exercise in maximizing both of these metrics. One of the drawbacks of relying solely on Recall and Precision is that electronic discovery practitioners often have to make tradeoffs which increases one metric while decreasing the other. For instance, in some cases using broader search criteria will yield more responsive documents which raises Recall, but at the same time will increase the number of nonresponsive items as well which will decrease Precision. This can make it challenging to gauge whether a set of Recall and Precision numbers is better or more accurate than a corresponding set of Recall and Precision measures for another case or even the same case. The F1 or F-measure is a statistical metric that is used to measure predictive coding accuracy and evaluate these tradeoffs. 6 While it is not an arithmetic average of Recall and Precision, F-measure can generally be thought of as a balance between these two numbers. In the above example, if one plugs in 25 percent Recall and 50 percent Precision into a calculator, the resulting F-measure is 33.3 percent. Comparing the F-measure provides reviewers with an objective standard by which to assess review accuracy. When academic studies reference the accuracy of linear review versus predictive coding, they are usually referring to F-measure. What is a good Recall, Precision, or F-measure? There is no standard measure that indicates when a review team has achieved an accurate result. Courts have left it up to the parties involved in discovery to agree to a reasonable percentage based on the time, cost, and risk tradeoffs. When using predictive coding, achieving higher accuracy generally requires higher costs to review more documents as part of the iterative training process. One viewpoint is that once reviewers using predictive coding on a case reach a level of accuracy equal to or greater to their best efforts using linear review the court and opposing party should find this sufficient. Other organizations may seek to reach 80 percent or even 90 percent Recall on high risk maters but perhaps only 60 percent or 70 percent Recall for other matters. Using sampling to measure accuracy cost-effectively If a review team performed linear review in parallel with predictive coding, it would be fairly straightforward to populate Table 1 and calculate the Recall, Precision, and F-measure of the case. However, the goal of predictive coding is to reduce review cost by reducing the manual effort spent on review. This problem is not unique to electronic discovery. An automotive assembly plant produces thousands of cars each day and all of them need to meet certain safety and quality standards. It would be impractical to perform a crash test on all of these vehicles to ensure they were safe for customers. The same is true of many other manufactured goods. 6. Refers to the balance or harmonic mean between Precision and Recall. The F-measure is used to measure accuracy

6 Acceptance testing was developed to address the challenges of measuring product quality, which relies on measuring the quality of a few units from a statistically random sample. Mathematically, as long as the sample size is statistically valid and items are chosen randomly, the sample represents the characteristics of the larger population with a known margin of error. Whether companies are manufacturing products like cars or bullets where the stakes for product quality are very high, or performing document review where the stakes can be equally high, random sampling provides a way to test accuracy cost-effectively. In order to test the accuracy of predictive coding using sampling, review teams calculate the appropriate sample size required for accuracy testing based on criteria such as the population size and desired margin of error. Documents are selected at random and set aside in a control set. Once reviewed by the review team and the predictive coding engine, Recall, Precision, and F-measure can be calculated. For example, if the results show that the predictive coding has achieved a Precision of 80 percent with a margin of error of +/-three percent, then mathematically these results can be applied to the population of documents in the entire case. This approach allows reviewers to measure accuracy in a reliable and cost-effective way. Challenges with existing approaches As discussed earlier, a sample size must be statistically-valid in order to apply the results from sampling to the entire population. First generation predictive coding solutions have relied on basic statistical calculators for determining sample size. These calculators typically use just three inputs for determining sample size: population, confidence level, and margin of error, and overlooking other factors that are rarely needed to calculate sample size. While these calculators often work in many different domains, there are situations where they are inappropriate due to the type of data being measured. In fact, these simple calculators often do not work for electronic discovery data sets. Basic sample size calculators assume that the characteristic being measured is the majority of the population. In car manufacturing, cars that meet the quality standards are typically the majority of the cars being tested. That means the sample size suggested by a basic sample size calculator is valid for quality testing. In electronic discovery, however, the characteristic being measured is responsive documents. In most matters, responsive documents are not the majority of documents and often they comprise only a small percentage of documents in the case. A case with a low percentage of responsive documents is a case with low yield and a basic calculator will not work. In these situations, the margin of error increases beyond what is predicted by a basic sampling calculator. As a result, the accuracy measures from sampling are not useful to show results to the court or opposing counsel because the margin of error can be in the double digits. To understand this, let s look at a real-life example. Assume we are tasked with estimating the number of 7 tall individuals in the U.S. We know intuitively that the vast majority of population is not 7 tall, and there are very few people this tall. A basic statistical calculator would suggest a sample size of 2401 given the input of 312 million population size, 95 percent confidence-level, and 2 percent margin of error. However, it is very likely that in the randomly selected 2401 individuals, none is 7 tall. But, since the sample is assumed to be statistically-valid, one could project the results on the U.S. population and conclude that there are no 7 tall individuals in the country, which we know is not the case. Thus, the margin of error on the projections in this case is 100 percent, dramatically increasing from 2 percent (specified at the time of creating the sample). Many legal teams would find it difficult to show to the court that Recall is between 0 and 80 percent. 6

7 In order to confidently use sampling in situations with high stakes such as electronic discovery, it is necessary to look at all of the variables which can impact sample size and margin of error. The Veritas Solution Working with expert statisticians at Stanford and electronic discovery practitioners at leading enterprises, law firms, and service providers, the Veritas ediscovery Platform, was developed to deliver a robust set of review quality control capabilities. This Veritas solution provides an easy-to-use, built-in sampling workflow to verify review accuracy, whether performed using linear review, predictive coding, or a combination of the two. Taking into account all factors that make sampling statistically sound, the ediscovery Platform provides reviewers with the ability to leverage sampling without a background in statistics or the need to hire a statistician for each case. Figure 1: Veritas takes into account all factors affecting sample size in order to minimize margin of error. Key benefits of review quality control include: Advanced Sampling Using a two-step patent pending approach to sampling, the ediscovery Platform automatically calculates the statistically valid sample size taking into account all variables that impact sampling, including yield and desired F-measure. This capability goes beyond basic sample size calculators to ensure the results meet the defensibility requirements of electronic discovery. Random document selection The ediscovery Platform automatically selects documents randomly from across the case based on the unique sample size required for accuracy testing. This takes the guesswork out of picking documents used to test predictive coding or linear review accuracy. Automatic accuracy calculation Once review teams login and tag documents in the control set, the ediscovery Platform automatically calculates accuracy metrics including Recall, Precision, and F-measure. Tracking accuracy also allows reviewers to estimate the cost of achieving higher accuracy. Reporting for third parties Once accuracy has been established using the review quality control capabilities of the ediscovery Platform, review teams can export reports detailing all steps used in testing accuracy and associated metrics. These reports are useful for demonstrating the results of predictive coding to the court or opposing counsel. 7

8 Figure 2: Advanced Sampling: Intuitive statistical sampling tools help users select an appropriate random sample based on the yield of the case documents. With a real-time view on predictive coding accuracy, legal teams can make informed decisions to improve the accuracy of review and have confidence in the results of their review process. Having visibility into the accuracy of review and the cost of achieving higher levels of accuracy also enables organizations to make informed arguments for proportionality. Conclusion While predictive coding is a promising technology to address the time and cost of electronic discovery, recent cases demonstrate the need to measure review accuracy by following statistically sound sampling practices. There are a number of sampling nuances that can derail even the most prepared legal teams, making it challenging to ensure the defensibility of review without hiring a third-party expert. Veritas has worked with statisticians and practitioners to develop an easy to use quality control workflow that incorporates statistical sampling best practices. Using the Review Quality Control capabilities of the ediscovery Platform, review teams can measure review accuracy, whether they use linear review, predictive coding, or in a combination of the two. The Veritas sampling tools help ensure the highest degree of defensibility, allowing legal teams to experience the cost, time and accuracy benefits of predictive coding while having confidence in the results. 8

9 About Veritas Technologies Corporation Veritas Technologies Corporation enables organizations to harness the power of their information, with solutions designed to serve the world s largest and most complex heterogeneous environments. Veritas works with 86 percent of Fortune 500 companies today, improving data availability and revealing insights to drive competitive advantage. For specific country offices and contact numbers, please visit our website. Veritas World Headquarters 500 East Middlefield Road Mountain View, CA (650) Veritas Technologies Corporation. All rights reserved. Veritas and the Veritas Logo are trademarks or registered trademarks of Veritas Technologies Corporation or its affiliates in the U.S. And other countries. Other names may be trademarks of their respective owners /2015 9

Predictive Coding Defensibility and the Transparent Predictive Coding Workflow

Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive coding is one of the most promising technologies to reduce the high cost of review by

More information

Predictive Coding Defensibility and the Transparent Predictive Coding Workflow

Predictive Coding Defensibility and the Transparent Predictive Coding Workflow WHITE PAPER: PREDICTIVE CODING DEFENSIBILITY........................................ Predictive Coding Defensibility and the Transparent Predictive Coding Workflow Who should read this paper Predictive

More information

Cost-Effective and Defensible Technology Assisted Review

Cost-Effective and Defensible Technology Assisted Review WHITE PAPER: SYMANTEC TRANSPARENT PREDICTIVE CODING Symantec Transparent Predictive Coding Cost-Effective and Defensible Technology Assisted Review Who should read this paper Predictive coding is one of

More information

Top 10 Best Practices in Predictive Coding

Top 10 Best Practices in Predictive Coding Top 10 Best Practices in Predictive Coding Emerging Best Practice Guidelines for the Conduct of a Predictive Coding Project Equivio internal document " design an appropriate process, including use of available

More information

Veritas ediscovery Platform

Veritas ediscovery Platform TM Veritas ediscovery Platform Overview The is the leading enterprise ediscovery solution that enables enterprises, governments, and law firms to manage legal, regulatory, and investigative matters using

More information

Judge Peck Provides a Primer on Computer-Assisted Review By John Tredennick

Judge Peck Provides a Primer on Computer-Assisted Review By John Tredennick By John Tredennick CEO Catalyst Repository Systems Magistrate Judge Andrew J. Peck issued a landmark decision in Da Silva Moore v. Publicis and MSL Group, filed on Feb. 24, 2012. This decision made headlines

More information

REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - PRIORITIZATION FOR 100% REVIEW. Bill Tolson Sr. Product Marketing Manager Recommind Inc.

REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - PRIORITIZATION FOR 100% REVIEW. Bill Tolson Sr. Product Marketing Manager Recommind Inc. REDUCING COSTS WITH ADVANCED REVIEW STRATEGIES - Bill Tolson Sr. Product Marketing Manager Recommind Inc. Introduction... 3 Traditional Linear Review... 3 Advanced Review Strategies: A Typical Predictive

More information

A Practitioner s Guide to Statistical Sampling in E-Discovery. October 16, 2012

A Practitioner s Guide to Statistical Sampling in E-Discovery. October 16, 2012 A Practitioner s Guide to Statistical Sampling in E-Discovery October 16, 2012 1 Meet the Panelists Maura R. Grossman, Counsel at Wachtell, Lipton, Rosen & Katz Gordon V. Cormack, Professor at the David

More information

Quality Control for predictive coding in ediscovery. kpmg.com

Quality Control for predictive coding in ediscovery. kpmg.com Quality Control for predictive coding in ediscovery kpmg.com Advances in technology are changing the way organizations perform ediscovery. Most notably, predictive coding, or technology assisted review,

More information

Software-assisted document review: An ROI your GC can appreciate. kpmg.com

Software-assisted document review: An ROI your GC can appreciate. kpmg.com Software-assisted document review: An ROI your GC can appreciate kpmg.com b Section or Brochure name Contents Introduction 4 Approach 6 Metrics to compare quality and effectiveness 7 Results 8 Matter 1

More information

Veritas Enterprise Vault for Microsoft Exchange Server

Veritas Enterprise Vault for Microsoft Exchange Server Veritas Enterprise Vault for Microsoft Exchange Server Store, manage, and discover critical business information Trusted and proven email archiving Veritas Enterprise Vault, the industry leader in email

More information

Managed Services: Maximizing Transparency and Minimizing Expense and Risk in ediscovery and Information Governance

Managed Services: Maximizing Transparency and Minimizing Expense and Risk in ediscovery and Information Governance Managed Services: Maximizing Transparency and Minimizing Expense and Risk in ediscovery and Information Governance January 18, 2013 Andrew Bayer, Director of Business Development Adam Wells, VP, Business

More information

www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence April 2013

www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence April 2013 www.pwc.nl Review & AI Lessons learned while using Artificial Intelligence Why are non-users staying away from PC? source: edj Group s Q1 2013 Predictive Coding Survey, February 2013, N = 66 Slide 2 Introduction

More information

The United States Law Week

The United States Law Week The United States Law Week Source: U.S. Law Week: News Archive > 2012 > 04/24/2012 > BNA Insights > Under Fire: A Closer Look at Technology- Assisted Document Review E-DISCOVERY Under Fire: A Closer Look

More information

Three Methods for ediscovery Document Prioritization:

Three Methods for ediscovery Document Prioritization: Three Methods for ediscovery Document Prioritization: Comparing and Contrasting Keyword Search with Concept Based and Support Vector Based "Technology Assisted Review-Predictive Coding" Platforms Tom Groom,

More information

E-discovery Taking Predictive Coding Out of the Black Box

E-discovery Taking Predictive Coding Out of the Black Box E-discovery Taking Predictive Coding Out of the Black Box Joseph H. Looby Senior Managing Director FTI TECHNOLOGY IN CASES OF COMMERCIAL LITIGATION, the process of discovery can place a huge burden on

More information

Technology Assisted Review of Documents

Technology Assisted Review of Documents Ashish Prasad, Esq. Noah Miller, Esq. Joshua C. Garbarino, Esq. October 27, 2014 Table of Contents Introduction... 3 What is TAR?... 3 TAR Workflows and Roles... 3 Predictive Coding Workflows... 4 Conclusion...

More information

Mastering Predictive Coding: The Ultimate Guide

Mastering Predictive Coding: The Ultimate Guide Mastering Predictive Coding: The Ultimate Guide Key considerations and best practices to help you increase ediscovery efficiencies and save money with predictive coding 4.5 Validating the Results and Producing

More information

Veritas Enterprise Vault.cloud for Microsoft Office 365

Veritas Enterprise Vault.cloud for Microsoft Office 365 TM Veritas Enterprise Vault.cloud for Microsoft Office 365 Assume control over your information ecosystem Benefits at a glance Satisfies email retention requirements by journaling an immutable copy of

More information

Five Features Your Cloud Disaster Recovery Solution Should Have

Five Features Your Cloud Disaster Recovery Solution Should Have Five Features Your Cloud Disaster Recovery Solution Should Have Content Executive summary... 3 Problems with traditional disaster recovery... 3 Benefits Azure and AWS bring to the data center... 4 5 Features

More information

The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review

The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review ediscovery & Information Management White Paper The Tested Effectiveness of Equivio>Relevance in Technology Assisted Review Scott M. Cohen Elizabeth T. Timkovich John J. Rosenthal February 2014 2014 Winston

More information

Symantec ediscovery Platform, powered by Clearwell

Symantec ediscovery Platform, powered by Clearwell Symantec ediscovery Platform, powered by Clearwell Data Sheet: Archiving and ediscovery The brings transparency and control to the electronic discovery process. From collection to production, our workflow

More information

Predictive Coding Helps Companies Reduce Discovery Costs

Predictive Coding Helps Companies Reduce Discovery Costs Predictive Coding Helps Companies Reduce Discovery Costs Recent Court Decisions Open Door to Wider Use by Businesses to Cut Costs in Document Discovery By John Tredennick As companies struggle to manage

More information

Making reviews more consistent and efficient.

Making reviews more consistent and efficient. Making reviews more consistent and efficient. PREDICTIVE CODING AND ADVANCED ANALYTICS Predictive coding although yet to take hold with the enthusiasm initially anticipated is still considered by many

More information

Recent Developments in the Law & Technology Relating to Predictive Coding

Recent Developments in the Law & Technology Relating to Predictive Coding Recent Developments in the Law & Technology Relating to Predictive Coding Presented by Paul Neale CEO Presented by Gene Klimov VP & Managing Director Presented by Gerard Britton Managing Director 2012

More information

The Evolution, Uses, and Case Studies of Technology Assisted Review

The Evolution, Uses, and Case Studies of Technology Assisted Review FEBRUARY 4 6, 2014 / THE HILTON NEW YORK The Evolution, Uses, and Case Studies of Technology Assisted Review One Size Does Not Fit All #LTNY Meet Our Panelists The Honorable Dave Waxse U.S. Magistrate

More information

ESI and Predictive Coding

ESI and Predictive Coding Beijing Boston Brussels Chicago Frankfurt Hong Kong ESI and Predictive Coding Houston London Los Angeles Moscow Munich New York Palo Alto Paris São Paulo Charles W. Schwartz Chris Wycliff December 13,

More information

2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500

2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500 2972 NW 60 th Street, Fort Lauderdale, Florida 33309 Tel 954.462.5400 Fax 954.463.7500 5218 South East Street, Suite E-3, Indianapolis, IN 46227 Tel 317.247.4400 Fax 317.247.0044 Presented by Providing

More information

Pros And Cons Of Computer-Assisted Review

Pros And Cons Of Computer-Assisted Review Portfolio Media. Inc. 860 Broadway, 6th Floor New York, NY 10003 www.law360.com Phone: +1 646 783 7100 Fax: +1 646 783 7161 customerservice@law360.com Pros And Cons Of Computer-Assisted Review Law360,

More information

A Modern Approach for Corporations Facing the Demands of Litigation

A Modern Approach for Corporations Facing the Demands of Litigation A Modern Approach for Corporations Facing the Demands of Litigation The first pure Software-as-a-Service (SaaS) e-discovery technology designed to help in-house legal teams face the increased risk and

More information

Best Practices for Streamlining Digital Investigations

Best Practices for Streamlining Digital Investigations Best Practices for Streamlining Digital Investigations Content Key Challenges Facing Digital Investigations Today 1... 3 Limitations of the Traditional Investigations Process... 3 Step 1: Collect Data

More information

E-Discovery Getting a Handle on Predictive Coding

E-Discovery Getting a Handle on Predictive Coding E-Discovery Getting a Handle on Predictive Coding John J. Jablonski Goldberg Segalla LLP 665 Main St Ste 400 Buffalo, NY 14203-1425 (716) 566-5400 jjablonski@goldbergsegalla.com Drew Lewis Recommind 7028

More information

Introduction to Predictive Coding

Introduction to Predictive Coding Introduction to Predictive Coding Herbert L. Roitblat, Ph.D. CTO, Chief Scientist, OrcaTec Predictive coding uses computers and machine learning to reduce the number of documents in large document sets

More information

E-Discovery in Mass Torts:

E-Discovery in Mass Torts: E-Discovery in Mass Torts: Predictive Coding Friend or Foe? Sherry A. Knutson Sidley Austin One S Dearborn St 32nd Fl Chicago, IL 60603 (312) 853-4710 sknutson@sidley.com Sherry A. Knutson is a partner

More information

analytics stone Automated Analytics and Predictive Modeling A White Paper by Stone Analytics

analytics stone Automated Analytics and Predictive Modeling A White Paper by Stone Analytics stone analytics Automated Analytics and Predictive Modeling A White Paper by Stone Analytics 3665 Ruffin Road, Suite 300 San Diego, CA 92123 (858) 503-7540 www.stoneanalytics.com Page 1 Automated Analytics

More information

Measurement in ediscovery

Measurement in ediscovery Measurement in ediscovery A Technical White Paper Herbert Roitblat, Ph.D. CTO, Chief Scientist Measurement in ediscovery From an information-science perspective, ediscovery is about separating the responsive

More information

Only 1% of that data has preservation requirements Only 5% has regulatory requirements Only 34% is active and useful

Only 1% of that data has preservation requirements Only 5% has regulatory requirements Only 34% is active and useful Page 1 LMG GROUP vs. THE BIG DATA TIDAL WAVE Recognizing that corporations, law firms and government entities are faced with tough questions in today s business climate, LMG Group LLC ( LMG Group ) has

More information

Viewpoint ediscovery Services

Viewpoint ediscovery Services Xerox Legal Services Viewpoint ediscovery Platform Technical Brief Viewpoint ediscovery Services Viewpoint by Xerox delivers a flexible approach to ediscovery designed to help you manage your litigation,

More information

Docketing or IP Asset Management Best Practices for In-House IP Teams

Docketing or IP Asset Management Best Practices for In-House IP Teams Docketing or IP Asset Management Best Practices for In-House IP Teams IPfolio Corporation 1900 Addison St #200 Berkeley, CA 94704, USA +1-510-981-1200 www.ipfolio.com info@ipfolio.com Table of Contents

More information

PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX?

PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX? Vol. 46 No. 3 February 6, 2013 PREDICTIVE CODING: SILVER BULLET OR PANDORA S BOX? The high costs of e-discovery have led to the development of computerized review technology by which the user may search

More information

The ediscovery Balancing Act

The ediscovery Balancing Act WHITE PAPER: THE ediscovery BALANCING ACT The ediscovery Balancing Act Striking the Right Mix of In-House and Outsourced Expertise The ediscovery Balancing Act Contents Introduction...........................................

More information

Veritas NetBackup With and Within the Cloud: Protection and Performance in a Single Platform

Veritas NetBackup With and Within the Cloud: Protection and Performance in a Single Platform Veritas NetBackup With and Within the Cloud: Protection and Performance in a Single Platform Content Highlights... 3 Cloud-enabled Backup and Recovery... 3 Integrating Veritas NetBackup with the Cloud....

More information

Data Sheet: Archiving Symantec Enterprise Vault Discovery Accelerator Accelerate e-discovery and simplify review

Data Sheet: Archiving Symantec Enterprise Vault Discovery Accelerator Accelerate e-discovery and simplify review Accelerate e-discovery and simplify review Overview provides IT/Legal liaisons, investigators, lawyers, paralegals and HR professionals the ability to search, preserve and review information across the

More information

Confidently Virtualize Business-Critical Applications in Microsoft

Confidently Virtualize Business-Critical Applications in Microsoft Confidently Virtualize Business-Critical Applications in Microsoft Hyper-V with Veritas ApplicationHA Who should read this paper Windows Virtualization IT Architects and IT Director for Windows Server

More information

Clearwell Legal ediscovery Solution

Clearwell Legal ediscovery Solution SOLUTION BRIEF: CLEARWELL LEGAL ediscovery SOLUTION Solution Brief Clearwell Legal ediscovery Solution The Challenge: Months Delay in Ascertaining Case Facts and Determining Case Strategy, High Cost of

More information

Litigation Solutions insightful interactive culling distributed ediscovery processing powering digital review

Litigation Solutions insightful interactive culling distributed ediscovery processing powering digital review Litigation Solutions i n s i g h t f u l i n t e r a c t i ve c u l l i n g d i s t r i b u t e d e d i s cove r y p ro ce s s i n g p owe r i n g d i g i t a l re v i e w Advanced Analytical Review Data

More information

Veritas Backup Exec 15: Protecting Microsoft SQL

Veritas Backup Exec 15: Protecting Microsoft SQL Veritas Backup Exec 15: Protecting Microsoft SQL Who should read this paper Technical White Papers are designed to introduce IT professionals to key technologies and technical concepts that are associated

More information

eops 2010: Electronic Discovery Operational Parameters Survey Executive Summary April, 2010

eops 2010: Electronic Discovery Operational Parameters Survey Executive Summary April, 2010 eops 2010: Electronic Discovery Operational Parameters Survey Executive Summary April, 2010 Better information will make E-Discovery more efficient. The multi-billion dollar electronic discovery market

More information

Amazing speed and easy to use designed for large-scale, complex litigation cases

Amazing speed and easy to use designed for large-scale, complex litigation cases Amazing speed and easy to use designed for large-scale, complex litigation cases LexisNexis is committed to developing new and better Concordance Evolution capabilities. All based on feedback from customers

More information

Renowned Law Firm Reduces Cost and Risk by Moving from Legacy Software to AccessData E-Discovery Suite

Renowned Law Firm Reduces Cost and Risk by Moving from Legacy Software to AccessData E-Discovery Suite LEGAL CASE STUDY Solomon Renowned Law Firm Reduces Cost and Risk by Moving from Legacy Software to AccessData E-Discovery Suite By: Introduction Solomon is a San Diego-based law firm that has provided

More information

Problem Solving and Data Analysis

Problem Solving and Data Analysis Chapter 20 Problem Solving and Data Analysis The Problem Solving and Data Analysis section of the SAT Math Test assesses your ability to use your math understanding and skills to solve problems set in

More information

INFORMATION CONNECTED

INFORMATION CONNECTED INFORMATION CONNECTED Business Solutions for the Utilities Industry Primavera Project Portfolio Management Solutions Achieve Operational Excellence with Robust Project Portfolio Management Solutions The

More information

Veritas Backup Exec : Protecting Microsoft SharePoint

Veritas Backup Exec : Protecting Microsoft SharePoint Veritas Backup Exec : Protecting Microsoft SharePoint Who should read this paper Technical White Papers are designed to introduce IT professionals to key technologies and technical concepts that are associated

More information

DSi Pilot Program: Comparing Catalyst Insight Predict with Linear Review

DSi Pilot Program: Comparing Catalyst Insight Predict with Linear Review case study DSi Pilot Program: Comparing Catalyst Insight Predict with Linear Review www.dsicovery.com 877-797-4771 414 Union St., Suite 1210 Nashville, TN 37219 (615) 255-5343 Catalyst Insight Predict

More information

Symantec Clearwell and Enterprise Vault EOOC Eating Our Own Cooking Initiative in ediscovery Enables Symantec to Save US$13 Million over Seven Years

Symantec Clearwell and Enterprise Vault EOOC Eating Our Own Cooking Initiative in ediscovery Enables Symantec to Save US$13 Million over Seven Years BUSINESS IMPACT STUDY Symantec Clearwell and Enterprise Vault EOOC Eating Our Own Cooking Initiative in ediscovery Enables Symantec to Save US$13 Million over Seven Years Executive Summary For years, Symantec

More information

Simplify SSL Certificate Management Across the Enterprise

Simplify SSL Certificate Management Across the Enterprise WHITE PAPER White Paper Simplify SSL Certificate Management Across the Enterprise Simplify SSL Certificate Management Across the Enterprise Contents introduction 1 A Platform for Single-Point Control and

More information

The case for statistical sampling in e-discovery

The case for statistical sampling in e-discovery Forensic The case for statistical sampling in e-discovery January 2012 kpmg.com 2 The case for statistical sampling in e-discovery The sheer volume and unrelenting production deadlines of today s electronic

More information

Strategies for Implementing an Effective and Defensible Legal Hold Workflow

Strategies for Implementing an Effective and Defensible Legal Hold Workflow Strategies for Implementing an Effective and Defensible Legal Hold Workflow Who should read this paper Corporate Counsel, IT/Legal Liaisons, and Messaging Administrators involved in the preservation of

More information

NightOwlDiscovery. EnCase Enterprise/ ediscovery Strategic Consulting Services

NightOwlDiscovery. EnCase Enterprise/ ediscovery Strategic Consulting Services EnCase Enterprise/ ediscovery Strategic Consulting EnCase customers now have a trusted expert advisor to meet their discovery goals. NightOwl Discovery offers complete support for the EnCase Enterprise

More information

Empower Decision-Making with Information Insight Veritas Information Governance Solutions

Empower Decision-Making with Information Insight Veritas Information Governance Solutions Empower Decision-Making with Information Insight Veritas Information Governance Solutions genius resides in the capacity True for evaluation of uncertain, hazardous, and conflicting information. Winston

More information

Data Sheet: Archiving Symantec Enterprise Vault Store, Manage, and Discover Critical Business Information

Data Sheet: Archiving Symantec Enterprise Vault Store, Manage, and Discover Critical Business Information Store, Manage, and Discover Critical Business Information Managing millions of mailboxes for thousands of customers worldwide, Enterprise Vault, the industry leader in email and content archiving, enables

More information

ediscovery WORKFLOW AUTOMATION IN SIX EASY STEPS

ediscovery WORKFLOW AUTOMATION IN SIX EASY STEPS NUIX INFORMATION PAPER ediscovery WORKFLOW AUTOMATION IN SIX EASY STEPS SUMMARY Many organizations rely on a small number of expert staff who have the process knowledge and technical expertise to deliver

More information

On-Demand CRM Executive Brief

On-Demand CRM Executive Brief On-Demand CRM Executive Brief Grow Your Business, Not Your Support Costs Creating a cost-effective, multi-channel support operation with On-Demand CRM www.tatacommunications.com/enterprise/saas/crm.asp

More information

Symantec Enterprise Vault for Microsoft Exchange

Symantec Enterprise Vault for Microsoft Exchange Symantec Enterprise Vault for Microsoft Exchange Store, manage, and discover critical business information Data Sheet: Archiving Trusted and proven email archiving Symantec Enterprise Vault, the industry

More information

Integrated Analytics. Simplified Case Administration

Integrated Analytics. Simplified Case Administration The Difference E-discovery s most complete document review and case management software. NR R Visual Review Ringtail combines powerful keyword search, concept clustering and e-discovery s best, and only,

More information

Report on App, Platform and Device Preferences from the Leader in Secure Mobility

Report on App, Platform and Device Preferences from the Leader in Secure Mobility RESEARCH REPORT GOOD TECHNOLOGY TM MOBILITY INDEX REPORT Q3 2014 Report on App, Platform and Device Preferences from the Leader in Secure Mobility This report is part of the Good Technology TM Mobility

More information

Creating an IT Infrastructure that Adapts to Your Business PLAYBOOK

Creating an IT Infrastructure that Adapts to Your Business PLAYBOOK Creating an IT Infrastructure that Adapts to Your Business PLAYBOOK F O R C H A N G E For decades, data centers have been over-provisioned two or even three times over in an attempt to plan for growth.

More information

Symantec Enterprise Vault for Microsoft Exchange Server

Symantec Enterprise Vault for Microsoft Exchange Server Symantec Enterprise Vault for Microsoft Exchange Server Store, manage, and discover critical business information Data Sheet: Archiving Trusted and proven email archiving performance and users can enjoy

More information

LexisNexis Concordance Evolution Amazing speed plus LAW PreDiscovery and LexisNexis Near Dupe integration

LexisNexis Concordance Evolution Amazing speed plus LAW PreDiscovery and LexisNexis Near Dupe integration LexisNexis Concordance Evolution Amazing speed plus LAW PreDiscovery and LexisNexis Near Dupe integration LexisNexis is committed to developing new and better Concordance Evolution capabilities. All based

More information

Veritas AdvisorMail. Email archiving, compliance, and ediscovery solution designed specifically for U.S. financial services companies

Veritas AdvisorMail. Email archiving, compliance, and ediscovery solution designed specifically for U.S. financial services companies Veritas AdvisorMail Email archiving, compliance, and ediscovery solution designed specifically for U.S. financial services companies Email compliance redefined Our new and improved version of redefines

More information

The Random Sampling Road to Reasonableness. Reduce Risk and Cost by Employing a Complete and Integrated Validation Process

The Random Sampling Road to Reasonableness. Reduce Risk and Cost by Employing a Complete and Integrated Validation Process The Random Sampling Road to Reasonableness Reduce Risk and Cost by Employing a Complete and Integrated Validation Process By: Michael R. Wade Planet Data Executive Vice President Chief Technology Officer

More information

Unified ediscovery Platform White Paper @LEGAL DISCOVERY, LLC. www.legaldiscoveryllc.com info@legaldiscoveryllc.com 1-877-215-9508

Unified ediscovery Platform White Paper @LEGAL DISCOVERY, LLC. www.legaldiscoveryllc.com info@legaldiscoveryllc.com 1-877-215-9508 Unified ediscovery Platform White Paper @LEGAL DISCOVERY, LLC www.legaldiscoveryllc.com info@legaldiscoveryllc.com 1-877-215-9508 Benefits of a Unified ediscovery Platform Litigators have often used technology

More information

The Importance of Data Quality for Intelligent Data Analytics:

The Importance of Data Quality for Intelligent Data Analytics: The Importance of Data Quality for Intelligent Data Analytics: Optimizing the Financial and Operational Performance of IT White Paper IT decisions are only as good as the data they re based on. And that

More information

Legal exchange. Total Legal Spend Management Solution for Corporate legal departments

Legal exchange. Total Legal Spend Management Solution for Corporate legal departments Legal exchange Total Legal Spend Management Solution for Corporate legal departments Delivering greater efficiency, insight and control of Legal Spend. That s Uniquely Bottomline. With a continued reliance

More information

GOOD TECHNOLOGY TM MOBILITY INDEX REPORT Q2 2014

GOOD TECHNOLOGY TM MOBILITY INDEX REPORT Q2 2014 RESEARCH REPORT GOOD TECHNOLOGY TM MOBILITY INDEX REPORT Q2 2014 Report on App, Platform and Device Preferences from the Leader in Secure Mobility This report is part of the Good Technology Mobility Index,

More information

Predictive Coding as a Means to Prioritize Review and Reduce Discovery Costs. White Paper

Predictive Coding as a Means to Prioritize Review and Reduce Discovery Costs. White Paper Predictive Coding as a Means to Prioritize Review and Reduce Discovery Costs White Paper INTRODUCTION Computers and the popularity of digital information have changed the way that the world communicates

More information

Copyright 2000-2007, Pricedex Software Inc. All Rights Reserved

Copyright 2000-2007, Pricedex Software Inc. All Rights Reserved The Four Pillars of PIM: A white paper on Product Information Management (PIM) for the Automotive Aftermarket, and the 4 critical categories of process management which comprise a complete and comprehensive

More information

Predictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review

Predictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review Predictive Coding: How to Cut Through the Hype and Determine Whether It s Right for Your Review ACEDS Webinar April 23, 2014 Sponsored by Robert Half Legal 1 2014 Robert Half Legal. An Equal Opportunity

More information

Predictive Coding: E-Discovery Game Changer?

Predictive Coding: E-Discovery Game Changer? PAGE 11 Predictive Coding: E-Discovery Game Changer? By Melissa Whittingham, Edward H. Rippey and Skye L. Perryman Predictive coding promises more efficient e- discovery reviews, with significant cost

More information

Business intelligence

Business intelligence Business intelligence with Microsoft Dynamics GP Microsoft Dynamics GP: The proven solution for efficiency and insight across your business. More than 40,000 customers use Microsoft Dynamics GP. And for

More information

Pros And Cons Of Statistical Sampling

Pros And Cons Of Statistical Sampling Portfolio Media. Inc. 860 Broadway, 6th Floor New York, NY 10003 www.law360.com Phone: +1 646 783 7100 Fax: +1 646 783 7161 customerservice@law360.com Pros And Cons Of Statistical Sampling Law360, New

More information

Real-time asset location visibility improves operational efficiencies

Real-time asset location visibility improves operational efficiencies Real-time asset location visibility improves operational efficiencies Offering smart capabilities for asset tracking to dramatically improve efficiency and lower cost Highlights Improve asset utilization

More information

INFORMATION MANAGED. Project Management You Can Build On. Primavera Solutions for Engineering and Construction

INFORMATION MANAGED. Project Management You Can Build On. Primavera Solutions for Engineering and Construction INFORMATION MANAGED Project Management You Can Build On Primavera Solutions for Engineering and Construction Improve Project Performance, Profitability, and Your Bottom Line Demanding owners, ineffective

More information

An Introduction to Sampling

An Introduction to Sampling An Introduction to Sampling Sampling is the process of selecting a subset of units from the population. We use sampling formulas to determine how many to select because it is based on the characteristics

More information

PMS 288 Blue or CMYK = C100-M85-Y0-C43 PMS 1255 Ochre / Yellow or CMYK = C0-M35-Y85-C30. Tax Technology

PMS 288 Blue or CMYK = C100-M85-Y0-C43 PMS 1255 Ochre / Yellow or CMYK = C0-M35-Y85-C30. Tax Technology PMS 288 Blue or CMYK = C100-M85-Y0-C43 PMS 1255 Ochre / Yellow or CMYK = C0-M35-Y85-C30 Tax Technology PO Ryan has provided BASF outstanding value by recovering overpaid taxes while identifying and implementing

More information

case 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION

case 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION case 3:12-md-02391-RLM-CAN document 396 filed 04/18/13 page 1 of 7 UNITED STATES DISTRICT COURT NORTHERN DISTRICT OF INDIANA SOUTH BEND DIVISION IN RE: BIOMET M2a MAGNUM HIP IMPLANT PRODUCTS LIABILITY

More information

Litigation Solutions. insightful interactive culling. distributed ediscovery processing. powering digital review

Litigation Solutions. insightful interactive culling. distributed ediscovery processing. powering digital review Litigation Solutions insightful interactive culling distributed ediscovery processing powering digital review TECHNOLOGY ASSISTED REVIEW Eclipse combines advanced analytic technology with machine learning

More information

Translation Management System. Product Brief

Translation Management System. Product Brief Translation Management System Product Brief Contents Who s Using Smartling Who s Using Smartling The world s leading businesses use Smartling s cloud-based software platform to create, manage, and deliver

More information

Simplify the e-discovery process by learning which tools to use and when to use them. CHAPTER 7. Proactive. Review tools. litigation hold tools.

Simplify the e-discovery process by learning which tools to use and when to use them. CHAPTER 7. Proactive. Review tools. litigation hold tools. THE WINDOWS MANAGER S GUIDE TO INSIDE: Reactive litigation hold tools Proactive litigation hold tools Review tools Enterprise search tools Archive systems CHAPTER Exploring e-discovery tools Simplify the

More information

Integrated Marketing Management Aprimo Marketing Studio On Demand

Integrated Marketing Management Aprimo Marketing Studio On Demand Integrated Marketing Management Aprimo Marketing Studio On Demand The cloud-based platform that adds new efficiency and effectiveness to all aspects of your marketing. A robust suite of marketing operations

More information

Making The Most Of Document Analytics

Making The Most Of Document Analytics Portfolio Media. Inc. 860 Broadway, 6th Floor New York, NY 10003 www.law360.com Phone: +1 646 783 7100 Fax: +1 646 783 7161 customerservice@law360.com Making The Most Of Document Analytics Law360, New

More information

Achieving & Maintaining E-discovery Fitness

Achieving & Maintaining E-discovery Fitness E-DISCOVERY W HITE PAPER Advice from Fortune 1000 E-discovery Experts: Achieving & Maintaining E-discovery Fitness By Ari Kaplan, Principal, Ari Kaplan Advisors Introduction Introduction Fitness is an

More information

Altiris IT Management Suite 7.1 from Symantec

Altiris IT Management Suite 7.1 from Symantec Altiris IT 7.1 Achieve a new level of predictability Overviewview Change is inevitable for IT and it comes from several sources: changing needs from lines of business, managing and supporting too many

More information

Symantec Enterprise Vault Discovery.cloud

Symantec Enterprise Vault Discovery.cloud Fact Sheet: Archiving and ediscovery Symantec Enterprise Vault.cloud is a cloud-based archiving service that helps organizations store, manage, and discover business-critical information. The service is

More information

Find, track, pipeline, and manage your highly-skilled talent.

Find, track, pipeline, and manage your highly-skilled talent. Jobvite Engage: High Tech Find, track, pipeline, and manage your highly-skilled talent. As competition heats up for hard-to-find skills across the tech industry everything from precious engineering and

More information

Background. The team. National Institute for Standards and Technology (NIST)

Background. The team. National Institute for Standards and Technology (NIST) This paper presents the results of the collaborative entry of Backstop LLP and Cleary Gottlieb Steen & Hamilton LLP in the Legal Track of the 2009 Text Retrieval Conference (TREC) sponsored by the National

More information

BUSINESS INTELLIGENCE

BUSINESS INTELLIGENCE BUSINESS INTELLIGENCE Microsoft Dynamics NAV BUSINESS INTELLIGENCE Driving better business performance for companies with changing needs White Paper Date: January 2007 www.microsoft.com/dynamics/nav Table

More information