E-Discovery Getting a Handle on Predictive Coding

E-Discovery Getting a Handle on Predictive Coding John J. Jablonski Goldberg Segalla LLP 665 Main St Ste 400 Buffalo, NY 14203-1425 (716) 566-5400 jjablonski@goldbergsegalla.com Drew Lewis Recommind 7028 Northridge Dr Nashville, TN 37221 (615) 916-0788 drew.lewis@recommind.com

John J. Jablonski, a partner at Goldberg Segalla LLP in Buffalo, New York, is the chair of its e-discovery practice group and co-chair of its cyber risk and social media practice group. An experienced trial lawyer, he consults with Fortune 500 companies about pre-litigation planning, record retention policies, and implementation of legal holds. Mr. Jablonksi is the author of the blog Legal Holds and Trigger Events and co-author of 7 steps for Legal Holds of ESI and Other Documents (ARMA International, 2009). Drew Lewis is e-discovery counsel at Recommind Inc. in Nashville, Tennessee. He consults with outside counsel and in-house legal departments about effective discovery strategies and the benefits of implementing predictive coding solutions. Previously, he worked as a commercial litigator with an international firm. Mr. Lewis has handled all aspects of discovery and, through his private practice, developed a streamlined approach to discovery that helped many clients resolve their litigation favorably, while still keeping costs in alignment.

E-Discovery Getting a Handle on Predictive Coding Table of Contents Presentation...139 E-Discovery Getting a Handle on Predictive Coding Jablonski and Lewis 137

Presentation COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING DREW LEWIS ediscovery Counsel Recommind Inc. and BILL TOLSON Sr. Product Marketing Manager Recommind Inc. E-Discovery Getting a Handle on Predictive Coding Jablonski and Lewis 139

1 The Same Old Way Is No Longer Good Enough... 3 2 Comparing Traditional Linear Review to Predictive Coding... 4 3 Calculating Predictive Coding Cost Savings... 4 4 What is Return on Investment (ROI)?... 7 5 Calculating Return on Investment (ROI) for Predictive Coding... 7 6 Conclusion...10 COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 2 140 Employment and Labor Law May 2013

1 THE SAME OLD WAY IS NO LONGER GOOD ENOUGH Traditional linear review including keyword search, a manual, expensive, time-consuming and error-prone process requires teams of legal professionals to review hundreds of thousands or millions of documents one page at a time to determine relevance to the case. This review step in the ediscovery process drives the largest cost of ediscovery - having legally trained people actually read documents that have been singled out by some method, usually a keyword search, to have some content that may be potentially responsive to the case. The obvious problem with the traditional review method is that keyword searches produce huge amounts of documents; hundreds of thousands, millions, tens of millions or more. These millions of documents all have to be read to determine responsiveness. In the last several years a new, more accurate and less costly ediscovery review process has been used successfully in many court cases including several high profile cases. Predictive Coding, a next generation document review and analysis technology, includes disruptive technology that has the ability to dramatically reduce document review time, measurably lower document review costs and raise accuracy well beyond traditional linear review techniques. Predictive coding provides a computer-generated judgment, throughout several iterations of machine learning, with an explicit confidence score about the relevance of each document. This capability allows counsel and their clients to dramatically expedite the actual document review process by automatically assessing the relevance of documents and prioritizing large numbers of documents as responsive while concurrently improving accuracy and lowering the risk of missing key documents. The end result is human reviewers actually read a much smaller percentage of the corpus. With that understanding in mind, it s useful to actually calculate the projected cost savings and return on investment (ROI) to show financial justification of the investment. To calculate predictive coding cost savings and ROI we first need to understand how traditional legal review is currently conducted and its cost. COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 3 E-Discovery Getting a Handle on Predictive Coding Jablonski and Lewis 141

2 COMPARING TRADITIONAL LINEAR REVIEW TO PREDICTIVE CODING Traditional linear review relies on creating a reviewable results set of potentially responsive documents by running keyword searches across the enterprise data repositories to compile massive sets of documents that must then be read by legally trained professionals to determine relevancy and privilege. As you can imagine, this can be extremely slow and costly and susceptible to risk. For example, consider an average ediscovery review situation (Figure 1) where 679,349 documents were returned from a keyword search originating from a larger data set of 2,072,282 documents - a 33% cull-down rate. Figure 1 3 CALCULATING PREDICTIVE CODING COST SAVINGS In a traditional linear review, all 679,349 documents would have to be read by attorneys to determine responsiveness. Figuring an attorney can review and make a decision on 45 to 55 documents per hour, the number of attorney hours needed to review all 679,349 documents would be 15,096 hours. Further calculating the cost per hour for the attorney(s) to review the documents is $50/hour, the total cost of this review would be $754,800. Using a predictive coding solution for the same example produces an immediate reduction in documents to be reviewed from 679,349 down to 175,018. Predictive coding further reduces the number of documents to be actually read to a small percentage of those 175,018 documents COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 4 142 Employment and Labor Law May 2013

thereby further reducing the total cost of review to $194,464. This reduction in documents to be reviewed produces a savings of $560,336 ($754,800 - $194,464) or 75%. Figure 2 Another example of the potential cost savings associated with predictive coding is seen in Figure 2. In this case, a company started off with a total document count of 1,150,000. Using traditional linear review including keyword search techniques, those 1.15 million documents were culled down to 690,000 documents or a 40% culling reduction. The total cost after review ended up at $448,000 or almost $4,000 per GB. Looking at that same example using predictive coding, the cull-down and review cost savings were much greater; a 95% cull-down rate or 57,500 documents (1,150,000 down to 57,750) and COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 5 E-Discovery Getting a Handle on Predictive Coding Jablonski and Lewis 143

a total cost after review of $37,000 or $321 per GB - a 92% reduction ($448,000 - $37,000) providing a total cost savings of $411,000 in the total cost over traditional linear review. Another benefit of using Predictive Coding for document review besides cost savings is the amount of actual review time you can save. Figure 3 below represents three different potential case sizes; small, medium and large. Looking at the relatively small IP Case, we see a cost savings of $100,500 and a time savings of 2,010 hours or 251 days based on a single reviewer. The medium sized Second Request Case produced a cost savings of $432,300 and 4.25 single reviewer years. Lastly, the Large Tort Case returned a cost savings of almost $2 million and a time savings of 39,592 hours or 19.63 single reviewer years. Figure 3 (Calculations include a 55 documents-per-hour review rate, an hourly reviewer rate of $50 per hour and an Axcelerate results set review reduction of 70%) The obvious revelation in the above comparison is the relationship between the number of documents you have to review to the number of hours spent reviewing those documents to the total cost of the review. If you do not have to review 70% of the documents (by letting the Predictive Coding solution review them for you), then you will see a 70% reduction in the time to review the documents and will realize a 70% cost reduction in the overall review of the corpus. Let s now look at how cost savings relate to return on investment. COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 6 144 Employment and Labor Law May 2013

4 WHAT IS RETURN ON INVESTMENT (ROI)? ROI is a performance measurement used to evaluate the efficiency of an investment expressed as a percentage. Simply put, if an ROI calculation is negative or less than another ROI comparison, then it s not a good investment. To calculate ROI, the benefit (return) of an investment is divided by the cost of the investment - the result is expressed as a percentage or a ratio. Predictive coding ROI calculations require the following data points: The total cost of the current ediscovery process used The total cost of the ediscovery process after the investment is in place The total cost of the ediscovery investment The actual ROI formula looks like this: 5 CALCULATING RETURN ON INVESTMENT (ROI) FOR PREDICTIVE CODING Using the above ROI formula, let s look at an example from an actual customer. ROIs can be calculated by individual case, annual discovery cost or over longer periods of time. In this example we will calculate the predictive coding ROI over a 1 year period COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 7 E-Discovery Getting a Handle on Predictive Coding Jablonski and Lewis 145

Table 1 To collect the data points mentioned above, we need to determine the average number of ediscovery requests the organization acts upon per year, the average number of custodians per case, the total data collected in GB, the estimated cull rate as a percentage, the average number of documents per GB, the estimated number of documents a legal reviewer can review per hour, and finally the average hourly rate legal reviewers will charge. With these basic data points we can begin to estimate the cost of a traditional linear review. Table 2 Using the total documents after culling number of 1,600,000 (400 GB X 40% = 160 GB X 10,000 documents per GB = 1,600,000) divided by the number of documents per hour a legal reviewer can work (50) produces 32,000 hours to review 1,600,000 documents. To calculate the cost per discovery event, we simply multiply 32,000 hours by the rate of $55/hour to get a cost of $1,760,000. Finally, to calculate the cost of discovery annually, we multiply the number of COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 8 146 Employment and Labor Law May 2013

discoveries per year by the average discovery cost per event (3 X $1,760,000) to arrive at $5,280,000 annually. The next step in calculating predictive coding ROI is to estimate the cost of discovery after the predictive coding solution is adopted. Table 3 Starting with the total documents to review after culling of 1,600,000, we apply a standard predictive coding seed set review and sampling percentage of 30% meaning you will actually only have to physically review 30% of the 1,600,000 documents or 480,000 (remember predictive coding automates much of the review process). Then using the Documents/Hour Review Rate and Hourly Billing Rate per Reviewer from Table 1, we can calculate the cost per discovery event after predictive coding; [480,000 / 50] * $55 = $528,000 per discovery event or $1,584,000 annually. The last step in calculating predictive coding ROI is to plug the calculated numbers into the ROI formula: [Cost of discovery Cost of discovery after predictive coding solution] Cost of predictive coding solution / Cost of predictive coding solution Or: [$5,280,000 - $1,584,000] - $400,000 / $400,000 = 8.24 or 824% COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 9 E-Discovery Getting a Handle on Predictive Coding Jablonski and Lewis 147

Table 4 Looking at this specific predictive coding ROI, 824% is wildly positive so this organization would be remiss not to adopt predictive coding technology to quickly reduce their cost of ediscovery review. In fact, taking these calculations a bit further, we can see that for this specific example, a breakeven time of 0.13 years, [the total cost of the predictive coding solution / the annual cost savings] * 1.2, or a little more than a month to pay for the predictive coding solution from the estimated savings. 6 CONCLUSION Return on investment is an often asked for but little understood financial measure. Many equate cost savings to ROI but cost savings is only a part of the equation. ROI also includes looking at the cost of the solution that produced the savings. ROI lets you compare returns from various investment opportunities to make the best investment decision. For example, if a savings was calculated to be $325,000, many would say that s great, let s do it. But if the cost of that solution was $6,000,000 rather than $400,000 then the ROI would be 82% not 824%. Still not bad but if that same $6,000,000 could be invested in something else that produced a higher ROI, then it wouldn t be seen as the best investment. Calculating the ROI of predictive coding is straight forward and usually will produce a largely positive ROI due mostly to the reduction in documents legally trained professionals must actually review. But, another reason to consider predictive coding for legal review is its increased accuracy and consistency. Increased accuracy provides you a more complete data set during early case assessment and greatly reduces your risk of incomplete discovery response. COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 10 148 Employment and Labor Law May 2013

Recommind s Axcelerate Review & Analysis solution is the most powerful and cost-effective document review methodology on the market, enabling attorneys to instantly find any and all documents relating to a particular person, timeframe, topic, communication, issue, or concept. Axcelerate s unique Predictive Coding workflow allows clients the option of reviewing some, most, or all of a collection in a fully defensible fashion the market's only solution with this nextgeneration capability. The result is the fastest and most cost-effective document review available with built-in quality control capabilities unmatched in the industry. Contact Us: Recommind, Inc. Tel: +1 415 394 7899 www.recommind.com/contact Copyright Recommind, Inc. 2000-2013. Recommind, Inc. s name and logo are registered trademarks of Recommind, Inc. Auto-File, Axcelerate, CORE, Decisiv, Decisiv Email, Insite Legal Hold, One-Click Coding, Perceptiv, Predictiv Contract Analysis, QwikFind, and SmartFiltering are trademarks or registered trademarks of Recommind, Inc., its subsidiaries in the United States and other countries. Other brands and names are the property of their respective owners. COST SAVINGS AND RETURN ON INVESTMENT OF PREDICTIVE CODING 11 E-Discovery Getting a Handle on Predictive Coding Jablonski and Lewis 149