Judge Peck Provides a Primer on Computer-Assisted Review By John Tredennick

Transcription

1 By John Tredennick CEO Catalyst Repository Systems

2

3 Magistrate Judge Andrew J. Peck issued a landmark decision in Da Silva Moore v. Publicis and MSL Group, filed on Feb. 24, This decision made headlines as being the first judicial opinion to approve the process of predictive coding, which is one of the many terms people use to describe computer-assisted coding. Well, Judge Peck did just that. As he hinted during his presentations at LegalTech 2012, this was the first time a court had the opportunity to consider the propriety of computer-assisted coding. Without hesitation, Judge Peck ushered us into the next generation of e-discovery review people assisted by a friendly robot. I recommend reading the decision (and its accompanying predictive-coding protocol) not for its result but for its reasoning. This is one of the best sources I have seen on the reasons for and processes underlying predictive coding. Indeed, Judge Peck provided a primer on how to conduct predictive coding that is must reading for anyone wanting to get up to speed on this process. What is? Judge Peck started by quoting from his earlier article in Law Technology News: By computer-assisted coding, I mean tools (different vendors use different names) that use sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e. training by) a human reviewer. As Judge Peck concluded: This judicial opinion now recognizes that computer-assisted review is an acceptable way to search for relevant ESI in appropriate cases info@catalystsecure.com 2

4 Why Do We Need? The answer for Judge Peck was simple: Other methods of finding relevant documents are expensive and less effective. As he explained: The objective of e-discovery is to identify as many relevant documents as possible while reviewing as few non-relevant documents as possible. Linear review is often too expensive. Despite being seen as the gold standard, studies show that computerized searches underlying predictive coding are at least as accurate as human review, if not more accurate. Studies also show a high rate of disagreement among human reviewers as to whether a document is relevant. In most cases, the difference is attributable to human error or fatigue. Key word searches to reduce data sets also miss a large percentage of relevant documents. The typical practice of opposing parties choosing keywords resembles a game of Go Fish, as Ralph Losey once pointed out. Key word searches are often over-inclusive, finding large numbers of irrelevant documents that increase review costs. They can also be under-inclusive, missing relevant documents. In one key study the recall rate was just 20%. Ultimately, Judge Peck reminded us of the goals underlying the Federal Rules of Civil Procedure. Perfection is not required. The goal is the just, speedy, and inexpensive determination of lawsuits. Judge Peck concluded that the use of predictive coding was appropriate in this case for the following reasons: The parties agreement. The vast amount of ESI (over 3 million documents). The superiority of computer-assisted review over manual review or keyword searches. The need for cost effectiveness and proportionality. The transparent process proposed by the parties. The last point was perhaps the most important factor leading to the decision: MSL s transparency in its proposed ESI search protocol made it easier for the Court to approve the use of predictive coding info@catalystsecure.com 3

5 How Does the Process Work? The court attached the parties proposed protocol to the opinion. While it does not represent the only way to do computer-assisted review, it provides a helpful look into how the process works. 1. The process in this case began with attorneys developing an understanding of the files and identifying a small number that will function as an initial seed set representative of the categories to be reviewed and coded. There are a number of ways to develop the seed set, including the use of search tools and other filters, interviews, key custodian review, etc. 2. Opposing counsel should be advised of the hit counts and keyword searches used to develop the seed set and invited to submit their own keywords. They should also be provided with the resulting seed documents and allowed to review and comment on the coding done on the seed documents. 3. The seed sets are then used to begin the predictive coding process. Each seed set (one per issue being reviewed) is used to begin training the software. 4. The software uses each seed set to identify and prioritize all similar documents over the complete corpus under review. Essentially, they review at least 500 of the computer-selected documents to confirm that the computer is properly categorizing the documents. This is a calibration process. 5. Transparency requires that opposing counsel be given a chance to review all non-privileged documents used in the calibration process. If the parties disagree on tagging, they meet and confer to resolve the dispute. 6. At the conclusion of the training process, the system then identifies relevant documents from the larger set. These documents are reviewed manually for production. In this case, the producing party reserved the right to seek relief should too many documents be identified. 7. Accuracy during the process should be tested and quality controlled by both judgmental and statistical sampling. 8. Statistical sampling involves a small set of documents randomly selected from the total files to be tested. That allows the parties to project error rates from the sample. 9. Here, the parties agreed on a series of issues that will, of necessity, vary on other cases. The key point is that the parties agree on the issues and test the coding during the process info@catalystsecure.com 4

6 Random Samples It is important to create an initial random sample from the entire document set. The parties used a 95% confidence level with an error margin of 2%. They determined that the sample size should be 2,399 documents. You can figure this out using one of the publicly available sample-size calculators such as Raosoft, which we often use. Seed Sets The protocol goes on to describe a number of ways to generate seed sets including: Agreed-upon search terms. Judgmental analysis. Concept search. The parties frequently sampled the results from searches to evaluate their effectiveness. Some computer-assisted coding systems like the one used for this case start their process with seed sets. The notion is that attorneys understand the cases, know what is and is not relevant and can train the system to recognize more relevant documents more effectively than starting with no seed documents. Others think this is a mistake. They believe that however well meaning, the attorneys will bias the system to find what they think is relevant and get self-reinforcing results. In this regard, they are suggesting that the attorneys will make the same mistakes found in key word searches thinking that you know which words will be most effective at finding your documents. Systems following this logic urge the user to start from scratch, telling the system what is and is not relevant based on reviewing documents. As you do that, the system begins developing its own profile of relevant documents and builds out the searches. The belief is that the system may create a better search through this process than it might if you bias it with your seed documents. There is a middle ground here as well. Many of the latter systems (no seed) will allow you to submit a limited number of seed documents as part of the training process. That may represent the best of both worlds or it may not, depending on your beliefs. The important point is that there are different approaches to computer-assisted processing. This protocol shows you one approach only info@catalystsecure.com 5

7 Training Iterations The process involves a number of computer runs to find responsive documents. The parties started with a first set of potentially relevant documents based on analysis of the seed set. After that review, the computer was asked to consider the new tagging and find a second set for testing. Then a third and a fourth. The protocol suggested that the parties run through this process seven times. The key is to watch the change in the number of relevant documents predicted by the system after each round of testing. Once that number dropped below a delta of 5%, the parties had the option to stop. The notion is that the system has become stable by that time, with further review unlikely to uncover many more relevant documents. Finishing the Process Once the training has completed and the system is stable, we move from computerassisted to human-powered review. At that point, the producing party reviews all of the potentially responsive documents and produces accordingly. Final QC Protocol As a final stage, the parties need to focus on the potentially non-responsive documents the ones the system says to ignore. The parties select a random sample (2,399 documents again) to see how many were, in fact, responsive. These same documents (non-privileged ones) must be produced to the opposing party for review. If that party finds too many responsive documents in the sample or otherwise objects, it is time for a meet-and-confer to resolve the dispute. Failing that, you can always go to the court and fight it out. Is This the Bible on Predictive Coding? Certainly not. There are a lot of ways to approach this process. However, first opinions on any topic carry a lot of weight. We chose a profession that is guided by precedent, and these are first tracks on this new and exciting subject. The suggested procedures make sense to me and provide a starting point for your predictive coding efforts. This opinion and its accompanying protocol are important reading whether you are proposing or opposing the process for your next case info@catalystsecure.com 6

8 About John Tredennick A nationally known trial lawyer and longtime litigation partner at Holland & Hart, John founded Catalyst in 2000 and is responsible for its overall direction, voice and vision. Well before founding Catalyst, John was a pioneer in the field of legal technology. He was editor-in-chief of the multi-author, two-book series, Winning With Computers: Trial Practice in the Twenty-First Century (ABA Press 1990, 1991). Both were ABA best sellers focusing on using computers in litigation technology. At the same time, he wrote, How to Prepare for Take and Use a Deposition at Trial (James Publishing 1990), which he and his co-author continued to supplement for several years. He also wrote, Lawyer s Guide to Spreadsheets (Glasser Publishing 2000), and, Lawyer s Guide to Microsoft Excel 2007 (ABA Press 2009). John is the former chair of the ABA s Law Practice Management Section. For many years, he was editor-in-chief of the ABA s Law Practice Management magazine, a monthly publication focusing on legal technology and law office management. More recently, he founded and edited Law Practice Today, a monthly ABA webzine that focuses on legal technology and management. Over two decades, John has written scores of articles on legal technology and spoken on legal technology to audiences on four of the five continents. You can contact John Tredennick at jtredennick@catalystsecure.com. About Catalyst Catalyst is a pioneer in providing secure, scalable multi-lingual document repositories for electronic discovery, litigation support, and other complex regulatory matters. For over a decade, corporations and their counsel have relied on Catalyst to control litigation costs and make review teams more effective. Our systems and supporting services cover the heart of the litigation lifecycle--from processing and search, to analytics, review, production and trial. For more information about Catalyst, visit: info@catalystsecure.com 7