Case 1:11-cv ALC-AJP Document 175 Filed 04/26/12 Page 1 of 5

Transcription

1 Case 1:11-cv ALC-AJP Document 175 Filed 04/26/12 Page 1 of 5 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK )( Monique Da Silva Moore; Maryellen O'Donohue; Laurie Mayers; Heather Pierce; and Katherine Wilkinson, on behalf of themselves and all others similarly situated, USDC:1Jl)Ny..DOCIJIII:Ift I &LIC1'IlOHI 'ALLY'" ~-'. DOell '...; DAD~ g.tisi-;\,...~ Plaintiffs, 11 Civ (ALC) (AJP) - against- Opinion & Order Publicis Groupe SA and MSLGroup, Defendants )( ANDREW L. CARTER, JR., United States District Judge: Before the Court are Plaintiffs' objections, filed pursuant to Rule 72(a) of the Federal Rules of Civil Procedure, to Magistrate Judge Andrew J. Peck's rulings rendered during a February 8, 2012 discovery conference and his February 24,2012 opinion and order (dkt. no. 96). Judge Peck's rulings and written order discussed, inter alia, the use of the predictive coding software, a computer assisted form of review. The Court assumes the parties' familiarity with the facts and the predictive coding method. Plaintiffs and Defendant MSLGroup entered into a stipulation governing MSLGroup's production of electronically stored information ("ESI protocol"). The substance of the ESI protocol resulted from a series of court conferences and party discussions, but the February 8 conference set the final parameters for the protocol. Judge Peck so-ordered the ESI protocol on February 22,2012. (Dkt. No. 92.) The last paragraph of the protocol provides that Plaintiffs object to the entire ESI protocol, but signed the agreement because it reflects Judge Peck's discovery rulings and Judge Peck rejected Plaintiffs' version of the ESI protocol.

2 Case 1:11-cv ALC-AJP Document 175 Filed 04/26/12 Page 2 of 5 Plaintiffs object to the February 8 discovery rulings, the ESI protocol, and the February 24 opinion and order, arguing, inter alia, that the predictive coding method contemplated in the ESI protocol lacks generally accepted reliability standards, that the use of such method violates Fed. R. Civ. P. 26 and Federal Rules of Evidence 702, that Judge Peck improperly relied on outside documentary evidence in his February 24 opinion and order, that MSLGroup s expert is biased because the use of the predictive coding method will reap financial benefits for the company, that Judge Peck failed to hold an evidentiary hearing, and that he adopted MSLGroup s version of the ESI protocol on an insufficient record. Plaintiffs request that the Court overturn the Magistrate Judge s rulings because they are erroneous and contrary to law. 1 Plaintiffs also submitted a letter requesting that Judge Peck recuse himself from the action, which Judge Peck denied on April 2, 2012, but allowed them to file a formal motion. Plaintiffs filed their recusal motion on April 13, 2012, incorporating similar arguments made in their Rule 72(a) objections. (Dkt. No. 169.) Rule 72(a) provides that for nondispositive orders issued by a magistrate judge, [t]he district judge in the case must consider timely objections and modify or set aside any part of the order that is clearly erroneous or is contrary to law. Fed. R. Civ. P. 72(a); see also 28 U.S.C. 636(b)(1)(A). Under this highly deferential standard of review, magistrates are afforded broad discretion in resolving [non-dispositive] disputes and reversal is appropriate only if their discretion is abused. AMBAC Fin. Servs., LLC v. Bay Area Toll Auth., No. 09 Civ. 7062, 2010 WL , at *2 (S.D.N.Y. Nov. 30, 2010) (citation omitted). A magistrate judge s ruling is considered contrary to law when it fails to apply or misapplies relevant statutes, case law, or rules of procedure. In re Comverse Tech., Inc. Sec. Litig., No. 06 Civ. 1825, 2007 WL , 1 Plaintiffs filed their objections to Judge Peck s February 8 rulings on February 22, 2012 and Judge Peck issued his opinion and order on February 24, (Dkt. Nos ) Judge Peck addressed some of Plaintiffs objections in his opinion and order. Plaintiffs had an opportunity to respond to the written order and they submitted their reply brief on March 19, (Dkt. Nos ) 2

3 Case 1:11-cv ALC-AJP Document 175 Filed 04/26/12 Page 3 of 5 at *2 (E.D.N.Y. Mar. 2, 2007). The reviewing court must be left with the definite and firm conviction that a mistake has been committed to overturn the magistrate judge's resolution of a nondispositive matter. AMBAC Fin. Servs., 2010 WL , at *2 (citation and internal quotation omitted). Matters concerning discovery generally are considered nondispositive of the litigation. Thomas E. Hoar, Inc. v. Sara Lee Corp., 900 F.2d 522, 525 (2d Cir. 1990). Mindful of this highly deferential standard of review, the Court adopts Judge Peck s rulings because they are well reasoned and they consider the potential advantages and pitfalls of the predictive coding software. The Court has thoroughly reviewed the ESI protocol along with the parties submissions. 2 At the outset, the Court notes that Plaintiffs and Judge Peck disagree about the scope of Plaintiffs acquiescence concerning the use of the method. Judge Peck s written order states that Plaintiffs have consented to its use, (Opinion and Order at 17 ( The decision to allow computer-assisted review in this case was relatively easy the parties agreed to its use (although disagreed about how best to implement such review. ))), while Plaintiffs argue that Judge Peck s order mischaracterizes their position (Pl. Reply, dated March 19, 2012, at 4-5). Nevertheless, the confusion is immaterial because the ESI protocol contains standards for measuring the reliability of the process and the protocol builds in levels of participation by Plaintiffs. It provides that the search methods will be carefully crafted and tested for quality assurance, with Plaintiffs participating in their implementation. For example, Plaintiffs counsel may provide keywords and review the documents and the issue coding before the production is made. If there is a concern with the relevance of the culled documents, the parties may raise the issue before Judge Peck before the final production. Further, upon the receipt of the production, if Plaintiffs determine that they are missing relevant documents, they may revisit the issue of 2 The predictive coding method is provided in pages of the ESI protocol. (Dkt. No. 92.) 3

4 Case 1:11-cv ALC-AJP Document 175 Filed 04/26/12 Page 4 of 5 whether the software is the best method. At this stage, there is insufficient evidence to conclude that the use of the predictive coding software will deny Plaintiffs access to liberal discovery. Plaintiffs arguments concerning the reliability of the method are also premature. It is difficult to ascertain that the predictive software is less reliable than the traditional keyword search. Experts were present during the February 8 conference and Judge Peck heard from these experts. The lack of a formal evidentiary hearing at the conference is a minor issue because if the method appears unreliable as the litigation continues and the parties continue to dispute its effectiveness, the Magistrate Judge may then conduct an evidentiary hearing. Judge Peck is in the best position to determine when and if an evidentiary hearing is required and the exercise of his discretion is not contrary to law. Judge Peck has ruled that if the predictive coding software is flawed or if Plaintiffs are not receiving the types of documents that should be produced, the parties are allowed to reconsider their methods and raise their concerns with the Magistrate Judge. The Court understands that the majority of documentary evidence has to be produced by MSLGroup and that Plaintiffs do not have many documents of their own. If the method provided in the protocol does not work or if the sample size is indeed too small to properly apply the technology, the Court will not preclude Plaintiffs from receiving relevant information, but to call the method unreliable at this stage is speculative. 3 There simply is no review tool that guarantees perfection. The parties and Judge Peck have acknowledged that there are risks inherent in any method of reviewing electronic documents. Manual review with keyword searches is costly, though appropriate in certain situations. However, even if all parties here were willing to entertain the notion of manually reviewing the documents, such review is prone to human error and marred with inconsistencies from the various attorneys determination of whether a document is responsive. Judge Peck 3 The Court adopts Judge Peck s analysis of Rule 26(g) and Fed. R. Evidence 702 for similar reasons provided in his written opinion. 4

5 Case 1:11-cv ALC-AJP Document 175 Filed 04/26/12 Page 5 of 5 concluded that under the circumstances ofthis particular case, the use of the predictive coding software as specified in the ESI protocol is more appropriate than keyword searching. The Court does not find a basis to hold that his conclusion is clearly erroneous or contrary to law. Thus, Judge Peck's orders are adopted and Plaintiffs' objections are denied. The Court also concludes that it should not reject Judge Peck's rulings concerning the production of W -2s. Judge Peck previously ruled that Plaintiffs may visit defense counsel's office and identify the W-2s they want. He also quashed Plaintiffs' third-party subpoena on Automatic Data Processing Services. Plaintiffs and MSLGroup dispute whether the W -2s that Plaintiffs identified have actually been produced. (Compare PI. Reply at 10 with Def. Opp. at 25.) To the extent that Plaintiffs are missing the W-2s that they identified to MSLGroup, the Court instructs them to send MSLGroup an noting the missing W -2s within two days of this order. MSLGroup should produce those specified W -2s within three business days of Plaintiffs' . If there are additional disputes about the missing W-2s, they should return to Judge Peck. Further, Judge Peck's decision to delay production of s from MSLGroup's chief executive officer until "Phase II" of the ESI protocol is not erroneous or contrary to law. Judge Peck did not make a final determination about whether MSLGroup should produce those s. The Court reminds the parties that it affords Judge Peck's non-dispositive rulings great deference and that magistrate judges generally have broad latitude with respect to discovery issues. Dated: New York, New York April 25, 2012 SO ORDERED. ~7~rJ- United States District Judge 5

6 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK MONIQUE DA SILVA MOORE, ) MARYELLEN O DONOHUE, ) LAURIE MAYERS, HEATHER ) PIERCE, and KATHERINE ) WILKINSON on behalf of themselves ) Civ No. 11-CV-1279 (ALC) (AJP) and all others similarly situated, ) ) PLAINTIFFS, ) ) v. ) ) PUBLICIS GROUPE SA and ) MSLGROUP, ) ) DEFENDANTS. ) ) PLAINTIFFS REPLY IN SUPPORT OF RULE 72(a) OBJECTION TO MAGISTRATE JUDGE PECK S FEBRUARY 8, 2012 DISCOVERY RULINGS

7 TABLE OF CONTENTS I. INTRODUCTION...1 II. ARGUMENT...3 A. Judge Peck s Adoption of MSL s Method Without Expert Evidence on the Record, and His Reliance on Other Improper Sources, Was Contrary to Law Judge Peck s Failure to Conduct an Evidentiary Hearing or Receive Expert Testimony Before Issuing His Opinion Was Contrary to the Federal Rules Judge Peck s Reliance on Secondary Sources and Other Information That is Outside the Record and/or Unreliable Was Contrary to Law...5 B. Judge Peck s Adoption of MSL s Method Without Statistically Sound Quality Assurance Measurements Was Contrary to Law...7 C. Judge Peck s Additional Sua Sponte Rulings Should Be Reversed...9 III. CONCLUSION...10 i

8 I. INTRODUCTION On February 24, 2012, two days after Plaintiffs filed objections to his ruling from the bench, Magistrate Judge Peck became the first judge to issue a written opinion approving the use of predictive coding. 1 The ESI protocol that Judge Peck approved ( MSL s Method ) was adopted virtually wholesale from Defendant MSLGroup. As Plaintiffs repeatedly warned, although the use of predictive coding may be appropriate under certain circumstances, the devil is in the details. This is where MSL s Method fails and where Judge Peck erred. Indeed, MSL s Method risks failing to capture a staggering 65% of the relevant documents in this case. Judge Peck has long been a vocal advocate of predictive coding. In the ten months between the filing of the Amended Complaint and the February 24 written opinion, Judge Peck authored an article and made no fewer than six public appearances espousing the use of predictive coding. 2 A frequent presence on these panels is Ralph Losey, Jackson Lewis e- discovery counsel in this case another outspoken predictive coding advocate whom Judge Peck know[s] very well. See Jan. 4, 2012 Tr. (Declaration of Siham Nurhussein in Support of 1 Judge Peck s written opinion, which expanded on his reasoning and included observations about Plaintiffs Objection, erroneously claimed to be the first to approve of computer-assisted review. In fact, keyword searches, which have been approved by numerous courts throughout the country, are also a type of computer-assisted review. 2 In July 2011, Judge Peck was a keynote speaker at the 2011 Carmel Valley ediscovery retreat, where he sanctioned the use of predictive coding. [ On October 1, 2011, Judge Peck published an article entitled Search Forward, espousing the virtues of predictive coding. On November 17-18, 2011, Judge Peck again discussed predictive coding at the Georgetown Law CLE 8 th Annual Advanced ediscovery Institute Conference. [ On January 18, 2012, Judge Peck participated in a CLE event with defense counsel Ralph Losey entitled e-discovery Judges in Charlotte, where predictive coding was discussed at length. [ On January 30, 2012, Judge Peck served on a panel with Mr. Losey at the LegalTech New York 2012 Conference entitled Man v. Machine: The Promise/Challenge of Predictive Coding and Other Disruptive Technologies, where both men promoted the use of predictive coding. [ On January 31, 2012, Judge Peck moderated another panel discussion at Legal Tech, in which Mr. Losey promoted the use of predictive coding. [ On January 31, 2012, Judge Peck also participated in a panel entitled Judicial Perspectives on Technology-Assisted Review, in which he promoted predictive coding and noted that Defendants in this case must have thought [they] died and went to heaven. [ 1

9 Plaintiffs Reply ( Nurhussein Decl. ), Ex. A) at In January 2012 alone, the month before Judge Peck issued his ESI opinion, the two men shared the stage at three panels where predictive coding was discussed extensively. 4 Moreover, defense counsel Losey and Judge Peck cited each other s positions on predictive coding with approval in their respective articles, which came out just four months before Judge Peck issued his ESI opinion. 5 At no point has either Judge Peck or MSL disclosed their extrajudicial activities on the very issues before the Court to Plaintiffs. Recommind, MSL s e-discovery vendor, has not been absent from these activities. 6 On April 26, 2011 (less than two weeks after Plaintiffs filed their Amended Complaint), Recommind patented its predictive coding technology. To promote its product, Recommind is a frequent sponsor of the e-discovery panels on which Judge Peck and Defense counsel Losey sit. Judge Peck s February 24 e-discovery ruling is expected to be a boon not only to the predictive coding industry, but also to Recommind s bottom line. 7 Computer-assisted review vendors have been eagerly eyeing the multi-billion e-discovery market currently devoted to document review. Given 3 Mr. Losey has been involved in this case as early as June 2011, when the parties participated in an ESI call. 4 See n. 2, supra. 5 In Search Forward, Judge Peck relied on a posting from Defense counsel Losey s personal blog to support the proposition that In too many cases, however, the way lawyers choose keywords is the equivalent of the child s game of Go Fish. (See Doc. 96 at 20 (citing Ralph C. Losey, Child s Game of Go Fish is a Poor Model for e- Discovery Search, in Adventures in Electronic Discovery (2011), available at com/ 2009/10/04/childs-game-of-go-fish-is-a-poor-model-for-e-discovery-search/).) In his blog posting entitled Judge Peck Calls Upon Lawyers to Use Artificial Intelligence and Jason Barn Warns of a Dark Future of Information Burn-Out If We Don t, defense counsel Losey in turn embraces Judge Peck s position on predictive coding, as conveyed in Search Forward. See 6 For example, Recommind has sponsored at least six e-discovery conferences at which Judge Peck spoke: the Georgetown Law CLE e-discovery conference in November 2009 (where defense counsel Losey was also a panelist), the same conference in November 2010, a panel at the Information Retention & E-Disclosure Management Summit in May 2011, the 2011 Carmel Valley ediscovery retreat in July 2011, the Georgetown Law CLE e- discovery conference in November 2011, and three panels at the LegalTech New York 2012 Conference in January Contrary to MSL s assertion in its response to Plaintiffs Objection, MSL did not inform Plaintiffs that it had retained Recommind or that it planned to employ predictive coding in June Plaintiffs first learned that MSL planned to use Recommind in September 2011, well into discovery. 7 See Carpenter, Craig (VP of Marketing at Recommind), Judge Peck s Predictive Coding Game-Changer, available at (Feb. 27, 2012) ( The sooner we as an industry begin to reap [the] benefits [of Judge Peck s opinion], the better. ) 2

10 the stakes, Judge Peck should have more closely scrutinized the many red flags in MSL s ESI protocol. Instead, Judge Peck rushed to issue an opinion for the benefit of the bar. Extrajudicial activities aside, what should matter is whether MSL s Method will ensure that MSL fulfills its obligations under Rule 26 to produce reasonable discovery. Here, the answer is a resounding no. Judge Peck s adoption of MSL s Method was contrary to law and/or clearly erroneous for two main reasons. First, Judge Peck adopted MSL s Method on an insufficient record; Judge Peck failed to hold an evidentiary hearing or obtain expert testimony as to its reliability and accuracy. Second, MSL s Method fails to meet basic standards for reliability; the protocol risks failing to capture up to 65% of the documents material to Plaintiffs case. Accordingly, Plaintiffs respectfully request that the Court reverse Judge Peck s ESI rulings. II. ARGUMENT A. Judge Peck s Adoption of MSL s Method Without Expert Evidence on the Record, and His Reliance on Other Improper Sources, Was Contrary to Law 1. Judge Peck s Failure to Conduct an Evidentiary Hearing or Receive Expert Testimony Before Issuing His Opinion Was Contrary to the Federal Rules Judge Peck s Opinion and MSL s opposition brief assert that Federal Rule of Evidence 702 and the established Daubert regime for judicially screening scientific and technical evidence and methodology have no application to predictive coding. (Doc. 96 at 4, 14-15; Doc. 104 at 13-14) The outright rejection of the principles of Daubert is contrary to the rulings of multiple courts including Judge Peck s own rulings which have previously held that resolving contested issues of whether a particular search and information retrieval method was appropriate... involves scientific, technical or specialized information. Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 260 n.10 (D. Md. 2008). 8 Hence, a comprehensive and searching 8 See also William A. Gross Constr. Assocs., Inc. v. Am. Mfg. Mut. Ins. Co., 256 F.R.D. 134, 136 (S.D.N.Y. 2009) (M.J. Peck); United States v. O Keefe, 537 F. Supp. 2d 14, 24 (D.D.C. 2008); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331, 333 (D.D.C. 2008). 3

11 judicial review is required; the trial judge must decide a method s appropriateness with the benefit of information from some reliable source whether an affidavit from a qualified expert, a learned treatise, or, if appropriate, from information judicially noticed. Id. A similar case debating the merits of competing search technologies including predictive coding is instructive. In Kleen Prods., LLC v. Packaging Corp. of Am., No. 10 C 5711 (N.D. Ill.), Magistrate Judge Nan R. Nolan required the parties to fully brief the issues and submit expert reports in advance of a full-blown evidentiary hearing that is anticipated to take at least two full days. See Docket Report of Kleen Prods. (Nurhussein Decl., Ex. B) at Doc. 277; see also id.at Docs. 266, 267, 274, , 301. A review of the types of evidence submitted in that case, as well as the questions raised by Judge Nolan, provides insight into what a Court is required to consider when applying the law in this area to the specific facts of a case. See generally Tr. of Feb. 21, 2012 Hearing in Kleen Prods. (Nurhussein Decl., Ex. C). Judge Peck demanded no such rigor here before adopting MSL s ESI protocol. Judges are required to rely on the aid of experts when resolving disputes as to the reliability or appropriateness of a novel methodology. See Victor Stanley, 250 F.R.D. at 260 n.10 (citing Am. Nat l Bank & Trust Co. v. Equitable Life Assurance Soc., 406 F.3d 867, 879 (7th Cir. 2005)). However, in his rush to be the first in line to approve predictive coding, Judge Peck did not elicit expert testimony or give the parties an opportunity to question or cross-examine the experts. Judge Peck attempted to distinguish Kleen Prods. by characterizing it as a more difficult case where [one] party wants to use computer-assisted review and the [other] party objects. (Doc. 96 at 18.) By contrast, he claims, the decision to allow predictive coding in the instant case was relatively easy because the parties agreed to its use. Id. at 17.This grossly mischaracterizes Plaintiffs position and the history of the parties negotiations. As evidenced by 4

12 the transcripts in this case, the letters submitted by Plaintiffs to Defense counsel and the Court over the last five months, and the declarations from the parties experts, Plaintiffs willingness to consider the use of predictive coding was contingent on certain critical safeguards being built into the ESI protocol. See Plaintiffs correspondence (Nurhussein Decl., Exs. D to H.) Plaintiffs indicated that they only would be willing to consider the use of an ESI protocol that employed predictive coding if it was established as reliable. In keeping with the Sedona Conference s cooperation principles, Plaintiffs attempted to work on a joint protocol with MSL, rather than dismiss predictive coding outright. However, Plaintiffs efforts to cooperate on the protocol were effectively treated as a waiver of all objections. In so doing, Judge Peck sets a dangerous precedent that is likely to deter future litigants from even considering predictive coding, lest they be bound by a protocol that contains no measure of reliability. 2. Judge Peck s Reliance on Secondary Sources and Other Information That is Outside the Record and/or Unreliable Was Contrary to Law Equally problematic is the information upon which Judge Peck did choose to rely in his opinion. Judge Peck s reliance on secondary sources such as outdated and/or non-peer reviewed articles (including one of his own); blog entries by Defense counsel; and untested, unverified information from Recommind, was contrary to law. The Second Circuit has held that when facts or opinions found in... secondary sources are disputed, it is error to accept the data (however authentic) as evidence, at least without affording an opposing party the opportunity to present information which might challenge the fact or the propriety of noticing it. MacNamara v. City of NY, 249 F.R.D. 70, 93 n.16 (S.D.N.Y. 2008)(citation omitted). Judge Peck implicitly and improperly took judicial notice of the facts and opinions in five secondary sources, without affording Plaintiffs an opportunity to challenge these facts and opinions. Because Plaintiffs 5

13 dispute the accuracy of the articles relied on it is contrary to law to take judicial notice of the articles for the truth of the matters asserted therein. See id. 9 Similarly, it is contrary to law for Judge Peck to rely on other factual information that he learned outside of this case for example, from seminars and conferences on e-discovery or ex parte conversations with Defense counsel. In order to rely on factual information outside of the record, the Court must take judicial notice pursuant to Fed. R. Civ. P Indeed, it is for this reason that Judge Peck may not serve as both expert and judge in this case. The facts noticed by Judge Peck (i.e. that keyword searches are ineffective and predictive coding is undoubtedly superior) are subject to reasonable dispute. See Fed. R. Evid. 201(b); Int l Star Class Yacht Racing Assoc., 146 F.3d at 70. The vast majority of federal courts in this 9 Indeed, the information within the articles is also unpersuasive when applied to the parties dispute. For example, Judge Peck relied on an unsourced, hyperbolic blog posting from Defense counsel s personal blog. See Doc. 96 at 20; n. 5, supra. Similarly, Judge Peck admitted that his own non-peer reviewed article posted on the website Legal Technology News, simply explained my understanding of computer-assisted review. (Doc. 96 at 3, emphasis added). For obvious reasons, an article explaining one judge s understanding of how predictive coding and other computer-assisted review technologies are supposed to work is not evidence that all versions of predictive coding will meet the reasonable search requirements of Rule 26, much less that MSL s preferred method will do so. Apart from his own writings and those of Defense counsel, Judge Peck cited to three articles. First, Judge Peck cited an article from 1985, in which the authors focused on the cost-benefit analysis of a technologically unsophisticated litigator s use of keyword searches as compared to the costs of a complete manual review of only 40,000 documents. See id. (citing David L. Blair & M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System, 28 Comm. ACM 289 (1985) available at edu/~sok/papers/b/blair-maron.pdf.) This reference does not support a conclusion that keyword searches conducted in 2012 would be largely ineffective. Second, Judge Peck cited an article in which the authors conclude only that two computer-assisted categorization processes had similar (but not superior) results to a manual review process. (Doc. 96 at 18 (quoting Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, 61 Am. Soc y for Info. Sci. & Tech. 70, 79 (2010).) There is no information in the article or the record regarding how the processes reviewed in the article worked, how many items were sampled and reviewed manually in order to train the computer-review system, or any other identifying features enabling a comparison to MSL s method. Third, Judge Peck quotes extensively from Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech. 11 (2011), (Doc. 96 at ) The facts in this non-peer reviewed piece, published by a lawyer at Wachtel, support an exceedingly narrow proposition: that a single study indicates that two types of computer-assisted review (neither involving predictive coding) appeared to yield more accurate search results than manual review. In fact, 8 of the 10 computer-assisted methods tested in the study were determined to be less accurate than manual review. This does not support Judge Peck s conclusions about predictive coding, much less bear upon the parties dispute regarding MSL s specific protocol. 10 See Int l Star Class Yacht Racing Assoc. v. Tommy Hilfiger U.S.A., Inc., 146 F.3d 66, (2d Cir. 1998) (vacating the district court opinion for relying on statements of fact from another opinion to establish prevailing trademark search practices; industry practice is subject to dispute, [Plaintiff] is entitled to have its day in court, and, through time-honored methods, test the accuracy of [Defendant s] submissions and introduce evidence of its own ) (internal quotation marks and citations omitted). 6

14 District and nationwide allow keyword searches, which is one of many types of computerassisted review, as a reliable discovery method under Rule Even Judge Peck conceded that no other court has found predictive coding superior to all alternatives. Thus, neither of these facts is generally known. Moreover, for all of the reasons set forth above, neither of the facts can be accurately and readily determined from the sources cited by Judge Peck, nor are the sources cited by Judge Peck sources whose accuracy cannot reasonably be questioned. See Fed. R. Evid In short, it was contrary to law for Judge Peck to judicially notice these facts. Finally, any potential reliance on information provided by MSL s supposed experts, employees of MSL s e-discovery vendor Recommind, is clearly misplaced. MSL s experts were never sworn-in and their declarations were submitted after Judge Peck had already adopted MSL s Method. More importantly, MSL s experts are far from disinterested; they have a large financial stake in whether the Court adopts MSL s Method, which is premised on Recommind s proprietary and patented technology. Plaintiffs recently learned of facts that further undermine the reliability of MSL s experts. MSL failed to note that Recommind was banned from participating in future Text Retrieval Conference (TREC) studies for improperly representing the results of the 2011 tests. See Declaration of Paul J. Neale ( Neale Decl. ) 19. B. Judge Peck s Adoption of MSL s Method Without Statistically Sound Quality Assurance Measurements Was Contrary to Law Judge Peck not only derived his analysis from improper sources but focused on the wrong question. Judge Peck primarily discussed, as a general matter, the propriety of computer-assisted 11 See, e.g., In re: Pilot Project Regarding Case Management Techniques for Complex Civil Cases in the Southern District of New York, Standing Order M at 21 (S.D.N.Y. Nov. 11, 2011) available at Ad Hoc Committee for Electronic Discovery, Default Standard For Discovery, Including Discovery of Electronically Stored Information at 5 (D. Del. Dec. 8, 2011) available at Seventh Circuit Electronic Discovery Committee, Principles Relating to the Discovery of Electronically Stored Information at Principle 2.05 (7th Cir. Aug. 1, 2010) available at 7

15 document review (no doubt meaning some variation of predictive coding), but the real question was whether MSL s specific protocol would adequately address its Rule 26 obligations. 12 It is clear error to find that, as adopted, MSL s Method has proper quality assurance measurements in place. 13 Focusing only on the training section, Judge Peck and MSL claim that MSL s Method is transparent. Specifically, they assert that the method is based on the following numbers: (a) an initial random sample of 2,399 documents coded by category for relevance, to determine the yield of relevant documents; (b) up to seven trainings with attorney review of at least 500 documents to teach the system; and (c) a final random sample of 2,399 documents of the deemed-irrelevant documents, again coded by category for relevance. Judge Peck and MSL erroneously argue that this last sample can be used to gauge the effectiveness of the system by providing a check to ensure that smoking gun-type documents are captured. Judge Peck s determination is clearly erroneous because the final random sample size of 2,399 documents is not statistically significant. In other words, it is far too small to obtain meaningful estimates of the system s recall. See Neale Decl. 26. A precise estimate of the system s recall is important so that the Court and the parties can determine the percentage of documents the system has successfully identified as relevant and irrelevant. Using MSL s most recent estimation that the likely yield of relevant documents is 1.5% of the roughly 2.5 million electronic documents, 14 one can predict that the system will identify 37,500 documents as relevant and deem 2,462,500 documents (98.5% of the 2.5 million documents) irrelevant. 12 Judge Peck dismissed Plaintiffs misgivings as premature; however, once MSL s protocol is adopted, the horse will have already left the barn. 13 As previously noted by Plaintiffs expert, MSL s method does not comply with generally accepted industry practices, which require that one set out preliminary metrics and definitions prior to the use and testing of a system. Predetermined standards increase trust in the system and prevent later accusations of bias; in other words, you set the rules before you play the game. 14 To the extent that the large volume of ESI in this case is being used to justify the adoption of MSL s method (See Doc. 96 at 22), Plaintiffs contend that MSL has purposefully performed the equivalent of a document dump to inflate the numbers. For example, MSL has claimed that a wide range of business documents ordinarily kept in hard copy (policies, procedures, organizational charts, etc.) are only accessible through an search. 8

16 Neale Decl. 24. It is from this set of 2,462,500 irrelevant documents that MSL s Method will, as its last quality assurance phase, randomly sample only 2,399 documents. Id. 23. Although 2,399 is a statistically significant sample size for estimating the yield, there is no formula that supports the use of the same number for estimating the system s recall. Id. 26. Prior to the February 8 hearing, Plaintiffs proposed using a minimum sample size of 16,555 documents at this final phase. This is a statistically significant number that will allow for a meaningful estimate of the system s recall. Id. 28. Although MSL originally agreed to this number, MSL unexpectedly and unilaterally imposed a seven-fold decrease without any explanation scientific or otherwise justifying the number 2,399. Since MSL s about-face, neither Judge Peck nor MSL has set forth any reasons to support the use of this number, or set forth evidence as to why it would be problematic to randomly sample 16,555 documents. Indeed, in the same preliminary study MSL relies on to tout the quality of the technology to be used in its predictive coding protocol, the technology s recall, was very low, on average 35%. See 2011 TREC Study Draft (Neale Decl., Ex. 1) Therefore, the Court can fairly assume that MSL s method will miss 65% of the relevant documents. Id. 16. This raises serious questions about the adoption of a novel discovery method, particularly without a scientifically sound way to measure its accuracy. See generally id. C. Judge Peck s Additional Sua Sponte Rulings Should Be Reversed 15 Plaintiffs also object to two additional rulings made by Judge Peck. First, Judge Peck s sua sponte decision to quash Plaintiffs subpoena seeking W-2s from ADP, a third party payroll provider, was clearly erroneous and contrary to law. The issue is not, as MSL suggests, moot; ten months after Plaintiffs served their document requests, and nearly six weeks after Judge Peck 15 Plaintiffs incorporate by reference all arguments made in their February 22, 2012 Objection. The omission of certain arguments from this reply brief should not be construed as a waiver. 9

17 quashed the subpoena, Plaintiffs have yet to receive all the requested W-2s from MSL. These documents could have been produced weeks ago, with minimal burden, by ADP. Second, in deciding to delay production of s from MSL s CEO, Olivier Fleurot, until Phase II of the ESI protocol, Judge Peck determined that s stored in France likely would be covered by the French privacy and blocking laws. (Doc. 96 at 8.) This sua sponte ruling, made without considering any evidence, is clearly erroneous and contrary to law. Judge Peck failed to engage in the required comity analysis, under which the vast majority of U.S. precedents have found that French law does not preempt discovery. 16 III. CONCLUSION For the foregoing reasons, Plaintiffs request that the Court grant Plaintiffs Objections and reverse Judge Peck s February 8, 2012 rulings. Because Plaintiffs willingness to consider predictive coding was conditioned on the development of an ESI protocol that contained sufficient quality control standards a condition that has not been met Plaintiffs ask that the Court reject MSL s use of predictive coding and require the parties to come up with a new ESI Protocol. To the extent that the Court needs further information about the failings of MSL s method, Plaintiffs request that the Court hold an evidentiary hearing or direct Judge Peck to do so. Alternatively, Plaintiffs request that the Court modify the Protocol in advance of its implementation to include (a) a statistically significant sampling of irrelevant documents to establish the recall percentage; and (b) a defined standard of recall and precision (which can be used to determine the system s accuracy). 16 See Societe Nationale Industrielle Aerospatiale v. U.S. Dist. Ct. for the S.D. of Iowa, 482 U.S. 522, (1987) (setting forth the comity factors to be considered by courts and holding that the French blocking law did not preempt the Federal Rules of Civil Procedure); True Position, Inc. v. LM Eriksson Telephone Co., No , 2012 U.S. Dist. LEXIS (E.D. Pa Mar. 6, 2012) (noting that other courts have held the French Blocking Statute does not subject defendant to a realistic risk of prosecution, and cannot be construed as a law intended to universally govern the conduct of litigation within the jurisdiction of a United States court. ) (citations omitted); In re Am. Int l Grp., Inc., 2008 Securities Litig., 2010 U.S. Dist. LEXIS , at *7-9 (S.D.N.Y. Dec. 1, 2010) (same); In re Air Cargo Shipping Svcs. Antitrust Litig., 2010 U.S. Dist. LEXIS 30598, at *58-60 (E.D.N.Y. Mar. 29, 2010) (same). 10

18 DATED: March 19, 2012 SANFORD WITTELS & HEISLER, LLP /s/ Janette Wipper Janette Wipper Esq. Jeremy Heisler, Esq. Steven L. Wittels, Esq. Siham Nurhussein, Esq. Deepika Bains, Esq. Attorneys for the Plaintiffs and the Class 11

19 CERTIFICATE OF SERVICE I, Siham Nurhussein, hereby certify under penalty of perjury that on this 19 th day of March true and correct copies of the foregoing Reply in Support of Rule 72(a) Objection to Magistrate Judge Peck s Rulings were served on all counsel of record by operation of the Court s ECF system. /s/ Siham Nurhussein 12

20 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK MONIQUE DA SILVA MOORE, ) MARYELLEN O DONOHUE, ) LAURIE MAYERS, HEATHER ) PIERCE, and KATHERINE ) WILKINSON on behalf of themselves ) Civ No. 11-CV-1279 (ALC) (AJP) and all others similarly situated, ) ) PLAINTIFFS, ) ) v. ) ) PUBLICIS GROUPE SA and ) MSLGROUP, ) ) DEFENDANTS. ) ) DECLARATION OF PAUL J. NEALE IN SUPPORT OF PLAINTIFFS REPLY IN SUPPORT OF RULE 72(a) OBJECTIONS TO MAGISTRATE JUDGE PECK S FEBRUARY 8, 2012 DISCOVERY RULINGS I, Paul J. Neale, declare as follows: 1. I am the Chief Executive Officer and a Managing Director of DOAR Litigation Consulting LLC and have been retained by Sanford Wittels and Heisler, LLP as a consultant and expert in the above-captioned matter. 2. I hold a Bachelor of Arts degree in criminal justice from Temple University. 3. I have advised lawyers and their clients on the management of information in litigation for over 20 years and am a nationally recognized expert on issues relating to the management and production of electronically stored information ( ESI ). 4. I am a frequent author, lecturer and CLE instructor regarding the proper management of ESI and on the evolving state of the law and technology as they relate to ESI issues. 5. As a Managing Director at DOAR Litigation Consulting, I am routinely called upon to render expert advice and provide expert testimony on behalf of clients on discovery issues such as ESI preservation, spoliation, cost-shifting, reasonableness, inaccessibility determinations, ESI sanctions and the use of alternative technologies in the analysis and review of ESI.

21 6. I submit this declaration to clarify Plaintiffs position and to address the misstatements and misrepresentations made in the declarations attached to Defendant MSL s brief, by two representatives of their vendor Recommind: Eric Seggebruch and Jan Puzicha. 7. Defendants, along with their experts at Recommind, obfuscate the flaws in the ESI protocol ( the protocol ) adopted by Judge Peck, by focusing on the training of the Axcelerate system (i.e. Recommind s proprietary technology that is used in the ESI protocol), and by relying on the accuracy of other systems designed to conduct computer-assisted review, to support the accuracy of Recommind s Axcelerate system. 8. The protocol s primary flaw is that it does not include a scientifically supported method for validating the results of the Axcelerate system s predictive coding process as modified by the Defendants and accepted by Judge Peck. As it currently stands, the Plaintiffs, the Defendants and the Court will never know whether the Defendants predictive coding process met any acceptable standard for the production of documents responsive to Plaintiffs document requests. 9. As stated during the February 8, 2012 hearing and cited in Judge Peck s opinion, I am a proponent of the use of predictive coding, when it can be validated as reliable. However, the use of predictive coding or any other computer-assisted review approaches (including the use of keyword searching, which is also a type of computer-assisted review) should include a proper validation of the process against some pre-established measure within the context of the specific use of that approach. 10. The Defendants and Judge Peck s reasoning that predictive coding is better than the alternatives, despite the lack of foundation, should not be a de facto validation of the Defendants specific use of Recommind s Axcelerate system in the instant action. There must be some requirement to validate the efficacy of the process. Misstatements& Misrepresentations about the 2011 TREC Study 11. Mr. Seggebruch s statements in his March 7, 2012 declaration (that were also echoed by Recommind in its widely distributed marketing material), which refer to Recommind s performance in the 2011 TREC study, are misleading and incomplete. 12. Mr. Seggebruch stated in paragraph 18 of his declaration: In one category, Recommind achieved F 1 scores over 60%. This is not a very high score; one could reasonably infer from it that 40% of all responsive documents are likely to be missed by the Axcelerate system. More concerning, the score he reported referred to overall accuracy, but excluded recall the proportion of responsive documents actually found. 2

22 13. In addition, Mr. Seggebruch failed to reveal to the Court in his declaration that the F 1 scores reported refer to Recommind s representation (not TREC s findings) of Recommind s hypothetical F 1 scores. The hypothetical scores refer to an afterthe-fact assessment of how the system would perform under the best possible circumstances, not how the system actually performed. 14. A draft version of TREC s report of the 2011 study (attached as Ex. 1) indicates that Recommind s actual F 1 scores were significantly lower than their hypothetical F 1 scores. 1 For example, for the same run in which Recommind received a 62.3% hypothetical F 1 score, their actual F 1 score is 24.7% and their recall was 25.8%. In other words, over 74% of all responsive documents were missed. (Ex. 1, at 13 Table 10.) 15. On average, Recommind s recall scores were approximately 35% in the 2011 TREC study. 16. Applying Recommind s recall scores from the 2011 TREC study to this case, Defendants would fail to produce 65 out of every 100 of the relevant documents. In other words, the system may incorrectly code the vast majority of relevant documents as irrelevant. This is illustrated as follows: 17. In my opinion, incorrectly identifying many relevant documents as irrelevant is an unacceptable result, even when you compare it to human review of documents as was done by Recommind and Defendant MSL in their brief, and Judge Peck in his written opinion. 1 This document was not released by TREC and is admittedly preliminary and subject to change. Due to Recommind s selective usage of the results, however, it is, absent Recommind s own disclosure, the only evidence of Recommind s performance during the 2011 TREC study. 3

23 18. Mr. Seggebruch s dismissal of the TREC studies is contradicted by his company s marketing material. Recommind s material states, In the final results stage where teams worked among themselves in a real-world scenario, Recommind had the best results (the highest accuracy) in all three topics, all by a wide margin. (Ex. 2 at 3 (emphasis added); see also generally Ex. 3.) Despite this publication, Mr. Seggebruch comfortably stated to the Court in paragraph 18 of his declaration, I believe that the reliance on TREC is wholly misplaced because TREC is an academic exercise, rather than a real-world review. 19. Mr. Seggebruch s statements referring to Recommind s 2011 TREC results are further undermined by the fact that Recommind has been banned from future participation in TREC studies due to violating their agreement with TREC by publicizing its preliminary results as compared to other participants in 2011 and prior years. Misstatements& Misrepresentations about the 2009 TREC Study 20. Defendant MSL improperly refers to the 2009 TREC study and to the article by Maura Grossman and Gordon Cormack that analyzes the results of that study as follows: 2 a. Recommind did not participate in the 2009 TREC study so the results in no way reflect their performance. b. The two systems that the Grossman & Cormack article determined to have been conclusively superior in the 2009 TREC study H5 and the University of Waterloo use technologies and methods that are distinctly different from Recommind Axcelerate system that Judge Peck adopted. 21. It is a misrepresentation of fact for Defendant MSL to imply that the results of these two participants in the 2009 TREC study a study in which Recommind did not participate can be used to support the reliability of the protocol in this case. The notion that if one computer-assisted review system performs well, then all computer-assisted review systems must perform well is akin to Toyota using BMW s and Audi s safety tests to validate the safety of Toyota s vehicles. TREC 2009 and the Grossman & Cormack article do not evaluate Recommind s system, and thus cannot be used to support the potential performance of the Axcelerate program generally or in the context of this case. 2 Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech. 11 (2011), (Doc. 100.) 4

24 Misstatements & Misrepresentations about the Quality Control of the ESI Protocol 22. The protocol as currently adopted by Judge Peck does not include a reliable measurement of the accuracy of the protocol s system, and specifically fails to include a scientifically supportable measure of recall, which is the metric that establishes what proportion of the responsive documents the system identified as relevant. 23. Instead, the protocol takes a random sample of only 2,399 documents at the final stage of the system s testing in the section Quality Control by Random Sample of Irrelevant Documents. This step in the predictive coding process is the final and only gauge as to whether Recommind s system is identifying as relevant an acceptable percentage of the responsive documents. 24. Using Defendant MSL s own numbers, a random sample of only 2,399 documents is not scientifically supportable. Brett Anders, counsel for Defendant MSL, stated at the January 4, 2012 hearing that his review of an initial random sample of documents indicated that the ultimate percentage of responsive documents would be 1.5% of the total population. (See Tr. Jan. 4, 2012 H ring (Nurhussein Decl. Ex. A) at 46.) Based on this 1.5% number, one can predict that, of the approximately 2.5 million documents subject to the ESI protocol, 37,500 documents would be responsive to Plaintiffs requests, and thus identified as relevant. Conversely, the remaining 2,462,500 documents would be identified as irrelevant by the system. 25. The assessment of how many responsive documents (i.e. relevant documents) were missed by the system allows for the measurement of recall. The Court should note that the smaller the percentage of responsive documents in a given population, the larger the sample size required to measure recall. 26. A sample size of 2,399 documents randomly selected from 2,462,500 documents is not a statistically valid sample size that will allow the parties or the Court to determine how many responsive documents were missed by the Recommind system. 27. Mr. Puzicha, in paragraph 9 of his declaration, states that a precise estimate of recall is irrelevant, however, as long as the estimation interval is within boundary of a standard accepted by the Court. The fact is, however, there has been no standard established by the Court. That is exactly what we are asking the Court to do. 28. In my opinion, a sample size of 16,555 documents during the Quality Control by Random Sample of Irrelevant Documents stage of the protocol is a statistically valid sample size, which is necessary given (1) the Defendants estimation of a low yield percentage (1.5%) of relevant documents; and (2) preliminary evidence of Recommind s poor recall results in the 2011 TREC study 5