Honorable Garrett E. Brown, U.S.D.J. (ret.)

Size: px
Start display at page:

Download "Honorable Garrett E. Brown, U.S.D.J. (ret.)"

Transcription

1 Civil Litigation Track Recent Legal and Technological Developments in E-Discovery: What Every State and Federal Practitioner Should Know Co-Sponsored by the Federal Practice Section Moderator: Honorable Garrett E. Brown, U.S.D.J. (ret.) Speakers: Honorable Tonianne J. Bongiovanni, U.S.M.J Jennifer Marino Thibodaux, Esq. Gibbons, PC (Newark) Matthew F. Knouff, Esq., General Counsel and E- Discovery Counsel Complete Discovery Source Fernando M. Pinguelo, Esq., Chair, Cyber Security & Data Collection Group Scarinci Hollenbeck, LLC (Lyndhurst) 2013 New Jersey State Bar Association. All rights reserved. Any copying of material herein, in whole or in part, and by any means without written permission is prohibited. Requests for such permission should be sent to the New Jersey State Bar Association, New Jersey Law Center, One Constitution Square, New Brunswick, New Jersey The material contained in these pages is for educational purposes only and not intended as a substitute for the professional services an attorney would normally provide to a client, including up to the minute legal research.

2 E-Discovery For Federal and State Litigators: Practice Pointers, A View From the Bench, Recent Case Law, and Computer-Assisted Review Federal Practice Section, NJSBA Convention, Atlantic City, May 16, 2013 Presented by: Honorable Tonianne J. Bongiovanni, U.S.M.J., Matthew Knouff, Esq., Fernando M. Pinguelo, Esq., and Jennifer Marino Thibodaux, Esq. Moderated by: Honorable Garrett E. Brown, U.S.D.J. (ret.) General E-Discovery Tab No. Material ediscovery Terms That You Should Know, Complete Discovery Source 2. Peace of Mind for Serious ediscovery, Complete Discovery Source 3. Computer Systems Checklist (Administrator), Complete Discovery Source 4. Fernando M. Pinguelo, Esq., Shannon Capone Kirk, Esq., & Sara Johnson Meyers, Esq., The Cloud, Server Consolidation, Ephemeral Data, and Information Security: (Manageable) E-Discovery Challenges Facing CIOs and InfoSec, Bloomberg BNA, Digital Discovery & e-evidence, December 6, 2012 Legal Hold Tab No. Material 5. The Sedona Conference Commentary on Legal Holds: The Trigger & The Process. Copyright The Sedona Conference. Reprinted With Permission. 6. Medeva Pharma Suisse A.G., et al. v. Roxane Laboratories, Inc., No (FLW) (TJB) (D.N.J. Jan. 28, 2011) # v

3 Meet-and-Confer Tab No Material 7. Local Civil Rule The Sedona Conference Jumpstart Outline, March 2011 Version. Copyright The Sedona Conference. Reprinted With Permission. 9. Meet and Confer Checklist, Complete Discovery Source Preservation & Production Tab No Material 10. Computer Systems Checklist (Employee), Complete Discovery Source 11. Chain of Custody Short Form, Complete Discovery Source 12. I-Med Pharma Inc. v. Biomatrix, Inc., et al., No (DRD) (D.N.J. Dec. 9, 2011) 13. Jennifer Marino Thibodaux, Esq., Not So Fast: 95 Million Reasons to Carefully Select and Limit Search Terms, Gibbons P.C. E-Discovery Law Alert, January 24, Goldmark v. Mellina, No. A T2 (N.J. App. Div. Jun. 18, 2012) 15. Kevin W. Weber, Esq., New Jersey State Courts Enter the E-Discovery Arena in Earnest; Award Sanctions for Spoliation, Gibbons P.C. E-Discovery Law Alert, June 29, Bull v. UPS, Inc., 665 F.3d 68 (3d Cir. 2012) 17. D Onofrio v. Borough of Seaside Park, No (AET) (TJB) (D.N.J. May 30, 2012) 18. Juster Acquisition Co., LLC v. N. Hudson Sewerage Authority, No (JLL)(MAH) (D.N.J. Jun. 6, 2012) # v

4 Discoverability of Social Media Tab No Material 19. Fernando M. Pinguelo, Esq. & Chris Borchert, Recent Developments in NJ Law: Social Media and More, Dictum NJSBA Young Lawyer Division Newsletter, March Romano v. Steelcase Inc., 907 N.Y.S.2d 650 (N.Y. Sept. 21, 2010) 21. Melissa C. DeHonney, Esq., Private Facebook and MySpace Postings Are Discoverable, Gibbons P.C. E-Discovery Law Alert, October 11, McCann v. Harleysville Ins. Co. of N.Y., 78 A.D.3d 1524 (N.Y. Nov. 12, 2010) 23. Robert D. Brown, Jr., New York s Appellate Division Finds Facebook Accounts Off-Limits When Discovery Demands Are Non-Specific, Gibbons P.C. E- Discovery Law Alert, January 31, Largent v. Reed, Civ. No (Pa. C.P. Franklin Nov. 8, 2011) 25. Hoy v. Holmes, No. S (Pa. C.P. Schuylkill Jan. 31, 2013) 26. Mailhoit v. Home Depot, U.S.A., Inc. et al., Civ. No (C.D. Cal. Sept. 7, 2012) 27. Mara E. Zazzali-Hogan, Esq., No Fishing Expeditions Allowed When it Comes to the Discovery of Social Media, Gibbons P.C. E-Discovery Law Alert, October 12, 2012 Social Media & Sanctions Tab No Material 28. The Kati Roll Co., Inc. v. Kati Roll Platters, Inc., et al., No (GEB) (D.N.J. Aug. 3, 2011) 29. Mara E. Zazzali-Hogan, Esq., Chief Judge Finds That Alteration of Facebook Page Can Lead to Spoliation Inference, Gibbons P.C. E-Discovery Law Alert, December 16, U.S. v. Suarez, No (JLL) (D.N.J. Oct. 21, 2010) 31. Mara E. Zazzali-Hogan, Esq. & Jennifer Marino Thibodaux, Esq., Federal Judge Rules Government Failed to Preserve Text Messages and Orders Adverse Inference Instruction in Criminal Case, Gibbons P.C. E-Discovery Law Alert, # v

5 November 2, Lester v. Allied Concrete Co., et al., Case No. CL and CL (Va. Cir. Ct. May 27, 2011) 33. Lester v. Allied Concrete Co., et al., Case No. CL and CL (Va. Cir. Ct. Sept. 23, 2011) 34. Mara E. Zazzali-Hogan, Esq., How a Stink Bomb and Its Proof That Facebook Pictures Were Deleted Might Have Blown Up a $10.6 Million Verdict, Gibbons P.C. E-Discovery Law Alert, August 8, Mara E. Zazzali-Hogan, Esq., Lester v. Allied Part 2: Clean Up of Compromising Social Media Evidence Can Result in Severe Sanctions, Gibbons P.C. E-Discovery Law Alert, February 21, Jennifer Marino Thibodaux, Esq., Ex-Juror Who Friended Defendant Faces Jail for Bragging on Facebook About Dismissal From Jury Duty, Gibbons P.C. E- Discovery Law Alert, March 22, 2012 Technology-Assisted Review Tab No Material 37. Da Silva Moore v. Publicis Group & MSL Group, No. 11 Civ (ALC) (AJP) (S.D.N.Y. Feb. 24, 2012) 38. Phillip J. Duffy, Esq., Taking the Plunge: Judge Peck Issues First Opinion Endorsing Computer-Assisted ESI, Gibbons P.C. E-Discovery Law Alert, March 2, Predictive Coding Lessons from Judge Peck s Opinion in Da Silva Moore, Complete Discovery Source 40. Dino Medina, The CAR Keys to Defensibility (Computer Assisted Review), Complete Discovery Source 41. Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E- Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J. L. & Tech. 11 (2011). Reprinted with Permission. 42. The Grossman-Cormack Glossary of Technology-Assisted Review, Federal Courts Law Review (2013). Reprinted with Permission. 43. Technology Assisted Review Workflow Graphic, Complete Discovery Source # v

6 52 ediscovery Terms That You Should Know A Admissible: Evidence that is allowable in court Analytics: The term used to refer to the various technologies used to provide multiple views into the data set Archive: Long term repository for the storage of records and files Attachment: A document or file that is connected to another document or file either externally, e.g. a document connected to an , or embedded, e.g. an image in a word processing document. Attachment Backup: Both the action of and the result of creating a copy of data as a precaution against the loss or damage of the original data. B Backup tape: Portable media used to store copies of data that are created as a precaution against the loss or damage of the original data Batch Processing: The processing of a large amount of ESI in a single step C Chain-of-Custody: The processing, tracking and recording the movement, handling and location of electronic evidence chronologically from collection to production. It is used to verify the authenticity of the ESI. Child Document: A file that is attached to another communication file. E.g. the attachment to an or a spreadsheet imbedded in a word processing document Container File: A single file containing multiple documents and/or files, usually in a compressed format; e.g. zip, rar, pst Custodian: Most often refers to the individual from whose file systems a group of records were extracted. This person is not necessarily the author of the documents. D Data Extraction: The process of parsing data from electronic documents to identify their metadata and body contents Data Mapping: The process of identifying and recording the location and types of ESI within an organization s network, and policies and procedures related to that ESI De-duplication: De-duping. The process of comparing the characteristics of electronic documents to identify and/or remove duplicate records to reduce review time and increase coding consistency De-NIST: The process of separating documents generated by a computer system from those created by a user. This automated process utilized a list of file extensions developed by the National Institute of Standards and Technology. Discovery: The process of identifying, securing, reviewing information that is potentially relevant to the matter and producing information that can be utilized as evidence in the legal process Document Family: All parts of a group of documents that are connected to each other for purposes of communication; e.g. an and its attachments E e-disclosure: The ediscovery process as it is practiced in the European Union Electronic discovery: ediscovery, e-discovery. The process of identifying, preserving, collection, preparing, reviewing and producing ESI in the context of a legal or investigative process Electronic evidence: Information that is stored in an electronic format. This is used to prove or disprove the facts of a legal matter. An electronic communication sent or received via a data application designed for that purpose (e.g. MS Outlook, Lotus Notes, Google Gmail) ESI: Electronically stored information F Filtering: The process of applying specific parameters to remove groups of documents that do not fit those parameters, in order to reduce the volume of the data set, e.g. date ranges and keywords Forensics: The handling of ESI including collection, examination and analysis, in a manner that ensures its authenticity, so as to provide for its admission as evidence in a court of law FRCP: Federal Rules of Civil Procedure, the rules that govern ediscovery and other aspects of the civil legal process. H Hash: An algorithm that generates a unique value for each document. It is referred to as a digital fingerprint and is used to authenticate documents and to identify duplicate documents. I Image (Drive): To make an identical copy of a drive including its empty space. mirror image Image (File): To make a picture copy of a document. The most common image formats in ediscovery are tiff and pdf. L Legacy Data: Data whose format has become obsolete. Legal Hold: A communication requesting the preservation of information that is potentially relevant to current or a reasonably anticipated legal matter and the resulting preservation. Peace of Mind for Serious ediscovery (855)

7 Load File: A file used to import data into an ediscovery system. It defines document parameters for imaged documents and often contains metadata for all ESI it relates to. M Media: The device used to store electronic information, e.g. hard drives, back up tapes, DVDs Metadata: Often referred to as data about data, it is the information that describes the characteristics of ESI, e.g. sender, recipient, author, date. Much of the metadata is not accessible to non-technical users. N Native Format: A file that is maintained in the format in which it was created. This format preserves metadata and details about the data that might be lost when the documents are converted to image format, e.g. pivot tables in spreadsheets. Near-duplicate: Two or more files that contain a specified percentage of similarity. Also, the process used to identify those nearly-identical files Normalization: Reformatting data so that it is stored in a standardized format O OCR: Optical Character Recognition is the process of converting images of printed pages into electronic text P Parent Document: A document to which other documents/files are attached Predictive Coding: A document categorization process that extrapolates the tagging decisions of an expert reviewer across a data set. It is an iterative process that increases accuracy with multiple training passes. Precision: In search results analysis, precision is the measure of the level of relevance to the query in the results set of documents. See Recall Processing: The ediscovery workflow which ingests data, extracts text and metadata, and normalizes the data. Some systems include the data indexing and de-duplication in their processing workflow. Production: The delivery, to the requesting party, of documents and ESI that meet the criteria of the discovery request R Recall: In search results analysis, recall is the measure of the percentage of total number of relevant documents in the corpus returned in the results set. See Precision Redact: To intentionally conceal, usually via an overlay, portions of a document considered privileged, proprietary or confidential S Search: The process of looking within a data set using specific criteria (a query). There are several types of search ranging from simple keyword to concept searches that identify documents related to the query even when the query term is not present in the document. Slack space: The unused portion of a disk that exists when the data does not completely fill the space allotted for it. This space can be examined for otherwise unavailable data. Spoliation: The destruction or alteration of data that might be relevant to a legal matter Structured data: Data stored in a structured format such as a database. Structured data can create challenges in ediscovery. See Unstructured data T TIFF: Tagged Image Format is a common graphic file format. The file extension related to this format is.tif. U Unallocated space: Most often, this is space created on a hard drive when a file is marked for deletion. This space is no longer allocated to a specific file. Until it is overwritten, it still contains the previous data and can often be retrieved. Unicode: The code standard that provides for uniform representation of character sets for all languages. It is also referred to as double-byte language. Unstructured data: Data that is not stored in a structured format such as word processing documents and presentations Complete Discovery Source (CDS Legal) is a leading provider of ediscovery services and a trusted advisor to more than 75% of the Am Law 100 and a quarter of the Fortune 500. CDS uses the most advanced, tested, and defensible technology to support all stages of the EDRM including planning, early case assessment, processing and production, review hosting, managed review, and cross-border global coverage. Peace of Mind for Serious ediscovery (855)

8

9

10 Peace of Mind for Serious ediscovery ediscovery is a critical part of your case. From proactive litigation-readiness to managed review, the process must be fast, efficient, and focused on reducing costs. Choosing the right partner to ensure a strategic and defensible approach gives you the first advantage. Having experts manage that course of action puts you one step ahead from start to finish. CDS is the first choice of more than 75% of Am Law 100 firms. CDS uses the most advanced, analytic technology available. Our hands-on experts personally shepherd your evidence through time-tested processes to provide clear strategies and superior outcomes. From accessible project managers, who you know by name, to our SAS 70 Type II audited, secure data centers, everything we do is about providing you with unmatched peace of mind. One quarter of the Fortune 500 partners with CDS. (855) Early strategies are critical to the success and path of work. Then those strategies need to be executed flawlessly. CDS is our first choice for reliable, defensible ediscovery. PARTNER, AM LAW 100 FIRM Litigation Readiness ediscovery requirements are changing fast. Companies and law firms must be prepared for today s complex discovery requests while anticipating what s next. CDS partners with legal teams to establish proactive information management through better data retention and destruction policies. Shrinking data volumes before a complaint is filed makes you faster out of the gate, reduces costs, and offers you better command of even the most demanding discovery requests. Early Case Assessment CDS proprietary data analysis system, indexes documents at lightning speed, so you can start evaluating data before most other systems have finished ingesting. Multithreading and our distributive architecture allow your CDS team to analyze multiple data sources simultaneously. Detailed reports, deduplication, and extensive filtering tools mean that the dataset can be drastically reduced prior to processing. delivers time and cost savings that are just one part of the ECA process. Processing and Production ediscovery that is fast, efficient, and defensible allows your litigation team to focus on more manageable volumes of meaningful evidence. Our project managers and engineers use best-inbreed hardware and software to expertly guide your evidence through the process. Size does not matter. We can scale our services from a single CD to terabytes of ESI. Time has direct implications on ediscovery cost and resources. CDS offers unmatched processing speeds at competitive prices. Our capacity and dedication turn most projects around in the time that it takes the competition to get started. High-speed multi-thread processing Real-time reporting De-duplication Full redaction tools Native and image file production Concordance or EDRM compliant XML load files Integrated production services continued on back NEW YORK WASHINGTON, DC CHICAGO SAN FRANCISCO ATLANTA

11 Peace of Mind for Serious ediscovery Review Hosting and Managed Review If you re working around the clock, so are we. We support our highly secure SAS 70 Type II data centers 24/7. And rest assured that we employ the most advanced technologies and protocols to protect sensitive data from damage by malice, accident or error. CDS partners with the industry s leading review tool developers and contract review staff to meet your case s specific needs. For complex projects, our experienced PMs and Director of ediscovery Strategy work with case teams to tailor workflows from reviewer training to trial preparation. CDS National Data Center is SAS 70 Type II audited. Our three locations are available 24/7. Global Coverage and Transborder ediscovery/edisclosure The world is getting smaller, but the demands of global electronic discovery are growing. Multinational corporations involved in U.S. litigation can be required to collect data that is subject to EU directives. CDS international edisclosure program merges EU-based edisclosure specialists with our proven U.S. based ediscovery firm, to seamlessly carry you through the process. And CDS Safe Harbor certification ensures that data which reaches the U.S. remains compliant with EU privacy regulations. From collection to production, our solutions are ready for your overseas needs. Our Unicode-compliant technology provides multi-lingual capabilities. It automatically identifies the languages within documents and provides detailed reports. Reviewers can search documents in Roman, Asian, and Cyrillic languages. (855) CDS Headquarters 345 Park Avenue New York, NY (212) CDS Washington 1050 Connecticut Avenue NW Washington, DC (202) CDS Chicago 70 West Madison St. Three First National Plaza Suite 1400 Chicago, IL (312) CDS has repeatedly proven its reputation as an industry leader, a partner guaranteed to tackle and meet any project and deadline in a cost effective manner. [Our] entire project had to be completed within two days time... and they completed the project early... CDS is a highly valuable, respected, and competitive asset in the e-discovery community. E-DISCOVERY PROJECT MANAGER, TOP 50 LAW FIRM CDS San Francisco 425 Market Street Suite 2200 San Francisco, CA (415) CDS has been a tremendous help and life saver... we have used them to migrate several databases from other platforms because of their stellar support and services... they have lived up to what they promised. LIT SUPPORT MANAGER, AM LAW 100 FIRM BEST IN SERVICE CERTIFIED RESELLER New York Law Journal Rankings Top Three Best ediscovery Provider Complete Discovery Source Schedule Contract 35F-0480R We self-certify compliance with U.S. EU SAFEHARBOR U.S. DEPARTMENT OF COMMERCE

12 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) COMPANY NAME IT ADMINISTRATOR JOB TITLE DEPARTMENT CONTACT NUMBER ADDRESS AVAILABILITY GENERAL INFORMATION 1. Where are the physical locations where target devices reside? 2. Are there any target locations outside the U.S.? [Yes / No] If yes, please list these other locations. 3. What are the operating systems of the servers to be collected (e.g., Unix, Linux, Windows, Macintosh OS, etc.)? 4. How many file servers are on the system (size and locations)? This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

13 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 5. Are there any proprietary systems or business servers? [Yes / No] If yes, please describe. 6. Does the company use SharePoint or other content management or data storage systems? [Yes / No] If yes, please describe. 7. Are there any structured data repositories that need to be collected (e.g., accounting data, human resources information, etc.)? [Yes / No] If yes, please describe. NOTE: This type of information, as well as information from other sources, could be subject to HIPAA or other regulations. 8. How many data storage devices and custodian computers are within the scope? Device Number Make Model Storage Desktop Laptop Mobile devices CDs DVDs Thumb drives This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

14 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 9. Does the company use any private or public cloud based services for data storage or content management? [Yes / No] If yes, please describe. 10. Can you briefly describe the ways in which your company secures company data? Are there any documented policies regarding access restrictions, device requisitioning and disposal, passwords, or any other such security and/or use policies? [Yes / No] If yes, please describe. Request copies of each policy. 11. Have any computers, servers, or other devices been replaced or decommissioned during the relevant time period? [Yes / No] If yes, please describe. Request copies of any records. 12. Are home drives available to employees? [Yes / No] If yes, please describe. Be sure to include information on how documents are shared internally within the organization. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

15 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 13. Do users have remote access to the network from outside the office? [Yes / No] If yes, please describe. DOCUMENT RETENTION POLICY 1. Is there an existing document retention policy? [Yes / No] If yes, request a copy. 2. How is the policy implemented and enforced? 3. Has the policy been suspended since this matter began? (i.e. litigation hold procedures) [Yes / No] If yes, please describe. SYSTEM 1. What type of system(s) is currently deployed (e.g., Microsoft Outlook, IBM Lotus Notes, QUALCOMM Eudora, etc.)? 2. Is the current system the same system that was used during the relevant time period? [Yes / No] If no, then please describe. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

16 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 3. Is the data center located on site or at a remote location? 4. Is there a size restriction for individual mailbox(s)? [Yes / No] If yes, what action is taken to reduce mailbox sizes? 5. How are s stored? Are there multiple stores? [Yes / No] If yes, how many and where are they located? 6. What system(s) is currently used to back up ? 7. In what format are s backed up? 8. How frequently are the s backed up (e.g., daily, weekly or monthly)? 9. Where are the archives stored? 10. How long are the archives maintained? 11. What type of storage media is used for backups? 12. Is the backup data accessible? [Yes / No] Whether yes or no, please describe. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

17 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 13. Is the backup data text searchable? [Yes / No] Whether yes or no, please describe. 14. Is the backup data separable by custodian? [Yes / No] Whether yes or no, please describe. 15. Is there a disaster recovery plan in addition to the backup procedure? [Yes / No] If yes, request a copy. 16. Does anyone use personal accounts for business? [Yes / No] If yes, please provide further details on access restrictions and acceptable use. DESKTOPS AND LAPTOPS 1. What is the estimated amount of data per custodian of interest? 2. What operating system(s) do the workstations generally run on? 3. Do any custodians use home computers for business purposes? [Yes / No] Is there a record of that information? [Yes / No] If yes, request a copy. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

18 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 4. What system(s) is currently used for workstation backup? 5. How are user workstations backed up? 6. How frequently are the workstations backed up (e.g., daily, weekly or monthly)? 7. Where are the workstation archives stored? 8. Is there an Acceptable Use Policy for personnel use of PCs, Laptops, the Internet, and/or any other devices? [Yes / No] If yes, request a copy. 9. Is there any encryption on the desktops and/or laptops? [Yes / No] If yes, please describe. Is there an Encryption Policy? [Yes / No] If yes, request a copy. BACKUP TAPES 1. Please describe your backup schedule(s) and procedures. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

19 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 2. What backup software packages and versions are currently being used? 3. Is there a documented backup tape rotation schedule? [Yes / No] Whether yes or no, please describe. 4. What types of tapes are/were used, including manufacturer, model, and size (e.g., QIC, DAT, DLT)? 5. Do the tapes contain e mail or file server data? [Yes / No / Both] Whether yes, no, or both, please describe. 6. What are the date ranges for the tapes? 7. Is there a historical records/master index of the backup tapes contents? [Yes / No] If yes, request a copy. 8. Is there an established tape labeling protocol? [Yes / No] If yes, please describe. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

20 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) REMOVABLE MEDIA 1. Do custodians store data on other media (e.g., USB drives, Zip drives, DVD, CD, Floppy, removable hard drive)? [Yes / No] If yes, please describe. Is there a policy regarding removable media? [Yes / No] If yes, request a copy. 2. Do custodians backup their own data? [Yes / No] If yes, please describe. DATABASES 1. What types of database server systems and client programs are currently in use? 2. What are the names of database software used? 3. Are there standard reports prepared from the databases? [Yes / No] If yes, please describe. LEGACY SYSTEMS 1. Are there legacy e mail, file handling, file storage, voic , or database systems? [Yes / No] If yes, please describe. 2. How is this data stored? Is this storage server within the scope of the preservation obligation? This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

21 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 3. Is the system active or archived? VOICE DATA 1. What voic system is in use? 2. Where is the voic data stored? 3. Can users store messages locally? [Yes / No] If yes, for how long? 4. Is voice data backed up? [Yes / No] If yes, how is this data backed up? MOBILE DEVICES (e.g., Blackberry, Palm, ipad, Treo, ipod, Cell Phones, etc.) 1. Does the organization issue or support any mobile devices? [Yes / No] If yes, which types of devices? Please include device type, make, model, storage sizes, and any other relevant information. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

22 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 2. Is the use of personal mobile devices permitted? [Yes / No] If yes, can they be connected to work issued devices (e.g., desktops, laptops, etc.)? [Yes / No] Are employees allowed to store data on mobile devices? [Yes / No] If yes to any of the above, please explain. 3. Is the same data stored on the device stored on shared drives at the organization? [Yes / No] If so, where are these shared drives located? 4. Is there an Acceptable Use Policy for personnel use of mobile devices? [Yes / No] If yes, request a copy. SOCIAL MEDIA AND NETWORKING 1. Are employees restricted from using social media or networking sites? [Yes / No] Please describe. If yes, is there a documented policy? [Yes / No] If yes, request a copy. 2. Are you aware of any employees using social media within the scope of their work related duties? [Yes / No] If yes, please describe. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

23 COMPUTER SYSTEMS CHECKLIST (ADMINISTRATOR) 3. Does the company itself maintain a presence on any social media or networking sites? [Yes / No] If yes, which sites? Who maintains these sites? What security precautions are taken with respect to these sites? 4. Are you aware of any social media, networking sites, or blogs related to the type of work that your company does or the industry in which your company operates? [Yes / No] If yes, please name these sites. ADDITIONAL SOURCES & COMMENTS This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

24 Digital Discovery & e-evidence Reproduced with permission from Digital Discovery & e-evidence, 12 DDEE, 12/06/2012. Copyright 2012 by The Bureau of National Affairs, Inc. ( ) BNA INSIGHT The authors explain why CIOs and CISOs must balance the sometimes competing needs of cost-effective technological solutions for business needs against legal preservation and collection requirements. The Cloud, Server Consolidation, Ephemeral Data, and Information Security: (Manageable) E-Discovery Challenges Facing CIOs and InfoSec BY FERNANDO M. PINGUELO, SHANNON CAPONE KIRK, AND SARA JOHNSON MEYERS I. Issues Facing CIOs and CISOs That May Impact E-Discovery T oday s CIOs and CISOs face challenges with a wide range of technological issues: cloud storage, server consolidation, retention of ephemeral data, data security, and privacy laws, to name a few. On top of that, electronic discovery (E-Discovery) 1 invades nearly every aspect of these issues. Why are CIOs and CISOs asked to balance the sometimes competing needs of cost-effective technological solutions for business needs against legal preservation and collection requirements (a.k.a. E-Discovery)? Simply: Because they have to. Cloud Computing. Today s CIOs are storing more data in more places than ever before. In 2011, 90.9 percent of data center sites surveyed by the Association for Computer Operations Management (AFCOM) used more storage space than they did three years ago. 2 During that same three-year period, 37 percent were able to 1 Electronic discovery, more commonly known as E-Discovery, is the component of U.S.-based legal actions whereby parties (and in some cases, non-parties) are compelled to exchange information (in particular electronically stored information (ESI)) relevant to the claims and defenses asserted in the lawsuit. The Federal Rules of Civil Procedure (FRCP), the Federal Rules of Evidence (FRE), and state and local procedural rules have been amended to address the unique challenges posed by ESI in the context of a lawsuit. For example, in December 2006, FRCP 26(a) was amended to add electronically stored information as its own category of discoverable information; FRCP 26(b)(2) sets up a two tier discovery process for accessible and inaccessible data, including procedures for cost shifting on inaccessible data. FRCP 26(b)(5) and FRE 502 both address the increased risk of inadvertent production of privileged documents given the volume of ESI to be reviewed, and FRCP 26(f) mandates that parties meet and confer prior to the commencement of discovery in order to agree on review and production protocols. 2 AFCOM Survey Examines Data Center, Cloud Computing Use. COPYRIGHT 2012 BY THE BUREAU OF NATIONAL AFFAIRS, INC. ISSN

25 2 reduce their staff, and 29 percent kept their staffing levels the same. 3 According to AFCOM CEO Jill Yaoz, this trend is in large part due to the development and implementation of new tools and processes that have allowed IT departments and data centers to store massive amounts of data efficiently and inexpensively. Specifically, IT departments and data centers are increasingly relying on cloud-based systems for the storage of big data. The advent of cloud-based storage systems has had a profound impact on the way businesses collect and store their information. By offering the ability to store large amounts of data remotely, cloud storage not only enables CIOs to drastically reduce their IT budgets, but also allows IT departments the freedom to concentrate on matters more central to business growth, such as high-value activities and innovation. The number of data centers implementing cloud-based storage is only expected to increase. According to Yaoz, [o]ur prediction is that 80 percent to 90 percent of all data centers will be adopting some form of cloud computing in the next five years. 4 Yet, E-Discovery concerns may require businesses to maintain comprehensive records on what data is stored in the cloud and where, precisely, that cloud exists. This information may become important in the context of E-Discovery if data is located in a foreign jurisdiction with privacy laws restricting or blocking the transfer and/or storage of data. 5 In addition, some cloud service providers employ subcontractors to store data. This may be problematic for businesses ordered to produce data in a timely manner, especially if the requested data is located in more than one place with more than one subcontractor. In other words, compared to pre-cloud storage, businesses may lose control over the timing and execution of the extraction of data. At a recent conference in Palo Alto, Google CIO Ben Fried addressed the importance of streamlining various cloud storage providers tapped by the same company to ensure easy and efficient integration and to prevent what he characterized as a widespread lack of good corporate practices in the context of remote data storage afcom-survey-examines-data-center-cloud-computinguse.htm. 3 IT Workloads Up As Data Center Technology Added Without More Staff. it-workloads-data-center-technology-added-without-morestaff Id. 5 See, e.g., Directive 95/46/EC, of the European Parliament and of the Council of 24 October 1995 on the Protection of Individuals with Regard to the Processing of Personal Data and on the Free Movement of Such Data [hereinafter Directive 95/ 46], 1995 O.J. (L281) (EC), available at LexUriServ/LexUriServ.do? uri=celex:31995l0046:en:html (prohibiting transfers of data to other countries that do not provide adequate levels of protection), and Working Document 1/2009 on Pre-Trial Discovery for Cross Border Civil Litigation, at 8 (Feb. 11, 2009) [hereinafter E.U. Working Document], available at ec.europa.eu/justice/policies/privacy/docs/wpdocs/2009/ wp158_en.pdf (noting that [t]he mere or unsubstantiated possibility that an action may be brought before the U.S. Courts may not be sufficient to impose preservation obligations.). 6 Top CIO Challenges: Cloud, BYOD, Big Data. Cloud-BOYD-Big-Data /. Bask Iyer, CIO of Juniper Networks, added that businesses must be mindful of employee use habits so as to ensure that employees utilize the company s chosen storage service providers. 7 [C]ompared to pre-cloud storage, businesses may lose control over the timing and execution of the extraction of data. Simply put, because of the ease associated with storing data in the cloud, businesses might assume the same is true for producing data from the cloud. This assumption, if untrue, might complicate a company s attempt to comply with E-Discovery demands: Data may be in a foreign location with privacy laws, manpower, and technology that impede the search, storage and extraction of data in the method or time required. Also, data for one custodian may not necessarily be grouped together on one server, and a group of custodians might be dispersed amongst many servers. Moreover, issues can arise with respect to the notification and imposition of a company s preservation obligations when information is stored in the cloud as opposed to on a company s own servers. Admissibility issues can also be triggered if the chain of custody for the various document sources is not accurately tracked and documented. In other words, the data is quite simply not all in one room; it may be in many shifting rooms, which brings us to our next topic: server consolidation. Server Consolidation At the same time that CIOs are looking to reduce spending by outsourcing data management and retention to the cloud, they may also be looking to consolidate internal servers to one location, or if it is a worldwide company, to one or several regions. 8 The analysis a CIO might consider may typically include the following factors: 1. Cost savings with consolidation will we reap them? 7 Id. 8 As noted above, foreign data privacy laws must be taken into consideration when implementing decisions about data storage. In the event of U.S. litigation, data stored in a foreign country will likely not be as accessible in response to U.S. discovery demands. Moreover, foreign data privacy laws may restrict the retention of certain types of data, such as personal information, and may impose restrictions on whether and how long data can be retained, making compliance with preservation obligations under U.S. case law problematic. See, e.g., the discussion of the Zubulake case, infra; Directive 95/46, supra and E.U. Working Document, supra, note 6, at 8. For a detailed analysis of the implications surrounding the imposition of U.S. litigation holds in foreign jurisdictions, see Kenneth N. Rashbaum, Matthew Knouff & Melinda Albert, U.S. Legal Holds Across Borders: A Legal Conundrum?, 13 N.C. JOLT 68 (Fall 2011) available at Rashbaum_Knouff_Albert_69_94.pdf COPYRIGHT 2012 BY THE BUREAU OF NATIONAL AFFAIRS, INC. DDEE ISSN

26 3 2. Business continuity is there a risk of consolidating i.e. if the servers in one location experience an interruption, how (and how long) will that impact the company? (Think power loss, storms/natural disaster, war/civil unrest.) 3. Data privacy laws how might they be impacted by consolidation? How can and should I align disparate laws and regulations? 4. Security how can I ensure it? It is difficult to answer these questions in a vacuum, especially because no two companies needs are the same and because local counsel in differing jurisdictions are usually required for full analysis. While no U.S. court is known to have directly weighed in on the pros and cons of server consolidation, one court did recently refer to server consolidation in the context of a larger E-Discovery dispute. In the patent infringement action, Apple v. Samsung, 9 the parties sought adverse inference jury instructions regarding spoliation of evidence. Responding to Apple s motion, the court considered whether Samsung took adequate steps to prevent the spoliation of relevant data after it should have reasonably anticipated litigation. At issue was Samsung s default system, called mysingle, which automatically deletes all s after a two-week period, unless otherwise specified by the employee user. According to Samsung, it relied on the automated deletion feature for several business reasons. First, automated deletion can help to prevent the misappropriation of confidential business information in the event a computer is lost or stolen. Second, a two-week deletion period is less expensive to implement than a 30-day deletion period. Third, the shorter window means that confidential business information is less likely to be disclosed to a third-party via or unauthorized access. Fourth, a two-week automated deletion period best complies with Korean privacy laws. Apple argued that Samsung s continued use of its biweekly destruction policy without any methodology for verifying whether Samsung employees complied with Samsung s litigation hold instructions demonstrated inadequate data retention methods resulting in spoliation of relevant evidence. The court agreed with Apple, and directed that an adverse inference instruction be used at trial against Samsung. 10 Although the opinion does not directly address the issue of server consolidation, the court does refer to the fact that all of Samsung s was consolidated on servers in Korea. [Samsung s ] contains a general guideline [that] calls for all s to be automatically deleted after the passage of two weeks. This functionality operates and stores company-wide in Korea [and] has no exceptions Without elaborating on its opinion of the server consolidation, the Court did take the time to make this further statement: [Samsung s platform] stores received and sent employee s on company-wide servers, as opposed to dividing the servers by business unit From these references, some might wonder whether the court intended to imply that a company might better manage its preservation duties by not consolidating all functionality in one location. In other words, is the court asking whether Samsung could have lifted the 14- day deletion policy for certain relevant business units and thereby complied with its preservation duties and also not had the burden of storing all s, companywide? 13 Ephemeral data such as temporary Internet files, frequently overwritten or automatically updated metadata..., log files..., RAM, and fragments of deleted files in unallocated (slack) computer space may be possible sources of responsive data. Ephemeral Data and Data Security. The management of cloud-based data and server consolidation is not the only consideration for a company in litigation. Ephemeral data such as temporary Internet files, frequently overwritten or automatically updated metadata (such as last-opened files), log files (such as firewall or other security related log data), RAM, and fragments of deleted files in unallocated (slack) computer space may be possible sources of responsive data. 9 Apple Inc. v. Samsung Electronics, BL (N.D. Cal. July 25, 2012); Motion for Relief from Judgment granted in part by Apple Inc. v. Samsung Electronics, 2012 BL (N.D. Cal. August 21, 2012); (see, 12 DDEE 304, 8/2/12) 10 Apple Inc. v. Samsung Electronics, 2012 BL, supra, at. Note however, that this instruction was never given because the presiding trial judge, Federal District Court Judge Lucy Koh, overruled Federal Magistrate Judge Paul S. Grewal s Order, and decided instead to give the same instruction with respect to each of the parties, i.e., that they had each failed to preserve relevant evidence after the duty to preserve arose and [w]hether this fact is important to you in reaching a verdict in this case is for you to decide. Apple Inc. v. Samsung Electronics, 2012 BL, supra at *. This was a significant victory for Samsung which had argued that Apple s duty to preserve must have arisen no later than Samsung s. Magistrate Judge Grewal had not reached Samsung s equal treatment argument initially because he had denied Samsung s motion as untimely. Ultimately, however, the identical adverse inference instructions were not given at trial because... the parties indicated at the August 20, 2012 hearing that if the Court decided to issue identical adverse inference instructions against both parties, they prefer that neither adverse inference instruction be given. Id. 11 Apple Inc. v. Samsung Electronics, 2012 BL, supra, at. 12 Id. 13 Although arguably it may seem to be less burdensome to implement a hold where, as here, servers are consolidated, Samsung filed affidavits claiming that to extend the retention policy for its employees would have cost an additional $35,983,193 per year. Id. at, note 43. However, Samsung did not estimate the cost of temporarily moving key custodians accounts to unique servers that do not biweekly destroy s, or the cost of temporarily moving... these custodians accounts to Microsoft Outlook. Id. In addition to cost, data privacy laws may also affect a company s data storage decisions. As noted above, Samsung averred that their consolidated storage system policy was the one that best complies with Korean privacy law. Id. at note 45. DIGITAL DISCOVERY & E-EVIDENCE REPORT ISSN BNA

27 4 Two recent cases address the issue of ephemeral data in the context of E-Discovery. In Nacco Materials Handling Group v. Lilly Company, 14 a lift truck manufacturer brought an action against a former dealer alleging that the dealer improperly accessed the manufacturer s secure website. The court held that the dealer had an obligation to initiate the preservation of relevant ESI including ephemeral data such as Internet browsing histories immediately upon receiving the manufacturer s complaint. Because the dealer failed to preserve the relevant data, the court imposed sanctions, specifically ordering the reimbursement of plaintiff s costs for forensic examination, as well as the cost of imaging and analysis of the manufacturer s hard drives. In Tener v. Cremer, 15 the court considered whether a non-party should be compelled to produce temporary user logs, which were deleted through the course of normal business operations. After finding that defendant posted defamatory statements using a computer owned and operated by New York University (NYU), plaintiff served the school with a subpoena seeking the names of all of the individuals who accessed the Internet on the date the statement originated. Plaintiff argued that NYU could use special software to retrieve the data, despite NYU s claim that the data had been overwritten numerous times and therefore could not be retrieved, at least not without undue burden or cost. The court articulated a two-pronged inquiry to determine whether NYU would be compelled to produce the requested data. First, the court considered whether the data sought is recoverable. Second, the court engaged in a cost-benefit analysis to determine if the needs of the case warrant the data sought. Here, the court remanded the case to determine whether the data sought was accessible and whether it could be produced without undue burden or cost. Although ephemeral data is certainly not necessary in most cases, considering the high costs associated with these forensic discovery methods, parties should consider whether ephemeral data is a source of information that might be responsive to a given litigation Nacco Materials Handling Group v. Lilly Company, 278 F.R.D. 395 (W.D. Tenn. 2011). 15 Tener v. Cremer, 931 N.Y.S.2d 552 (N.Y. App. Div. 2011); (see 12 DDEE 164, 4/26/12). 16 If so, consideration should be given to admissibility issues, particularly with respect to foreign ephemeral data, such as whether such data can be authenticated and/or would fall under the Business Records exception to the hearsay rule, FRE 803(6). The Sedona Conference Commentary on ESI Admissibility notes, for example, that when a system, instead of an individual, is the creator or owner of a document or file, as when logs of certain activity are automatically created by an operating system, it may be difficult to provide foundational evidence as to how and where the data was created. The Sedona Conference Commentary on ESI Evidence and Admissibility at (March 2008), available at https:// thesedonaconference.org/publication/the%2520sedona% 2520Conference%25C2%25AE%2520Commentary%2520on% 2520ESI%2520Evidence%2520%252526%2520Admissibility. Although ephemeral data is certainly not necessary in most cases... parties should consider whether ephemeral data is a source of information that might be responsive to a given litigation. Another nuance in the preservation of ephemeral data is the need to sometimes preserve firewall, antivirus, intrusion prevention and/or detection, network activity, and other logs of information that might show the path a hacker might have taken. The preservation, of course, might be for a forensic investigation or litigation needs, but may very well be different from the normal course of business. Parties faced with this type of preservation request are often asked if they have centralized logging, and if not, how quickly the overwriting of logging can be disabled. Any CISO concerned with being prepared for investigating security incidents might want to consider how to respond to an urgent request to preserve this type of data. 17 II. What Is the Legal Department Faced With? As noted in the discussion of the Samsung case above, several courts have held that it is counsel s duty to affirm that a party has complied with its applicable discovery obligations; several note that counsel s duty does not end with the issuance of a litigation hold. 18 Courts have also held that counsel cannot rely on a client s assertions that all relevant information has been preserved and collected. Therefore, counsel may seek to work directly with IT personnel, many times below the rank of CIO, CISO, manager or director, who have direct responsibility for and knowledge of relevant data sources, both on site and in the cloud. For example, in a data security matter, counsel may wish to speak directly with the person who manages the firewall logs. Then, Counsel and IT professionals might work in tandem to assist with the implementation of the litigation hold by identifying the appropriate custodians and other data sources. Case Law. In the seminal case on this subject, Zubulake v. UBS Warburg, 19 Judge Shira Scheindlin imposed sanctions on UBS for failing to guarantee that relevant data was both preserved and produced. Accordingly, she granted plaintiff s motions for sanctions in the form of an adverse inference instruction with re- 17 Again, as noted above, foreign data privacy laws may operate to restrict the ability to collect and preserve foreign ephemeral data, by for example, limiting that which is collected to that which is relevant and not excessive to a particular investigation. Directive 95/46, supra, note 6, at art. 6; E.U. Working Document, supra, note 6 at Under FRCP 26(g), counsel of record must certify that all disclosures are correct and complete and sanctions may be imposed for non-compliance with this requirement. 19 Zubulake v. UBS Warburg, LLC, 229 F.R.D. 422 (S.D.N.Y 2004) ( Zubulake V ) COPYRIGHT 2012 BY THE BUREAU OF NATIONAL AFFAIRS, INC. DDEE ISSN

28 5 spect to deleted s and the imposition of certain costs. In reaching this conclusion, the Zubulake court discussed counsel s role in ensuring the identification, preservation, and production of relevant information: A party s discovery obligations do not end with the implementation of a litigation hold to the contrary, that s only the beginning. Counsel must oversee compliance with the litigation hold, monitoring the party s efforts to retain and produce the relevant documents. Proper communication between a party and her lawyer will ensure (1) that all relevant information (or at least all sources of relevant information) is discovered, (2) that relevant information is retained on a continuing basis, and (3) that relevant nonprivileged material is produced to the opposing party. 20 In a recent decision, Nat l Day Laborer Org. Network v. United States Immigration & Customs Enforcement Agency, 21 Judge Scheindlin strongly admonished against the practice of self collection, whereby parties allow individual employees to search for relevant data and applied that caution against government agencies in the context of a Freedom of Information Act ( FOIA ) case. Nat l Day Laborer Org. Network dealt with plaintiffs requests for documents under the FOIA and their assertions that defendant government entities searches for such information were inadequate. Although this case involves an analysis of the reasonableness of such search efforts under the FOIA, Judge Scheindlin s opinion is applicable in the broader context of E-Discovery. Plaintiffs charged that the search efforts of the FBI and Department of Homeland Security were unreasonable. In the context of the FOIA, the court found that the scope of some of the government s searches were adequate while some were not, noting that it is impossible to evaluate the adequacy of an electronic search for records without knowing what search terms have been used. 22 Disagreeing with defendants argument that custodians should be trusted to run effective searches of their own (i.e., self-collect ), the court stated: 20 Id. at Nat l Day Laborer Org. Networkv. United States Immigration and Customs Enforcement Agency, 2012 BL WL (S.D.N.Y. July 13, 2012). 22 Id. at *1011. There are two answers to defendants question. First, custodians cannot be trusted to run effective searches, without providing a detailed description of those searches, because FOIA places a burden on defendants to establish that they have conducted adequate searches; FOIA permits agencies to do so by submitting affidavits that contain reasonable specificity of detail rather than merely conclusory statements. Defendants counsel recognize that, for over twenty years, courts have required that these affidavits set [ ] forth the search terms and the type of search performed. But, somehow, DHS, ICE, and the FBI have not gotten the message. So it bears repetition: the government will not be able to establish the adequacy of its FOIA searches if it does not record and report the search terms that it used, how it combined them, and whether it searched the full text of documents. The second answer to defendants question has emerged from scholarship and case law only in recent years: most custodians cannot be trusted to run effective searches because designing legally sufficient electronic searches in the discovery or FOIA contexts is not part of their daily responsibilities. Searching for an answer on Google (or Westlaw or Lexis) is very different from searching for all responsive documents in the FOIA or e-discovery context. 23 Judge Scheindlin concluded that it was unnecessary for her to evaluate the adequacy of defendants search terms. Instead, she ordered the parties to work cooperatively to agree on additional search terms and protocols and, if necessary, testing to evaluate and refine those terms.... Defendant agencies, in turn, will need to cooperate fully with plaintiffs. 24 National Day Laborer Org. Network deals with a number of relevant current E-Discovery issues, including the adequacy of keyword search terms and technology-assisted review. Importantly, from the perspective of a CIO, it also demonstrates the technological expertise required to assure the adequacy of a collection effort. A recent case in the Eastern District of Texas is instructive on this point. In Green v. Blitz, 25 plaintiff brought a product liability suit against a manufacturer of gasoline cans, alleging that a gas can had caused her husband s death. Her main theory of liability was premised on the fact that the can had no flame arrester, which defendants had not incorporated and which they argued did not work anyway. During the jury deliberations, the parties entered into a high-low settlement agreement under which the parties agreed that, notwithstanding the verdict, the case would be dismissed with prejudice and a release executed. The jury ultimately ruled in the plaintiff s favor, but under the parties agreement, the amount awarded triggered a payment by the defendant on the low end of the scale. Subsequently, counsel for plaintiff was involved in related litigation during which he learned of extremely relevant and material documents that had not been produced in the Green case. Plaintiff filed a motion to re-open the case and for sanctions arguing that defendant s discovery violations were in bad faith and had led to an unfair outcome. The court held that the defendant had committed several discovery abuses, including failing to investigate, search for, and produce relevant evidence, and never issuing a written litigation hold order. The sanctions were both steep and creative: The court ordered the defendant to pay plaintiff $250,000 in sanctions, to disclose the court s order to plaintiffs in every lawsuit proceeding against it, and to disclose the order in any case in which defendant is a party for the next five years (a $500,000 sanction would be returned upon compliance). What was the nature of the defendant s failings in Green that warranted these extreme sanctions? According to the court, in responding to discovery, the employee solely responsible for searching for and collecting documents relevant to the litigation, 26 and who also headed up the research and investigation around flame arresters, 27 failed to take several steps necessary to discharge a duty to preserve and investigate responsive documents: No litigation hold was issued, key 23 Id. at * Id. at * Green v. Blitz, 2011 BL (E.D. Tex, Mar. 1, 2011); (11 DDEE 168, 4/14/11). 26 Id. at * Id. DIGITAL DISCOVERY & E-EVIDENCE REPORT ISSN BNA

29 6 28 Id. at*7. 29 Id. at*9 30 Id. at*10 word searches were not run (a search for flame arrester was not run and would have garnered responsive information as it was in the subject line of s), and the individual did not contact defendant s IT personnel to coordinate with the search and extraction of relevant data. Making matters worse, this employee admitted that he was about as computer literate-illiterate as they get. 28 In light of the undisclosed documents, the court found that defendant s conduct was willful and plaintiff was prejudiced. The failure to search relevant data was compounded where no coordination with IT occurred and thus preservation was not ensured. Indeed, according to the opinion, a lead member in IT had repeatedly requested that employees delete their s during the precise time period that defendant was actively defending multiple products liability suits relating to its gas cans and had a duty to preserve evidence. 29 The court also addressed defendant s failure to halt the overwriting of back up tapes, concluding that it will never be known how much prejudice against the plaintiff was actually caused by Blitz s failure to preserve documents. 30 While obviously an extreme factual scenario, Green should serve as an example of how valuable coordination with IT professionals can be in the context of E-Discovery. III. E-Discovery Tips for the IT Professional In conclusion, the following considerations may prove useful in managing the E-Discovery issues posed by big data: 1. Consider anticipating E-Discovery issues in advance of litigation, taking precautions such as mapping the location and/or filepaths of relevant data (wherever located, e.g., on-site, in the cloud, etc.) 2. Consider reviewing your cloud storage vendor and subcontractor contracts and ensure that the appropriate contractual obligations are in place with respect to how and by whom stored data are to be preserved, collected, and produced. 3. Consider the above questions concerning server consolidation with appropriate counsel, along with other issues counsel may raise. 4. Consider if education among IT staff is necessary in conjunction with the legal department s needs. 5. Plan early for preservation in countries where relevant data reside, taking into account local laws that may affect implementation of litigation holds. 6. Document data collection, review and chain of custody, particularly for data in the cloud and outside the U.S., in order to lay a foundation for admissibility. Fernando M. Pinguelo is the Chair of the Cyber Security & Data Protection Group at Norris McLaughlin & Marcus; Shannon Capone Kirk is E-Discovery Counsel at Ropes & Gray; Sara Johnson Meyers is a Senior Discovery Attorney at Ropes & Gray. The views set forth in this article are those of the authors alone and do not reflect the views of their respective firms or clients. Nothing in this article should be viewed as attorney advice. Any individuals seeking advice on the matters presented herein should seek individual and specific advice, as facts and application of law change for every matter. The authors may be contacted as follows: COPYRIGHT 2012 BY THE BUREAU OF NATIONAL AFFAIRS, INC. DDEE ISSN

30 The Sedona Conference Commentary on Legal Holds: The Trigger & The Process Information is the lifeblood of the modern world, a fact that is at the core of our litigation discovery system. The law has developed rules regarding the manner in which information is to be treated in connection with litigation. One of the principal rules is that whenever litigation is reasonably anticipated or pending against an organization or an organization contemplates initiating litigation that organization has a duty to take reasonable steps to preserve relevant information. The duty to preserve information includes an obligation to identify, locate, and maintain information that is relevant to specific, predictable, and identifiable litigation. When preservation of electronically stored information ( ESI ) is required, the duty to preserve supersedes records management policies that would otherwise result in the destruction of ESI. A legal hold program defines the processes by which information is identified, preserved, and maintained when it has been determined that a duty to preserve has arisen. The basic principle that an organization has a duty to preserve relevant information in anticipation of litigation is easy to articulate. However, the precise application of that duty can be elusive. Every day, organizations apply the basic principle to real-world circumstances, confronting the issue of when the obligation is triggered and, once triggered, what is the scope of the obligation. Recent court decisions, most notably Pension Committee v. Banc of America Securities, LLC, 685 F. Supp. 2d 456 (S.D.N.Y. Jan. 15, 2010) and Rimkus Consulting v. Cammarata, 688 F. Supp. 2d 598 (S.D. Tex. Feb. 19, 2010), illustrate the importance of understanding preservation obligations on the consequences of ignoring them. This updated 2010 Commentary is intended to provide guidance on those issues and is divided into two parts: the trigger and the process. Part I addresses the trigger issue and provides practical guidelines for making a determination as to when the duty to preserve relevant information arises. What should be preserved and how the preservation process should be undertaken including the implementation of legal holds is addressed in Part II. The guidelines are intended to facilitate reasonable and good faith compliance with preservation obligations. The guidelines are meant to provide the framework an organization can use to create its own preservation procedures. In addition to the guidelines, suggestions as to best practices are provided along with several illustrations as to how the guidelines and best practices might be applied under hypothetical factual situations. Guideline 1: Guideline 2: Guideline 3: Guideline 4: Guideline 5: Guideline 6: A reasonable anticipation of litigation arises when an organization is on notice of a credible probability that it will become involved in litigation, seriously contemplates initiating litigation, or when it takes specific actions to commence litigation. Adopting and consistently following a policy or practice governing an organization s preservation obligations are factors that may demonstrate reasonableness and good faith. Adopting a process for reporting information relating to a probable threat of litigation to a responsible decision maker may assist in demonstrating reasonableness and good faith. Determining whether litigation is or should be reasonably anticipated should be based on a good faith and reasonable evaluation of relevant facts and circumstances. Evaluating an organization s preservation decisions should be based on the good faith and reasonableness of the decisions undertaken (including whether a legal hold is necessary and how it should be executed) at the time they are made. The duty to preserve involves reasonable and good faith efforts, taken as soon as is practicable and applied proportionately, to identify and, as necessary, notify persons likely to have relevant information to preserve the information. p 1

31 The Sedona Conference Commentary on Legal Holds: The Trigger & The Process Guideline 7: Guideline 8: Guideline 9: Guideline 10: Guideline 11: Factors that may be considered in determining the scope of information that should be preserved include the nature of the issues raised in the matter, the accessibility of the information, the probative value of the information, and the relative burdens and costs of the preservation effort. In circumstances where issuing a legal hold notice is appropriate, such a notice is most effective when the organization identifies the custodians and data stewards most likely to have relevant information, and when the notice: a) Communicates in a manner that assists persons in taking actions that are, in good faith, intended to be effective b) Is in an appropriate form, which may be written c) Provides information on how preservation is to be undertaken d) Is periodically reviewed and, when necessary, reissued in either its original or an amended form, and e) Addresses features of relevant information systems that may prevent retention of potentially discoverable information. An organization should consider documenting the legal hold policy, and, when appropriate, the process of implementing the hold in a specific case, considering that both the policy and the process may be subject to scrutiny by opposing parties and review by the court. Compliance with a legal hold should be regularly monitored. Any legal hold policy, procedure, or practice should include provisions for releasing the hold upon the termination of the matter at issue so that the organization can adhere to policies for managing information through its useful lifecycle in the absence of a legal hold. Copyright 2010, The Sedona Conference Reprinted courtesy of The Sedona Conference The full text of this document is available free for personal use from The Sedona Conference website at p 2

32 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 1 of 36 PageID: 4275 UNITED STATES DISTRICT COURT DISTRICT OF NEW JERSEY MEDEVA PHARMA SUISSE A.G., et al., Civil Action No (FLW) Plaintiffs, v. MEMORANDUM OPINION ROXANE LABORATORIES, INC. Defendant. BONGIOVANNI, Magistrate Judge This matter comes before the Court upon Plaintiffs Medeva Pharma Suisse A.G., Warner Chilcott Pharmaceuticals Inc. and Warner Chilcott Company, LLC s (collectively, Medeva ) motion to preclude Defendant Roxane Laboratories, Inc. ( Roxane ) from disputing the location and duration of release of mesalamine from Roxane s generic product and Asacol in the human body. The Court has fully reviewed and considered all of the papers submitted in support of and in opposition to Medeva s motion as well as the arguments set forth by counsel during the November 23, 2010 hearing. For the reasons set forth more fully below, Medeva s motion is GRANTED in part and DENIED in part. I. Procedural History and Background Medeva filed the instant Hatch-Waxman patent infringement case against Roxane on October 26, 2007 after Roxane filed an Abbreviated New Drug Application ( ANDA ) and Paragraph IV Certification and Notice letter seeking approval to commercially manufacture and market a generic version of Medeva s Asacol product, which is marketed under United States Patent No. 5,541,170 (the 170 patent ) to treat mild to moderate ulcerative colitis. The 170 patent is owned by Plaintiff Medeva Pharma Suisse A.G. Prior to October 30, 2009, the 170

33 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 2 of 36 PageID: 4276 patent was licensed exclusively to Proctor & Gamble Pharmaceuticals, Inc. ( P&G ), now known as Warner Chilcott Pharmaceuticals Inc. On October 30, 2009, Warner Chilcott Company, LLC acquired P&G. Warner Chilcott Company, LLC is now the exclusive licensee of the 170 patent. The 170 patent relates to the formulation of the active ingredient mesalamine, by which a core of mesalamine is specially coated for release to the afflicted gastrointestinal tissues. The 170 patent claims a tablet formulation of mesalamine which includes an acrylicbased resin coating that releases the entire dose of mesalamine to the right side of the colon so that it acts topically to relieve symptoms. The infringement issue in this matter relates to whether Roxane s proposed generic product releases its mesalamine dose in accordance with the claim. The validity issues concern whether the claimed invention would have been obvious to a person of ordinary skill in the art. At issue in the present motion is Medeva s claim that Roxane is responsible for the spoliation of evidence. Specifically, Medeva argues that Roxane (1) destroyed highly-relevant documents concerning Roxane s product development efforts as well as its testing to determine where and when Asacol releases mesalamine and (2) manufactured a secret batch of its generic product in order to conduct testing in humans after persuading the Court to preclude Medeva from conducting the same kind of testing without prior Court approval. Both of these arguments as well as Roxane s opposition thereto are discussed in more detail below. As a result of the alleged spoliation, Medeva requests that sanctions be imposed. Initially, Medeva sought to preclude Roxane from disputing the location and duration of release of mesalamine from Roxane s generic product and Asacol in the human body. Medeva then expanded its request for 2

34 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 3 of 36 PageID: 4277 sanctions to include additional and lesser sanctions such as the spoliation inference, reopening discovery, waiver of the attorney-client privilege and the imposition of attorney s fees and costs. A. The Destruction of Documents 1. Medeva s Position Medeva claims that Roxane impermissibly destroyed relevant documents after it anticipated litigation and was under a duty to preserve same. In this regard, Medeva argues that Roxane had a duty to preserve documents as early as July 31, 2001, which corresponds to the date of an entry on Roxane s privilege log that asserts work product protection; Medeva notes that the work product doctrine does not apply unless the document at issue was created in anticipation of litigation. Further, Medeva claims that Roxane certainly had a duty to preserve documents no later than February 2002 as the evidence establishes that Roxane knew at that time that it intended to file an ANDA with a Paragraph IV Certification regarding its generic product. Nevertheless, Medeva claims that Roxane failed to issue a document preservation notice (i.e., a litigation hold) until July Additionally, Medeva claims that Roxane s normal document retention policies did not sufficiently protect against spoliation. Indeed, Medeva argues that Roxane s normal document retention policies confirm that Roxane breached its duty to preserve documents because they show that documents were actually destroyed in 2002 and thereafter. Further, Medeva argues that the testimony obtained in this case also confirms that Roxane had annual file clean-up days 3

35 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 4 of 36 PageID: during which documents were destroyed. Medeva therefore contends that Roxane failed to take adequate steps to prevent spoliation between July 2001 and July Further, Medeva argues that the evidence establishes that relevant documents were actually destroyed. For example, Medeva claims that Roxane destroyed documents relating to its first pharmacokinetic study involving Asacol (the Anapharm study ). Medeva claims that Roxane failed to produce (1) the protocol for the Anapharm study, which Roxane s Director of Regulatory and Medical Affairs testified would have shown Roxane s objectives and hypotheses for the study and which Roxane confirmed no longer exists; (2) Dr. Kramer s (Roxane s consultant) final report from the Anapharm study and (3) any subsequent correspondence regarding the study, despite the fact that Dr. Kramer sent an to Roxane asking whether any changes regarding the study should be made. Moreover, Medeva argues that Roxane cannot shift its blame for failing to preserve documents related to the Anapharm study by arguing that Medeva could have, but did not subpoena Anapharm for the missing information. Medeva argues that Roxane had an independent responsibility to preserve the missing information. Likewise, Medeva claims that it was not in a position to pursue discovery related to the Anapharm study because it was not listed in Roxane s Rule 26(a) disclosures, it is only 1 Medeva also argues that Roxane s reliance on a document retention policy which established that documents such as laboratory notebooks, technical reports and developmental and analytical records were preserved for at least 20 years is inadequate because that policy is dated May 28, As a result, Medeva contends that Roxane cannot establish that it took reasonable steps to preserve all relevant information between July 2001 and July 2007 when Roxane finally issued a litigation hold. Roxane, however, subsequently produced document retention policies dating back to 1998, which show that it was its normal business practice to retain technical information for 30 years. Medeva, however, notes that none of the document retention policies relied on by Roxane specifically reference Roxane s generic mesalamine product. 4

36 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 5 of 36 PageID: 4279 referenced in a single , Roxane s witnesses did not mention it in their testimony, Roxane did not produce its research agreement with Anapharm and Roxane would not admit that the study existed. Indeed, Medeva claims that by the time it learned of the Anapharm study, it was the fall of 2009 and fact discovery had closed. As such, Medeva claims that it did not have the opportunity to subpoena Anapharm and that even if it were granted leave to do so out of time, it did not believe that it would get a response from Anapharm in time to make use of it here. In addition, Medeva argues that Roxane has only produced several documents in traditional paper file form and has failed to produce the corresponding electronic versions of these s and documents. Such documents relate to analysis of pharmacokinetic studies involving Roxane s prototype formulations as well as correspondence regarding the design of Roxane s clinical study created to test the alleged clinical effectiveness of Roxane s generic product against both Asacol and a placebo. Specifically, Medeva takes issue with Roxane s production of electronic documents and s from several key custodians. (Medeva Reply Br. at 4). For example, Medeva claims that no s or other electronic documents of any kind were collected from Eric Spiller, Roxane s Associate Director of Product Development and the project leader responsible for developing Roxane s generic product. During oral argument, Medeva clarified that it is not arguing that Roxane should have produced documents in both paper and electronic format, but instead, that the lack of electronic documents establishes that information was destroyed. Medeva also argues that Roxane failed to produced signed final reports for studies related to Roxane s development and testing of its generic product. Medeva contends that it knows that these signed final reports exist because it obtained them from third parties, even though Roxane 5

37 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 6 of 36 PageID: 4280 failed to produce same. Similarly, Medeva argues that discovery obtained from Quintiles, a contractor retained by Roxane to conduct a clinical study, supports the fact that Roxane destroyed documents. In this regard, Medeva points out that Quintiles, but not Roxane, produced a number of s sent between Quintiles and Roxane employees between 2005 and 2006 that concerned Roxane s evaluation of U.K. Asacol as well as Roxane s representations to foreign regulatory authorities concerning the formulation characteristics of its tablets. Medeva claims that because Roxane did not suspend its routine destruction of documents, there is no way of knowing what other documents and s were destroyed before Roxane implemented its litigation hold. Medeva also argues that what is clear form the sheer number of destroyed documents is that Roxane is liable for spoliation. Further, Medeva contends that the destroyed documents are relevant, indeed critical, to this litigation and that it has been prejudiced by their loss. With respect to relevance, Medeva argues that the destroyed documents are relevant to (1) Roxane s efforts to develop its formulation; (2) Roxane s claims of non-infringement; and (3) Roxane s challenge to the validity of the 170 patent. Indeed, Medeva argues that the destroyed documents bear on critical issues such as whether Roxane s generic product releases mesalamine to the right side of the colon and whether Asacol releases mesalamine to the right side of the colon, a fact that Roxane has claimed is central to its defense of this matter. For example, Medeva contends that the Anapharm study, which Medeva claims evaluated the release of Asacol in the gastrointestinal tract, is obviously relevant to Roxane s allegations that Asacol is not a commercial embodiment of the 170 patent and Roxane s claims of non-infringement. 6

38 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 7 of 36 PageID: 4281 With respect to prejudice, Medeva claims that the fact that relevant evidence was destroyed establishes prejudice. Indeed, Medeva argues that all it has to show to establish prejudice is that its ability to effectively prepare a full and complete trial strategy has been impeded. Medeva argues that its ability to effectively prepare such a trial strategy has in fact been impeded because Roxane destroyed documents that concern the location of the release of mesalamine as well as Roxane s development of its generic product. Given the destruction, Medeva does not have access to the missing documentary evidence and has been prevented from questioning Roxane s witnesses regarding same. What is more, Medeva claims that the destroyed documents appear to support the conclusion that Asacol releases at the terminal ileum, as claimed by the 170 patent. Further, Medeva claims that the destruction of the aforementioned documents is prejudicial because Roxane keeps altering its position regarding the location of where its generic product releases in the body. Medeva claims that because the cite of release has been a moving target, it is prejudiced by the destruction of documents that bear on where and how Roxane designed its product to release in the gastrointestinal tract (i.e., the types of documents that would shed the most light on Roxane s design decisions and testing of its product and Asacol). In addition, Medeva argues that the prejudice imposed on it has been increased because, not only were relevant documents destroyed, but Roxane sought to evade discovery through deposition testimony and interrogatory responses on the subject matter contained in the destroyed documents. For example, Medeva claims that despite the fact that the FDA was interested in where Roxane s generic product releases in the body and despite the fact that Roxane made statements in documents submitted to the FDA regarding where its product releases mesalamine, 7

39 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 8 of 36 PageID: Roxane refused to produce discovery regarding where its generic product releases mesalamine. Indeed, Medeva argues that no deponent, including Roxane s Rule 30(b)(6) witness, was able to testify regarding where Roxane s generic product releases mesalamine in the gastrointestinal tract or the basis for Roxane s characterization of its generic product s release characteristics that is found in Roxane s ANDA, proposed labeling, Paragraph IV Notice letter or patent application. In addition, Medeva claims that Roxane s supplemental answers to Medeva s interrogatories were insufficient. For example, Medeva s Interrogatory No. 11 sought details regarding the release of mesalamine from Roxane s generic product within the human gastrointestinal tract. Roxane s response pointed to its Paragraph IV Notice letter and its employee Vinay Shukla as sources of responsive information. Medeva contends that Roxane s response is insufficient because, as previously stated, Roxane s witnesses testified that Roxane had no knowledge of the factual bases for the assertions contained in Roxane s Paragraph IV Notice letter and Mr. Shukla specifically testified that he was not the expert in the in vivo area as to where Roxane s generic product releases and explicitly refused to answer questions about Roxane s Paragraph IV Notice letter that addressed Roxane s generic product s release characteristics. Given the alleged magnitude of Roxane s misconduct, Medeva argues that the sanction of preclusion is justified even if such a sanction is tantamount to dismissal of Roxane s claim. As previously stated, Medeva argues that Roxane not only destroyed documents but sought to evade other efforts to obtain the information contained in the destroyed documents. Moreover, Medeva 2 Such documents include Roxane s proposed labeling submitted with its ANDA, its ANDA, its Paragraph IV Notice letter and its revised proposed labeling regarding where its product releases mesalamine 8

40 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 9 of 36 PageID: 4283 claims that Roxane, whose business model is to challenge the intellectual property rights of innovative drug companies, knew what it was doing when it failed to issue a timely litigation hold and destroyed relevant evidence because Roxane s. Medeva, therefore, asks the Court to sanction Roxane by precluding Roxane from disputing where its generic product releases and where Asacol releases in the human body. Medeva also argues that if the Court were to consider additional and/or lesser remedies, then the following may be be appropriate: (1) imposition of adverse inferences regarding where Roxane s generic product and Asacol release; (2) reopening of fact discovery and requiring Roxane to restore all back-up tapes from 2001 through the present; (3) finding that Roxane s conduct constitutes a waiver of work product protection from 2001 through 2007; and (4) imposition of all fees and costs associated with the instant motion to preclude as well as those incurred by Medeva s experts in testing and preparing reports on Lot No Roxane s Position Roxane opposes Medeva s motion to preclude as well as its request for additional and/or lesser remedies arguing that there has been no spoliation of evidence. In the first instance, Roxane claims that no documents were actually or intentionally destroyed. Roxane argues that from June 2002 forward it had a formal document retention policy in place that required laboratory notebooks, technical reports and developmental and analytical records to be retained for 20 years. Indeed, Roxane argues that from , its document retention policy required Roxane to preserve technical information for at least 30 years. Further, Roxane claims that its document retention policy did not permit documents to be purged without safeguards; instead, on the file clean-up days that occurred under Roxane s document retention policies, employees paid 9

41 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 10 of 36 PageID: 4284 close attention to the document retention policies to ensure that no technical information, including all such information related to mesalamine, would have been destroyed. In addition, Roxane contends that the evidence shows that no documents relating to mesalamine were destroyed. In this regard, Roxane notes that its Rule 30(b)(6) witness on document retention issues explicitly testified that any and all documents concerning mesalamine were not destroyed. Roxane also argues that the litigation hold put in place in 2007 merely modified the ongoing document retention policy to suspend any potential destruction of documents. (Roxane Opp. Br. at 4). Roxane claims that it would be unreasonable to have required Roxane to institute a litigation hold in 2001, four years before its first ANDA product was ever manufactured. Moreover, Roxane claims that such a litigation hold would only have continued the preservation of documents already being preserved under Roxane s document retention policies. With respect to the allegedly destroyed documents identified by Medeva, Roxane argues that the documents are not critical to this matter and also claims that Medeva has failed to show that Roxane actually and intentionally destroyed same. For example, with respect to the Anapharm study, Roxane argues that while witnesses testified that a protocol for the study must have existed, nothing in the record establishes that Roxane itself had the protocol. Further, Roxane notes that while a protocol describing how the Anapharm study would be performed was not produced, Roxane did produce other documents relating to the study, including Dr. Kramer s principal and supplemental reports, which detail how the Anapharm study was actually carried out. Moreover, Roxane claims that there is no evidence that Dr. Kramer prepared other documents related to the study or that the study results were ever memorialized in any other format. 10

42 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 11 of 36 PageID: 4285 In addition, Roxane claims that the Anapharm study is not relevant to this litigation, and certainly not of critical importance, because it does not relate to Roxane s generic product or a prototype thereof, but instead relates to Medeva s Asacol product and Roxane never asked Dr. Kramer to examine where Asacol releases mesalamine in the gastrointestinal tract. Further, Roxane argues that Medeva s conduct establishes that the Anapharm study is not important to this litigation. In this regard, Roxane notes that Medeva did not pursue discovery on this study directly from Anapharm. Roxane also notes that in its deposition of Dr. Kramer, Medeva asked almost nothing about the Anapharm study and did not even inquire into the existence of a protocol. Further, Roxane claims that despite the fact that Medeva had the results of the Anapharm study and could have had its experts analyze same, it did not. Roxane also contends that it was not obligated to make a duplicate production of documents that were produced in the traditional paper file format. Indeed, Roxane argues that it was not required to produce both paper and electronic versions of documents and s. Consequently, Roxane argues that the fact that certain documents and s were only produced in paper format and were not electronically produced does not support the conclusion that these documents and s were destroyed. In addition, with respect to Eric Spiller s files, Roxane claims that the evidence shows that Mr. Spiller maintained all of his mesalamine related documents in a specific file and that this file was produced to counsel. Indeed, Roxane argues that there is no testimony that either Mr. Spiller or any other employee ever deleted or destroyed a single document concerning mesalamine. More importantly, Roxane claims that there is no evidence that the allegedly missing documents are important to this case. For example, Roxane argues that there is no evidence that 11

43 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 12 of 36 PageID: 4286 the allegedly missing product development documents would have contained information about where either Roxane s generic product or Asacol releases. Further, Roxane notes that there could have been no such documents concerning its generic product until April 2005 because Roxane s generic product was not manufactured until that time. Similarly, Roxane claims that the allegedly missing documents concerning Roxane s prototype formulations are not relevant to this matter and would not have contained information regarding the release of either its generic product or Asacol. Roxane also argues that the record does not support Medeva s contention that Roxane ever possessed the allegedly missing final study reports related to Roxane s development and testing of its generic product that Medeva obtained from Quintiles. Indeed, Roxane notes that Elizabeth Ernst, Roxane s Director of Regulatory Medical, testified that she did not recall ever receiving the final study reports. In addition, with respect to the remainder of the Quintiles production, Roxane claims that the vast majority of same is made up of raw data generated from a clinical study that was conducted in a dozen countries and which was never forwarded to Roxane. (Roxane Opp. Br. at 6). Roxane claims that the fact that a stray or two was produced by Quintiles, but not Roxane, does not support Medeva s claim of spoliation. (Id.) In addition, Roxane argues that it did not violate any discovery orders entered by this Court, nor did it engage in discovery misconduct. For example, Roxane claims that it did not evade discovery on where its generic product releases mesalamine; instead, it simply did not know the answer to Medeva s questions. Roxane contends that prior to expert discovery it had no information regarding where its generic product releases mesalamine because it never conducted any experiment or study that would provide such information as it did not design its 12

44 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 13 of 36 PageID: 4287 product to target a specific site of action. Indeed, Roxane claims that the specific site of release was not important to its development of its generic product or to the FDA s evaluation of same. Instead, Roxane claims that the only matter of importance was whether Roxane s generic product showed equivalent efficacy to Asacol. Roxane claims that the fact that certain documents submitted to the FDA comment on where its generic product releases mesalamine are of no moment because the statements were not purposefully made. For example, Roxane contends that the fact that its original draft label submitted to the FDA contained the statement delays release of mesalamine until it reaches the terminal ileum and beyond is an artifact of the fact that Roxane initially copied Asacol s label verbatim. Roxane notes that it made amendments to the label for its generic product as the application process proceeded. Similarly, Roxane argues that its interrogatory responses regarding where its generic product releases mesalamine are not deficient because they incorporate Roxane s expert reports which describe in detail where its product releases mesalamine. Further, Roxane argues that it cannot be held responsible for any issues Medeva had in obtaining discovery from Quintiles because Quintiles is a third party and Roxane was not involved in its production of documents. Further, Roxane argues that Medeva has not established that it has been prejudiced by Roxane s failure to produce the allegedly missing documents. Indeed, Roxane claims that Medeva does not explain how the missing documents would impede its ability to put on its infringement and validity case. Roxane argues that that is because Medeva is fully informed on Roxane s efforts to develop its generic product and both Roxane s claims of non-infringement as well as its challenge to the validity of the 170 patent. Roxane notes that previously in this litigation, Medeva represented that it was ready to go to trial and that that ability has not been 13

45 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 14 of 36 PageID: 4288 impaired. In this regard, Roxane notes that no expert referenced the subject matter of the missing documents in their opening expert reports and argues that Medeva has made no showing that the missing documents, including the Anapharm protocol, would have been favorable to its case or that they would have been any more useful than the information already produced. As a result, Roxane argues that there is no basis for the Court to enter sanctions against it. First, Roxane argues that preclusion is an extreme sanction that is only granted in rare circumstances that do not exist here. Consequently, Roxane claims that it should not be precluded from disputing the location and duration of release of mesalamine from either its generic product or Asacol in the human body. Second, Roxane claims that the adverse inferences sought by Medeva, that Roxane destroyed documents that established that Roxane s generic product releases to the right side of the colon and that Roxane destroyed documents that established that Asacol releases to the right side of the colon, bear no relation to the content of the allegedly missing documents and are not warranted. For example, Roxane claims that the missing Anapharm protocol cannot speak to where Roxane s generic product releases because the study was conducted on Asacol. Further, Roxane argues that it would be unfair to use the missing Anapharm protocol to support an adverse inference regarding Asacol because there is no evidence that Roxane possessed the protocol or that it intentionally destroyed it. Roxane argues that this is particularly true given the fact that it produced other information from the Anapharm study, including the test results. Indeed, Roxane claims that there is no evidence in the record that the allegedly missing documents might or would have been unfavorable to Roxane. Similarly, Roxane claims that Medeva has not established that the missing documents have impeded its ability to put on either 14

46 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 15 of 36 PageID: 4289 its infringement or validity case. As a result, Roxane claims that there is no basis to impose either of the requested adverse inferences on it. Third, Roxane argues that there is no reason for the Court to reopen discovery or to permit Medeva to obtain back-up tapes from Roxane. In this regard, Roxane claims that there has been no spoliation of evidence and consequently Medeva has no need for more discovery. Roxane also notes that Medeva s request for additional discovery is insincere because Medeva has claimed that additional discovery will not cure the alleged spoliation. Further, Roxane argues that the parties entered a stipulation agreeing that back-up tapes not be produced and Medeva s request for the wholesale restoration of same is unfairly burdensome, unnecessary, duplicative, time-consuming, expensive and would needlessly delay this matter and open the door for additional discovery motions. Fourth, Roxane argues that there is no basis for finding a wholesale waiver of Roxane s assertion of the attorney-client privilege and work product doctrine. Roxane claims that Medeva has not challenged a single entry on Roxane s privilege log and therefore it would be inappropriate to find that Roxane s conduct amounts to the broad waiver requested by Medeva. In fact, Roxane argues that the only basis for a claim of waiver is Roxane s nondisclosure and failure to log a claim of privilege regarding the existence of Lot No As a result, Roxane argues that there is no reason to give Medeva unfettered access to all of Roxane s documents containing attorney work product and attorney-client communications. (Roxane s Sur-Reply at 9). Moreover, as explained below in more detail, while Roxane denies that it waived work product protection regarding Lot No , even if the Court were to find that such a waiver 15

47 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 16 of 36 PageID: 4290 occurred, no case law supports Medeva s request that such a limited waiver supports the wholesale production of all of Roxane s protected information. Finally, Roxane argues against the imposition of fees and costs. Roxane argues that the imposition of fees and costs is not warranted because Roxane has not engaged in any improper behavior. Roxane argues that it properly produced documents and that it properly invoked work product to protect the existence of the Concealed Lot, which was made at the direction of outside counsel. As a result, Roxane claims there is no need for the Court to punish Roxane or deter future misconduct by imposing fees and costs. B. The Concealment of Lot No Medeva s Position Medeva argues that Roxane impermissibly concealed the existence of a batch of its generic product, Lot No (the Concealed Lot ), that it manufactured in November Roxane used the Concealed Lot to conduct gamma scintigraphy testing on humans and intended 3 to use the results of that testing as evidence of non-infringement. Medeva argues that it sought discovery regarding all batches of mesalamine tablets manufactured by Roxane, including samples thereof. Medeva claims that Roxane initially refused to provide Medeva with any samples of its product and did so only after the Court entered an Order precluding Medeva from conducting human testing using the samples produced. Despite Medeva s request for all samples, Medeva argues that Roxane only produced tablets from three lots, Lot No , 3 After this motion was fully briefed, Roxane learned that there was something wrong with the Concealed Lot. The Concealed Lot was not manufactured properly and, as such, the tablets did not work as Roxane intended. Consequently, while initially intending to rely on its experts testing of the Concealed Lot, Roxane is no longer doing so. 16

48 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 17 of 36 PageID: 4291 which pertained to expired tablets, and Lot Nos and , which pertained to unexpired tablets; Roxane did not provide any tablets from the Concealed Lot. Further, Medeva claims that despite having requested information regarding all samples of Roxane s mesalamine product, Roxane neither referenced the Concealed Lot in its discovery responses nor explicitly claimed that information concerning same was being withheld based on the attorney-client privilege or work product doctrine. For example, Medeva notes that Interrogatory No. 5 directed Roxane to: Identify by batch number, lot number, date, location, and number any quantities of any Delayed Release Dosage Form containing Mesalamine, including without limitation the Generic Product made by or for You or on Your behalf, or any quantities of any tablets, capsules, or other dosage form containing a coating component or other agent intended for use with any Delayed Release Dosage Form containing Mesalamine. (Decl. of Paul A. Ainsworth in Support of Medeva s Motion ( Ainsworth Decl. ), Ex. 32 at 3). Medeva argues that neither Roxane s response nor supplemental response to this interrogatory, which follow, disclosed the existence of the Concealed Lot or claimed that any information regarding same was being withheld based on the attorney-client privilege or work product doctrine: RESPONSE TO INTERROGATORY NO. 5 Roxane objects to this interrogatory as vague. To the extent understood, Roxane also objects to this Interrogatory as overly broad, unduly burdensome and not calculated to lead to the discovery of admissible evidence in that it seeks information concerning products and formulations other than the product proposed in ANDA No

49 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 18 of 36 PageID: 4292 Subject to the foregoing objections and its General Objections, Roxane states that information responsive to this interrogatory may be ascertained from documents that will be produced pursuant to Federal Rule of Civil Procedure 33(d). SUPPLEMENTAL RESPONSE TO INTERROGATORY NO. 5 Roxane incorporates its original response and further responds as follows: Information responsive to this Interrogatory may be ascertained from documents already produced.... Roxane reserves the right to supplement its response further at a future time. (Id. at 4) Medeva argues that the Concealed Lot represents Roxane s ANDA product because Roxane s documents establish that these tablets were for product development use and manufactured to support in-vivo studies. (Medeva Reply Br. at 25 (emphasis in original)). As such, Medeva argues that Roxane should have disclosed the Concealed Lot. In addition, Medeva argues that to the extent Roxane claims that the Concealed Lot and information related thereto is protected by the work product doctrine, Roxane waived that protection when it failed to include such a claim on its privilege log and when it did not assert same in response to Medeva s numerous direct questions regarding the existence of Roxane s tablets. Medeva also argues that, even if properly logged, the Concealed Lot and certain information pertaining thereto would not be entitled to work product protection. For example, Medeva claims that the mere existence of the Concealed Lot is not protected by the work product doctrine and contends that the tablets themselves do not disclose any attorney s mental impressions. Medeva also argues that the fact that none of the documents relating to the Concealed Lot refer to litigation and the fact that Roxane did not disclose the existence of the Concealed Lot on its privilege log weigh against Roxane establishing that the 18

50 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 19 of 36 PageID: 4293 work product doctrine applies. In this regard, Medeva notes that in its documents concerning the Concealed Lot: (1) Roxane described the testing of the Concealed Lot as being for product development purposes ; (2) Roxane never informed the Institutional Review Board/Independent Ethics Committee ( IRB/IEC ), the entity that approved Roxane s testing of the Concealed Lot in humans, or the prospective study participants that the study would be used for litigation; and, (3) when informing the IRB/IEC as well as prospective study participants about who may receive the results of same, Roxane never identified this Court, Medeva, or Medeva s experts as possible 4 recipients, despite having identified various other people and entities, including the FDA. Further, Medeva argues that the case law relied upon by Roxane to support their nondisclosure of the Concealed Lot and information related thereto is either inapposite or supports a finding in Medeva s favor. First, Medeva claims that only two of the cases relied upon by Roxane (Novartis Pharm. Corp. v. Abbott Lab., 203 F.R.D. 159 (D. Del. 2001) and Vardon Gold Co. v. BBMG Golf Ltd., 156 F.R.D. 641 (N.D. Ill. 1994)) discuss product testing and in those cases the issue was the disclosure of the testing methods and test results, not the actual product. Moreover, Medeva argues that in six of the seven cases cited by Roxane, the non-disclosing party raised timely privilege and work product objections to the disclosure of the concealed information (see Medeva Reply Br. at 14, n.20), and in the remaining case, the court determined that a waiver had occurred due to the party s failure to timely raise a claim of privilege. (See Id. at 14-15). 4 Medeva does not argue that Roxane should have disclosed its testing protocol or test results related to the Concealed Lot prior to its experts reliance thereon. Medeva does, however, claim that Roxane was obligated to disclose, at a minimum, the existence of the tablets. 19

51 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 20 of 36 PageID: 4294 Medeva also claims that the parties stipulation regarding expert discovery does not justify Roxane s nondisclosure of the Concealed Lot. The parties joint stipulation regarding expert discovery provides that: 4. Pursuant to this agreement, the following categories of data, information, or documents need not be disclosed by any party, and are outside the scope of permissible discovery, except to the extent required by 5, and except to the extent required by FED.R.CIV.P. 26(a)(2): *** b. draft reports, draft studies, or draft workpapers; preliminary calculations, computations, or data runs; or other preliminary or draft materials prepared by, for, or at the direction of an expert witness[.] (Decl. of David H. Silverstein in Support of Roxane s Opp. to Medeva s Motion (the Silverstein Decl. ), Ex. EE 4). Medeva argues that there was nothing draft or preliminary about the final tablets manufactured by Roxane that make up the Concealed Lot and, as such, the parties stipulation did not protect their disclosure. In addition, Medeva claims that Roxane cannot shield otherwise discoverable information by forwarding it to an expert witness. Further, to the extent Roxane attempts to use the fact that Medeva drafted the parties joint stipulation to suggest that Medeva should have more precisely drafted same to make clear that information like the Concealed Lot was not protected by the stipulation, Medeva notes that Roxane also had an active hand in drafting the stipulation and argues that Roxane, being the party who knew it would be manufacturing the Concealed Lot, could have sought an express stipulation excluding such tablets from the scope of permissible discovery. Additionally, Medeva takes issue with the fact that Roxane conducted human testing using the Concealed Lot. Medeva argues that based on ethical concerns regarding the 20

52 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 21 of 36 PageID: 4295 uncontrolled administration of an unapproved drug product, Roxane convinced the Court to prohibit Medeva from conducting human testing using expired samples of Roxane s generic product produced in discovery. Medeva further claims that Roxane then insisted that Medeva also agree not to conduct human testing with unexpired samples of its generic drug product before producing same. Medeva argues that Roxane obtained the aforementioned prohibition knowing full well that it intended to conduct human testing using the Concealed Lot. Medeva contends that given the Court Order, Roxane s testing in humans was inappropriate because the same ethical considerations applied to Roxane as Medeva. Finally, Medeva argues that it has been prejudiced by Roxane s impermissible withholding of the Concealed Lot. While Medeva acknowledges that some of the harm caused by Roxane s nondisclosure of the Concealed Lot has been alleviated, given Roxane s admission that the Concealed Lot is not representative of its ANDA product as well as Roxane s decision not to rely on its testing of same in this matter, Medeva also argues that it nevertheless has been prejudiced by Roxane s failure to timely disclose the existence of the Concelaed Lot. In this regard, Medeva argues that Roxane s late disclosure of the Concealed Lot resulted in Medeva being forced to seek an extension of expert discovery and conduct expedited testing on the Concealed Lot over Christmas and New Years at considerable expense in order to establish that the Concealed Lot was not representative of Roxane s generic ANDA product. Further, Medeva argues that but for this costly expedited testing, this case would have been tried with Roxane relying on the Concealed Lot as being representative of its ANDA product. As a result, Medeva requests that the Court sanction Roxane. In addition to seeking reimbursement of its expert fees and costs incurred in testing the Concealed Lot, Medeva also seeks the same sanctions described 21

53 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 22 of 36 PageID: 4296 above with respect to Roxane s alleged destruction of documents. Medeva claims that such sanctions are appropriate especially in light of the fact that Roxane s nondisclosure of the Concealed Lot further establishes Roxane s distortion of discovery. 2. Roxane s Position Roxane argues that the Concealed Lot was protected from disclosure by the work product doctrine and was not discoverable until its experts relied on the results of the testing conducted using the Concealed Lot. In support of this argument, Roxane claims that the Concealed Lot was developed at the request of outside counsel for use by outside counsel s retained experts in conducting an experimental study for use at trial in this litigation. Specifically, Roxane claims that the Concealed Lot, which differs from Roxane s ANDA product because it contains a very small amount of samarium oxide, was developed to permit Roxane s experts to use gamma scintigraphy to examine where Roxane s generic product releases in the gastrointestinal tract. (Roxane Opp. Br. at 19). Roxane contends that the Concealed Lot and Roxane s testing thereof were protected by the work product doctrine because the Concealed Lot was prepared for use at trial by Roxane. Roxane argues that regardless of whether the actual tablets that make up the Concealed Lot disclose Roxane s attorneys mental impressions, they are still protected attorney work product because the work product doctrine is not limited to the protection of counsel s mental impressions. Moreover, Roxane claims that the tablets, themselves, do contain its attorneys mental impressions because the only purpose of adding samarium oxide to the tablets is to enable a scientist to conduct gamma scintigraphy testing. For this reason, Roxane argues that even disclosing the existence of the Concealed Lot would have disclosed Roxane s attorneys 22

54 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 23 of 36 PageID: 4297 selection of the specific test for which the Concealed Lot was created. Indeed, Roxane claims that the facts surrounding the creation, manufacture and existence of the Concealed Lot are facts created during the pendency of this litigation at the direction of counsel for use at trial and were thus protected from disclosure. In addition, Roxane claims that the fact that certain labeling uses the phrase product development neither changes the fact that the Concealed Lot was made at the direction of counsel, nor indicates that the Concealed Lot was not made in anticipation of litigation. Roxane argues that it is clear that the Concealed Lot was not made in the ordinary course of Roxane s business both because the FDA never requested that Roxane perform gamma scintigraphy testing and because Roxane has not indicated that it will submit the gamma scintigraphy test results to the FDA. Further, Roxane argues that even Medeva s experts recognized that the Concealed Lot was not created in Roxane s ordinary course of business as they described the Concealed Lot as being created in anticipation of litigation in their expert reports. As such, Roxane claims that it is clear that the Concealed Lot and related testing are protected by the work product doctrine. Moreover, Roxane contends that it did not waive work product protection by failing to include the Concealed Lot and related documents on its privilege log or by failing to assert a privilege regarding same during its employees depositions. In this regard, Roxane argues that it was not obligated to log the existence of the Concealed Lot or any documents concerning same on its privilege log because the Concealed Lot was created after the Complaint in this matter was filed and there is no rule requiring an ongoing privilege log. Roxane also claims that it was not bound to assert work product protection during its employees depositions because the questions asked by Medeva involved Roxane s prototype formulations and generic ANDA product, not 23

55 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 24 of 36 PageID: 4298 litigation tablets that contained samarium oxide. Similarly, Roxane claims that it did not need to assert that any documents were being withheld based on the work product doctrine in its response to Medeva s Interrogatory No. 5 because Roxane only agreed to produce proposed ANDA batches meaning that the Concealed Lot fell outside the scope of discovery that Roxane agreed to provide. (Roxane Opp. Br. at 23, n.16). Further, Roxane argues that it was not obligated to include the Concealed Lot or any related documents on its privilege log because the parties joint stipulation regarding expert discovery put same outside the scope of permissible discovery until Roxane s experts relied on Roxane s testing using the Concealed Lot. In this regard, Roxane claims that the tablets that made up the Concealed Lot as well as its testing represented preliminary [] materials and draft studies within the meaning of the parties joint stipulation and thus were outside of the scope of permissible discovery and not required to be logged. Roxane also argues that Medeva s conduct shows that no privilege log entry was required. Specifically, Roxane claims that Medeva s privilege log does not contain entries regarding expert witnesses. Further, Roxane claims that Medeva s previous complaints about Roxane s privilege log did not involve Roxane s failure to identify documents related to its expert witnesses. In addition, Roxane argues that it did not violate any Court Orders when it conducted human testing using the Concealed Lot. Roxane notes that the Court entered an Order precluding Medeva from conducting human testing using Roxane s expired generic product without prior Court approval, approval which Medeva never sought. Roxane argues that the Court did not limit Roxane s ability to test its own product and that Roxane was entitled to assume any risks 24

56 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 25 of 36 PageID: 4299 inherent in its own testing of that product. As such, Roxane argues that it did not engage in any misconduct when it tested a modified version of its generic product in humans under stringent study conditions and the supervision of a medical monitor, and with prior approval of a medial board. Furthermore, Roxane claims that even if it should have disclosed the Concealed Lot and related documents to Medeva, either in total or on its privilege log, Medeva has not been prejudiced. In this regard, Roxane claims that because the Court extended expert discovery, Medeva has not been denied the opportunity to conduct additional testing. Moreover, Roxane claims that this issue is now moot since Roxane is no longer relying on its testing of the Concealed Lot. Additionally, Roxane argues that Medeva could have conducted human testing using Roxane s expired and unexpired generic product if it had sought the Court s permission to do so, but Medeva never did. Roxane contends that Medeva s failure to request permission to conduct human testing had nothing to do with Roxane s obstructing Medeva s ability to conduct same. As a result, Roxane claims that the imposition of any sanctions is unwarranted. II. Analysis A. Standard of Review Spoliation is the destruction or significant alteration of evidence, or the failure to preserve property for another s use as evidence in pending or reasonably foreseeable litigation. Zubulake v. UBS Warburg LLC, 229 F.R.D. 422, 430 (S.D.N.Y. 2004) (internal quotation marks and citations omitted). It occurs when a party has intentionally or negligently breached its duty to preserve potentially discoverable evidence[.] Kounelis v. Sherrer, 529 F.Supp.2d 503, (D.N.J. 2008). The duty to preserve evidence arises when a party reasonably believes that 25

57 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 26 of 36 PageID: 4300 litigation is foreseeable and, as such, may arise many years before litigation commences[.] Micron Technology, Inc. v. Rambus Inc., 255 F.R.D. 135, 148 (D.Del. 2009). Evidence of spoliation may give rise to sanctions, which include: dismissal of a claim or granting judgment in favor of a prejudiced party; suppression of evidence; an adverse inference 5 (i.e., the spoliation inference); fines; and attorney s fees. Mosaid Tech., Inc. v. Samsung Elec. Co., Ltd., 348 F.Supp.2d 332, 335 (D.N.J. 2004). The imposition of sanctions is warranted when there is evidence that a party s spoliation of evidence threatens the integrity of th[e] Court. Id. Spoliation sanctions serve the following three functions: remedial, punitive and deterrent. They level the playing field so that the prejudiced party is restored to the position it would have been in absent the spoliation. They punish the spoliator for its misconduct and they warn other potential litigants that spoliation of evidence will not be tolerated and will be dealt with by the Court if necessary. See Id. The Court has authority to impose spoliation sanctions pursuant to both the Federal Rules of Civil Procedure and its inherent authority, and the choice of which sanction should be imposed rests in the sound discretion of the Court. Id. Of the sanctions that can be imposed, dismissal and suppression of evidence are considered the two most drastic and should only be levied in the most extraordinary of circumstances. Id. In determining whether such drastic sanctions are warranted, the Court considers: (1) the degree of fault of the party who altered or destroyed the evidence; (2) the degree of prejudice suffered by the opposing party; and (3) whether there is a lesser sanction that will avoid substantial unfairness to the opposing party and, where the 5 The imposition of sanctions in patent cases is controlled by regional circuit law. Monsanto Co. v. Ralph, 382 F.3d 1374, 1380 (Fed. Cir. 2004). Therefore the Court relies on Third Circuit law in analyzing what, if any, sanctions should be imposed on Roxane. 26

58 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 27 of 36 PageID: 4301 offending party is seriously at fault, will serve to deter such conduct by others in the future. Schmid v. Milwaukee Elec. Tool Corp., 13 F.3d 76, 79 (3d Cir. 1994). In order to establish prejudice, the party seeking spoliation sanctions must come forward with plausible, concrete suggestions as to what [the missing] evidence might have been (Id at 80), and must show that its ability to prepare effectively a full and complete trial strategy has been impeded. Ware v. Rodale Press, Inc., 322 F.3d 218, 222 (3d Cir. 2003). Generally, however, absent a showing of bad faith and substantial prejudice, dispositive sanctions should not be imposed. See Schmid, 13 F.3d at 80. Of the lesser sanctions often considered by courts faced with spoliation is the spoliation inference. The spoliation inference is an adverse inference that permits a jury to infer that destroyed evidence might or would have been unfavorable to the position of the offending party. Mosaid, 348 F.Supp.2d at 336 (quoting Scott, 196 F.R.D. at 248). In order for the spoliation inference to apply, the party seeking the adverse inference must establish that the following four factors have been satisfied: First, it is essential that the evidence in question be within the party s control. Second, it must appear that there has been actual suppression or withholding of the evidence. Third, the evidence destroyed or withheld was relevant to claims or defenses. And fourth, it was reasonably foreseeable that the evidence would later be discoverable. Mosaid, 348 F.Supp.2d at 336 (internal quotation marks and citations omitted); see also Aurelio v. Bd. of Educ. of the Borough of Carteret, Civil Action No (JLL), 2009 WL , *9 (D.N.J. June 23, 2009) (indicating that burden fell on Plaintiff, party seeking adverse inference, to show that all four factors were met)). 27

59 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 28 of 36 PageID: 4302 With respect to the second factor (i.e. actual suppression ), negligent destruction of relevant evidence can be sufficient to give rise to the spoliation inference. Id. at 338. A party need not show that its adversary intentionally or knowingly destroyed or withheld evidence. Id. at 337. This is true because [i]f a party has notice that evidence is relevant to an action, and either proceeds to destroy that evidence or allows it to be destroyed by failing to take reasonable precautions, common sense dictates that the party is more likely to have been threatened by the evidence and, regardless of the offending party s culpability[,]... it cannot be denied that the 6 opposing party has been prejudiced. Id. at 338. In addition, with respect to the third factor, that the destroyed or withheld evidence is relevant, while not explicitly addressed by the Third Circuit or the District of New Jersey, typically, where the culpability of the offending party is negligence, gross negligence or even recklessness (as opposed to willful or knowing) the party seeking the spoliation inference must establish not only that the destroyed or withheld evidence is probative under Fed.R.Evid. 401, but also must adduce sufficient evidence from which a reasonable trier of fact could infer that the destroyed or unavailable evidence would have been of the nature alleged by the party affected by its destruction. Zubulake, 229 F.R.D. at 431, n. 67 (internal quotation marks and citation omitted). In other words, the party seeking the spoliation inference must make a sufficient showing from which a fact finder could reasonably determine that the destroyed or withheld evidence would have been favorable to the movant. 6 As described above, where one of the more drastic sanctions such as dismissal or suppression of evidence is sought, a greater showing of culpability may be required. Schmid, 13 F.3d at 80; see Mosaid, 348 F.Supp.2d at 338, n

60 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 29 of 36 PageID: 4303 B. The Destruction of Documents The Court finds that spoliation sanctions are not warranted for Roxane s alleged destruction of documents. While the court cannot fathom why Roxane failed to institute a litigation hold prior to July 2007 when it clearly anticipated litigation and had a duty to preserve information at least as early as February 2002, when it knew it would be filing a Paragraph IV Certification, and perhaps as far back as July 31, 2001 if its privilege log claiming work product protection over a document with that date is to be credited, the Court finds that Roxane adequately maintained its mesalamine files pursuant to its normal document retention policies. These policies required Roxane to retain technical information such as laboratory notebooks, technical reports and developmental and analytical records for a minimum of twenty years. Further, while those policies did not specifically reference Roxane s ANDA product, the record establishes that documents concerning same were in fact retained. This is true despite the fact that Roxane held file clean-up days prior to instituting a litigation hold and after it had a duty to preserve evidence. Indeed, the quantity and more importantly quality of documents produced by Roxane in this litigation establish that Roxane diligently preserved mesalamine related information. As such, this matter is easily distinguishable from Sanofi-Aventis Deutschland GmbH v. Glenmark Pharm. Inc., USA, Civil Action No. 07-CV-5855 (DMC-JAD), *9 (D.N.J. July 1, 2010), where the Court imposed an adverse inference against the defendants after determining that they practiced systematic document destruction when no litigation hold was in place. While Medeva points to several categories of documents that are allegedly missing, Medeva simply has not established that Roxane intentionally or negligently destroyed the vast 29

61 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 30 of 36 PageID: 4304 majority of these documents. Indeed, of all of the documents identified by Medeva as having been destroyed by Roxane, the Court finds that the record at best supports the conclusion that Roxane at one point possessed but failed to produce the Anapharm contract and protocol and two s produced by Quintiles. Moreover, the Court finds that Roxane s failure to produce these documents does not warrant the imposition of spoliation sanctions. In this regard, the Court finds that Medeva has not been prejudiced by Roxane s failure to produce the Anapharm contract and protocol and the two s produced by Quintiles. With respect to the Anapharm documents, the Court notes that the Anapharm study did not deal with Roxane s ANDA product or any prototypes thereof, but instead examined Medeva s product. As such, the Anapharm study was not submitted to the FDA and was not important to Roxane s ANDA. Further, while the Anapharm study addressed Medeva s product, Medeva s experts do not appear to have relied upon or even considered the study in rendering their opinions. More importantly, while Roxane failed to produce its contract with Anapharm or the Anapharm protocol, which would have outlined what the study was going to entail, Roxane did produce the principal and supplemental reports related to the Anapharm study which set forth how the study actually proceeded and the results of the study, including a full data analysis with tables, graphs and explanatory text. Given Roxane s production of these reports coupled with the fact that neither Medeva nor Roxane s experts rely on the Anapharm study, the Court simply cannot see how Medeva s ability to prepare effectively a full and complete trial strategy has 30

62 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 31 of 36 PageID: 4305 been impeded by Roxane s failure to produce its contract with Anapharm or the Anapharm protocol. Ware, 322 F.3d 218, 222 (3d Cir. 2003). 7 The Court likewise does not believe that Medeva s ability to effectively prepare a full and complete trial strategy has been impeded by Roxane s failure to produce the two s that it once possessed and were produced by Quintiles. In this regard, the Court notes that while Roxane did not produce these s, Quintiles did. As such, Medeva has obtained the information missing from Roxane s production. Moreover, while the fact that these s were missing from Roxane s production may, under certain circumstances, raise a serious concern over what other information is missing from Roxane s production, such a concern is simply not present here where the breadth and quality of Roxane s document production establishes that Roxane met its duty to preserve information. Further, the substance of the two missing s establishes that they are not of critical importance to this matter: one discussed whether the Russian equivalent to the FDA wanted to receive information in writing or via telephone and the second discussed where Asacol could be purchased in Europe. Under these circumstances, the Court finds that spoliation sanctions are not warranted. Indeed, the record does not support the conclusion that Roxane s destruction/non-production of documents has threatened the integrity of this Court. See Mosaid, 348 F.Supp.2d at 335. The Court finds that Medeva s ability to fairly participate in this matter and effectively prepare a full and complete trial strategy has not been impeded by Roxane s alleged destruction of documents. See Ware, 322 F.3d at 222. As such, the Court finds no reason to punish Roxane for its conduct 7 While not necessary for the Court to reach this conclusion, the fact that Medeva never sought to obtain the missing Anapharm documents directly from Anapharm, reinforces the Court s determination that Medeva has not been prejudiced by their nonproduction. 31

63 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 32 of 36 PageID: 4306 or to deter other potential litigants from engaging in same. See Mosaid, 348 F.Supp.2d at 335 (describing three functions served by imposing spoliation sanctions). C. The Concealment of Lot No Unlike with Roxane s alleged destruction of documents, the Court finds that sanctions are warranted for Roxane s concealment of Lot No In coming to this conclusion, the Court does not reach the issue of whether the Concealed Lot was entitled to attorney-client privilege and/or work product protection, but, instead, assumes for purposes of this motion that it was. As such, the Court focuses its inquiry on whether the existence of the Concealed Lot should have been disclosed by Roxane on its privilege log and whether Roxane s failure to log same warrants the imposition of sanctions. The Court finds that both questions should be answered in the affirmative. With respect to Roxane s decision not to include the existence of the Concelaed Lot on its privilege log, the Court finds Roxane s reasons for failing to do so completely unavailing. In this regard, the Court finds that Roxane has not provided any legal support that stands for the proposition that it was not required to include the existence of the Concealed Lot on its privilege log. To the extent, Roxane relies on Grider v. Keystone Health Plan Cent., Inc., 580 F.3d 119 (3d Cir. 2009) to argue that it was not required to do so because there is no requirement to maintain an ongoing privilege log, the Court finds Roxane s reliance on the Grider decision to be misplaced. First, Grider does not hold that litigants are not required to maintain ongoing privilege logs. Instead, in a footnote, the Court in Grider merely acknowledges that a privilege log may not be required for communications with counsel that take place after the filing of a law suit. Id. at 140, n.22. Second, what is clear from the context of the footnote, the Grider case 32

64 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 33 of 36 PageID: 4307 did not contemplate the requirement or lack thereof to log the existence of a product, as opposed to a communication, created at the request of counsel. Id. Further, none of the other cases relied upon by Roxane supports its contention that it had no duty to include the existence of the Concealed Lot on its privilege log. In this regard, the Court notes that Roxane did not cite to a single case in which the Court affirmed a company s decision to refrain from including a product manufactured at its facilities on the advice of counsel on its privilege log. The closest cases cited by Roxane are Novartis, 203 F.R.D. 159 and Vardon,156 F.R.D. 641, which address claims of privilege raised over the methods used and results obtained from product testing conducted for use in pending litigation; they did not address the nondisclosure of the product being tested. More importantly, in both cases claims of privilege were timely made by the party from whom discovery concerning the testing was sought. Indeed, in the only case cited by Roxane in which a party withheld information it deemed to be protected by the work product privilege without logging the privilege, the court determined that a waiver had occurred given the responding party s failure to include an index of the withheld documents on its privilege log. Feacher v. Intercontinental Hotels Group, Civ. Action No. 3:06- cv-877 (TJB/DEP), 2007 WL , *5-6 (N.D.N.Y. Oct. 22, 2007). Similarly, neither Roxane s Response to Medeva s Interrogatory No. 5 nor the parties joint stipulation regarding expert discovery exempted it from having to include a notation regarding the existence of the Concealed Lot on its privilege log. With respect to Medeva s Interrogatory No. 5, Roxane s statement that it objects to this interrogatory as overly broad, unduly burdensome and not calculated to lead to the discovery of admissible evidence in that it seeks information concerning products and formulations other than the product proposed in 33

65 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 34 of 36 PageID: 4308 ANDA No [,] is insufficient. Information regarding the existence of the Concealed Lot fell within the scope of Medeva s Interrogatory No. 5 and clearly is relevant to this litigation. While the composition of the Concealed Lot differs slightly from Roxane s ANDA product in that it contains a very small amount of samarium oxide, a material that can be made radioactive, which would permit Roxane to do testing to determine where its product releases in the gastrointestinal tract, Roxane intended the Concealed Lot to be functionally representative of its ANDA product. (Roxane Opp. Br. at 19). As such, if Roxane intended to withhold information regarding the existence of the Concealed Lot based on the work product doctrine, it had an obligation to specifically notify Medeva that it was doing so by including an entry in its privilege log regarding same; the aforementioned objection was simply inadequate. Likewise, the parties joint stipulation regarding expert discovery, which put outside the scope of permissible discovery... draft reports, draft studies, or draft workpapers; preliminary calculations, computations, or data runs; or other preliminary or draft materials prepared by, for, or at the direction of an expert witness does not excuse Roxane s failure to make note of the existence of the Concealed Lot on its privilege log. (Silverstein Decl., Ex. EE 4). First, the Court agrees with Medeva that there is nothing preliminary or draft about the final tablets manufactured by Roxane that make up the Concealed Lot. Second, according to the Court s reading of the parties joint stipulation regarding exert discovery, it does not appear that a lot of tablets falls within the types of materials intended to be covered by the stipulation. If Roxane had such an intent, then it should have made sure that the joint stipulation was written more clearly to disclose same. Thus, the Court finds no justification for Roxane s failure to include a notation on its 34

66 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 35 of 36 PageID: 4309 privilege log regarding the Concealed Lot. To be clear, the Court does not find that Roxane had to disclose the purpose behind the Concealed Lot. Indeed, Roxane had no obligation to disclose that the Concealed Lot contained samarium oxide or that Roxane intended to use the Concealed Lot to conduct gamma scintigraphy testing in order to determine where Roxane s ANDA product releases in the gastrointestinal tract. Instead, Roxane simply had to disclose that it was withholding samples of and identifying information regarding a batch of tablets created at the direction of counsel. Such a notation would have put Medeva on notice that information was being withheld and would have allowed Medeva to have filed a timely petition with the Court to determine whether Roxane s privilege claim was appropriate. Roxane s failure to appropriately log its claim of work product protection regarding the Concealed Lot effectively precluded Medeva from challenging Roxane s claim of privilege and forced Medeva, perhaps unnecessarily, to engage in costly, expedited expert testing on the Concealed Lot when Roxane finally disclosed the existence of same at the end of expert discovery. Further, Roxane s failure precipitated the filing of the instant motion. As such, it not only harmed Medeva, but also thwarted the Court s efforts to resolve this case in an efficient and economic fashion by preventing the Court from addressing Roxane s claim of privilege in the first instance and necessitating the Court s decision on this otherwise unnecessary motion. As such, the Court finds that the imposition of sanctions is warranted for Roxane s unjustified failure to include a notation on its privilege log regarding the existence of the Concealed Lot. In this regard, the Court finds that the imposition of Medeva s reasonable expert fees and costs associated with its expedited testing of the Concealed Lot along with its reasonable attorneys fees and costs incurred in briefing and arguing the portion of the instant motion related 35

67 Case 3:07-cv FLW -TJB Document 223 Filed 01/28/11 Page 36 of 36 PageID: 4310 to Roxane s nondisclosure of the Concealed Lot are an appropriate sanction. The Court believes that the imposition of the aforementioned costs and fees will fairly compensate Medeva for the harm it suffered from Roxane s nondisclosure of the existence of the Concealed Lot and will also effectively deter Roxane and future litigants from further engaging in this type of conduct. The Court notes that under other circumstances in lieu of imposing all of the fees and costs noted above, or perhaps even in addition to same, it may have deemed Roxane to have waived privilege with respect to the Concealed Lot. The Court, however, finds that such a sanction would be ineffectual here given the fact that Roxane no longer intends to rely on its testing of the Concealed Lot in this litigation. III. Conclusion For the reasons stated above, Medeva s motion for the imposition of spoliation sanctions is GRANTED in part and DENIED in part. An appropriate Order follows. Dated January 28, 2011 s/tonianne J. Bongiovanni HONORABLE TONIANNE J. BONGIOVANNI UNITED STATES MAGISTRATE JUDGE 36

68 LOCAL CIVIL AND CRIMINAL RULES OF THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF NEW JERSEY With Revisions as of June 22, 2012

69 TABLE OF CONTENTS INTRODUCTION Revision FOREWORD Revision LOCAL CIVIL RULES PAGE 1.1 Rules of Procedure; Scope of these Rules Definitions Service of Process Service and Filing of Pleadings and Other Papers Electronic Service and Filing Documents Protective Orders and Public Access Under CM/ECF Extensions of Time and Continuances Application and Motion Practice Affidavits and Briefs Pleading Damages Special Matters - Review of Social Security Matters Special Matters - Admiralty and Maritime Rules for the United States District Court for the District of New Jersey Local Patent Rules Form of Pleadings Signing of Pleadings Verification of Petitions Applications for Fed. R. Civ. P. 11 Sanctions Pretrial Conferences; Scheduling; Case Management Notice of Claim of Unconstitutionality Statutory Court Discovery Depositions for Use in a Foreign Country Letters Rogatory Interrogatories Requests for Production of Documents Requests for Admission Discovery Motions Applications for Fed. R. Civ. P. 37 Sanctions Jury Demand Allocation and Assignment of Cases Dismissal of Inactive Cases Consolidation of Cases Seal Petit Jurors Assessment of Jury Costs Civil Jury Taking of Civil Verdict ii-

70 52.1 Oral Opinions Costs Compensation for Services Rendered and Reimbursement of Expenses Prepayment of Clerk's and Marshal's Fees Summary Judgment Motions Entry of Judgments and Orders Applications for Emergency Relief Security and Sureties Receiverships Deposit in Court Marshal's Vouchers United States Magistrate Judges Civil Trials by Consent before United States Magistrate Judges Court Session Motion Days Custody of Original Papers, Records and Exhibits Briefs Part of Public Record Entry of Satisfaction of Judgments and Decrees Filing of Mandate Clerk to Maintain List of Official Newspapers Transcripts Naturalization Habeas Corpus and Motions under 28 U.S.C In Non Death Penalty Cases Habeas Corpus and Motions under 28 U.S.C In Death Penalty Cases Adoption and Amendment of Local Rules Relaxation or Modification of Local Rules Procedure in The Absence of Rule or Statutory Provision Title Admission of Attorneys Withdrawal of Appearance Judicial Ethics and Professional Responsibility Discipline of Attorneys Extrajudicial Statements Arbitration Mediation Media Coverage Possession and use of Electronic Equipment LOCAL CRIMINAL RULES 1.1 Scope and Applicability United States Magistrate Judges Grand Jurors Motions under Fed. R. Crim. P Assignment of Criminal Cases Selection and Impanelment of Trial Jurors Probation Motions under Fed. R. Crim. P iii-

71 44.1 Formal Written Appearance - Criminal Matters Release from Custody Conduct in the Courtroom Record of Proceedings Custody and Disposition of Exhibits Proceedings in Misdemeanor and Petty Offense Cases Title Extrajudicial Statements in Criminal Proceedings CONVERSION TABLES General Rule to Local Civil/Criminal Rule Local Civil Rule to General Rule Local Criminal Rule to General Rule COURT'S APPENDICES A1. Client's & Supervising Attorney's Authorizations for Appearance by Law Student A2. Form for Designating Compliance with the Student Practice Rule B. Criminal Case Appearance Form C. Affidavit by Owner of Cash Security D. Order Granting Motion to Deposit Sum of Money with the Court E. List of Petty Offenses and Minimum Fines Applicable Thereto Pursuant to L.Cr.R. 58.1(c) F. Transcript Rates G. Official Newspapers H. Appointment of Attorneys in Pro Se Civil Actions I. Plan of Implementation of the United States District Court for the District of New Jersey Pursuant to the Criminal Justice Act of 1964, as Amended J. Plan for Prompt Disposition of Criminal Cases K. Schedule of Fees L. Application for Extension of Time to Reply M. Guidelines for Arbitration N. RESERVED O. Optional RICO Case Order P. In Forma Pauperis Affidavit and Order Q. Guidelines for Mediation R. Guidelines for Litigation Conduct S. Discovery Confidentiality Order T. Procedures for Patent Pilot Project Cases -iv-

72 is unconstitutional. If memoranda have been served discussing the constitutional question, two copies of each memorandum shall be forwarded with the notification. (b) If, at any time prior to the trial of an action in which neither the State of New Jersey nor any officer, agency or employee thereof is a party, a party to the action questions the constitutionality of any State statute, such party (to enable the Court to comply with 28 U.S.C. 2403(b)) shall forthwith, upon the filing of any pleading which raises the question, notify the Judge to whom the action is assigned, in writing, of the existence of said question identifying: (1) the title and docket number of the action; (2) the statute challenged; and (3) why it is claimed that the statute is unconstitutional. If memoranda have been served discussing the constitutional question, two copies of each memorandum shall be forwarded with the notification. Source: G.R. 32. Civ. RULE 24.2 STATUTORY COURT Where, pursuant to law, an action must be heard by a District Court composed of three Judges, two from this Court and one from the Third Circuit, the procedure to be followed by counsel in filing pleadings and submitting briefs will be as follows: (a) All pleadings are to be filed with the Clerk in quadruplicate, the original becoming part of the Clerk's file, the three copies to be distributed by the Clerk to the members of the Statutory Court. (b) Six copies of briefs are to be submitted. Unless otherwise directed by the Court, they are to be delivered to the Clerk for distribution to the members of the Statutory Court. Source: G.R. 33. Civ. RULE 26.1 DISCOVERY (a) Discovery - Generally All parties shall conduct discovery expeditiously and diligently. (b) Meeting of Parties, Discovery Plans, and Initial Disclosures (1) The requirements currently codified in Fed. R. Civ. P. 26(a) and (f) pertaining to required disclosures, meetings of parties, and submission of discovery plans, shall apply to all civil cases filed after December 1, 1993 and to all civil cases pending on December 1, 1993 that have not had their initial scheduling conference prior to January 20, 1994; except that these requirements shall not apply to those civil cases described in L.Civ.R. 72.1(a)(3)(C) in which scheduling conferences are not normally held, unless the judicial officer otherwise directs. The judicial officer may modify or suspend these requirements in a case for good cause. (2) The initial meeting of parties as required in Fed. R. Civ. P. 26(f) shall be convened at least 21 days before the initial scheduling conference, and the proposed discovery plan under Fed. R. Civ. P. 26(f)(3) shall be generated at that meeting and delivered to the Magistrate Judge within -49-

73 14 days after the meeting of parties. The parties shall submit their Fed. R. Civ. P. 26(f) discovery plan containing the parties' views and proposals regarding the following: (a) Any changes in timing, form, or requirements of mandatory disclosures under Fed. R. Civ. P. 26(a); (b) The date on which mandatory disclosures were or will be made; (c) The anticipated substantive scope of discovery, including both discovery relevant to the claims and defenses and discovery relevant to the subject matter of the dispute; (d) Whether any party will likely request or produce computer-based or other digital information, and if so, the parties' discussions of the issues listed under the Duty to Meet and Confer in L. Civ. R. 26.1(d)(3) below; (e) The date by which discovery should be completed; (f) Any needed changes in limitations imposed by the Federal Rules of Civil Procedure, local rule, or standing order; (g) Any orders, such as data preservation orders, protective orders, etc., which should be entered; (h) Proposed deadline for joining other parties and amending the pleadings; (i) Proposed dates for filing motions and for trial; (j) Whether the case is one which might be resolved in whole or in part by voluntary arbitration (pursuant to L. Civ. R or otherwise), mediation (pursuant to L. Civ. R or otherwise), appointment of a special master or other special procedure. The parties shall make their initial disclosures under Fed. R. Civ. P. 26(a)(1) within 14 days after the initial meeting of the parties, unless otherwise stipulated or directed by the Court. Such discovery plans and disclosures shall not be filed with the Clerk. (c) Discovery Materials (1) Initial and expert disclosure materials under Fed.R.Civ.P.26(a)(1) and 26(a)(2), transcripts of depositions, interrogatories and answers thereto, requests for production of documents or to permit entry onto land and responses thereto, and requests for admissions and answers thereto shall not be filed until used in a proceeding or upon order of the Court. However, all such papers must be served on other counsel or parties entitled thereto under Fed.R.Civ.P.5 and 26(a)(4). (2) Pretrial disclosure materials under Fed.R.Civ.P.26(a)(3) shall be incorporated by reference into the order entered after any final pretrial conference under Fed.R.Civ.P.16(d). (3) In those instances when such discovery materials are properly filed, the Clerk shall place them in the open case file unless otherwise ordered. -50-

74 (4) The party obtaining any material through discovery is responsible for its preservation and delivery to the Court if needed or ordered. It shall be the duty of the party taking a deposition to make certain that the officer before whom it was taken has delivered it to that party for preservation and to the Court as required by Fed. R. Civ. P. 30(f)(1) if needed or so ordered. (d) Discovery of Digital Information Including Computer-Based Information (1) Duty to Investigate and Disclose. Prior to a Fed. R. Civ. P. 26(f) conference, counsel shall review with the client the client s information management systems including computer-based and other digital systems, in order to understand how information is stored and how it can be retrieved. To determine what must be disclosed pursuant to Fed. R. Civ. P. 26(a) (1), counsel shall further review with the client the client s information files, including currently maintained computer files as well as historical, archival, back-up, and legacy computer files, whether in current or historic media or formats, such as digital evidence which may be used to support claims or defenses. Counsel shall also identify a person or persons with knowledge about the client s information management systems, including computer-based and other digital systems, with the ability to facilitate, through counsel, reasonably anticipated discovery. (2) Duty to Notify. A party seeking discovery of computer-based or other digital information shall notify the opposing party as soon as possible, but no later than the Fed. R. Civ. P. 26(f) conference, and identify as clearly as possible the categories of information which may be sought. A party may supplement its request for computer-based and other digital information as soon as possible upon receipt of new information relating to digital evidence. (3) Duty to Meet and Confer. During the Fed. R. Civ. P. 26(f) conference, the parties shall confer and attempt to agree on computer-based and other digital discovery matters, including the following: (a) Preservation and production of digital information; procedures to deal with inadvertent production of privileged information; whether restoration of deleted digital information may be necessary; whether back up or historic legacy data is within the scope of discovery; and the media, format, and procedures for producing digital information; (b) Who will bear the costs of preservation, production, and restoration (if necessary) of any digital discovery. Amended: March 14, 2001, October 6, 2003 Source: L.Civ.R. 26.1(a) - G.R. 15.E.1; L.Civ.R. 26.1(b) - G.R. 15.B.1-2; L.Civ.R. 26.1(c) - G.R. 15.G. Civ. RULE 27.1 DEPOSITIONS FOR USE IN A FOREIGN COUNTRY (a) A person desiring to take the deposition of a witness who resides or may be found within the District for use in a judicial proceeding pending in a foreign country may apply ex parte to the Court for an appropriate order. If the deposition is to be taken upon written interrogatories, a copy of the interrogatories shall be annexed to the application. If the court of the foreign country has appointed a person to take the deposition, the order shall designate that person commissioner unless there be good cause for withholding such designation. If no such appointment has been -51-

75 T H E S E D O N A C O N F E R E N C E W O R K I N G G R O U P S E R I E S SM THE SEDONA CONFERENCE JUMPSTART OUTLINE : Questions to Ask Your Client & Your Adversary to Prepare for Preservation, Rule 26 Obligations, Court Conferences & Requests for Production A Project of The Sedona Conference Working Group on Electronic Document Retention & Production (WG1) MARCH 2011 VERSION Copyright 2011, The Sedona Conference

76 THE SEDONA CONFERENCE JUMPSTART OUTLINE A Project of The Sedona Conference Working Group on Electronic Document Retention & Production (WG1) Author: The Sedona Conference Editor-in-Chief: Ariana J. Tadler March 2011 Version This outline was initially prepared by Ariana Tadler for The Sedona Conference Institute s SM program entitled Getting Ahead of the ediscovery Curve: Strategies to Reduced Costs & Meet Judicial Expectations held March 13-14, 2008, at the Westin Horton Plaza San Diego, San Diego, CA, as an example of a tool to assist counsel in dealing with electronic discovery issues. It was updated recently. The opinions expressed in this publication, unless otherwise attributed, represent consensus views of the members of The Sedona Conference Working Group on Electronic Document Retention & Production. They do not necessarily represent the views of any of the individual participants or their employers, clients, or any other organizations to which any of the participants belong, nor do they necessarily represent official positions of The Sedona Conference. Thanks go to all who participated in the dialogue that led to this Commentary. In addition, we thank all of our Working Group Series SM Sustaining and Annual Sponsors, whose support is essential to our ability to develop Working Group Series SM publications. For a listing of our sponsors just click on the Sponsors Navigation bar on the homepage of our website. COMMENTS & SUGGESTIONS should be directed to: Ken Withers at The Sedona Conference 5150 North 16 th Street, Suite A-215 Phoenix, AZ Copyright 2011 The Sedona Conference All Rights Reserved. REPRINT REQUESTS: Requests for reprints or reprint information should be directed to Richard Braman, Board Chair of The Sedona Conference, at

77 The Sedona Conference Jumpstart Outline March 2011 Introduction The Sedona Conference Jumpstart Outline This outline sets forth, by way of example only, a series of topics and questions to ask your client and your adversary as you prepare for meeting obligations related to preservation, requests for production, court conferences, and Fed. R. Civ. P. 26. The answers to these questions will guide you in (i) instructing your client about its preservation and production obligations and (ii) understanding your adversary s systems and preservation efforts to date, and (iii) structuring and tailoring your discovery requests addressed to your adversary. This is a simplified outline to assist, in particular, those people who have had only limited experience in dealing with electronic discovery. As those with extensive experience in this arena know, the process of questioning and even the questions themselves are iterative in scope. With each answer you elicit, inevitably additional questions must be asked. Hopefully, having an outline like this within easy reach will serve as a jumpstart to encourage transparency and dialogue in the discovery process, as contemplated by the Rules and The Sedona Conference Cooperation Proclamation. 1. Document Retention Policy 1.1 Do you have a document retention (or records management) policy? Is it a written policy? If yes, when was the policy implemented? If yes, is the policy enforced? By whom? How? If yes, did the policy change during [insert relevant time period]? If yes, are you willing to produce the policy/policies? 2. Key Custodians of Potentially Relevant Information 2.1. Given the facts of the case, who are the key custodians of potentially relevant information? Who is responsible for maintaining/administering the company s electronic systems? 2.2. To what extent has information in the possession, custody, or control of the key custodians been preserved? (Discuss what those efforts have been to date and what, if any, additional efforts are underway.) If conferring with your client, address efforts to date and further efforts that need to be made If conferring with your adversary, discuss efforts to date and, if insufficient, request that further efforts be made, if appropriate. p 1

78 The Sedona Conference Jumpstart Outline March Disclosure of identities of key custodians In representing your client, consider disclosing to your adversary the identities of the key custodians for whom information has been/will be preserved If you are a requesting party, consider identifying those people who you believe are key custodians to memorialize your request for preservation of their information Are there any third parties that may hold potentially relevant information? To what extent has information in the possession, custody, or control of third parties been preserved? (Discuss what those efforts have been to date and what, if any, additional efforts are underway.) If conferring with your client, address efforts to date and further efforts that need to be made with respect to third parties If conferring with your adversary, discuss efforts to date and, if insufficient request that further efforts, be made, if appropriate In representing your client, consider disclosing to your adversary the identities of the third parties for whom information has been/will be preserved If you are a requesting party, consider identifying those people who you believe are third parties that may have relevant data to memorialize your request for preservation of their information. NOTE: This is an iterative process. You should plan to confer with your adversary on a recurring basis so that you can continue to update your adversary on any additional key custodians. 3. Network Servers The questions below concern current and former database and file servers on any potentially relevant network that now store or previously stored discoverable electronic data (hereinafter referred to as network servers ). These questions should be asked of both your client and your adversary Do you use, for any purpose, a network-based system? If yes, please describe Do you have a system that serves to back up the information managed and/or stored on the network(s)? If yes, do you have at least one computer (i.e., non-incremental) backup of each of your network servers for each month for the period [insert relevant time period]? If not, for which months do you/do you not have at least one complete backup? p 2

79 The Sedona Conference Jumpstart Outline March For those months, if any, for which you do not have a complete backup, do you have incremental backups or other backups from which a full backup can be created of all data as of a given date in each such month? If so, please describe the nature of such incremental or other backups and identify the months for which you have them Can specific files contained on network backups be selectively restored? How? By what means? Have you ever done this before? In what context? Is the context such that the data restored might be deemed relevant in the context of the current litigation? 3.4. As a matter of firm policy, do you overwrite, reformat, erase, or otherwise destroy the content of the backups of your network servers on a periodic basis? If so, under what circumstances? If so, what is the rotation period? If the rotation period has changed since [insert date], please describe the changes Do you maintain a company-wide intranet or other database accessible to any employees that provides/stores potentially relevant information? [Consider being more specific, e.g., regarding [a particular subject]. ] 3.6. Do you maintain network servers at any of the company s divisions/business units/locations/offices/subsidiaries that exist separately from or in addition to companywide server(s)? If yes, to what extent do any of those servers store any potentially relevant information in the context of this litigation? Ask follow-up questions consistent with the network server-based questions above. 4. Servers The questions below concern the current or former servers on your network ( servers ) that now or previously stored discoverable electronic internal or external peer-to-peer messages, including , third party sources, and instant messages (collectively, ) Identify the systems (client and server-side applications) used for and the time period for the use of each such system, including any systems used at any [overseas] facilities. p 3

80 The Sedona Conference Jumpstart Outline March Do you maintain servers at any or all of the company s divisions/business units/locations/offices/subsidiaries that exist separately or in addition to the company-wide server(s)? Are the systems the same/different from those identified in Question 4.1 above? Discuss any differences Are end-user s that appear in any of the following folders stored on (i) the end-user s hard drive, (ii) an server, or (iii) a server of a third party application service provider: In Box? Sent Items? Delete or trash folder? End user stored mail folders? 4.4. If any of your systems have changed since [insert relevant period], identify any legacy systems, the current system(s), and the date of the last backup made with each relevant legacy system Do you have at least one complete (i.e., non-incremental) backup of each of your servers for each month [for the period to ]? If not, for which months do you not have at least one complete backup? For those months, if any, for which you do not have a complete backup, do you have incremental or other backups from which a full backup can be created of all data as of a given date in each such month? If so, please describe the nature of such incremental or other backups and identify the months for which you have them Does each complete backup contain all messages sent or received since creation of the immediately prior complete backup? Do your backups contain the messages that are in each employee s In Box as of the time such backup is made? Do your backups contain the messages that are in each employee s Sent Items folder as of the time such backup is made? Do your backups contain the messages that are in each employee s delete or trash folder as of the time such backup is made? p 4

81 The Sedona Conference Jumpstart Outline March Do your backups contain the messages that are in each employee's stored mail folders as of the time such backup is made? Do your backups contain the messages that have been stored to each employee s hard drive? 4.7. Can specific boxes contained on backups be restored selectively? Does the company have or maintain an index or mapping resource that would serve as a reference to identify which employees is stored on particular backups? 4.8. As a matter of firm policy, do you overwrite, reformat, erase, or otherwise destroy the content of the backups of your servers on a periodic basis? If so, what is the rotation period? If the rotation period has changed since [insert date], describe the changes Did you, at any time, have a system that maintained electronic copies of all s sent or received by certain of your employees? Do you have such a system now? If so, describe the system(s) and the date(s) of first use If so, does such system(s) contain copies of all s captured from the date of first use until the present? If so, does such system(s) capture a copy of all s sent and/or received by employees in [identify relevant departments/groups that might be relevant]? 5. Hard Drives The questions below concern the current and former local or non-network drives contained in current or former employees laptop and desktop computers or workstations As a matter of firm policy, are employees desktop and laptop hard drives backed up in any way? If so, under what circumstances? If so, how long are such backups retained? Please describe the backup system As a matter of firm policy, are employees permitted to save files, s, or other data(excluding system- and application-generated temporary files) to their desktop or laptop hard drives? p 5

82 The Sedona Conference Jumpstart Outline March Since [insert relevant date], has it been technically possible for firm employees to save files, s, or other data (excluding system and application generated temporary files) to their desktop or laptop hard drives? 5.4. Do you implement technical impediments to minimize the opportunity for employees to save files, s, or other data (excluding system and application generated temporary files) to their desktop or laptop hard drives? Is it possible for employees to override such impediments? 5.5. To what extent has a search been done to determine the extent to which any of the key custodians in this litigation did, in fact, save files, s, or other data to their desktop or laptop hard drives? Flash drives? 5.6. As a matter of firm policy, are employees desktop and laptop hard drives erased, wiped, scrubbed, or reformatted before such hard drives are, for whatever reason, abandoned, transferred, or decommissioned? If so, are, as a matter of firm policy, files, s, or other data stored on such hard drives copied to the respective employee s replacement drive, if any? If so, as a matter of firm policy, are such files, s, or other data copied on a bitby-bit basis? 6. Non-Company Computers 6.1 Does firm policy permit, prohibit, or otherwise address employee use of computers not owned or controlled by the company to create, receive, store, or send work-related documents or communications? If so, what is that policy? 6.2. Is there any technical impediment to employees using computers not owned or controlled by the firm to create, receive, store, or send work-related documents or communications? p 6

83 The Sedona Conference Jumpstart Outline March 2011 The Sedona Conference Working Group Series SM & WGS SM Membership Program The Sedona Conference Working Group Series SM ( WGS SM ) represents the evolution of The Sedona Conference from a forum for advanced dialogue to an open think-tank confronting some of the most challenging issues faced by our legal system today. DIALOGUE DESIGNED TO MOVE THE LAW FORWARD IN A REASONED & JUST WAY. The WGS SM begins with the same high caliber of participants as our regular season conferences. The total, active group, however, is limited to instead of 60. Further, in lieu of finished papers being posted on the website in advance of the Conference, thought pieces and other ideas are exchanged ahead of time, and the Working Group meeting becomes the opportunity to create a set of recommendations, guidelines or other position piece designed to be of immediate benefit to the bench and bar, and to move the law forward in a reasoned and just way. Working Group output, when complete, is then put through a peer review process, including where possible critique at one of our regular season conferences, hopefully resulting in authoritative, meaningful and balanced final papers for publication and distribution. The first Working Group was convened in October 2002, and was dedicated to the development of guidelines for electronic document retention and production. The impact of its first (draft) publication The Sedona Principles: Best Practices Recommendations and Principles Addressing Electronic Document Production (March 2003 version) was immediate and substantial. The Principles was cited in the Judicial Conference of the United State Advisory Committee on Civil Rules Discovery Subcommittee Report on Electronic Discovery less than a month after the publication of the public comment draft, and was cited in a seminal e-discovery decision of the Southern District of New York less than a month after that. As noted in the June 2003 issue of Pike & Fischer s Digital Discovery and E-Evidence, The Principles...influence is already becoming evident. The WGS SM Membership Program was established to provide a vehicle to allow any interested jurist, attorney, academic or consultant to participate in Working Group activities. Membership provides access to advance drafts of Working Group output with the opportunity for early input, and to a Bulletin Board where reference materials are posted and current news and other matters of interest can be discussed. Members may also indicate their willingness to volunteer for special Project Team assignment, and a Member s Roster is included in Working Group publications. We currently have active Working Groups in the areas of 1) electronic document retention and production; 2) protective orders, confidentiality, and public access; 3) the role of economics in antitrust; 4) the intersection of the patent and antitrust laws; (5) Markman hearings and claim construction; (6) international e-information disclosure and management issues; and (7) e-discovery in Canadian civil litigation. See the Working Group Series SM area of our website for further details on our Working Group Series SM and the Membership Program. p 49

84 Copyright 2011, The Sedona Conference All rights reserved. Visit

85 MEET AND CONFER CHECKLIST AT THE OUTSET OF LITIGATION / TRIGGERING OF PRESERVATION DUTY TASK Review any document retention policy or policies currently in place. Interview relevant personnel (e.g., information technology professionals, in-house counsel, individual custodians, etc.) regarding any steps taken to preserve or delete documents prior to retaining counsel. Build a team to oversee the litigation hold process. Make sure that the roles and responsibilities of the various team members are clearly defined and understood. Assemble a team to assist with and manage the ediscovery process (e.g., in-house counsel, litigation support, outside counsel, service providers, other experts, etc.). Early decisions can and will impact the entire process. Proactive, competent team members are the foundation of any defensible process. Build factual support for any electronically stored information that may have been lost due to the routine, good-faith operation of an electronic information system. (See FRCP 37(e)). Ensure that all retention policies and processes are appropriately suspended to facilitate compliance with your duty to preserve relevant information (e.g., automated archiving/deletion procedures, backup tape rotation schedules, filters, etc.). Review the entire information infrastructure. How is information created, transferred, accessed, stored, and discarded? Account for all physical locations globally (e.g., headquarters, field offices, home offices, etc.), cloud and other third-party environments, backup and disaster recovery resources, social media, mobile devices, personal devices and habits (e.g., CFO regularly backs up her to DVD, consistently works remotely, and uses personal for work purposes), etc. COMPLETED? YES/NO NOTES This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

86 MEET AND CONFER CHECKLIST Inquire about the presence of any structured data repositories, content management systems, or other data storage systems. Review all backup tape rotation, recycling, and retention procedures. Understand how data travels through the IT environment. What are the policies in place that govern how data moves through the data ecosystem during the entire information lifecycle (e.g., Acceptable Use policies, BYOD policies, encryption, etc.)? If possible, chart out the information infrastructure with a data map. Information infrastructures are extremely dynamic so any attempt at data mapping must leverage an iterative process that develops in tandem with discovery. Develop a list of all potential custodians of relevant data including former employees and third parties. Interview document custodians and educate them on their preservation obligations. Develop a list of all potential sources of relevant data. Consider grouping data according to specific categories to promote organization (e.g., HR databases, sales team s, etc.). Determine the number and type of data assets that have been decommissioned during the relevant time period and the specific data custodians which would have had the ability to utilize said assets to generate, transmit, store, received, discard, or otherwise manipulate data in any fashion. Understand the technological systems used for generating, transmitting, storing, and deleting information (e.g., What system is used? Is stored locally or remotely? Are there size restrictions on individual mailboxes?). Develop a comprehensive protocol to implement the litigation hold(s). Make sure to have well defined procedures in place to monitor compliance with and update the hold(s) periodically according to changes in scope. It is important to also consider hold termination to avoid the unnecessary preservation of data that can be deleted in good faith. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

87 MEET AND CONFER CHECKLIST Distribute litigation hold notices. 1 Consider issuing a broad hold immediately upon the triggering event and then refine as understanding develops. Understand the types of data generated by and stored in various locations. What is their format? Are they proprietary in nature? What type of metadata do they generate? Determine if there are any uniquely sensitive data types maintained within the systems (e.g., ephi, PII, trade secrets, technical data under ITAR, data covered by any foreign privacy regulation, etc.). Begin to build factual support for data that is reasonably accessible vs. not reasonably accessible. (See FRCP 26(b)(2)(B)). Establish a budget for discovery and get input and approval from the appropriate parties. Obtain contact information for all necessary IT administrators and other personnel. Consider establishing a liaison to facilitate ediscovery tasks between team members and stakeholders. Evaluate all resources (e.g., technological, in-house and out-sourced personnel, and processes) in order to build an effective team and ediscovery process. Consider creating a defensibility binder whereby all importance decisions and considerations surrounding the ediscovery processes are identified and tracked. Understand all applicable rules and regulations, including any hyper-local rules that a specific court may impose on ediscovery related tasks. Make sure that ethical obligations associated with ediscovery, social media, and relevant technology are clearly understood. Read and understand FRE 502. Consider entering into a FRE 502(d) order. 1 This checklist is not meant to be a comprehensive guide to successfully implementing defensible legal holds. However, at the very least, a hold must inform employees of their duty to identify and preserve any information that may be relevant to a pending, current, or anticipated litigation or investigation. It should be distributed in writing to key players and those with potentially relevant information. It should clearly describe the target subject matter, contain specific instructions pertaining to the types of information that should be preserved, and include contact information for relevant members of the hold implementation team. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

88 MEET AND CONFER CHECKLIST Set the stage for good faith cooperation by sending a brief communication to opposing parties foreshadowing or highlighting various ediscovery issues. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

89 MEET AND CONFER CHECKLIST PREPARING FOR A MEET & CONFER TASK If data sources outside of the United States are involved, familiarize yourself with any applicable privacy regulations and/or blocking statutes. Consider engaging experts in cross-border ediscovery as well as local counsel to assist with incountry culling & filtering and cross-border data transfers. Determine if any foreign language documents are going to play a role in the matter. Make a list of key data custodians. Make a list of all target devices and data sources (e.g., computers, shared networks, servers, personal computers, cloud (e.g., Gmail, Yahoo, Hotmail, etc.), mobile devices, social media, etc.). Determine if any of the devices are encrypted. Determine the operating systems for all servers and devices (e.g., Unix, Linus, Windows, Macintosh OS, etc.). Determine if there are redundant sources of data within the system (i.e. all s generated on a user s workstation are also backed up onto on a network server). Consider that data located outside of the United States may also be located on domestic storage media. Estimate collection volume based on the number of custodians and sizes of the various devices. Develop a collection timeline. COMPLETED? YES/NO NOTES Determine standard data types that will be collected (e.g., PST/NSF files, MS Office files, Bloomberg Messages, audio files, etc.). Determine if there is any sensitive data that needs to be specifically identified and isolated (e.g., ephi, PII, trade secrets, technical data under ITAR, etc.). This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

90 MEET AND CONFER CHECKLIST Determine if there are any storage systems that may present unique ediscovery challenges (e.g., SharePoint, voic , audio tapes, etc.). Determine if any legacy data is within the scope of the matter and needs to be collected. Determine if proprietary data needs to be collected and if licenses are required for processing. Determine the method of collection and the personnel best suited to perform the collection (e.g., data custodian, forensic collections experts, in-house IT personnel, etc.). Determine the specifications for data processing. Consider a priority custodian list for processing purposes to streamline review efforts. Determine the level of deduplication (e.g., custodian level or global level). Will near deduplication be a consideration? Define any other culling filters that may be used to reduce the scope of data for review (i.e. file extensions, domains, date ranges, keywords, etc.). Developing these culling filters is an iterative process that should involve careful selection, sampling, testing, cooperation, and possible expert assistance. Determine what search and retrieval methodologies will be efficacious for the matter. Note that desired functionality may require a specific index to be built (i.e. keyword search, DT Index for proximity searching in certain platforms, analytics index, etc.). Determine if any advanced analytics or other specialized processes will be utilized (e.g., near deduplication, threading, conceptual searching, automatic clustering and categorization, Technology Assisted Review (TAR), etc.). 2 2 Please see the Grossman Cormack Glossary of Technology Assisted Review for definitions of various terms related to TAR. Available at This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

91 MEET AND CONFER CHECKLIST If TAR is being considered, build a protocol for developing seed sets or other system-training intelligence (e.g., random sampling, judgmental sampling, multi-modal approach, etc.). Determine if you want to review a statistically valid sample set or some other threshold. Determine your confidence level/interval (%). Consider the use of a statistician to develop metrics around precision and recall. Consider proportionality when engaging such an expert as such assistance may only be beneficial when the TAR process is challenged. Define an acceptable overturn rate. Determine if redaction will be necessary and the methods used to redact. Determine resources needed for performing the relevancy review (i.e. online review platforms and other technologies, physical review locations, etc.). Consider if contract attorneys are required and the difference in hourly rates across various markets in the U.S. Will reviewers be near-shored or offshored? Determine the timeframe to complete the relevancy review. Determine resources needed to complete the privilege review. Determine the timeframe to complete the privilege review. Decide the format in which you want to receive productions as well as the format in which you want to produce. Determine if there has been any spoliation or lost data and how best to support/defend any gaps in data. Understand the rules in your jurisdiction regarding spoliation and sanctions. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

92 MEET AND CONFER CHECKLIST Assess whether you will request cost-shifting or how to best defend a cost-shifting request from opposing counsel. Assess whether you might move for a protective order or to compel discovery. How might you best respond to opposition from opposing counsel? Decide if you want an IT representative or other expert to attend the conference(s) with you. Determine what reports and other metrics you need to measure and track the process. Evaluate these against the actual reports and metrics that are available to you based on your team and technology and then reconcile any discrepancies. Decide if you want your IT representative(s) to communicate directly with the IT representative(s) of the opposing party. Decide on the 30(b)(6) witness or witnesses who will be able to testify about the information infrastructure as well as document retention policies. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

93 MEET AND CONFER CHECKLIST DURING A MEET & CONFER TASK Discuss the scope of the matter. COMPLETED? YES/NO NOTES Data custodians Key witnesses Geographic scope Temporal scope Discuss the preservation and/or production of unreasonably accessible data. Discuss deduplication procedures. Discuss how to handle document families (e.g., parent s and attachments). Discuss keyword search terms or conceptually based search queries. Discuss the efficacy of advanced analytics and TAR. If Technology Assisted Review is considered, discuss the TAR process (e.g., methods for developing seed sets/training documents, access to seed sets/training documents and other such work product, acceptable overturn/error rates, etc.). Discuss production format (e.g., native, image only, images with extracted text and load files, hybrid formats such as searchable images with native Excels, paper, online access through a platform, shared online repository, etc.). Discuss whether metadata will be produced. Discuss a production timeline and if rolling productions will be required. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

94 MEET AND CONFER CHECKLIST Discuss bates numbering and any endorsing. Consider protocols for bates numbering native documents. Discuss whether cost-shifting/sharing will be necessary. Agree on how to handle any sensitive data (e.g., ephi, PII, trade secrets, confidential business information, technical data under ITAR, etc.). Decide whether IT representatives from each side need to talk to one another about the data. Determine protocols surrounding the clawback of inadvertently produced privileged material. SEE FRE 502. Determine if a quick peek agreement would be efficacious. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

95 MEET AND CONFER CHECKLIST AFTER A MEET & CONFER TASK Routinely audit compliance with the litigation hold. COMPLETED? YES/NO NOTES Reissue the litigation hold(s) notice as necessary to remind custodians of their obligations, include new custodians or data requests, and update procedures and other information. Maintain detailed records of all litigation hold efforts. Document chain of custody. Test, analyze, sample, redefine and update keyword search terms or conceptual categories. Memorialize the conference and any points of agreement and disagreement. Consider updating a defensibility binder. Creating a track record of the various attempts made to cooperate can support claims that good faith efforts were made to cooperate when moving for an order to compel discovery under FRCP 37(a) or a protective order under FRCP 26(c). Schedule additional conferences as the matter progresses. Cooperation is a process. If you have any questions please contact Matthew Knouff, Esq. at: Complete Discovery Source, Inc. US Headquarters Direct: Park Avenue Mobile: New York, NY Fax: This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

96 COMPUTER SYSTEMS CHECKLIST (EMPLOYEE) COMPANY NAME EMPLOYEES NAME JOB TITLE DEPARTMENT CONTACT NUMBER ADDRESS 1. COMPUTERS Please identify any computer systems in possession or shared by the custodian indicated above (including home computers and laptops). For each computer system that is used, please answer the following. No Computer Device Type Ownership Name 1 (Company or personal) Device Location (Office, home or remote) Operating System Comments: 1 Enter a unique designation that distinguishes this system from all others. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

97 COMPUTER SYSTEMS CHECKLIST (EMPLOYEE) 2. DATA STORAGE Besides the internal hard drive(s) in the above system(s), please list other portable storage media related to your position. Note that backups will be treated separately in the next section. No Type (e.g., flash drive, external drive, CDs, DVDs, fileserver, etc.) Company / personal property Device Location (e.g., Office, home or remote) Was this data transferred to any other device? Comments: 3. BACKUP If applicable, please enter type, location and schedule of your backups. No Type (Local tape, DVD, external drive, etc.) Location Backup Schedule This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

98 COMPUTER SYSTEMS CHECKLIST (EMPLOYEE) Comments: 4. MAIL SERVICES No Service (e.g., corporate service, MSN, Yahoo, Gmail, etc.) Function Company use / personal use or both User Account or address (e.g., Message stored locally INTERNAL DOMAINS No Domain Names (e.g., Comments: This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

99 COMPUTER SYSTEMS CHECKLIST (EMPLOYEE) 5. COLLABORATIVE SYSTEMS No System (e.g., SharePoint, Extranet Content Management System, TextFlow, etc.) Location (e.g., URL, Archives, etc.) Messages Stored Locally? Which computer(s)? Comments: 6. DATABASES (IF APPLICABLE) No Database System Type (e.g., Oracle, MS SQL Server, MySQL, SYSBASE, etc.) Database Name Comments: This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

100 COMPUTER SYSTEMS CHECKLIST (EMPLOYEE) 7. MOBILE DEVICES (IF APPLICABLE) No Device Name 2 Device Type (e.g., Blackberry, mobile phone, camera phone, tablet devices, ipod, etc.) Ownership (Company or personal) Acceptable Use of Device? SIM Card Available? / Device Synched with any other Device? Comments: 8. SOCIAL MEDIA AND NETWORKING (IF APPLICABLE) No 1 Site Name (e.g., Facebook, MySpace, YouTube, Twitter, LinkedIn, Ning, Blogs Username Password Acceptable Use of Site (e.g., was the site used for work, personal, accessed from work, etc.?) Description of Activity (e.g., frequent user, inactive user, etc.) 2 Enter a unique designation that distinguishes this system from all others. This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

101 COMPUTER SYSTEMS CHECKLIST (EMPLOYEE) Comments: 9. ADDITIONAL SOURCES & COMMENTS This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be discussed with or distributed to any other parties without prior written consent from CDS Complete Discovery Source, Inc.

102 Chain of Custody Short Form Case/Project Information Case Name Matter Name Client/Matter No. CDS Contact Information TITLE NAME OFFICE PHONE NO. CELL PHONE NO. Media Description Date Received Quantity Chain of Custody Date & Time Action Taken Release By Received By Chain of Custody Date & Time Action Taken Release By Received By Chain of Custody Date & Time Action Taken Release By Received By Chain of Custody Date & Time Action Taken Release By Received By sign & print name sign & print name sign & print name sign & print name sign & print name sign & print name sign & print name sign & print name This document is proprietary and confidential. It is for the exclusive use of the intended recipient. It may not be reproduced in any format nor discussed with any parties without prior written consent of Complete Discovery Source, Inc.

103 I-MED PHARMA INC. v. Biomatrix, Inc., Dist. Court, D. New Jersey Google Sch... Page 1 of 6 I-MED PHARMA INC., Plaintiff, v. BIOMATRIX, INC., GENZYME CORPORATION and GENZYME BIOSURGERY, a Division of Genzyme Corporation. Defendants. Civ. No (DRD). United States District Court, D. New Jersey. December 9, KREISBERG & MAITLAND, LLP, by: Gary Maitland, Esq., New York, New York, Attorneys for Plaintiff. BINGHAM MCCUTCHEN LLP, by: Jessica S. Boar, Esq., New York, New York, by: Thane D. Scott, Esq., by: Carol E. Head, Esq., by: Shuan Lue, Esq., Boston, Massachusetts, Attorneys for Defendant. OPINION DICKINSON R. DeBEVOISE, District Judge. This case highlights the dangers of carelessness and inattention in e-discovery. The underlying dispute arises from the alleged breach of two medical distribution contracts between Plaintiff I-Med Pharma, Inc. ("I-Med") and Defendant Biomatrix, Inc. [1] Plaintiff alleges that Defendant, in conjunction with its related entities, entered into contracts under which I-Med would be the exclusive Canadian distributor of several medical products produced by Biomatrix. Plaintiff further claims that after a merger with Genzyme, Defendant failed to live up to its obligations to produce the licensed products for distribution. Presently before the Court is Defendants' Appeal from Magistrate Judge Shipp's September 9, 2011 order. The September 9 order modified the terms of a prior January 14, 2011 order that required Plaintiff to produce the results of a forensic examination of its computer system. Under the terms of the new order, Plaintiff would not be required to produce documents recovered from any unallocated space files found on its system. Defendants claim that Judge Shipp abused his discretion by refusing to order Plaintiff to conduct a costly privilege review of the 95 million pages of documents recovered from the unallocated space files. Plaintiff contends that this appeal is a bad-faith effort by Defendants to force it to spent significant resources on a review of irrelevant data. For the reasons set forth below, Magistrate Judge Shipp's September 9, 2011 order is AFFIRMED. I. BACKGROUND Defendants are biotechnology firms who manufacture a variety of medicines and medical devices. This case concerns a distribution agreement concerning several medical products manufactured by Defendants. The products in question 3/27/2013

104 I-MED PHARMA INC. v. Biomatrix, Inc., Dist. Court, D. New Jersey Google Sch... Page 2 of 6 are the Biomatrix Hylashield ("Hylashield"), the Hylan Surgical Shield ("HSS"), the Hylashield Nite ("Nite"), and the Hylashield Lite ("Lite"). [2] On or about August 4, 1994, Plaintiff and Biomatrix entered into an agreement (the "1994 Agreement") whereby Biomatrix was appointed the exclusive distributor of Hylashield and HSS in Canada, the Bahamas, and the English speaking Caribbean Island nations. The initial term of the agreement was five years, and the agreement provided that if it was not terminated in writing by either party at least 90 days prior to its expiration date it would be automatically renewed for an additional five years. Similarly, on or about October 4, 1995, Plaintiff and Biomatrix entered into an agreement (the "1995 Agreement") appointing Plaintiff the exclusive distributor in Canada for Nite and Lite. That agreement also was for an initial term of five years, and it also provided for an automatic five year renewal if neither party gave notice of termination. Neither party terminated either agreement prior to renewal. In connection with the Agreements, Plaintiff took steps necessary to obtain certain product registrations and licenses from Canadian authorities. These included licenses from Health Canada for products distributed under the 1994 Agreement and under the 1995 Agreement, and trademark registrations. In late 2000, Biomatrix entered into a business combination with Defendant Genzyme. Plaintiff claims that it was assured on "numerous occasions" that "no changes would be made" that would affect the business conducted under the Agreements. However, in April 2001, Defendants allegedly closed a facility in Pointe -Claire, Quebec for the production of the distributed products. Defendants then changed the name of Hylashield Lite to Hylashield CL, and stopped delivery of the product to Plaintiff under either name. The Second Amended Complaint alleges that Defendants did not ship sufficient product under the Agreements or use their best efforts (as mandated by the Agreements) to fulfill Plaintiff's requirements for products. (Doc. No. 44). It also alleges that after the Genzyme combination Defendants failed or refused to supply products or otherwise perform under the Agreements. Id. Plaintiff pleads various species of breach of contract and fraud. Id. The instant dispute concerns data retrieved from a forensic investigation of Plaintiff's computer system. Pursuant to a May 27, 2010 stipulation between the parties, Defendants hired an expert to conduct a keyword search of I-Med's computer network, servers, and related storage devices. (Doc. No. 182). The search terms run were as follows: [3] back order* product* Blephagel expir* Chalifour and/or allet* Larm loss Jean-Guy Grenon revenue* vet* profit* Domareki and/or agreement Oasis reputation Wes contract* credit Bengl and/or refund Agerup Hyl* discount Pagano refriger* CL rebate Balazs and/or Bandi repine* HS* return Riggs and/or Rofy FDA HsS pro forma* unidose Health Canada shield* sample monodosc TPP Lite Biomatrix visco* Nite Genzym Laboratories 3/27/2013

105 I-MED PHARMA INC. v. Biomatrix, Inc., Dist. Court, D. New Jersey Google Sch... Page 3 of 6 inventor* I-Drop Unither manufactur* I-Lid Excelvision minimum* HA claim quota* Siccafluid complaint The search was not limited to targeted document custodians or relevant time periods. Indeed, the search was not even limited to active files. The expert was instructed to run the search terms across all data on the computer system, including so-called "unallocated space" areas of computer memory in which there is no write-protection and in which deleted and partially deleted files and other temporary data may often be found. The results should come as no surprise. The broad search terms hit millions of times across the large data set. In the unallocated space alone, the terms generated 64,382,929 hits. These hits represent an estimated 95 million pages of data. [4] Plaintiff balked at the prospect of conducting a privilege review of this material and petitioned Magistrate Judge Shipp for relief from the stipulation. (Doc. No. 219). On September 9, 2011, Judge Shipp conducted a telephone conference in which the parties explained their positions on the matter. (Doc. No. 236) ("Tr."). After permitting both sides a full opportunity to be heard, Judge Shipp issued an order permitting Plaintiff to withhold the data found in the "unallocated space" of its computer systems. (Doc. No. 235). Cogniscent of the costs Defendants had incurred in extracting and searching this data, the order permitted Defendants to seek reimbursement of those costs from Plaintiff. Id. In connection with this order, Judge Shipp made several findings on the record. First, he found that "good cause" existed to modify his prior order as the "burden on I-Med" would "outweigh any potential benefit that may result." Tr. 32:10, 32:21-24 ("Tr.") (Doc. No. 236). Second, he found that Genzyme had not met its burden of demonstrating the complete relevancy of the information sought, noting that for all of Defendants' complaints about spoliation, it had not actually identified any information actually destroyed by Plaintiff. Tr. 32: Finally, he found that the overbroad search terms made the likelihood of finding relevant information that would be admissible at trial "minimal." Tr. 33:2-8. Defendants now appeals from Judge Shipp's order. II. DISCUSSION A. Standard of Review A Magistrate Judge's decision is to be overturned only to the extent that the ruling is "clearly erroneous or contrary to law." L.Civ.R.72.1(c)(1)(A). The burden of showing that a ruling is "clearly erroneous or contrary to law rests with the party filing the appeal." Marks v. Struble, 347 F. Supp. 2d 136, 149 (D.N.J. 2004). A finding is clearly erroneous "when although there is evidence to support it, the reviewing court on the entire evidence is left with the definite and firm conviction that a mistake has been committed." Dome Petroleum Ltd. v. Employers Mut. Liab. Ins. Co., 131 F.R.D. 63, 65 (D.N.J.1990) (quoting United States v. U.S. Gypsum Co., 333 U.S. 364, 395 (1948)). Moreover, "[a] ruling is contrary to law if the magistrate judge has misinterpreted or misapplied applicable law." Pharmaceutical Sales and Consulting Corp. v. J.W.S. Delavau Co., Inc. 106 F.Supp.2d 761, 764 (D.N.J. 2000). Where an appeal seeks review of a matter within the purview of the Magistrate Judge, such as a discovery dispute, an even more deferential standard, the "abuse of discretion standard" must be applied. Kounelis v. Sherrer, 529 F.Supp.2d 503, 518 (D.N.J. 2008) "[w]here a magistrate judge is authorized to exercise his or her discretion, the 3/27/2013

106 I-MED PHARMA INC. v. Biomatrix, Inc., Dist. Court, D. New Jersey Google Sch... Page 4 of 6 decision will be reversed only for an abuse of discretion."; see also Holmes v. Pension Plan of Bethlehem Steel Corp., 213 F.3d 124, 138 (3d Cir. 2000) (discovery orders reviewed for abuse of discretion). An abuse of discretion occurs "when the judicial action is arbitrary, fanciful or unreasonable, which is another way of saying that discretion is abused only where no reasonable man would take the view adopted by the trial court. If reasonable men could differ as to the propriety of the action taken by the trial court, then it cannot be said that the trial court abused its discretion." Lindy Bros. Builders v. American Radiator & Standard Sanitary Corp., 540 F.2d 102, 115 (3d Cir. 1976). There is "particularly broad deference given to a magistrate judge's discovery rulings." Farmers & Merchants Nat. Bank v. San Clemente Financial Group Securities, Inc., 174 F.R.D. 572, 585 (D.N.J. 1997). Federal Rule of Civil Procedure 26(b)(2)(C) grants the Court considerable authority to limit a party's pursuit of otherwise discoverable information where the burden of a discovery request is likely to outweigh the benefits. The Court is directed to "limit the frequency or extent of discovery otherwise allowed by these rules or by local rule if it determines that: (i) the discovery sought is unreasonably cumulative or duplicative, or can be obtained from some other source that is more convenient, less burdensome, or less expensive; (ii) the party seeking discovery has had ample opportunity to obtain the information by discovery in the action; or (iii) the burden or expense of the proposed discovery outweighs its likely benefit, considering the needs of the case, the amount in controversy, the parties' resources, the importance of the issues at stake in the action, and the importance of the discovery in resolving the issues." See also Bayer AG v. Betachem, Inc., 173 F.3d 188, 191 (3d Cir. 1999) ("Although the scope of discovery under the Federal Rules is unquestionably broad, this right is not unlimited and may be circumscribed... The Federal Rules of Civil Procedure expressly allow a district court to use its discretion and deny discovery requests if the material sought is `unreasonably cumulative.') (internal citations omitted). For the reasons described in detail below, this Court finds Judge Shipp's order to be a reasonable exercise of his considerable discretion in managing the scope of permissible discovery. B. The Unallocated Space Files Defendants contend that Judge Shipp abused his discretion by applying the wrong legal standard to determine whether to permit relief from the stipulation. Defendants claim that "the Magistrate Judge was required to find, that `exceptional circumstances' would cause I-Med to suffer `manifest injustice' if required to perform its obligations." (Def. Br. 6). In support of his position, Defendants cite Waldorf v. Shuta, 142 F.3d 601, (3d Cir. 1998) for the proposition that no discovery stipulation can be modified absent a showing of "manifest injustice." This reading of Waldorf is overbroad, simplistic, and incorrect. Waldorf involved litigation over a catastrophic auto accident. The Waldorf plaintiff was a passenger who suffered debilitating injury after a collision that occurred near a defective traffic light. Waldorf, 142 F. 3d at 606. The plaintiff sued the drivers, the city that had installed the traffic light, and various city officials. Id. at 607. After an initial $8 million jury verdict was reversed on appeal, the city defendant proposed a deal. Id. It would stipulate to its liability on the condition that the trial be bifurcated and the damages trial against all defendants be tried first. Id. The lower court accepted this stipulation and tried the case as to damages. Id. The result was a $16 million verdict. Id. The city appealed and obtained a new trial. Id. The city then attempted to vacate the verdict and back out of the stipulation. Id. The trial court refused to permit it. Id. On review, the Court of Appeals upheld the trial court's decision to enforce the stipulation, finding that "exceptional circumstances" justifying relief from the validly executed stipulation did not exist. Id. at 617. In determining whether "manifest injustice" would result unless the city was relieved from its stipulation the court examined four factors: "(1) 3/27/2013

107 I-MED PHARMA INC. v. Biomatrix, Inc., Dist. Court, D. New Jersey Google Sch... Page 5 of 6 the effect of the stipulation on the party seeking to withdraw the stipulation; (2) the effect on the other parties to the litigation; (3) the occurrence of intervening events since the parties agreed to the stipulation; and (4) whether evidence contrary to the stipulation is substantial...." Id. at (internal citations omitted). The court concluded that when the city "made the stipulation prior to the second trial, it had a full understanding of the legal rights it was relinquishing, and had clear knowledge of the consequences of its stipulation" and that to permit the city to withdraw the stipulation would be "manifestly unfair to [the plaintiff]...." Id. at The instant dispute bears little resemblance to the one in Waldorf. The court in Waldof did not speak to the issue of discovery stipulations and orders. Rather, it addressed the legal standard for vacating a stipulation of liability. [5] While both might be broadly termed stipulations, they have very different legal significance. A party seeking to modify the scope of document production before the close of discovery is in a very different position than a party seeking to withdraw a crucial admission of wrongdoing post-trial. During discovery, the parties are still actively uncovering the evidence needed to bring a case to trial and have ample opportunity to modify and adjust their litigation strategy to any important developments. Clearly a court has the power to modify stipulations concerning discovery terms and deadlines while discovery is still ongoing without the showing of manifest injustice. A court could not effectively perform its duty to fairly and efficiently manage discovery if every minor change to a stipulated briefing schedule or deposition date required a showing of "exceptional circumstances" or "substantial and real harm." While courts should not casually discard agreements between the parties, nor should they abrogate their duty to balance both burden and the likelihood of uncovering relevant evidence merely because a party made an improvident agreement. And even if the Court were to apply the Waldorf standard to this case, the factors identified by the Court of Appeals would still support Judge Shipp's order. The first factor, the effect of the stipulation on the party seeking to modify it, weighs heavily in favor of modification. A privilege review of 65 million documents is no small undertaking. Even if junior attorneys are engaged, heavily discounted rates are negotiated, and all parties work diligently and efficiently, even a cursory review of that many documents will consume large amounts of attorney time and cost millions of dollars. At oral argument, Defendants' counsel argued that a less costly privilege review could be done by reviewing only documents containing the word "privileged" and producing everything else. This argument is not persuasive. [6] Even when dealing with intact files, potentially privileged information may often be found in s, memoranda, presentations, or other documents that are not explicitly flagged as privileged or confidential. And since the data searched here is likely to contain fragmented or otherwise incomplete documents, it is entirely possible for privileged material to be found without its original identifying information. The second factor, the impact on Defendants, does not weigh heavily against modification. Judge Shipp found that Defendants had failed to demonstrate a likelihood that relevant non-duplicative information would be found in the unallocated space files. While Defendants claim to have spent thousands of dollars obtaining the data from Plaintiff's computer system, the September 9 order permits them to seek reimbursement of this expense from Plaintiff. Even if it had not, this amount pales in comparison to the millions of dollars in additional expenditure that would be required for Plaintiff to adequately review the material for responsiveness or privilege. Defendants have failed to show that they will suffer any great prejudice as the result of the modification. The third factor, the occurrence of intervening events, points weakly in favor of modification. While the precise number of hits produced was not known in advance and Plaintiff argues that it could not have predicted the volume of material that the search would uncover, it should have exercised more diligence before stipulating to such broad search terms, particularly given the scope of the search. In evaluating whether a set of search terms are reasonable, a party should consider a variety of factors, including: (1) the scope of documents searched and whether the search is restricted to specific computers, file systems, or document custodians; (2) any date restrictions imposed on the search; (3) whether 3/27/2013

108 I-MED PHARMA INC. v. Biomatrix, Inc., Dist. Court, D. New Jersey Google Sch... Page 6 of 6 the search terms contain proper names, uncommon abbreviations, or other terms unlikely to occur in irrelevant documents; (4) whether operators such as "and", "not", or "near" are used to restrict the universe of possible results; (5) whether the number of results obtained could be practically reviewed given the economics of the case and the amount of money at issue. Taking all of the factors [7] together, the Court finds that even under the more stringent Waldorf rule, the September 9, 2011 order would be a proper exercise of Judge Shipp's considerable discretion in this case. While Plaintiff should have known better than to agree to the search terms used here, the interests of justice and basic fairness are little served by forcing Plaintiff to undertake an enormously expensive privilege review of material that is unlikely to contain non-duplicative evidence. And as the routine administration of discovery matters do not even require this heightened level of attention, the September 9, 2011 order is doubtlessly proper. III. CONCLUSION For the foregoing reasons, Magistrate Judge Shipp's September 9, 2011 order is AFFIRMED. [1] According to the allegations in the Complaint, the other Defendants are entities related to Biomatrix Defendant Genzyme Corp. having purchased or merged with Biomatrix in or about December 2000, creating a new entity, Defendant Genzyme Biosurgery. [2] Hylan, a chemical derivative of hyaluronan, is a proprietary gel manufactured by Biomatrix. It has numerous medical applications and is found in many of Defendants' products. The products at issue here are varieties of eye drops that contain Hylan fluid. [3] In addition, many of the terms were also to be run in French. [4] The Court has no choice but to take the parties at their word concerning these numbers. However it is troubling that the parties refer to the number of raw hits as though each represented a separate document. Given the volume of hits and search terms used, this is essentially impossible statistically speaking terms like "profit," "loss," "revenue," and "profit" frequently occur together, and it stands to reason that at least some files mentioning product lines would make reference to more than one at the same time. Consequently, the Court is left to wonder whether the total hit and estimated page numbers are genuinely correct. [5] Indeed, the lower court was clear that by "stipulation" it meant "an admission which `cannot be disregarded or set aside at will.'" Waldorf v. Borough of Kenilworth, 878 F.Supp. 686, 691 (D.N.J. 1995) (emphasis added) quoting Wheeler v. John Deere Co., 935 F.2d 1090, 1097 (10th Cir.1991). The stipulation at issue here admits nothing, and neither Waldorf opinion suggests that the holding was meant to apply to routine discovery orders. [6] This suggestion was prompted by a question from the Court asking how counsel for the Defendants would conduct a privilege review of the size and nature of the one that they sought to impose on Plaintiff. In spite of the answer given, it is difficult to believe that lawyers from Bingham McCutchen regularly disclose large quantities of information from their client's files without examining it. [7] The fourth factor identified by Waldorf is inapplicable in this case. Save trees - read court opinions online on Google Scholar. 3/27/2013

109 Not So Fast: 95 Million Reasons to Carefully Select and Limit Search Terms : E-Discover... Page 1 of 2 E-Discovery Law Alert Posted at 1:31 PM on January 24, 2012 by Jennifer Marino Thibodaux Not So Fast: 95 Million Reasons to Carefully Select and Limit Search Terms It has become commonplace for parties engaged in electronic discovery to discuss and agree upon keyword searches in an effort to limit the overall scope of discovery. A recent decision in the District of New Jersey, I-Med Pharma, Inc. v. Biomatrix, Civ. No (DRD), (D.N.J. 2011), demonstrates the pitfalls that arise when the parties too eagerly agree to conduct a search for electronically stored information using an overly broad set of keywords. The case also demonstrates a court s willingness to engage in proportionality analysis to cabin broad discovery. Biomatrix involved a dispute over two medical distribution contracts, with the plaintiff alleging that the defendant breached certain exclusivity provisions. During the course of the parties meet and confer obligations, the plaintiff s counsel agreed to allow the defendant s expert to conduct a keyword search of more than 50 terms on the plaintiff s computer network, servers, and related storage devices. Counsel should have known better than to agree to such keywords -- without limits as to time, custodian, or active file status -- that would almost certainly result in millions of hits. In this case, the agreed upon search yielded more than 64 million hits, approximating 95 million pages of data. Despite agreeing to conduct such a broad search as part of a previous court order, the plaintiff was forced to seek court approval to have the prior discovery order modified to further narrow the discovery inquiry. Not surprisingly, the defendants sought to hold the plaintiff to their initial deal, and also sought costs associated with the search. Magistrate Judge Shipp found that (1) good cause existed for modification because the plaintiff s privilege review of the documents would be unduly burdensome, (2) the defendants did not demonstrate the relevancy of the documents, and (3) the parties overbroad search terms were unlikely to yield relevant, admissible information. He thus amended the pre-existing discovery order, but held that the defendants could seek reimbursement of the costs associated with extracting and searching the data on the plaintiff s computer system. District Judge Dickinson R. Debevoise ultimately affirmed Magistrate Judge Michael Shipp s modification of the existing discovery order -- finding not only good cause to do so, but even manifest injustice if the order was not modified. See Waldorf v. Shuta, 142 F.3d 601, 1998 U.S. App. (3d Cir. 1998). Judge Debevoise also took the opportunity to provide guidance for parties to keep in mind when discussing and proposing search terms. In particular, the Court reasoned that the parties should have been more diligent before agreeing to the broad search terms, and listed the following factors for parties to consider when evaluating proposed search terms: 1. the scope of the documents searched and whether the search is restricted to specific computers, 3/27/2013

110 Not So Fast: 95 Million Reasons to Carefully Select and Limit Search Terms : E-Discover... Page 2 of 2 file systems, or document custodians; 2. any date restrictions imposed on the search; 3. whether the search terms contain proper names, uncommon abbreviations, or other terms unlikely to occur in irrelevant documents; 4. whether operators such as and, not, or near are used to restrict the universe of possible results; and 5. whether the number of results obtained could be practically reviewed given the economics of the case and the amount of money at issue. As a practical matter, litigants should pay attention to Judge Debevoise s guideposts before agreeing to a broad set of search terms. An eagerness to agree up front to a discovery plan and avoid a fight may only delay the inevitable if the search terms picked are so broad as to result in an unduly burdensome stack of material to review. Moreover, courts have become more willing to examine what s at stake in a case before ordering broad-based discovery; and seeking a more limited discovery order early in the case can avoid unnecessary expense later on. Jennifer Marino Thibodaux is an Associate on the Gibbons E-Discovery Task Force. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 3/27/2013

111 LLOYD GOLDMARK and ELIZABETH GOLDMARK, v. NOT FOR PUBLICATION WITHOUT THE APPROVAL OF THE APPELLATE DIVISION Plaintiffs-Respondents, THOMAS MELLINA, BETH MELLINA, STEVEN NEEDLE and NEEDLE POINT HOMES, LLC, and Defendants, BRACH EICHLER, L.L.C., Appellant. SUPERIOR COURT OF NEW JERSEY APPELLATE DIVISION DOCKET NO. A T2 PER CURIAM Submitted 0D\²'HFLGHG-XQH Before Judges Fisher, Nugent and Carchman. On appeal from the Superior Court of New Jersey, Law Division, Union County, Docket No. L Brach Eichler, appellant pro se (Bob Kasolas, of counsel and on the brief). Wacks & Hartmann, attorneys for respondents (Christopher E. Hartmann, on the brief).

112 In this appeal, Brach Eichler, L.L.C., contends that the trial judge abused her discretion in imposing a sanction against the firm in the amount of $5,502.50, based on the firm's failure to timely produce certain documents sought in discovery. Because the record supports the judge's conclusion that the firm failed to take adequate precautions to prevent the temporary disappearance of two critical s, we reject Brach Eichler's arguments and affirm. In 2007, the parties entered into a contract by which plaintiffs Lloyd and Elizabeth Goldmark (hereafter "purchasers") agreed to purchase, for $4,100,000, a home in Westfield to be constructed by defendants Thomas Mellina, Beth Mellina, Steven Needle, and Needle Point Homes, LLC (hereafter "sellers"); purchasers provided a $1,000,000 deposit. Disputes arose during construction. Sellers' counsel at the time, Robert Kraus, Esq., wrote to his counterpart on February 6, 2008, suggesting that purchasers buy the property in its existing stage and complete the construction with their own suppliers and contractors. Purchasers rejected this but the parties met to discuss settlement and settlement documents were later drafted. The 2 A T2

113 alleged settlement, however, was never finalized, and Kraus was terminated by sellers and replaced by current counsel. 1 Purchasers commenced this suit, claiming their construction disputes had been settled and, pursuant to their settlement agreement, purchasers were entitled to the return of their deposit; sellers resisted this contention and claimed purchasers breached the contract. Prior to trial, the property was sold to another buyer for a lower price than that which the purchasers agreed to pay; sellers returned the deposit to the purchasers. The matter proceeded to a non-jury trial. Among other things, Judge Frederic Kessler was required to decide: whether the matter had been settled and whether the settlement should be enforced; and, if not, whether the contract was breached or repudiated and, if so, by whom. The trial judge found that a settlement agreement had not been reached but also held that sellers had repudiated and breached the contract, thereby entitling purchasers to retain the deposit that had been returned prior to trial. Of relevance to the matter at hand, Judge Kessler relied upon and quoted extensively two s, between Kraus and the 1 Current counsel was, at the time, affiliated with another law firm. It is not clear when Brach Eichler assumed the role of counsel for sellers; however, since Brach Eichler does not claim that the other law firm is responsible for the imposed sanctions, the involvement of the other firm is not relevant. 3 A T2

114 sellers, dated March 4 and 31, Although these documents did not persuade the judge that the parties' disputes had been amicably settled, Judge Kessler, as explained in his thorough written opinion, greatly relied on the two s in finding that sellers cancelled the contract. The efforts expended by purchasers in seeking these two e- mails in discovery lie at the heart of the present controversy. Early in this suit's history, the purchasers served a subpoena on Kraus, seeking the non-privileged portions of his file. Cross-motions resulted in the rescheduling of Kraus's deposition and an order requiring the production of a privilege log, pursuant to Rule 4:10-2(e). Prior to his deposition, Kraus forwarded his file to sellers' current counsel, who reviewed the file and returned it to Kraus. Because Kraus was seriously ill, his deposition was videotaped. The log was provided when the deposition commenced; it identified, among other documents, the two March s exchanged by Kraus and defendant Beth Mellina (hereafter "Mellina"). Because sellers insisted these s and other identified documents were privileged, they were not turned over at the deposition. 4 A T2

115 Subsequent motions led first to the entry of an order requiring an in camera inspection. 2 In her written decision of November 4, 2009, Judge Kathryn A. Brock observed that the two March s were not in the materials provided for inspection. Sellers' current counsel responded that he did not make copies of Kraus's file when previously provided for his review and the alleged privileged documents were not in his firm's possession; he also represented to the court that Mellina did not have possession of copies of the two s. In the course of these proceedings, sellers' counsel first asserted that he did not "discuss [with Kraus] what ought to be in the privilege log." Upon further questioning by the judge, counsel acknowledged that he was sure he "discussed the log" with Kraus, and then asserted that what occurred took place a year earlier and he could not "possibly remember" what he discussed with Kraus. In light of the uncertainty about the location of the two s, purchasers were permitted to amend their complaint to add a claim of fraudulent concealment, and discovery was reopened. When re-deposed, Mellina claimed she deleted all sent and received s and, therefore, no longer had copies of the s. Purchasers hired a forensic specialist to search the 2 By this time, Kraus had passed away. 5 A T2

116 hard drive on Kraus's computer and the network at Kraus's former law firm; the s were eventually found on that firm's server back-up tapes. The s were then delivered to the judge, who, after an in camera review, ordered that they be turned over to purchasers. The judge also granted purchasers' motion to impose sanctions on Brach Eichler and Mellina because their failure to preserve the s resulted in additional and unnecessary efforts by purchasers. In her August 27, 2010 order, Judge Brock directed Brach Eichler and Mellina to pay the forensic specialist's $900 fee, as well as reasonable attorneys fees to be quantified after trial. The twelve-day non-jury trial before Judge Kessler took place in October and November As noted earlier, the two s greatly influenced the judge's determination of the issues. The judge quoted extensively from both s. Kraus's summarized for sellers "the negotiated settlement." Mellina's response revealed to Kraus that she had spoken to her husband and that they were "in accord with your summary of the negotiated settlement." Judge Kessler also found Mrs. Mellina's attempt to explain away this was preposterous. She essentially claimed that no agreement was reached, that everything Mr. Kraus wrote [in his ] was incorrect, and that when she spoke to her husband, they were both opposed to the 6 A T2

117 terms contained in [Kraus's ] -- yet she sent her because Mr. Kraus was terminally ill and needed their support. According to Mrs. Mellina, she told Mr. Kraus that she accepted the settlement when in fact she was opposed to it, and she also lied to him about having obtained her husband's approval. Not only is this wholly implausible, but my observations of Mrs. Mellina's demeanor during this testimony added to my skepticism. I likewise did not believe the testimony of Mr. Mellina that he opposed the settlement and told his wife so. Rather, I find that the from Mrs. Mellina accurately reflected the view of the Mellinas on March 4 -- that they were in favor of the settlement and wanted the agreement with the Goldmarks terminated so they could find another buyer and "get on to something else." Whether Mrs. Mellina forgot about her or simply did not expect it to surface (since she had deleted her copy, and Mr. Kraus'[s] copy had disappeared), it appears that she did not anticipate that it would be produced. After trial, Judge Brock considered purchasers' request for the quantification of the fees referred to in the August 27, 2010 order; Brach Eichler and Mellina sought reconsideration. The judge denied reconsideration and awarded purchasers $11,005, which consisted of $10,075 in attorneys fees, $30 in costs, and $900 for the forensic specialist. The judge allocated the award equally, so that Mellina and Brach Eichler were held responsible to pay to purchasers $5, each. 7 A T2

118 Mellina has not appealed, but Brach Eichler has, arguing: 3 I. [SELLERS'] COUNSEL FULLY COMPLIED WITH R. 4:10-2(e)(1) BY ROBERT KRAUS PREPARING A PRIVILEGE LOG FOR INSPECTION BY PLAINTIFFS, WHICH PRECLUDES THE AWARD OF SANCTIONS BECAUSE IT DISCLOSED THE PRIVILEGED MATERIALS IN QUESTION, COMPLIED WITH THE SEPTEMBER 12, 2008 CONSENT ORDER, ALLOWED THE COURT TO PERFORM AN IN CAMERA REVIEW TO DETERMINE IF THE DOCUMENTS SHOULD BE DISCLOSED, AND BECAUSE R. 4:10-2(e) DOES NOT REQUIRE THE COPYING OR MAINTENANCE OF A THIRD-PARTY'S FILE. II. [SELLERS'] COUNSEL FULLY COMPLIED WITH R. 4:18-1(a) BY KRAUS PRODUCING THE PRIVILEGED DOCUMENTS IN HIS POSSESSION, CUSTODY AND/OR CONTROL AS THEY WERE KEPT IN THE ORDINARY COURSE OF BUSINESS, AND [THEY WERE] ORGAN[IZ]ED AND LABELLED BY CATEGORIES IN RESPONSE TO THE SUBPOENA SERVED UPON ROBERT KRAUS. III. NOT ONLY DID [SELLERS'] COUNSEL COMPLY WITH THE COURT RULES, BUT MR. KRAUS CONFIRMED UNDER OATH HIS ENTIRE FILE WAS IN HIS POSSESSION AND THAT HE WOULD KEEP THE FILE SAFEGUARDED IN RESPONSE TO [PUR- CHASERS'] COUNSEL'S REQUEST TO DO SO. IV. NO CASE LAW SUPPORTS THE COURT'S AWARD OF SANCTIONS SINCE [SELLERS'] COUNSEL DID NOTHING DELIBERATELY IMPROPER TO CONDUCT DISCOVERY OF THE S. V. JUDGE BROCK HERSELF FOUND THAT [SELLERS] DID NOTHING IMPROPER OR WRONG THAT RESU[LT]ED IN THE TWO S MISSING, BUT STILL ERRO[NE]OUSLY AND WITHOUT LEGAL BASIS SANCTIONED COUNSEL SIMPLY BECAUSE HE DID NOT COPY MR. KRAUS'[S] FILE. 3 We have omitted Point I, which solely discusses the standard of appellate review, and renumbered the remaining arguments. 8 A T2

119 VI. [PURCHASERS] LOST THE ISSUE OF WHETHER A SETTLEMENT EXISTED AT TRIAL BASED UPON THE FACE OF THE NON-PRIVILEGED DOCUMENTS IN THE RECORD, AND ARE THUS NOT ENTITLED TO ANY SANCTIONS OR FEES DUE TO THEIR WASTEFUL EFFORT PURSUING PRIVILEGED DOCUMENTS TO PROVE A SETTLEMENT THAT NEVER EXISTED. VII. THE TRIAL COURT ERRED IN AWARDING $5, TO [PURCHASERS] AS SANCTIONS SINCE THE ONLY FEES [PURCHASERS] SHOULD HAVE INCURRED IN SUBPOENAING THE TWO EMA[I]LS FROM [KRAUS'S LAW FIRM] WAS ONE HOUR OF THEIR COUNSEL'S TIME TO PREPARE AND SERVE A SUBPOENA, AND $900 FOR THE COMPUTER FORENSIC SPECIALIST TO OBTAIN THE S FROM [KRAUS'S LAW FIRM'S] HARD DRIVE/BACKUP TAPES. We find no merit in these arguments and affirm. The discovery issue that gave rise to the controversy now at hand stems from the assertion of an attorney-client privilege over documents germane to the claims asserted in the trial court. Purchasers asserted, among other things, that the parties reached a settlement prior to the commencement of the suit and that sellers took steps to terminate their contract. Thus, communications between the parties and their attorneys prior to suit regarding the alleged settlement and the positions taken by the parties, through their counsel, were highly relevant. Sellers, however, also had a colorable claim that the communications between them and their attorney were insulated by the attorney-client privilege. Thus, a privilege log was 9 A T2

120 necessary to protect the parties' competing interests in the disposition of those documents and the judge properly directed its preparation. Upon taking the position that relevant material is privileged or subject to protection from discovery, a litigant has the obligation of maintaining and eventually disclosing the material as directed by the court. See, e.g., R. 4:10-2(e)(1). It would make a mockery of our discovery rules to allow a party or its counsel -- after identifying privileged information -- to destroy or carelessly lose or misplace the materials in question. 4 Here, the judge was not required to find that Brach Eichler knowingly allowed the disappearance of the s in order to impose a sanction. The matter was clearly one of interest to the purchasers, and counsel was obligated to protect against what either deliberately or accidentally occurred here. Even if our court rules do not expressly provide a remedy for these particular circumstances, our judges possess the inherent authority to require a party or the party's attorney to compensate a party damaged in this manner. See Lang v. Morgan's Home Equip. Corp., 6 N.J. 333, (1951); Seacoast Builders 4 Sellers' counsel's obligation to preserve those documents pending the court's further direction arose at that moment, if not sooner, see RPC 3.4(a), and was particularly enhanced in this case because one of the parties to those communications was terminally ill. 10 A T2

121 Corp. v. Rutgers, The State Univ., 358 N.J. Super. 524, 549 (App. Div. 2003); Summit Trust Co. v. Baxt, 333 N.J. Super. 439, 450 (App. Div.), certif. denied, 165 N.J. 678 (2000). As Chief Justice Vanderbilt wrote for the Court in Lang, supra, 6 N.J. at 338, our liberal discovery procedures are "essential to any modern judicial system in which the search for truth in aid of justice is paramount and in which concealment and surprise are not to be tolerated." Sanctions to enforce conduct that runs counter to these important principles are "inherent in our courts" and limited "only to the requirement that they be just and reasonable in the circumstances." Id. at After careful consideration, we are satisfied that the imposition of sanctions, as well as their amount, were just, reasonable and well-deserved. We find Brach Eichler's other arguments to be without sufficient merit to warrant discussion in a written opinion. R. 2:11-3(e)(1)(E). Affirmed. 11 A T2

122 New Jersey State Courts Enter the E-Discovery Arena in Earnest; Award Sanctions for E... Page 1 of 2 E-Discovery Law Alert Posted at 9:07 AM on June 29, 2012 by Gibbons P.C. New Jersey State Courts Enter the E-Discovery Arena in Earnest; Award Sanctions for Spoliation On June 18, 2012, an Appellate Court in New Jersey issued Goldmark v. Mellina, which held that asserting the attorney-client privilege does not excuse counsel and parties from their obligation to preserve relevant s or other documents. There, the Court upheld the trial judge s award of $5, in sanctions against a prominent New Jersey law firm because it had failed to timely produce electronic documents, which had temporarily disappeared, even though the lapse was not knowing. Because there were virtually no prior opinions (published or unpublished) addressing e-discovery in this jurisdiction, Goldmark is an important first-step towards providing e-discovery guidance to New Jersey practitioners. At issue in this real estate construction litigation was whether the underlying construction dispute was settled as the purchasers contended or whether the purchasers breached the contract, which was the basis of the sellers claim. Although the Trial Court determined that no settlement agreement had been consummated, it also held that the sellers had both repudiated and breached the contract. In reaching that determination, the judge relied upon and quoted extensively two [March 2008] s between the sellers and their attorney. Where the sellers and their subsequent attorneys, who were ultimately sanctioned erred, was in thwarting the purchasers efforts to obtain those s during discovery. Although the sellers attorneys properly produced a privilege log that listed those two s, they failed to provide them to the judge for an in camera inspection in the context of a discovery motion. As a result, the trial judge allowed discovery to be re-opened, and the sellers testified that the s had long been deleted. After the purchasers hired a forensic specialist to inspect the hard drive of the sellers former attorney and the network of his former firm (as he passed away while the litigation was pending), the s were found on that firm's server back-up tapes. The s were turned over to the judge, examined in camera and ordered to be turned over to the sellers. The judge awarded sanctions against the sellers and their counsel because their failure to preserve the s resulted in additional and unnecessary efforts by purchasers. The judge split the sanctions evenly between the sellers and their attorneys. As noted by the Appellate Division, [i]t would make a mockery of our discovery rules to allow a party or its counsel -- after identifying privileged information -- to destroy or carelessly lose or misplace the materials in question. While the court s opinion in Goldmark does not give the level of practical guidance that numerous federal cases have offered such as the seminal case in e-discovery, Zubulake v. UBS Warburg, LLC, it 3/27/2013

123 New Jersey State Courts Enter the E-Discovery Arena in Earnest; Award Sanctions for E... Page 2 of 2 does represent a first step by the Appellate Division towards accepting some of the general ESI preservation principles outlined in Zubulake and its progeny. Hopefully, New Jersey case law and court rules will soon take the next step and specifically adopt these principles without reservation so that e- discovery obligations are clear. As we discussed in a prior post, New York state courts appear to be moving in that direction as evidenced by the decision in U.S. Bank N.A. v GreenPoint Mtge. Funding, Inc. As the U.S. Bank court held, We are now persuaded that the courts adopting the Zubulake standard are moving discovery, in all contexts, in the proper direction. Kevin W. Weber is an Associate in the Gibbons Business & Commercial Litigation Department. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 3/27/2013

124 PRECEDENTIAL UNITED STATES COURT OF APPEALS FOR THE THIRD CIRCUIT No LAUREEN BULL, Appellant v. UNITED PARCEL SERVICE, INC. On Appeal from the United States District Court for the District of New Jersey (D.C. No cv ) District Judge: The Honorable Dennis M. Cavanaugh ARGUED SEPTEMBER 15, 2011 BEFORE: SLOVITER, SMITH, and NYGAARD, Circuit Judges (Filed: January 4, 2012)

125 David Zatuchni, Esq. [Argued] Zatuchni & Associates 287 South Main Street Lambertville, NJ Counsel for Appellant Michael T. Bissinger, Esq.[Argued] Day Pitney One Jefferson Road Parsippany, NJ Counsel for Appellee NYGAARD, Circuit Judge OPINION OF THE COURT After declaring a mistrial, the District Court dismissed Laureen Bull s state-law employment discrimination case as a sanction for failing to produce originals of certain medical notes requested by United Parcel Service. Bull maintains that the District Court abused its discretion by ordering this sanction. We agree and will reverse and remand for retrial. I. A. Our review is fact-intensive. However, the basic contours of the case can be summarized as follows. Bull, a 2

126 part-time employee since 1986, injured her neck and shoulder on the job in late December Though she promptly reported the incident and requested medical attention the following day, there was a two-week delay before she had access to a company doctor. She also alleges a number of instances in which her supervisors ignored, downplayed and misrepresented her injury to superiors. The company doctor diagnosed her with contusions and strains to her shoulder and neck, and restricted her lifting to twenty-five pounds. She was referred to an orthopedic specialist who lowered the lifting restriction to twenty pounds. She was also given physical therapy for approximately two months. UPS placed her on a temporary work assignment that did not involve lifting, but at the end of the 29-day assignment she stopped working and began receiving workers compensation. On March 29, 2006, the orthopedic specialist opined that though Bull was only 70 percent recovered she had reached maximum medical improvement. The doctor restricted her overhead lifting to 10 pounds, but did not mention other types of lifting. Bull returned to work and presented the specialist s note. She received a new work assignment, but after five days her new supervisor told her that her medical restrictions made it impossible for UPS to continue assigning work to her. UPS advised her to seek permanent disability. B. Bull wished to be reinstated and asked her union representative for help. The representative advised her to get a second opinion from her own doctor. Through her doctor s 3

127 referral, another orthopedic specialist, Dr. Farber, examined her on June 13, Farber gave her a note that said among other things: patient is capable of lifting 50 pounds or more. D.C. No cv-02291, ECF No , 10. The collective bargaining agreement requires that employees be capable of lifting 70 pounds. Bull faxed the note to the union representative who, in turn, faxed it to UPS. UPS, however, found numerous inconsistencies with the note, and told this to the union representative. 1 Bull s union representative then advised Bull to get another note, and to get more information to satisfy UPS s issues. Bull called Dr. Farber s office and requested another note. She then faxed a second note from Dr. Farber s office, dated August 14, UPS also found multiple problems with the second note. 2 These two notes from Dr. Farber s office have become central to this appeal. 1 UPS found that the Farber note, dated June 13, 2006, listed conflicting dates and provided contradictory answers on whether the medical condition was work related. Moreover, the note ambiguously stated that Bull could lift 50 pounds or more. UPS also deemed other portions of the note as illegible and observed that part of the note was cut off. ECF No. 18-1, UPS asserts that the August 14, 2006 Farber note had numerous problems, including: inconsistent dates; a signature differs from the June note; inconsistent answers on the issue of whether the medical condition was work-related; and, an ambiguous instruction that Patient is not able to lift over 70 pounds. The note was also cut off at the bottom. Finally, there were portions of the note that were illegible. ECF No. 18-1,

128 On September 27, 2006, UPS sent a letter to Bull s union representative, saying in part: As you know, we received two notes from Dr. Farber s office regarding Ms. Bull s ability to return to work; both notes (dated June 13, 2006 and August 14, 2006) indicate restricted duty.... The Company also requests that Ms. Bull produce the original notes from Dr. Farber s office due to the fact that the notes received to date are blurry and in some cases illegible. ECF No , 9. The representative contacted Bull, and requested again a new doctor s note and more information. Bull never responded. Instead, she filed a Workers Compensation lawsuit and contacted the Equal Opportunity Employment Commission. She then filed the instant claim in April During discovery, Bull turned over new copies of the Farber notes to UPS in response to their general discovery requests. C. At the March 2010 trial, during Bull s direct examination, her counsel sought to introduce copies of the June 13, 2006 note from Dr. Farber. UPS objected on the basis of best evidence. During the sidebar that followed, the District Court asked Bull s attorney where the original June 13, 2006 note from Dr. Farber was. He responded: we don t 5

129 have - - it doesn t exist any more. All we have is a copy. ECF No. 56-4, 21: He also pointed out that Dr. Farber had authenticated the note. UPS responded that they had documented letters asking for the originals, and that, during this litigation, we have asked for the originals, and we have never seen them. Id. at 23:4-5, 23: The District Court ultimately decided to overrule UPS s objections, concluding that the argument against admission went to the weight of the evidence, rather than its authenticity. Moments later, as Bull s counsel was about to request that the note be admitted into evidence, the District Court interrupted and said to Bull in open court: Well, before we do that: Where s the original of this note? She answered: The original note is in my home.... Id. at 28:6. Surprised by his client s response, Bull s counsel immediately said: Your honor, I understand what she just said. I ve been asking her for the originals since the very beginning when Mr. Bissinger [UPS s counsel] has been asking me for the originals. She just kept telling me that she doesn t have them, she s looked for them but she doesn t - - can t find those notes anymore, they don t exist any more. 3 3 In an affidavit accompanying Bull s response to UPS s motion for sanctions, counsel revisits this statement. He says: When queried as to original Farber medical notes, Ms. Bull indicated that she was unable to locate them and believed 6

130 Id. at 28: A few moments later, after a brief sidebar, Bull s attorney sought to clarify her statement. The following exchange occurred in open court. Q. Laureen, I understood have you looked for the original? Have you actually been able to find the original note? A. I have looked for the original note. Many things going on in my life with the paperwork, and the sale got damage to my apartment. I can try to find this note. they may have been lost due to flooding in her home. ECF No. 58-1, 2. He, then, goes on to say: The Court should be aware that this is undersigned counsel s recollection of what transpired in 2007, but not Ms. Bull s. Ms. Bull has recently advised her counsel that she does not recall being asked by undersigned for the original Farber medical notes and that she would have conducted a search for these documents back in 2007 is [sic] she had been required to do so. Id. Moreover, at oral argument, Bull s counsel stated emphatically that Bull did not lie about possessing these documents. Therefore, without a finding by the District Court, or indeed any testimony from Bull on this topic, we can neither speculate upon what Bull understood or intended with regard to the production of the Farber note originals, nor whether her responses were credible. 7

131 Q. Have I asked you to look for the original? The Court: Well, wait a minute. Wait a minute. In response to my question, you said the original was at your home. There was no hesitation. The Witness: Correct. The Court: Is the original of that note at your home? The Witness: It should be. The Court: As your attorney stated, this case has been going on for years. There were years of discovery. This note was asked for. Is there some reason - - have you made a search for it previously? The Witness: No, sir. Id. at 30:3-21. In the sidebar that followed, the District Court questioned counsel for Bull and UPS on the appropriate response to this revelation. UPS advocated excluding the originals and any copies other than those originally faxed to them by the union, specifically requesting that the jury be permitted to see only the document presented to UPS. The District Court brushed aside UPS s suggestions and instead 8

132 decided to declare a mistrial and invited UPS to file a motion for sanctions. Bull sent the original June 13 and August 14 notes from Dr. Farber to the District Court five days after the mistrial. UPS filed a motion for sanctions, and in October 2010 the District Court ordered the case dismissed with prejudice to sanction Bull s conduct. This appeal followed. II. A. The District Court ruled that Bull s failure to produce originals of the medical notes was spoliation and it invoked its inherent authority to order the case dismissed with prejudice as a sanction. See Schmid v. Milwakee Elec. Tool Corp., 13 F.3d 76, 78 (3d Cir. 1994). 4 Two questions arise: first, generally, whether the production of facsimiles and copies in place of the originals can be considered spoliation; and, second, whether Bull s specific acts or omissions in this case provided a reasonable basis to rule that she spoliated evidence, warranting dismissal with prejudice. Sanctions for spoliation of evidence are reviewed for an abuse of discretion. In re Hechinger Inv. Co. of Delaware, Inc., 489 F.3d 568, 574 (3d Cir. 2007). B. 4 We have jurisdiction over this appeal pursuant to 28 U.S.C

133 We first look at whether, generally, failing to produce original documents can be considered spoliation. Spoliation is usually referenced in instances where evidence has been altered or destroyed. See Micron Technology, Inc. v. Rambus Inc., 645 F.3d 1311, 1320 (Fed. Cir. 2011) (Spoliation occurs when evidence is destroyed or altered, or when a party fails to preserve evidence in instances where litigation is pending or reasonably foreseeable.). We have described it more broadly. For instance, in a context not involving a mistrial or dismissal, but while discussing a district court s decision to instruct the jury with an adverse inference for spoliation, we said the following: When the contents of a document are relevant to an issue in a case, the trier of fact generally may receive the fact of the document's nonproduction or destruction as evidence that the party that has prevented production did so out of the well-founded fear that the contents would harm him. Gumbs v. International Harvester, Inc., 718 F.2d 88, 96 (3d Cir.1983); United States v. Cherkasky Meat Co., 259 F.2d 89 (3d Cir. 1958). Brewer v. Quaker State Oil Refining Corp., 72 F.3d 326, 334 (3d Cir. 1995) (emphasis added). Indeed, a party s failure to produce a document can have the same practical effect as destroying it and we reaffirm that, under certain circumstances, nonproduction of evidence is rightfully characterized as spoliation. 10

134 Here, though Bull failed to produce the originals, she did provide UPS with facsimiles and photocopies of the documents. The District Court concluded that, in spite of producing copies, Bull s conduct was spoliation because, although UPS had not requested a forensic analysis of the originals, such tests might have yielded information directly relevant to the question of whether the documents were authentic. Such an analysis might not be possible on just copies. We are persuaded that, in some instances, original documents might yield relevant evidence that is simply not available from copies. As a result, we conclude with the District Court that theoretically producing copies in instances where the originals have been requested may constitute spoliation if it would prevent discovering critical information. With that said, we reach a very different conclusion when we turn to the question of whether spoliation occurred here. C. Spoliation occurs where: the evidence was in the party s control; the evidence is relevant to the claims or defenses in the case; there has been actual suppression or withholding of evidence; and, the duty to preserve the evidence was reasonably foreseeable to the party. Id. 5 Two 5 The District Court essentially merged its deliberation on spoliation with its analysis of spoliation sanctions. Though there is some overlap between the two, there are distinctive elements of each. We are focusing upon the spoliation analysis first. However, we note that the sanctions analysis includes the following factors: (1) the degree of fault of the 11

135 of these factors pose little controversy in this case. There is no question that the documents were in Bull s control since she admitted this on the stand and produced the documents five days after the District Court declared the mistrial. We also have no doubt that the Farber notes were relevant to both the claims and defenses of this case since the notes discuss a central issue in this case: Bull s capacity to work. The remaining factors, though, merit closer attention. D. With regard to whether Bull intentionally withheld the original notes from Dr. Farber s office, the District Court summarized the facts upon which it made its decision: Although Plaintiff would have this Court believe that originals of the documents in question were never requested during discovery, that contention is disingenuous in light of the record, which documents that at least as early as September 27, 2006, Defendant UPS had requested that Ms. Bull produce the original notes from Dr. Farber s office.... party who altered or destroyed the evidence; (2) the degree of prejudice suffered by the opposing party; and (3) whether there is a lesser sanction that will avoid substantial unfairness to the opposing party, and, where the offending party is seriously at fault, will serve to deter such conduct by others in the future. Schmid, 13 F.3d at

136 The District Court also said despite numerous requests, both formal and informal, to produce the disputed documents, Plaintiff never responded. As a result, the District Court concluded that: Common sense and reasonableness in light of this record, and the numerous and repeated references to the fact that Plaintiff never accounted for the absence of the originals, point to nothing less than purposeful withholding of critical evidence that can not [sic] be countenanced or overlooked by this Court. Although a District Court has discretion to draw inferences from the record on a party s intent, it strays beyond the bounds of its discretion when, as here, there is no factual basis to do so. The central problem is that the District Court accepted, without any critical examination, misrepresentations of the record promulgated by UPS. We start with the District Court s foundational premise: UPS hounded Bull for the originals over the course of years, but got no response. 6 To the contrary, we count a 6 UPS misstated as fact the following to the District Court: We have documented letters asking for the originals. ECF No. 56-4, 23:4-5. I have never been able to get the originals because they won t give them to me.... Id. at 24:7-9. I sent the trial subpoena. It was very clear I want the originals 13

137 total of only two requests by UPS for the original documents in the entire span of this case dating back to pre-litigation communications and neither of these inquiries were discovery requests. 7 The first communication was UPS s September 27, 2006, pre-litigation letter requesting the originals. This letter was addressed to Bull s union representative, not Bull. UPS dismisses the distinction because it says they have sworn testimony from the union representative that he told Bull that UPS wanted the originals. The representative made no such statement. He only requested that Bull provide new notes, new information, or maybe better or clearer copies. 8 The of the notes. Why - - and we have Rule 26, we have all of these things, and I still don t have it, and now we re three days into trial. Id. at 37: I don t know what to do here, Judge. We go through three days of trial. I asked for the originals notes. I sent a valid subpoena. She apparently has had them for three years, four years. Id. at 45:23-46:1. 7 UPS s request for production of documents in this case asked for copies of the documents described herein.... It did not request originals of these documents. 8 The full text of this portion of the Cherney deposition is as follows: Q. Did you ask or attempt to get better copies of the notes for UPS in response to [UPS s request]? A. Yes, I did. 14

138 union representative knew that UPS wanted the originals, but we have no evidence that Bull knew, and we do not have any basis to impute the union representative s knowledge of UPS s request to Bull. It does not end there, however. UPS also repeatedly stresses that, from the beginning of discovery, Bull had the September 27 letter from UPS. The implication is that Bull knew about this pre-litigation request for the originals at least by the early phase of this litigation. Although it may be Q. Do you recall what you did response to that? A. Can you repeat the question? Q. Sure. I asked you what did you do to get better copies of the notes or words to that effect. You said yes you did. I m asking did you go to Ms. Bull and ask her to get something, did you go to Dr. Farber? What did you do? A. I had conversations with Laureen in effect that there was issues with ambiguities with the note and that to further clarify it, to get notes that - - a note that would release her back to full duty. Q. Did you talk to her about the fact that the notes in some cases were - - or the company felt that there was some illegible parts of the notes or that they were blurry and we needed clearer copies? A. Yes. Robert Cherney Deposition, 57:14-58:12 (emphasis added). 15

139 technically accurate to say that the letter was in Bull s constructive possession, UPS fails to note that it produced the letter in response to Bull s general discovery request. Producing such a letter to Bull in response to her general requests for documents is not the same thing as presenting Bull with a specific discovery request for the original notes from Dr. Farber s office. Moreover, we would expect that had UPS intended this letter to result in Bull turning over the original notes in question it would have made at least some attempt to pursue their production when they did not materialize. Yet, UPS never raised the nonproduction of the originals in a motion to compel, or in any other communication. As a result, any assertion that the prelitigation letter of September 27, 2006 to the union representative put Bull on notice of UPS s request for originals fails. UPS s second request for the originals came on March 3, 2010 five days before trial commenced in the form of an . Though UPS refers to it as a trial subpoena, the District Court noted that it did not meet the requirements of Fed. R. Civ. P. 45. We agree. Indeed, there is no evidence that the document that UPS calls a subpoena was issued by the court. As such, given that discovery was closed and that the is not a subpoena, we cannot treat this communication as anything more than a reminder to Bull s counsel that, at trial, the best evidence rule would apply to the notes from Dr. Farber s office. Moreover, the record does not contain any evidence that Bull was ever aware of this . Therefore, like the September 27, 2006 letter, the fails to provide support for an inference that Bull knew of UPS s request and still inadvertently or intentionally withheld the original notes from them. 16

140 We conclude from this that the District Court was flatly wrong when it declared that UPS made multiple requests of Bull for the original notes. We must also conclude that any inference of Bull s intent to obstinately withhold the originals also fails for a lack of any factual foundation that she actually knew UPS wanted them. The District Court also makes much of UPS s challenge to the authenticity of the Farber notes in its motion for summary judgment, in a pretrial motion, and during its opening argument. It infers that Bull should have understood from UPS s challenge to the authenticity of the notes that UPS wanted and needed the originals. However, these, alone, imply no such thing. It is largely counsel s role to assess which evidence best serves the case. It was up to UPS s counsel to press for the originals if it needed them, and it was up to Bull s counsel to diligently investigate all possible sources for the originals if he needed them to rebut the argument. If neither counsel pressed Bull for the originals to make their case, we cannot impute to Bull a sophisticated knowledge of trial strategy. Indeed, it is obvious from UPS s arguments at sidebar that it was content to try the case with the copies. Moreover, UPS s challenges to the notes authenticity were largely confined to information that was readily available on the note copies. Bull, and her counsel for that matter, had no reason to place a high priority on locating the original notes. 9 Therefore, without anything else, Bull could not be deemed to be on notice that UPS wanted or needed the originals merely from their authenticity argument. 9 See supra notes 1 and 2. 17

141 Moving beyond UPS s requests for originals, the District Court references a statement made by Bull s counsel at trial as a basis to infer her intent. In the instant after Bull revealed that the original notes from Dr. Farber s office were likely in her possession, her counsel said: Your honor, I understand what she just said. I ve been asking her for the originals since the very beginning when Mr. Bissinger [UPS s counsel] has been asking me for the originals. She just kept telling me that she doesn t have them, she s looked for them but she doesn t - - can t find those notes anymore, they don t exist any more [sic]. ECF No. 56-4, 28: Bull disputes any memory of this request, and her attorney has moved away from saying that he made multiple requests of Bull. 10 Yet, even if we accept counsel s version of events as told at trial, we do not see how this exchange between counsel and client can be used to ground a conclusion that Bull committed, as the District Court said, flagrant violations of the most basic code of judicial propriety and honest dealing. Absent the backdrop of UPS s unsubstantiated portrayal of Bull as an obstinate stonewaller, this conclusion is dramatically overblown. 10 See supra note 3. 18

142 This is not to say that we take Bull s failure to turn over the originals to her counsel lightly. To be sure, Bull put her counsel in a terrible position. Her counsel represented in a side-bar to the District Court moments before her opencourt revelation that we don t have - - it [the Farber note original] doesn t exist any more. All we have is a copy. ECF No. 56-4, 21: He also unwittingly violated the best evidence rule. Fed. R. Evid There is no question that the late revelation concerning the originals had impacts. The key issue, however, is whether the discrepancy between her statements (if in fact there is a discrepancy) was an intentional misrepresentation or as her counsel insists inadvertence. The record does not answer this question. To be sure, the District Court was remiss in its failure to examine this issue more closely and in its failure to make findings of fact on inadvertence and misrepresentation. Yet, mindful that it is UPS s burden to prove Bull s bad faith conduct, there are strong reasons favoring a presumption of inadvertence. Counsel attempted to examine Bull on the record before the mistrial was declared to get her explanation for her late revelation, but the District Court inexplicably would not allow it. 11 There is no evidence that either Bull or her 11 Bull s counsel said at sidebar: Bull s Counsel: I would like to pursue [the reason for Bull not searching for the Farber notes] with Ms. - - and see ifthere can be a clarification with her that you can ask her questions and she can explain. You asked her a question and she gave you a one-word response, okay? Maybe there s more to it that would clarify. 19

143 counsel wished to be anything less than candid about this topic. They were simply prevented from doing so by the District Court who passed up the opportunity to take testimony and make findings of fact. Moreover, Bull s counsel stated emphatically in oral argument that Bull did not lie to him about the originals, seemingly ruling out bad faith. Since the entire case for Bull s bad faith seems to rest on counsel s words, we must take all of his words on the topic into account, including these. Moreover, even if Bull did obfuscate on the request from her attorney (a conclusion that ignores Bull s counsel s statements, and the later representations made by Bull to her counsel), there is still no evidence that Bull knew UPS wanted the originals, since she had already produced copies. In light of UPS s utter failure to produce any evidence of its own to ground the conclusion that Bull acted in bad faith, all of this supports abiding by a presumption of inadvertence. We must keep in mind that at issue is whether the District Court was reasonable in ruling that Bull s late revelation about the existence of the originals was a flagrant violation and that there is: [N]o plausible explanation other than Plaintiff s misconduct that explains the withholding of the original Farber notes. Despite The Court: No, she gave me a one-word response. Yes, I have it, it s at home, and I ve never even tried to find it. ECF No. 56-4, 42:

144 numerous requests both formal and informal, to produce the disputed documents, Plaintiff never responded. On this point, our review of the record has left us with a very different set of conclusions from the District Court. First, we do not see any basis for the District Court s characterization of UPS as persistently hounding Bull for the originals. Indeed, it did not. Second, as a result of this, there is not one instance in which we can verify that Bull actually knew that UPS wanted the original notes. Third, lacking such evidence there is no basis to characterize Bull as one who lied or obfuscated to prevail in her attempt to intentionally withhold the originals. We conclude from all of this that the District Court abused its discretion in ruling that, within its spoliation analysis, Bull intentionally withheld the original documents from UPS. E. We turn, finally, to the issue of whether Bull had a foreseeable duty to preserve and turn over the originals of the notes from Dr. Farber s office. The Court of Appeals for the Federal Circuit said that the question of reasonable foreseeability is a flexible fact-specific standard that allows a district court to exercise the discretion necessary to confront the myriad factual situations inherent in the spoliation inquiry. Micron Technology, Inc., 645 F.3d at Here, the District Court ruled that it is hard to imagine evidence that could have been more... foreseeable. We conclude that the District Court was within its discretion when it 21

145 determined that there was a foreseeable duty here, but we have some reservations. The record is scattered with support for the District Court s conclusion. 1.) Bull initiated both an EEOC proceeding and the instant litigation within a year of UPS s employment action. 2.) Bull was aware that she and UPS fundamentally disagreed on the meaning of the notes from Dr. Farber s office. 3.) Though she denies it, her counsel says that on at least one occasion he asked her for the originals. 4.) Both UPS s motion for summary judgment and a pre-trial motion mention the fact that UPS had never seen the originals. UPS also raised this fact in its opening argument. 5.) Finally, Bull had a duty under Rule 1002 of the Federal Rules of Evidence to produce the original documents before the notes could be introduced into evidence at trial. With that said, the duty issue is not so clear cut. We lack evidence that counsel for either party made any appreciable effort to induce Bull to search for and produce the original Farber notes. Moreover, it is clear that UPS s challenge to the authenticity of the notes rested upon information that was available on the copies of the documents. This leads us to wonder whether a lay-person like Bull, ignorant of the Rules of Evidence, might have concluded that copies of the notes were sufficient. Nonetheless, the question before us is not whether a particular scenario is possible, but rather whether the duty was objectively foreseeable. Under the circumstances of this case, we will assume that the District Court acted within its discretion in determining that the litigation and the future 22

146 need to provide access to the original notes from Dr. Farber s office were foreseeable. 12 F. 12 This highlights a growing concern for us that is not directly implicated in this case. As electronic document technology progresses, the concept of an original document is becoming more abstract. Moving from the more easily distinguishable photocopy or facsimile to documents created, transmitted and stored in an electronic form means that it will be increasingly difficult to ascertain where the boundary of an objectively reasonable duty to preserve such documents lies. There are and increasingly will be circumstances in which the foreseeability of a duty to preserve the information contained in a particular document is distinguishable under an objective analysis from the need to preserve that information in its original form or format. Indeed, arriving at a common understanding of what an original is in this context is challenging enough. Although it does, and always will rest with the courts to preserve the distinction between an objectively foreseeable duty and actual knowledge of such a duty, there is a concomitant obligation that counsel must assume to clearly and precisely articulate the need for parties to search for, maintain, and where necessary produce original or source documents. This case gives us one more opportunity to highlight our position that clarity in communications from counsel that establish a record of a party s actual knowledge of this duty will ensure that this technology-driven issue does not consume an unduly large portion of the court s attention in future litigation. 23

147 In summarizing the Brewer spoliation factors, we conclude that the District Court was within its discretion in determining that Bull had the original notes from Dr. Farber s office in her possession, that these documents were relevant to the case s claims and defenses, and we will assume that she had a reasonably foreseeable duty to preserve and when requested turn-over these documents. Yet, we conclude that the District Court abused its discretion in determining that Bull intentionally withheld these documents from UPS. In Brewer we discussed the connection between a finding of sanctionable spoliation and a ruling on bad faith, stating the following: For the [spoliation] rule to apply... it must appear that there has been an actual suppression or withholding of the evidence. No unfavorable inference arises when the circumstances indicate that the document or article in question has been lost or accidentally destroyed, or where the failure to produce it is otherwise properly accounted for. See generally 31A C.J.S. Evidence 156(2); 29 Am.Jur.2d Evidence 177 ( Such a presumption or inference arises, however, only when the spoliation or destruction [of evidence] was intentional, and indicates fraud and a desire to suppress the truth, 24

148 and it does not arise where the destruction was a matter of routine with no fraudulent intent. ). Brewer, 72 F.3d at 334 (emphasis added). Therefore, a finding of bad faith is pivotal to a spoliation determination. This only makes sense, since spoliation of documents that are merely withheld, but not destroyed, requires evidence that the documents are actually withheld, rather than for instance misplaced. Withholding requires intent. 13 As a result, we must be convinced that the District Court, on sufficient evidence, found that Bull intended to actually withhold the original documents from UPS before we can conclude that sanctionable spoliation occurred. There is no such finding of record here. As we have concluded that the District Court abused its discretion in ruling that Bull acted in bad faith, we must rule that the District Court abused its discretion in determining that Bull committed sanctionable spoliation. Accordingly, the District Court s sanction of dismissal with prejudice, which is grounded in the conclusion that Bull spoliated evidence, is also an abuse of discretion. III. 13 We note that the verb withhold is defined as To keep from doing something, to keep in check or under restraint; to hold back, restrain. Oxford English Dictionary. hhold#eid, last viewed December 5, It inherently expresses an intentional act. 25

149 A. Our decision to reverse the District Court s ruling on spoliation eliminates the need for a specific review of the District Court s sanction of a dismissal with prejudice, since it arose from its determination that Bull spoliated evidence. Yet, the District Court s rationale for the sanction contained in its opinion differs markedly from its analysis when it granted the mistrial and invited a motion for sanctions. When it granted the mistrial, the District Court based its decision on a perceived discovery violation that it characterized as spoliation. In its opinion it retreats from that basis and instead relies on its inherent power to sanction. The inherent power of the District Court to sanction parties conduct is, of course, not limited to instances of spoliation. Therefore, in an abundance of caution, we will review the record to ascertain whether the circumstances of this case, generally construed, provide any basis on which the District Court could substantiate dismissing the case with prejudice. Generally, [w]hile we defer to the District Court's discretion, dismissal with prejudice is only appropriate in limited circumstances and doubts should be resolved in favor of reaching a decision on the merits. Emerson v. Thiel College, 296 F.3d 184, 190 (3d Cir. 2002). Dismissals with prejudice are drastic sanctions. Poulis v. State Farm Fire and Cas. Co., 747 F.2d 863, 867 (3d Cir. 1984). Moreover, district courts ordinarily balance six factors in assessing the propriety of an involuntary dismissal with prejudice: (1) the party's personal responsibility; (2) the prejudice to the adversary; (3) a history of dilatoriness; (4) willfulness or bad faith; (5) the availability of alternative sanctions; and (6) the merit of the claim or defense. Doe v. Megless, 654 F.3d 26

150 404, 411 (3d Cir. 2011) (citing Poulis, 747 F.2d at 868). As the District Court acknowledged, some of the Poulis factors duplicate the spoliation analysis. Nonetheless, we will review each one separately. B. We begin by looking at Bull s personal responsibility for withholding these documents. As we noted earlier, the District Court concluded that there is: [N]o plausible explanation other than Plaintiff s misconduct that explains the withholding of the original Farber notes. Despite numerous requests, both formal and informal, to produce the disputed documents, Plaintiff never responded. As we already detailed, however, the District Court s inference that Bull intentionally withheld documents is not grounded in the record. Although Rule 1002 placed responsibility for producing the originals with Bull and her counsel, negligence remains a reasonable explanation for everything that happened. The same can also be said for her possible misrepresentation of the documents whereabouts to her attorney. The record simply does not have evidence, nor a factual finding supporting bad faith intent. Moreover, had the District Court found bad faith conduct on this evidence, it would have clearly erred. This leads us to conclude that, though the personal responsibility factor of the Poulis analysis must weigh against Bull because of the duty she had 27

151 under Rule 1002, this factor cannot be given great weight because there is neither evidence of, nor a finding that she intended to withhold the documents. C. The District Court concluded that the next factor, prejudice, also weighed heavily in favor of dismissal. It said: Had Defendant been able to prove that the document was tampered with, that would obviously have had significant bearing on the case. The fact that [UPS] could not conduct forensic analysis on the original was certainly prejudicial, even if the outcome of such an analysis remains unknown. It also ruled that this, combined with the costs of trial preparation and UPS s inability to account for the originals in the development of its trial strategy, resulted in significant prejudice. Examples of prejudice are the irretrievable loss of evidence, the inevitable dimming of witnesses' memories, or the excessive and possibly irremediable burdens or costs imposed on the opposing party. Scarborough v. Eubanks, 747 F.2d 871, 876 (3d Cir. 1984). We have also said: prejudice for the purpose of Poulis analysis does not mean irremediable harm, the burden imposed by impeding a party's ability to prepare effectively a full and complete trial 28

152 strategy is sufficiently prejudicial. Ware v. Rodale Press, Inc., 322 F.3d 218, 222 (3d Cir. 2003). Certainly, the lack of access, coupled with UPS s unreimburseable costs of preparing for trial, and the disclosure of UPS s trial strategy all support the District Court s use of its discretion in ruling that UPS suffered prejudice. For this reason, we conclude that the District Court did not abuse its discretion here. Again, however, we are dubious about the District Court s characterization of the prejudice as severe and irreparable, since we do not know the probative value of the originals (as compared to the copies) and are puzzled by UPS s lackadaisical effort in obtaining them given how critical they now say they were to their case. D. With regard to dilatoriness, the District Court ruled that it weighs extremely against Plaintiff. Yet, as we addressed above, there is no evidence that Bull was ever aware that UPS specifically requested the Farber note originals. Moreover, the first request that UPS made for the documents during the litigation was literally on the eve of trial. As a result, we do not find any reasonable basis for the conclusion that Bull was dilatory as to the production of the originals to UPS. With regard to her attorney, we must rely on counsel s representations that the delay between his own request for the original documents and her production of them five days after mistrial was due to inadvertence or misunderstanding. On 29

153 this basis, we cannot characterize her conduct as dilatory because there is no evidence of a willful delay. E. The fourth factor, bad faith, already has been discussed in detail above. Suffice to say, in spite of the District Court s strong, negative characterization of Bull on this element, we conclude that there is not a reasonable basis in the record to conclude that she willfully withheld the documents or that her conduct was in bad faith. F. With regard to the availability of lesser sanctions, the District Court said that it had considered them long and seriously, but, ultimately, concluded that anything less than dismissal with prejudice would condone flagrant violations of the most basic code of judicial propriety and honest dealing. It went on to say: [a] sanction other than dismissal would be insufficient to remedy the prejudice caused by Plaintiff s intentional withholding of evidence and failure to provide appropriate notice to Defendant. Finally, it concluded that there is no way around the fact that Plaintiff s conduct has irreparably prejudiced Defendant. There are numerous problems with this conclusion. As we already have said, the record does not support a ruling that Bull intentionally withheld documents, nor that she was conducting herself dishonestly in this litigation. Moreover, any prejudice arising from UPS s lack of access was remediable because UPS could have obtained the originals five days after the mistrial was declared. 30

154 Second, before the District Court declared a mistrial, UPS originally argued against a mistrial and strongly advocated that the District Court remedy the situation by barring Bull from introducing the originals into evidence. 14 Unlike Ware, where the exclusion of evidence necessitated a dismissal of the claim, the originals comprised only a portion of the evidence and UPS was at least originally content to proceed with trial using the copies of the notes that UPS had always had in its possession. Finally, though the District Court rejected an adverse inference instruction as an appropriate remedy, it gave no solid reason for this. In light of the fact that such an instruction would have bolstered the very portion of UPS s case that it says suffered because it lacked the originals, we fail to see how an adverse inference though itself a severe sanction would not have been preferable to a dismissal with prejudice. Given that dismissal with prejudice is a remedy of last resort, we are puzzled by the District Court s decision. For all of these reasons, we conclude that justification for the dismissal was overblown, and that sanctions other than 14 UPS made the following statements at sidebar, following the discovery of the Farber originals. I m fine with this jury. I don t know that we need a mistrial, Your Honor. But using this note is - - is clearly - - I don t know what to do. ECF No. 56-4, 34: I still say, Judge, the appropriate remedy is to let the jury rely upon what UPS relied upon. Id. at 38:1-2. I think the appropriate remedy again is to allow the trial to proceed with the faxed versions of the notes. Id. at 42:25-43:1. 31

155 dismissal with prejudice were available to adequately address any impacts suffered by UPS and the judicial process. The District Court abused its discretion in concluding that dismissal with prejudice was the only appropriate remedy. G. Finally, in assessing the sixth Poulis factor, the District Court concluded that the merits of Bull s claim are utterly lacking. Yet, the District Court appears to have misunderstood the subject of this portion of the analysis to be Bull s defense against sanctions. Poulis directs the District Court to examine the merits of the underlying claim: which, in this case, would be Bull s employment discrimination claim. Poulis, 747 F.2d at Moreover, even if the discrimination claim was judged to be meritless, Bull also advanced a second claim of workplace harassment that rested on a factual foundation that is distinct from the Farber notes. For all of these reasons, we conclude that the District Court abused its discretion in concluding that this factor weighed in favor of dismissal because it did not even address the merits of Bull s claim. H. To summarize, then, we conclude that two of the Poulis factors personal responsibility and prejudice weigh in favor of dismissal and four factors dilatoriness, bad faith, availability of lesser sanctions, and the merits of the underlying claim weigh against a dismissal. We are left with the question of whether the District Court s decision to dismiss the case with prejudice could be regarded as within its discretion when it is based primarily upon the prejudice 32

156 UPS suffered. We conclude that it was an abuse of discretion. Earlier, we referenced a statement in dicta from Scarborough in which we provided a list of the types of events that would support a ruling that a party s prejudice alone should result in a dismissal. In that portion of the opinion, we said: If there has been true prejudice to a party by its adversary s failure to file a timely or adequate pleading, discovery response, or pretrial statement, that factor would bear substantial weight in support of a dismissal or default judgment. Examples of such prejudice are the irretrievable loss of evidence, the inevitable dimming of witnesses' memories, or the excessive and possibly irremediable burdens or costs imposed on the opposing party. Scarborough, 747 F.2d at 876. We acknowledge that this list is exemplary and not exhaustive. Yet, it points to the scenario that we view as typical in a dismissal: the non-responsible party s case is severely impaired because it lacked the information that was not produced. Although UPS had every right to receive the original Farber notes, there is nothing in the record to even suggest that it asked for them in discovery, or that its case was crippled because it lacked them. Moreover, UPS itself suggested early in the deliberations, that 33

157 sanctions short of mistrial (and therefore short of dismissal) were sufficient to address its prejudice. We conclude from all of this that UPS s prejudice is not sufficiently weighty to support the District Court s sanction of dismissal with prejudice. Moreover, since all sanctions originate from the realm of equity, we note that UPS s representations to the District Court, and this Court, were less than candid. It is fair to say that UPS expended no small effort in obfuscating to the District Court and this Court the details of its requests for the originals. We acknowledge that it was the District Court s responsibility to plumb the record to learn that UPS did not, as it repeatedly asserted, hound Bull for the originals. Yet, UPS s counsel crossed a line between effective advocacy and its duty as an officer of the court to accurately present the record, and in so doing it encouraged the District Court s misunderstanding of the record Based upon the following statement at sidebar during trial, the District Court plainly was unaware of the fact that UPS never asked Bull directly for the notes, and that there is no evidence that she was aware of any such request made to anyone else; yet, UPS never made any effort to correct the misunderstanding. So all they did was in my view what would be a reasonable request. We find out she wouldn t give those [Farber notes] to UPS, she wouldn t give them to the people that are the union leaders that are supposed to be assisting her. Then all discussions stop, a lawsuit starts, and then all of a sudden, after three years of discovery, in the third day of trial, we find out that these may exist but nobody ever looked for them. Now come on....this is egregious. ECF No. 56-4, 49:

158 We conclude that, apart from the merits of the appeal, without the benefit of clean hands here, UPS should not be the beneficiary of a sanction that we are, under most circumstances, already loathe to affirm. IV. For all of these reasons, we conclude that the District Court abused its discretion in ordering that Bull be sanctioned by dismissing her case with prejudice. Therefore, we will reverse the order of the District Court and remand the cause for a re-trial. 35

159 NOT FOR PUBLICATION UNITED STATES DISTRICT COURT DISTRICT OF NEW JERSEY STEPHEN D ONOFRIO, Civil Action No (AET) Plaintiff, v. MEMORANDUM OPINION BOROUGH OF SEASIDE PARK, et al., Defendants. BONGIOVANNI, Magistrate Judge As the parties are aware, currently pending before the Court are several formal motions as well as a number of informally briefed issues. In this Memorandum Opinion, the Court 1 addresses only the Borough Defendants motion for discovery to reclaim inadvertently produced privileged documents pursuant to FED.R.CIV.P. ( Rule ) 26(b)(5)(B) ( motion for discovery ) [Docket Entry No. 134]. Plaintiff Stephen D Onofrio ( Plaintiff ) opposes the Borough Defendants motion. The Court has fully reviewed all arguments made in support of and in opposition to the Borough Defendants motion. The Court considers the Borough Defendants The term the Borough Defendants refers collectively to: the Borough of Seaside Park; 1 Former Mayor Robert W. Matthies; Mayor Thomas E. Connors; the Seaside Park Borough Council; Susan Maday; Maryanne Palmisano; John Jack Moyse; Benjamin J. Kaiser; Sharon Pratico; Nancy Koury; James Jablonski; Robert Brennan; John Coughlin; Seaside Park Police Department; William A. Beining, III; Edward C. Dickson; Daniel Fitzgerald; Lt. Francis Murphy Larkin; James Leone; Brian McKay; Stephen Shadiack; Brian Jankowski; Seaside Park Municipal Planning Board; Faith Liguori; Kathleen Hughes; Anthony DiCaro; Robert Bellantoni; Seaside Park Municipal Zoning Board; Andrew Sbordone; Martin Wilk, Jr.; Kenneth Deshay; Michael Giuliano; Francis Losey; Michael Tierny; Raymond Sites; Geoffrey N. Schwartz; Seaside Park Code Enforcement Department; James Anderson; Robert Nora; Rejean Laliberte; Gary Swirczynski; Richard Barbarise; and Charles Hollins.

160 motion without oral argument pursuant to Rule 78. For the reasons set forth more fully below, the Borough Defednants motion for discovery is DENIED. I. Background The parties and the Court are all familiar with the facts underlying this litigation. As such, they are not restated at length herein. Instead, the Court focuses mainly on the facts relevant to the pending motion for discovery. Plaintiff is the former owner and operator of an establishment located on the boardwalk in the Borough of Seaside Park known as the SawMill Café (the SawMill ). In this matter, Plaintiff has sued numerous individuals and entities associated with the Borough of Seaside 2 Park. Essentially Plaintiff claims that these individuals and entities inappropriately engaged in 2 Following is a complete list of all of the defendants initially sued in this matter. As noted, two have since been terminated from this case: (1) the Borough of Seaside Park; (2) Former Mayor Robert W. Matthies; (3) Mayor Thomas E. Connors; (4) the Seaside Park Borough Council; (5) Susan Maday (Borough Council Member); (6) Maryanne Palmisano (Borough Council Member); (7) John Jack Moyse (Borough Council Member); (8) Robert Martucci (Borough Council Member); (9) Benjamin J. Kaiser (Borough Council Member); (10) Sharon Pratico (Borough Council Member); (11) Nancy Koury (Borough Council Member); (12) James Jablonski (Borough Council Member); (13) Frank Fritz McHugh; (14) Robert Brennan (Borough Council Member); (15) John Does 1-20 (Seaside Park Council Members); (16) Jay Delaney, Jr. (Former Clerk and Borough Administrator); (17) John Coughlin (Former Acting Clerk and Borough Administrator); (18) Julie Keizer (Clerk and Borough Administrator); (19) CMX Engineering, Inc.; (20) Robert K. Forsyth (P.E.); (21) John Maczuga (P.E.); (22) John Does (Seaside Park Council, Consultants, Agents, Representatives, Employees and/or Attorneys); (23) Seaside Park Police Department; (24) William A. Beining, III (Former Chief); (25) Edward C. Dickson (Lt. and/or Chief); (26) Alex Condos (Sergeant); (27) James Citta (Former Sergeant); (28) James Boag (Sergeant); (29) Daniel Fitzgerald (Patrolman); (30) Lt. Francis Murphy Larkin; (31) James Leone (Patrolman); (32) Brian McKay (Sergeant); (33) Stephen Shadiack (Patrolman); (34) Brian Jankowski (Patrolman); (35) John Does (Members of Seaside Park Police Department); (36) Seaside Park Municipal Planning Board; (37) Joseph J. Stack (Planning Board Chairman); (38) John A. Krayesky (Planning Board Vice- Chairman) (Terminated 4/23/2010); (39) Donald Kadlac (Planning Board Member) (Terminated 5/27/2010); (40) Faith Liguori (Planning Board Member); (41) Kathleen Hughes (Planning Board Member); (42) Patricia Degutis (Planning Board Member); (43) Anthony DiCaro 2

161 tortious, fraudulent and extortionate conduct which violated Plaintiff s right to control and operate the SawMill. Plaintiff specifically asserts violations of the Equal Protection and Due th st Process Clauses of the 14 Amendment, the 1 Amendment, 42 U.S.C of the Civil Rights Act; 42 U.S.C of the Civil Rights Act; the Racketeer Influenced and Corrupt Organizations Act ( RICO ), 18 U.S.C et seq.; and N.J.S.A. 2C:41-1 et seq. ( New Jersey RICO ), which is the New Jersey statute that prohibits racketeering activity. Defendants deny all of Plaintiff s allegations. As part of discovery in this matter, the Borough Defendants reviewed the contents of 14 boxes of documents for possible production to Plaintiff. Of these 14 boxes, 8 came directly from the Borough Defendants (the Borough documents ). The other 6 boxes (the Ryan/McKenna (Planning Board Member); (44) Robert Bellantoni (Planning Board Member); (45) John Vanna (Planning Board Member); (46) John Does (Seaside Park Planning Board Members, Consultants, Agents, Representatives, Employees and/or Attorneys); (47) Seaside Park Municipal Zoning Board; (48) Andrew Sbordone (Zoning Board Chairman); (49) Martin Wilk, Jr. (Zoning Board Member); (50) Kenneth Deshay (Zoning Board Member); (51) Michael Giuliano (Zoning Board Member); (52) Francis Losey (Zoning Board Member); (53) Michael Tierny (Zoning Board Member); (54) Edward Gallagher (Former Zoning Board Member); (55) Raymond Sites (Former Zoning Board Member); (56) Geoffrey N. Schwartz (Zoning/Code Enforcement Officer); (57) John Does (Seaside Park Zoning Board Members, Consultants, Agents, Representatives, Employees and/or Attorneys); (58) Seaside Park Code Enforcement Department; (59) William Schultz (Construction Official); (60) James Anderson (Subcode Official); (61) Robert Nora (Subcode Official); (62) Rejean Laliberte (Subcode Official); (63) Gary Swirczynski (Subcode Official); (64) Richard Barbarise (Subcode Official); (65) John Does (Seaside Construction Code Enforcement Department and/or Building Department Members, Consultants, Agents, Representatives, Employees and/or Attorneys); (66) Taxpayers Association of Seaside Park; (67) John Tweed (Director, Officer, Trustee and/or Member of Taxpayers Association of Seaside Park); (68) Thomas Hourihan (President of Taxpayers Association of Seaside Park); (69) John Does (Taxpayers Association of Seaside Park, Members, Consultants, Agents, Representatives, Employees and/or Attorneys); (70) Seaside Park Democratic Club; (71) Seaside Park Republican Club; (72) ABC Corporations 1-20; (73) XYZ Partnerships 1-20; (74) John/Jane Does ; (75) Charles Hollins (Planning Board Member); and (76) Ryan Stack (Patrolman). 3

162 documents ) came from the files of Guy P. Ryan, Esq. and Michael J. McKenna, Esq., two attorneys who represented the Borough Defendants in several matters underlying this litigation. Before producing the information contained in the Borough documents and Ryan/McKenna documents counsel for the Borough Defendants ( Brown & Connery or counsel ) undertook a multi-attorney review of same during which counsel spent approximately 250 hours reviewing 3 the documents for responsiveness to Plaintiff s requests and privilege. While associates from Brown & Connery were responsible for reviewing a significant portion of the Borough documents, a partner undertook the review of the Ryan/McKenna documents for privilege. In conducting this review, the partner personally reviewed each of the Ryan/McKenna documents for privilege, made notes concerning the documents flagged as privileged and then confirmed all privileged notations with Mr. Ryan or Mr. McKenna. The partner then instructed a clerical employee to separate the privileged Ryan/McKenna documents from the non-privileged Ryan/McKenna documents based on the designations included in her detailed notes. The partner further instructed the clerical employee to have the privileged and non-privileged documents separately Bates stamped and to have the non-privileged documents burned onto a disc to be produced to Plaintiff. The privileged documents were to be withheld and set aside so that they could later be identified on a proper privilege log. Believing that the aforementioned process had been followed, on August 12, 2010, the Borough Defendants produced what at the time they believed to be the non-privileged Borough documents and non-privileged Ryan/McKenna documents to Plaintiff. While counsel did review 3 For purposes of this Opinion, unless otherwise defined, the term privilege refers to the attorney-client privilege and includes attorney work product. 4

163 the final form of the disc of non-privileged documents prior to producing same to Plaintiff, counsel did not conduct a page-by-page review of the disc. Counsel argues that such a review was not required and would have been impracticable given the volume of documents being produced. Indeed, counsel notes that approximately 30,000 pages of material were contained on the August 12, 2010 disc. Unfortunately, unbeknownst to counsel, the clerical employee tasked with separating the privileged Ryan/McKenna documents from the non-privileged Ryan/McKenna documents only did so for the first 2 of the 6 boxes of Ryan/McKenna documents. As a result, all of the information the partner designated as privileged from the latter 4 boxes of Ryan/McKenna documents was not withheld as intended, but instead was produced to Plaintiff. Again, counsel did not recognize that privileged information was included on the disc produced to Plaintiff because counsel did not conduct a page-by-page review of the disc before producing same. After the disc was produced on August 12, 2010, the clerical employee who assisted with the production of the Ryan/McKenna documents was reassigned and a new secretary began to work for the partner at Brown & Connery. As part of assisting the partner in this matter, the new secretary reviewed the format of the disc produced on August 12, As part of this review, the secretary realized two things. First, she noticed that some of counsel s electronic comments, which were entitled to be withheld as privileged, appeared on the documents produced on the disc. Second, she realized that the information contained on the disc could be more efficiently organized as originally produced each document and sometimes individual pages of a single, larger document were scanned as separate and distinct pdf files. 5

164 In light of these discoveries, the Borough Defendants recalled the August 12, 2010 disc. The partner instructed her new secretary to create a new disc, sans attorney comments and with merged pdf files for a cleaner production. Instead of having numerous individual pdf files, the new disc would contain two pdf files, one for the non-privileged Borough documents and the second for the non-privileged Ryan/McKenna documents. The Borough Defendants produced this new and more efficiently organized disc on November 4, The Borough Defendants claim that at the time they produced the November 4, 2010 disc they again did not recognize that privileged Ryan/McKenna documents were contained on same because counsel did not substantively re-review the documents being produced. Instead, counsel merely directed its clerical staff to remove the electronic comments accidentally included in the original disc and to merge the numerous individual pdf files to create a cleaner, more efficient production. As a result, counsel contends that it did not become aware of the inadvertent production of privileged Ryan/McKenna documents in producing the November 4, 2010 disc. Counsel similarly argues that it did not become aware of the disclosure of privileged information in December 2010 when Plaintiff informed the Borough Defendants that the November 4, 2010 disc was unreadable. Indeed, counsel claims that the minor technical difficulty identified by Plaintiff on December 9, 2010, i.e. that the disc was unreadable, did not necessitate a substantive re-review of the information produced. Instead, the Borough Defendants simply needed to reburn the November 4, 2010 disc, which they did and produced same to Plaintiff on December 10, 2010, the day after Plaintiff informed them that the second disc was unreadable. Further, while counsel did conduct a quality control audit of the documents being reproduced on the December 10, 2010 disc, counsel argues that this audit was performed 6

165 simply to confirm that the same documents were being produced on this disc as were produced on the August 12, 2010 disc and which were attempted to be produced on the November 4, 2010 disc. Counsel claims that this review was not substantive in nature and was delegated to a clerical employee. Counsel emphasizes that the audit was not substantive in nature and did not entail a re-review for privilege. In late November 2010, counsel for the Borough Defendants began creating the privilege logs for the documents initially produced on August 12, An associate at Brown & Connery was responsible for creating the logs and before they were produced to Plaintiff, a partner reviewed the logs for description information. (Cert. of Karen A. McGuinness [Docket Entry No ] (the McGuinness Cert. ) 16). In generating and producing the aforementioned privilege logs, counsel did not realize that there were fewer privilege entries for the Ryan/McKenna documents than there should have been based on the partner s review of said documents. Counsel argues that they did not detect this discrepancy both because of the amount of time that had passed between when the Ryan/McKenna documents were reviewed and when the privilege logs were generated and because of the shear volume of documents produced by the Borough Defendants. On March 15, 2011, Plaintiff informed the Borough Defendants that there was a problem with the information contained on the third disc, i.e. the disc produced on December 10, Specifically, Plaintiff noted that it appeared that the documents burned on the third disc were out of order. In response to Plaintiff s complaints, counsel for the Borough Defendants reviewed the information as produced on the December 10, 2010 disc against the information as originally scanned. From this review, counsel determined that Plaintiff was correct: documents on the 7

166 December 10, 2010 disc were produced out of order. In reaching this conclusion, counsel did not review the entire contents of the December 10, 2010 disc. Instead, counsel simply reviewed a small sampling from the Borough documents, not the Ryan/McKenna documents. Counsel claims that this sampling was all that was necessary to confirm that a problem existed with the December 10, 2010 disc. After confirming that there was, in fact, a problem with the December 10, 2010 disc, counsel for the Borough Defendants requested that personnel from the firm s Information Technology Department ( IT ) identify the cause of the problem as well as which portions of the disc were affected so that the problem could be rectified. IT reviewed the December 10, 2010 disc and determined that the problem with the disc was limited to only the Borough documents. The Ryan/McKenna documents were not affected. Apparently, the problem was attributable to a computer glitch due to the manner in which the Borough documents were scanned by the vendor and then uploaded onto [counsel s] system and subsequently bate stamped. (McGuinness Cert. 30). The Ryan/McKenna documents were processed differently and thus unaffected. Counsel for the Borough Defendants corrected the problem identified by Plaintiff on March 15, 2011 by restoring the affected documents to their original order and reproducing those documents on a fourth disc in April Counsel argues that in correcting the problem with the December 10, 2010 disc, they never intended to re-review any of the documents for privilege. Counsel, however, states that in reorganizing the documents, the partner working on this matter identified that certain documents that had been previously flagged as privileged had not, in fact, 8

167 4 been withheld, but had mistakenly been produced. As a result, the partner added said documents to the Borough Defendants privilege logs and did not include them on the April 2011 disc. Counsel claims that the fact that certain privileged information from the Borough documents had been mistakenly produced to Plaintiff did not trigger a re-review of the Ryan/McKenna documents because the partner had personally reviewed all of the Ryan/McKenna documents for privilege whereas a substantial portion of the Borough documents had been reviewed by associates. As a result, counsel argues that they had no reason to believe that privileged Ryan/McKenna documents had been produced. On January 12, 2012, Plaintiff filed a reply brief in further support of his motion for a preliminary injunction [Docket Entry No. 121]. Plaintiff attached numerous exhibits to that briefing, including Exhibits G, H & I, documents first produced by the Borough Defendants on the August 12, 2010 disc. On January 13, 2012, the Borough Defendants began their review of Plaintiff s reply brief. During that review, counsel saw that Plaintiff attached Exhibits G, H & I to his reply. Exhibits G, H & I were Ryan/McKenna documents that the partner at Brown & Connery had marked as privileged. Based on this disclosure, counsel for the Borough Defendants re-reviewed its production and recognized that approximately 1000 pages of privileged Ryan/McKenna documents had been produced. Consequently, on January 18, 2012, counsel for the Borough Defendants informed all other parties of this inadvertent production and asked that they discard and not use said documents pursuant to Rule 26(b)(5)(B). 4 In light of the fact that only the Borough documents had been affected by the computer glitch, there was no need for the Ryan/McKenna documents to be reorganized. 9

168 II. The Parties Arguments A. The Borough Defendants Under these circumstances, the Borough Defendants argue that they should be permitted to reclaim the privileged Ryan/McKenna documents because their disclosure of same was inadvertent. In the first instance, the Borough Defendants claim that the documents they seek to reclaim are clearly privileged. In this regard, the Borough Defendants argue that all of the documents they seek to reclaim represent attorney-client communications made in Messrs. Ryan and/or McKenna s representation of the Borough in the disputes underlying this lawsuit or protected attorney work product generated in connection with same. The Borough Defendants additionally claim that none of the documents were discussed in open meetings or produced in the underlying disputes. Further, they note that their privilege log produced on February 10, 2012 sets forth their specific claims of privilege. The Borough Defendants next turn to the alleged inadvertent nature of their disclosure. In this regard, the Borough Defendants claim that the production of the privileged Ryan/McKenna documents was clearly unintentional and therefore inadvertent. Further, the Borough Defendants argue that their attorneys took reasonable steps to prevent the disclosure of privileged information. In this regard, the Borough Defendants note that counsel engaged in a multiple-attorney privilege review of the documents prior to their production. Indeed, the Borough Defendants note that approximately 250 hours of attorney time was spent reviewing the documents prior to their production. In addition, the Borough Defendants claim that the disclosure of the privileged Ryan/McKenna documents was not unreasonably large. Specifically, the Borough Defendants 10

169 claim that as part of discovery in this litigation, they produced nearly 100,000 pages of documents. They note that their disclosure of the privileged Ryan/McKenna documents, which involved approximately 1,000 pages of documents, amounts to only 1% of their entire production. The Borough Defendants also contend that after they learned of the inadvertent disclosure they acted promptly to rectify the problem. In this regard, the Borough Defendants claim that within five days of learning of the inadvertent disclosure, they notified all other parties of same and requested that the inadvertently produced information be destroyed or sequestered. Moreover, for the reasons set forth above, the Borough Defendants argue that they had no reason to be believe that privileged Ryan/McKenna documents had been inadvertently produced until Plaintiff filed Exhibits G, H and I in connection with his reply brief in further support of his motion for a preliminary injunction. Further, the Borough Defendants argue that the interests of justice favor a finding that no waiver of privilege has been made. First, the Borough Defendants note that they have always maintained the inadvertently produced, privileged Ryan/McKenna documents as privileged. Indeed, they point out that additional copies of several of the inadvertently produced documents were in fact logged as privileged in other instances. Second, the Borough Defendants argue that finding a waiver would derogate the policy of full and open attorney-client communications which gives rise to the existence of the privilege in the first place. Third, the Borough Defendants argue that a finding of waiver would prejudice their ability to defend against Plaintiff s claims in this matter because Plaintiff will continue to use and cite to the documents outside of their proper context. Fourth, the Borough Defendants claim that Plaintiff has no 11

170 legitimate interest in keeping the inadvertently produced documents. In this regard, the Borough Defendants argue that there is no inherent fairness interest in permitting Plaintiff to keep and use the documents. The Borough Defendants claim that this is particularly true in light of the fact that some of the documents were marked attorney-client privileged and, as such, Plaintiff knew or should have known that the documents were inadvertently produced. Fifth, the Borough Defendants argue that Plaintiff breached his ethical duties under New Jersey s Rule of Professional Conduct ( RPC ) 4.4(b)by failing to alert them to their inadvertent disclosure. B. Plaintiff In contrast, Plaintiff argues that the Borough Defendants should not be permitted to reclaim the allegedly privileged Ryan/McKenna documents that they produced. As an initial matter, Plaintiff claims that the Borough Defendants have not established that any of the documents they seek to reclaim are, in fact, privileged. Further, Plaintiff argues that the Borough Defendants disclosure of the allegedly privileged Ryan/McKenna documents was not inadvertent. In this regard, Plaintiff contends that inadvertent does not simply equate to unintentional. Instead, Plaintiff argues that the Borough Defendants, as the party resisting waiver, must demonstrate that they took reasonable precautions to avoid the inadvertent disclosure of the allegedly privileged documents. Here, Plaintiff claims that the Borough Defendants did not take reasonable steps to prevent the disclosure of the allegedly privileged Ryan/McKenna documents. Indeed, Plaintiff notes that the Borough Defendants produced said documents on three separate occasions, the last of which occurred after the Borough Defendants conducted an audit of the information contained on the disc being produced. Moreover, Plaintiff argues that the Borough Defendants inexplicably 12

171 failed to conduct a second review of the produced Ryan/McKenna documents after they learned that on three separate occasions they had produced 728 pages of privileged Borough documents. Further, Plaintiff claims that the documents currently in dispute represent a significant portion of the Borough Defendants production. In this regard, Plaintiff notes that, in total, the Borough Defendants produced 12,318 pages of Ryan/McKenna documents and that the disputed documents make up 872 of these pages or 7.1% of said production. In addition, Plaintiff argues that the Borough Defendants failed to act promptly to address the disclosure of the allegedly privileged Ryan/McKenna documents because the Borough Defendants should have known long before January 13, 2012 that these documents had been disclosed and the trigger for promptness is when the party seeking to avoid waiver knew or should have known that the problem existed. Here, Plaintiff argues that the Borough Defendants should have learned of their disclosure in November 2010 or at the latest in March 2011 when they discovered that other privileged information contained on the same disc as the allegedly privileged Ryan/McKenna documents was produced. Finally, Plaintiff argues that the interests of justice favor the denial of the Borough Defendants motion. In this regard, Plaintiff notes that at the time the Borough Defendants initially sought to reclaim the allegedly privileged Ryan/McKenna documents on January 18, 2012, he had already relied on the documents for over a year and a half. III. Analysis A. Standard of Review Motions to reclaim documents that have allegedly been inadvertently produced are governed by FED.R.EVID. ( FRE ) 502. This Rule provides as follows: 13

172 Rule 26(b)(5)(B) states that: Rule 502. Attorney-Client Privilege and Work Product; Limitations on Waiver (b) Inadvertent Disclosure. When made in a federal proceeding or to a federal office or agency, the disclosure does not operate as a waiver in a federal or state proceeding if: (1) the disclosure is inadvertent; (2) the holder of the privilege or protection took reasonable steps to prevent disclosure; and (3) the holder promptly took reasonable steps to rectify the error, including (if applicable) following Federal Rule of Civil Procedure 26(b)(5)(B). If information produced in discovery is subject to a claim of privilege or of protection as trial-preparation material, the party making the claim may notify any party that received the information of the claim and the basis for it. After being notified, a party must promptly return, sequester, or destroy the specified information and any copies it has; must not use or disclose the information until the claim is resolved; must take reasonable steps to retrieve the information if the party disclosed it before being notified; and may promptly present the information to the court under seal for a determination of the claim. The producing party must preserve the information until the claim is resolved. In determining whether inadvertently produced documents should be returned, the Court engages in a two-step analysis. First, the Court must determine whether the documents at issue are, in fact, privileged. Peterson v. Bernardi, 262 F.R.D. 424, 427 (D.N.J. 2009). Second, if privileged, then the Court must determine whether the aforementioned three prongs of FRE 502(b) have been met. Id. The disclosing party has the burden of proving that the elements of FRE 502(b) have been satisfied. Id. Similarly, the burden of proving that a privilege exists rests 14

173 with the party asserting the privilege. Louisiana Mun. Police Employees Ret. Sys. v. Sealed Air Corp., 253 F.R.D. 300, (D.N.J. 2008). FRE 502 does not alter the placement of same. Peterson, 262 F.R.D. at 427. The approach set forth in FRE 502(b) is essentially the same as that used in Ciba-Geigy Corp. v. Sandoz Ltd., 916 F.Supp. 404, 411 (D.N.J. 1995). Under the approach outlined in Ciba- Geigy, the Court engages in a multi-factor analysis to determine whether a waiver occurred. Factors considered are (1) the reasonableness of the precautions taken to prevent inadvertent disclosure in view of the extent of the document production; (2) the number of inadvertent disclosures; (3) the extent of the disclosure; (4) any delay and measures taken to rectify the disclosure; and (5) whether the overriding interests of justice would or would not be served by relieving the party of its error. Id. (quoting Advanced Medical, Inc. v. Arden Medical Sys., Inc., Civ.A. No , 1988 WL at *2 (E.D. Pa. July 18, 1988)). B. Discussion As noted above, the Court begins its analysis by determining whether the inadvertently produced Ryan/McKenna documents are privileged. In this regard, the Court notes that the Borough Defendants did not go document by document or category of documents by category of documents through the Ryan/McKenna documents they seek to reclaim to explain why said documents are privileged or otherwise protected from discovery. Instead, the Borough Defendants generally argue that the documents are protected from discovery because they are all either attorney-client communications made in Messrs. Ryan and McKenna s representation of the Borough in the disputes underlying this litigation or protected work product generated in 15

174 connection with same. The Borough Defendants also attach their privilege log dated February 10, 2012 that corresponds to the Ryan/McKenna documents they seek to reclaim. While the Borough Defendants showing is not the strongest, the Court agreed to review all of the documents subject to this motion in camera. Based on the information provided by the Borough Defendants as well as the Court s review of the Ryan/McKenna documents subject to this motion, the Court makes the following findings: 1. No privilege has been asserted with respect to SP 24327; SP 24963; SP 25008; and SP Indeed, the Nature of Privilege/Redaction column on the Borough Defendants February 10, 2012 privilege log indicates that the aforementioned documents have been RELEASED. As such, to the extent the Borough Defendants seek to reclaim these documents, that request is denied. 2. No privilege has been asserted with respect to SP ; SP ; SP ; SP ; SP ; SP ; and SP Instead, the Nature of Privilege/Redaction column on the Borough Defendants February 10, 2012 privilege log states that these documents are NOT RELEVANT. While parties to a litigation need only produce relevant information (see Rule 26(b)(1)), nothing in FRE 502 or the pertinent case law interpreting same suggests that the Rule was designed to permit a party to reclaim documents that are merely irrelevant to the pending matter. In fact, FRE 502 explicitly provides that [t]he following provisions apply, in the circumstances set out, to disclosure of a communication or information covered by the attorney-client privilege or work-product protection. (Emphasis added). Similarly, Rule 26(b)(5)(B) pertains to information produced in discovery that is subject to a claim of privilege or of protection as trial-preparation material[.] (Emphasis added). Here, no such claims were made. Instead, the documents are merely classified as NOT RELEVANT. The Court finds the documents alleged lack of relevance to be an insufficient basis on which to permit the Borough Defendants to reclaim same. As a result, the Borough Defendants request to clawback these documents is denied. 3. It does not appear the any privilege has been asserted with respect to SP ; SP ; ; ; or Instead, the Nature 5 In conducting Its in camera review of the subject Ryan/McKenna documents, the Court was unable to locate SP It appears that this document was not submitted to the Court. Nevertheless, even without specifically reviewing this particular document, the Court is 16

175 of Privilege/Redaction column on the Borough Defendants February 10, 2012 privilege log for these documents reads POLICE RECORDS - PROTECTIVE ORDER. The Court notes that on January 19, 2012, It entered a Discovery Confidentiality Order ( DCO ) [Docket Entry No. 125] that had been agreed upon by the parties. Pursuant to the terms of the DCO, the DCO applies only to police related documents. (DCO 2). All of the aforementioned documents designated as POLICE RECORDS - PROTECTIVE ORDER appear to qualify as police related documents that, absent some other issue, would be subject to the DCO. Here, while the Borough Defendants do not explicitly so state, the Court assumes that they do not seek to actually reclaim the aforementioned documents, but 6 instead seek to ensure that said documents are subject to the DCO. The Court additionally assumes that the Borough Defendants are doing so because Plaintiff 7 will not agree to the retroactive protection of these documents under the DCO. In this regard, the Court notes that pursuant to its explicit terms, the DCO applies only to police related documents produced by Defendants after its entry or police related documents produced by Defendants before the entry of the DCO that were identified within 15 days of the date of the entry of the DCO and which Plaintiff agrees will be retroactively subject to the DCO. (DCO 4). The documents at issue here were produced approximately 17 months before the DCO was entered. As a result, absent Plaintiff s approval or Court Order, the aforementioned documents would not be subject to the protection afforded by the DCO. Despite the fact that SP ; SP ; ; ; and were produced nearly a year and a half before the DCO was entered, the Court shall require that these documents be afforded retroactive protection under same. As a result, while the Borough Defendants shall not be permitted to reclaim confident with Its determinations as expressed herein. 6 To the extent the Court is incorrect and the Borough Defendants seek to reclaim these documents, the Borough Defendants request is denied. The Borough Defendants have proffered no arguments to suggest that these police records are privileged or subject to attorney work product protection and, as explained with respect to the documents identified in category 2, FRE 502 and Rule 26(b)(5)(B) only apply where claims of attorney-client privilege and or attorney work product protection have been made. 7 Neither party has explicitly stated that this is the case; instead, the Court is inferring same from the information presented in the Borough Defendants motion and Plaintiff s opposition thereto. 17

176 these documents, Plaintiff, in using these documents, must treat them as protected material under the DCO. 4. The Borough Defendants claim that the following documents are protected by the attorney-client privilege: SP ; SP ; SP 19669; SP 19674; SP 19673; SP 19682; SP 24327; SP 24963; SP 25008; SP 25049; SP 24901; SP 25005; SP 25065; SP 25066; SP 25067; SP 25072; and SP (collectively the Clive Samuels documents ). These documents all involve communications with engineers at Clive Samuels and Associates, Inc. ( Clive Samuels ). The Borough Defendants claims of privilege regarding the Clive Samuels documents turn on whether Clive Samuels was the Borough s expert in one of the underlying disputes that forms the basis for part of this case or an independent courtappointed expert. The Borough Defendants argue that Clive Samuels was their expert whereas Plaintiff contends that Clive Samuels was an independent, courtappointed expert. The Court has reviewed the information presented by the parties on this issue and finds that Clive Samuels was the Borough s expert. This conclusion is based largely on this Court s independent review of the transcript of proceedings that occurred on October 6, 2006 before Hon. Frank A. Buczynski, Jr., J.S.C., as well as the Order entered by Judge Buczynski immediately after the hearing. In light of the Court s decision that Clive Samuels was the Borough s expert, the Court finds that the aforementioned documents are privileged. 5. The Borough Defendants claim that all of the remaining documents entered on the February 10, 2012 privilege log are protected by the attorney-client privilege. These documents include: SP 1422; SP ; SP ; SP 16668; SP ; SP ; SP ; SP ; SP 16770; SP 16771; SP ; SP ; SP 16738; SP ; SP 16788; SP ; SP ; SP ; SP ; SP ; SP 16850; SP ; SP ; SP ; l SP ; SP ; SP ; SP 19507; SP 19512; SP ; SP ; SP ; SP ; SP ; SP 20106; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP 17638; SP 17651; SP 17652; SP 17653; SP ; SP ; SP ; SP 18039; SP 18985; SP 18087; SP 18086; SP 18095; SP 18096; SP 18097; SP 18373; SP ; SP ; SP 18583; SP 18592; SP ; SP ; SP ; SP ; SP ; SP ; SP ; SP 18874; SP ; SP 19125; SP 19372; SP 19373; SP ; SP 19390; SP 19461; SP 19475; SP 19895; SP 19903; SP 19498; SP 19508; SP 19509; SP19533; SP 19536; SP 19537; SP 19556; SP 19614; SP 20088; SP 19687; SP 19688; SP 20090; SP 20091; SP 20092; SP 18

177 20093; SP 20094; SP ; SP 20161; SP 20162; SP ; SP ; SP ; SP ; SP ; SP ; SP 20933; SP ; SP ; SP 20955; SP ; SP 21141; SP 24327; SP 24963; SP 25008; SP 25049; SP 24832; SP 24901; SP 25005; SP 25058; SP 25072; SP 25081; SP 25241; SP 25338; SP 25425; SP 25431; SP ; SP ; SP 25439; SP 25443; SP ; SP 25450; SP 26104; SP 26133; SP ; SP 26157; SP ; SP 26226; SP ; SP 26261; SP 26262; SP 16845; SP 17557; SP ; SP 17981; SP ; SP 18033; SP ; SP ; SP ; SP ; SP 18271; SP 19893; SP ; SP ; SP 20104; SP 20720; SP 21165; SP 24976; SP 14654; SP 14657; SP 14658; SP ; SP 14673; SP 14674; SP ; SP 14682; SP 14686; SP 14689; SP 14692; SP 14693; SP 14698; SP 14735; SP 14763; SP 14764; SP 14773; SP 14790; SP 14794; SP 14803; SP 14808; SP 15208; SP 15214; SP 15217; SP 15222; SP 17853; SP 17858; SP 17861; SP 17862; SP 17864; SP 17950; SP 18046; SP 18407; SP 18408; SP 18436; SP 18445; SP 18466; SP 19061; SP 19077; SP 19406; SP 19611; SP 19700; SP 19734; SP 19906; SP 21162; SP 21166; SP 21172; SP 21349; SP 21466; SP ; SP 21494; SP 21500; SP 21504; SP 21597; SP ; SP 21625; SP 22068; SP 22362; SP ; SP ; SP ; SP 22390; SP 22394; SP 22397; SP ; SP 22406; SP 22442; SP 24296; SP 24316; SP 24322; SP 24935; SP 25123; and SP (the Remaining documents ). As noted above, the Court has reviewed all of these documents in camera. Based on the Court s in camera review, the Court finds that these documents are properly 10 identified as privileged. 8 The Court notes that an entry for this particular document does not appear on the February 10, 2012 privilege log submitted by the Borough Defendants. 9 The remaining documents including the foregoing were not withheld in their entirety but produced in redacted format. 10 The Court notes that in his motion to compel, Plaintiff raises additional reasons, other than waiver through their production to him, why some of these documents are not properly subject to the Borough Defendants claims of privilege. Examples of such reasons include, but are not limited to, the crime-fraud exception to the attorney-client privilege, the fact that the Borough Defendants allegedly placed certain of these otherwise protected communications at issue and that a waiver of the privilege occurred because the communications were shared with unnecessary third-parties. The Court has not considered these arguments in considering the Borough Defendants motion for discovery, but shall do so in deciding Plaintiff s motion to compel. As a result, the findings described above are made without prejudice to the Court s later consideration of Plaintiff s motion to compel. The same holds true with respect to the Court s determination concerning the Clive Samuels documents. 19

178 Having determined that the Clive Samuels documents and the Remaining documents are privileged, the Court next addresses whether these categories of documents were inadvertently produced under Rule 502(b) and the multi-factor test outlined in Ciba-Geigy. Because the Court has determined that the other categories of Ryan/McKenna documents subject to this motion are not privileged, the Court need not reach the question of whether they were inadvertently produced. Peterson, 262 F.R.D. at 427 (holding that [i]t is axiomatic that FRE 502 does not apply unless privileged or otherwise protected documents are produced. ) The Court, however, notes that for practical purposes the remaining analysis would apply equally to these other categories of documents had the Court determined that they were privileged. The Court has no doubt that the Borough Defendants production of the Clive Samuels and Remaining documents was inadvertent in the sense that their production was unintentional. However, while necessary to a finding under FRE 502(b) that the production of these documents does not amount to a waiver, because every inadvertent disclosure is an unintentional disclosure[,] the fact that the Borough Defendants have convinced the Court that said disclosure was unintentional does not go far in establishing the absence of waiver. Ciba-Geigy, 916 F.Supp. at 412. Instead, the other factors would also have to point to an absence of waiver. Peterson, 262 F.R.D. at 429 (citing Ciba-Geigy, 916 F.Supp. at 411) (noting that producing party s subjective intent is not controlling. ); Sensient, 2009 WL , at *4 (same). As to these other factors, the Court finds that the Borough Defendants initially took reasonable steps to prevent the disclosure of their privileged information. In this regard, the Court notes that counsel for the Borough Defendants engaged in a multi-attorney review of the Borough documents and Ryan/McKenna documents devoting approximately 250 hours of 20

179 attorney time to reviewing these documents for privilege. Moreover, given the likelihood that privileged information would be contained in the Ryan/McKenna documents, a partner at Brown & Conner personally reviewed all of the Ryan/McKenna documents for privilege. That partner took detailed notes concerning which documents she believed were privileged, confirmed the privilege designations with counsel in the matters underlying this lawsuit and marked in her notes all privileged documents that were to be withheld from production. This privilege review was more than adequate and certainly indicates that reasonable precautions were taken to prevent the inadvertent disclosure of privileged information. Cf. Peterson, 262 F.R.D. at 429 (finding that conclusory statement that privilege review was conducted without any specific information concerning when review occurred, how much time was spent reviewing documents, what documents were reviewed or other basic details of review process is not informative and does little to establish reasonableness of precautions taken to avoid disclosure). Also reasonable was counsel s decision to delegate to a non-attorney, clerical employee the task of separating the flagged, privileged Ryan/McKenna documents from the rest of the production. Indeed, the Court finds no reason why, after counsel invested substantial resources in reviewing the Borough and Ryan/McKenna documents for privilege, the task of physically separating the documents flagged as privileged from the non-privileged documents and sending the non-privileged information to a vendor to be scanned on a disc for production would also have to be performed by an attorney or paralegal. Instead, this is exactly the type of task that could reasonably be assigned to clerical staff. Moreover, the Court finds that the Borough Defendants were not obligated to review the disc before producing same to Plaintiff. Indeed, as explicitly noted in FRE 502(b) s Explanatory 21

180 Note, [t]he rule does not require the producing party to engage in a post-production review to determine whether any protected communication or information has been produced by mistake. Thus, having implemented reasonable steps to prevent the inadvertent disclosure of privileged information, the Borough Defendants were permitted to rely on same, at least until there were any obvious indications that a protected communication or information ha[d] been produced inadvertently. Explanatory Note, FRE 502(b). Turning next to the number and extent of the disclosures, the Court finds these factors to be neutral. As part of this litigation, the Borough Defendants have produced approximately 100,000 pages of documents, which represents a relatively substantial document production. See Sensient, 2009 WL , at *4 (finding that document production consisting of 47,000 documents reviewed with 45,000 documents (135,000 pages) produced was substantial). Given the size of the Borough Defendants production, mistakes were bound to occur. Id. At issue here is the disclosure of 872 pages of documents, which represents only about 1% of the Borough Defendants total production, which on a total percentage basis is notably small. Nevertheless, the Court finds that 872 pages of documents is not insignificant. Moreover, the Court does not necessarily agree with the Borough Defendants that the extent of their disclosures should be based on their total document production. While that method was apparently used in Sensient, 2009 WL , at *4, and Peterson, 262 F.R.D. at 429, the Court in Ciba-Geigy focused not on the defendants total production but on the 681 documents counsel had designated for use in two depositions. 916 F.Supp. at 407, 414. If the Court focuses more narrowly on the total number of documents produced on August 12, 2010 (approximately 26,000 pages) or even more so on only the Ryan/McKenna documents 22

181 (approximately 12,000 pages), the extent of the disclosure increases to approximately 3% or 7% respectively, more significant figures. Furthermore, even were the Court to focus solely on the Borough Defendants disclosures relative to the Borough Defendants total document production, i.e., the 1% figure pressed by the Borough Defendants, the Court would be mindful of the fact that the disclosures at issue in this motion are not the only unintentional disclosures made by the Borough Defendants in this case. Instead, though not at issue here, as part of their initial August 12, 2010 production, the Borough Defendants also produced 728 pages of privileged Borough documents. In addition, as in Peterson, the Court would also be mindful of the nature of the disclosures at issue. 262 F.R.D. at 429. Here, the disclosures in question largely involve communications between the Borough Defendants and their counsel in the legal issues underlying this lawsuit. As in Peterson, these communications warranted a significant level of scrutiny. Id. (finding that documents representing exchanges between plaintiff and his attorney warranted significant level of scrutiny). Thus, even assuming that the extent of the Borough Defendants disclosures was quite small - 1% of their total document production - this fact would be balanced by the nature of the disclosures at issue. For these reasons, the Court finds that the number and extent of the Borough Defendants disclosures are neutral factors. As to the Borough Defendants efforts to rectify their disclosure of the privileged Ryan/McKenna documents, the Court finds that the Borough Defendants did not take reasonable steps to remedy their error. In this regard, the Court finds that the Borough Defendants should have been aware that something was amiss with their document production long before Plaintiff 23

182 relied on three privileged documents as part of his reply brief filed in further support of his motion for a preliminary injunction on January 12, While FRE 502(b) does not require the producing party to engage in a post-production review to determine whether any protected communication or information has been produced by mistake[,] it does require the producing party to follow up on any obvious indications that a protected communication or information has been produced inadvertently. Explanatory Note, FRE 502(b). As a result, once a producing party is on notice that something [i]s amiss with its document production and privilege review[,] then that party has an obligation to promptly re-assess its procedures and re-check its production. Sensient, 2009 WL , at * 5. The Borough Defendants simply did not do so here, despite being provided with clear signs that something went awry with its document production and privilege review. While a number of the seemingly unending problems the Borough Defendants experienced with their document production did not involve the disclosure of protected information, several did. Indeed, as early as November 2010, the Borough Defendants were aware that protected information in the form of counsel s electronic comments had been produced to Plaintiff on the August 12, 2010 disc. Yet this disclosure did not prompt the Borough Defendants to review the entirety of their August 12, 2010 production for other mistakes. The Borough Defendants seem to suggest that such a review was unnecessary; though they do no provide much detail as to why. The Court assumes that the Borough Defendants position is based primarily on the fact that the inclusion of counsel s protected electronic comments would in no way indicate that actual documents that had been identified as privileged had also been erroneously included in their production. 24

183 The reasonableness of the Borough Defendants decision not to re-review its August 12, 2010 production depends in part on the circumstances surrounding the inclusion of the electronic comments in the production. The Court, however, has been provided with scant information in this regard. Indeed, the Court does not know how counsel s electronic comments came to be produced on the August 12, 2010 disc, who was responsible for removing same prior to production, etc. This information is certainly relevant because if the same clerical employee responsible for sorting the privileged Ryan/McKenna documents from the non-privileged documents was also responsible for removing the electronic comments, then the Borough Defendants would have had more reason to suspect that other mistakes occurred in the August 12, 2010 production based on the disclosure of counsel s electronic comments. Regardless, the Court is willing to assume for purposes of this motion that the disclosure of counsel s electronic comments in the August 12, 2010 production would not, standing alone, reasonably put the Borough Defendants on notice that actual privileged documents may have been mistakenly produced to Plaintiff. The disclosure of counsel s electronic comments, however, does not stand alone. On December 3, 2010, the Borough Defendants produced their privilege log relevant to their production of the Borough and Ryan/McKenna documents. That log included entries for 366 pages of privileged Ryan/McKenna documents. Missing, however, were the 872 pages of unintentionally disclosed documents that are the subject of the instant motion. In other words, approximately 70% of the material that should have been included on the Ryan/McKenna portion of the December 3, 2010 log was missing. 25

184 The Borough Defendants claim that they did not realize that their privilege log was missing fewer entries than should have been included given the sheer volume of their production and the passage of time from when counsel reviewed the Ryan/McKenna documents and when the privilege log was generated. In the first instance, the Court notes that the privilege log is not simply missing fewer entries than should have been included; instead, the bulk of the 11 entries are missing. While the volume of production and the passage of several months could certainly interfere with an attorney s recollection of exactly how many documents should have been logged, the Court is somewhat perplexed how counsel missed the fact that approximately 70% of the information that should have been included was not. The Court is also somewhat perplexed how the simple difference between the number of pages logged for the Borough documents as compared to the number of pages logged for the Ryan/McKenna documents did not raise any flags for the Borough Defendants. While the December 3, 2010 privilege log contains entries for 1389 pages of Borough documents, it only contains entries for 366 pages of Ryan/McKenna documents. This disparity strikes the Court as odd in light of the fact that, while larger than the Ryan/McKenna production, the Borough production consisted of only 2 additional boxes of documents and, more importantly, the Ryan/McKenna documents were obtained directly from counsel involved in the legal matters 11 Even if the Court were to consider the December 3, 2010 log in its entirety, as it pertains to both the Borough documents and Ryan/McKenna documents, a significant amount of information would be missing from same. Indeed, the December 3, 2010 log included entries for 1753 pages of information, 1389 of which pertained to the Borough documents and 366 of which pertained to the Ryan/McKenna documents. Missing from the log were 1600 pages of privileged information, 728 pages of privileged Borough documents and the 872 pages of Ryan/McKenna documents at issue here. Consequently, as produced the December 3, 2010 log was missing almost half of the information that should have been included on same. 26

185 underlying this litigation, suggesting that they were equally likely, and, in reality, probably much more likely to involve privileged communications than the Borough documents. Further, the Court notes that this disparity in number of pages logged would have been self evident regardless of the passage of time or the volume of the Borough Defendants document production. The Court also has questions about how the Borough Defendants December 3, 2010 privilege log was generated. The Court understands that the partner who reviewed the Ryan/McKenna documents was not responsible for drafting the December 3, 2010 privilege log even as to the portion of the log pertaining to the Ryan/McKenna documents. Instead, associates were responsible for creating the log in its entirety. The Borough Defendants, however, do not provide much detail concerning the exact process used by the associates to create the privilege log. For example, in generating the log entries pertaining to the Ryan/McKenna documents, did the associates reference the partner s detailed notes, which were initially used by the clerical employee to sort the privileged Ryan/McKenna documents from their non-privileged counterparts? If not, why? It would appear to the Court that those notes would have been invaluable to the creation of the December 3, 2010 privilege log. Moreover, if the notes were utilized, how then could counsel possibly not realize that 872 pages of privileged Ryan/McKenna documents were mistakenly disclosed. Based on the foregoing, the Court finds the Borough Defendants position with respect to the December 3, 2010 privilege log, namely that nothing in their generation of same indicated that there was anything wrong with their document production or privilege review, to be somewhat questionable. Nevertheless, as with the Borough Defendants disclosure of their 27

186 counsel s electronic comments, the Court is willing to assume, for purposes of this motion, that nothing in the creation or the content of the December 3, 2010 privilege log, or more specifically the fact that numerous log entries were missing from same, would have put the Borough Defendants on notice that something was amiss with their document production and privilege review. Consequently, the Court shall not find a waiver based on the Borough Defendants failure to reexamine the August 12, 2010 production based on their generation of the December 3, 2010 privilege log. Again, however, this does not represent the last discovery mishap experienced by the Borough Defendants involving the disclosure of protected information. Instead, in March 2011, after having been informed by Plaintiff of organizational issues with the manner in which their 12 documents appeared on the December 10, 2010 disc, the Borough Defendants both realized that (1) Plaintiff was correct, the documents produced on the December 10, 2010 disc were disorganized; and (2) 728 pages of privileged Borough documents had been inadvertently disclosed. It is this disclosure of 728 pages of privileged documents that is pertinent here. Even completely discounting the fact that the December 3, 2010 privilege log was missing a significant portion of its entries, the Court finds that the combination of the inadvertently produced attorney electronic comments and 728 pages of privileged Borough documents should have put the Borough Defendants on notice that something had gone profoundly awry with their document production and privilege review. Unlike the Borough 12 The Court notes that the December 10, 2010 disc was the disc generated by the Borough Defendants after being informed that their disc produced on November 4, 2010, which contained the Borough and Ryan/McKenna documents initially produced on August 12, 2010 albeit theoretically more efficiently organized and sans attorney electronic comments, was unreadable. 28

187 Defendants, the Court does not believe that it was reasonable for them to fail to fail to rereview the Ryan/McKenna portion of the December 10, 2010 disc [b]ecause the partner knew she had personally reviewed each of the Ryan/McKenna documents for privilege, because that group of documents was unaffected by the glitch, and because Plaintiff had not advised of any problems with that group of documents[.] (Borough Def. Reply at 4). In reaching this conclusion, the Court notes that the fact that the computer glitch only affected the portion of the December 10, 2010 disc that contained the Borough documents is largely irrelevant as the glitch itself did not cause the Borough Defendants privileged information to be disclosed. Instead, the glitch merely caused the Borough Defendants to take a slightly closer look at their production of the Borough documents. In doing so, the Borough Defendants realized that, despite their multi-attorney review during which counsel spent approximately 250 hours reviewing the Borough documents and Ryan-McKenna documents for responsiveness to Plaintiff s requests and privilege, 728 pages of privileged Borough documents had been disclosed. In addition, the fact that the partner had personally reviewed each of the Ryan/McKenna documents for privilege whereas associates had done a significant part of the review of the Borough documents does not excuse the Borough Defendants failure to recheck the entire production. First, while associates may have conducted a significant portion of the review of the Borough documents, the partner also had some involvement with that review and there is no evidence that the partner was not responsible for reviewing any of the mistakenly disclosed Borough documents. Second, and more importantly, the disclosure of the 728 pages of privileged Borough documents was not the first privilege-related error experienced by the 29

188 Borough Defendants with respect to information produced on the disc containing the Borough documents and Ryan/McKenna documents. Instead, as previously stated, at the time the Borough Defendants learned that they had inadvertently produced 728 pages of privileged Borough documents (first on August 12, 2010, then again on November 4, 2010 and lastly on December 10, 2010), they had already also discovered that they had mistakenly produced counsel s protected electronic comments in the same production; though the electronic comments had been removed since November 4, The Court finds that the problems experienced by the Borough Defendants with their production of the Borough documents and Ryan/McKenna documents would have spurred a reasonable person to recheck the entire contents of the December 10, 2010 disc. Their failure to do so necessitates a finding that the Borough Defendants did not take prompt and reasonable steps to rectify its error. This is true despite the fact that Plaintiff s counsel may have violated their ethical obligations under RPC 4.4(b) by failing to inform the Borough Defendants that privileged information had been produced. Regardless of whether Plaintiff s counsel had an ethical responsibility under RPC 4.4(b) to inform the Borough Defendants that privileged information had been disclosed, once a party is on notice of an error in its document production [that party] should not wait for its adversary to discover its error and then claim protection under FRE 502(b). Sensient, 2009 WL , at *5, n.12. Instead, once a party is on notice that something [i]s amiss with its document production and privilege review[,] that party must promptly re-assess its procedures and re-check its production. Sensient, 2009 WL , at *5. 30

189 The Court finds that how a party receives notice of its error is largely irrelevant: whether a party is informed by its adversary that privileged information has been inadvertently produced as in Sensient or whether the circumstances surrounding a party s production indicates that something has gone awry, as is the case here, is of little import. Instead, the key is that once a party has notice that something is amiss with its production and privilege review[,] that party has an obligation to promptly re-assess its procedures and re-check its production. Id. at *5. As explained above, the Borough Defendants, despite being on notice that something was amiss with their production of the Borough documents and Ryan/McKenna documents, failed to promptly re-check their production of same. Finally, the Court finds that under the circumstances of this case, the interests of justice favor finding that a waiver occurred. While the Court agrees with the Borough Defendants that the relevance of the mistakenly produced Ryan/McKenna documents is not dispositive of the waiver issue (Ciba-Geigy, 916 F.Supp. at 414 (finding [t]hat the documents are relevant and probative does not determine the waiver questions (internal quotation marks and citation omitted)), the Court also finds however, much like in Ciba-Geigy, the interests of justice would be served by a finding of waiver, where, as here, a party s negligence has resulted in the inadvertent production of a privileged document. 916 F.Supp. at 414; see also Peterson, 262 F.R.D. at 429 (same) and Sensient, 2009 WL at *5, n.12 (same). In reaching this conclusion, the Court is aware that, here, Plaintiff did not inform the Borough Defendants that they had produced privileged material. The Court is also aware that the Borough Defendants claim that, in failing to do so, Plaintiff s counsel violated their ethical obligations under RPC 4.4(b). 31

190 RPC 4.4(b) provides as follows: [a] lawyer who receives a document and has reasonable cause to believe that the document was inadvertently sent shall not read the document or, if he or she has begun to do so, shall stop reading the document, promptly notify the sender, and return the document to the sender. Here, as the Borough Defendants note, there were obvious indications that at least some of the documents at issue in this motion were privileged. Indeed, from the Court s in camera review it appears that at least 23 documents consisting of approximately 60 of the 872 pages of documents at issue in this motion bore a legend indicating that they were PRIVILEGED AND CONFIDENTIAL or ATTORNEY/CLIENT 13 CONFIDENTIAL communications. In light of these explicit designations, the Court finds that Plaintiff should have known that he received privileged information from the Borough Defendants. As a result, Plaintiff s counsel had an obligation under RPC 4.4(b) to, at a minimum, stop using the marked documents and return them to the Borough Defendants. Plaintiff s counsel did not do so. Plaintiff s counsel never addresses the Borough Defendants argument concerning RPC 4.4(b). Indeed, Plaintiff s counsel does not even attempt to explain why he failed to direct the Borough Defendants to their disclosure of privileged information. The Court is aware that Plaintiff in his pending motion to compel claims that several documents over which the Borough Defendants have made privilege claims, including a number of the Ryan/McKenna documents at issue here, have been improperly marked/withheld as privileged. As a result, it is possible that Plaintiff reasonably believed that the Ryan/McKenna documents marked as PRIVILEGED 13 The legends may not have appeared exactly as depicted above, but were substantially similar to the two legends provided. 32

191 AND CONFIDENTIAL and ATTORNEY/CLIENT CONFIDENTIAL had been purposefully disclosed by the Borough Defendants. However, without a specific argument in this regard, the Court cannot excuse Plaintiff s counsel s failure to notify the Borough Defendants of their disclosure. Consequently, it would appear that Plaintiff s counsel violated their duty under RPC 4.4(b). This fact obviously weighs against finding that a waiver occurred. Nevertheless, the Court on balance finds that this fact does not overcome the Borough Defendants failure to rereview their entire production after it became clear that there was something amiss with their document production and privilege review. Sensient, 2009 WL at *5. As described above, at least as of March 2011 the Borough Defendants should have known that something went awry with their production and review of the documents initially produced on August 12, At a minimum, at that point in time the Borough Defendants knew that the information initially produced on August 12, 2010, which included the Borough documents and Ryan/McKenna documents, not only contained their attorneys electronic comments, but also 728 pages of privileged Borough documents. As explained above, these disclosures should have spurred the Borough Defendants to re-review their entire production of the Borough documents and Ryan/McKenna documents. The Borough Defendants failure to do so was unreasonable. As a result, everything considered, the Court finds that the interests of justice favor finding that a waiver occurred. The Court, however, also finds that the interests of justice favor imposing a waiver that is narrow in scope. While the Court has determined that the Borough Defendants disclosure of the privileged Ryan/McKenna documents was not inadvertent, meaning the Borough Defendants shall not be permitted to reclaim same, the Court has also determined that the Borough 33

192 Defendants disclosure was unintentional. Clearly, the Borough Defendants in unintentionally disclosing over 800 pages of otherwise privileged information were not attempting to use the attorney-client privilege as both a sword and a shield. Indeed, this does not represent a case where a purposeful, selective disclosure was made. As a result, the Court shall limit the waiver to the actual documents disclosed and shall not require a broad subject matter waiver. Koch Materials Co. v. Shore Slurry Seal, Inc., 208 F.R.D. 109, (D.N.J. 2002) (noting that [t]he general rule that a disclosure waives not only the specific communication but also the subject matter of it in other communications is not appropriate in the case of inadvertent disclosure, unless it is obvious that a party is attempting to gain an advantage or make offensive or unfair use of the disclosure and finding that full subject matter waiver was not warranted) (internal quotation marks and citation omitted). III. Conclusion For the reasons set forth above, the Borough Defendants motion for discovery to reclaim inadvertently produced privileged documents is DENIED. An appropriate Order follows. Dated: May 30, 2012 s/tonianne J. Bongiovanni HONORABLE TONIANNE J. BONGIOVANNI UNITED STATES MAGISTRATE JUDGE 34

193 NOT FOR PUBLICATION UNITED STATES DISTRICT COURT DISTRICT OF NEW JERSEY : JUSTER ACQUISITION CO., LLC, : : Plaintiff, : Civil Action No (JLL) : v. : : NORTH HUDSON SEWERAGE : OPINION AUTHORITY, : : Defendant. : : HAMMER, United States Magistrate Judge I. INTRODUCTION This matter comes before the Court by way of defendant North Hudson Sewerage Authority s ( defendant or NHSA ) motion for a protective order regarding the sixty-seven (67) word searches of electronically stored information ( ESI ) demanded by plaintiff Juster Acquisition Co., LLC ( plaintiff or Juster ). See Nov. 28, 2012, Motion for Discovery Fees, ECF No. 14. Should the Court deny defendant s motion for a protective order, NHSA seeks, in the alternative, an order for fee-shifting on plaintiff s requested electronic discovery. If the Court determines that fee-shifting is in fact warranted, plaintiff submits that the Court must also require defendant to reimburse it for the electronic discovery costs it has incurred in the course of this litigation. For the reasons set forth herein, the Court denies with prejudice defendant s motion for a protective order and request for discovery fees, and as such, plaintiff s request for discovery fees is rendered moot. 1

194 II. FACTUAL BACKGROUND Juster filed a four-count Complaint on June 6, 2012, against NHSA to recover compensatory damages, costs and pre-judgment interest, attorneys fees and costs, and such other relief as the Court deems equitable and just. (Compl., June 6, 2012, ECF No. 1). Juster is a New York limited liability company that invests in various business opportunities such as complex financing transactions. (Id. 1, ECF No. 1). NHSA is a political subdivision and public body politic, formed pursuant to the Sewerage Authorities Law of New Jersey, N.J.S.A. 40:14A-1 et seq., for the purpose of acquiring, constructing, maintaining, and operating facilities for collection, treatment, and disposal of sewage. (Id. 2). The parties began their business relationship when NHSA sought to refinance and recapitalize the bulk of its primary debt obligations. (Id. 6). NHSA determined it would need help from a private party who possessed the requisite knowledge to execute the transaction, and Juster responded to NHSA s Request for Qualifications. (Id.). Juster alleges that the two parties negotiated and executed a term sheet ( Term Sheet ), which contained a binding Exclusivity Provision stating that NHSA would neither solicit other offers nor disclose any of the terms herein for a period of 18 months from the date of execution of this Term Sheet. (Id. 7). NHSA denies the existence of this Exclusivity Provision, or that the Term Sheet was ever executed. (Answer 7, July 13, 2012, ECF No. 7). Pursuant to the execution of the Term Sheet, Juster alleges that it devoted substantial time and effort developing and implementing a plan for NHSA to refinance its primary debt obligations ( Juster Work Product ). (Compl. 10, ECF No. 1). NHSA, on the other hand, 2

195 alleges that Juster undertook such action at its own expense in the hope of becoming NHSA s financing agent. (Answer 10, ECF No. 7). Accordingly, NHSA denies that it ever agreed to or has any obligation to reimburse Juster. (Id.). Juster alleges that on January 20, 2012, NHSA instructed Juster s financial advisors to put a halt to the transaction. (Compl. 11, ECF No. 1). It contends that NHSA indefinitely suspended the closing, and in the months following the suspension, NHSA refused to negotiate in good faith with Juster. (Id. 12). On or about May 24, 2012, NHSA took the Juster Work Product and proceeded to close the contemplated transaction without Juster, in violation of the Exclusivity Provision. (Id. 15). NHSA denies these allegations, but admits that it proceeded to closing on May 24, 2012, using a Certificates of Participation ( COP ) structure after determining that such a transaction structure would better meet its requirements for the refinancing. (Answer 15, ECF No. 7). NHSA alleges that Juster had the opportunity to participate but declined to do so. (Id.). Juster claims that NHSA s actions have damaged Juster in an amount not less than $41 million. (Compl. 16, ECF No. 1). III. PROCEDURAL HISTORY As set forth above, plaintiff filed a four-count Complaint against NHSA on June 6, 2012 (ECF No. 1), and defendant filed its Answer on July 13, 2012 (ECF No. 7). The Court ordered that Fed. R. Civ. P. 26 disclosures were to be exchanged on or before October 12, 2012, with fact discovery to remain open through May 22, (Pretr. Sched. Order 1 2, Oct. 5, 2012, ECF No. 10). The parties served written discovery, and agreed that all responses would be served on or before November 5, 2012, and any privilege logs would be served on or before November 12, (Id. 3). Plaintiff s counsel wrote to the Court on November 5, 2012, with intentions of requesting a revised scheduling order. (ECF No. 12). After a telephone status conference, the 3

196 Court issued an Order extending the deadline for all responses to written discovery to be served on or before December 10, 2012, and any responsive privilege log to be served on or before December 17, (Order, Nov. 13, 2012, ECF No. 13). Plaintiff issued its First Request for the Production of Documents on August 15, 2012, which included forty-nine (49) requests for documents related to defendant s answer and affirmative defenses and other relevant documents, as well as an additional list of sixty-seven (67) proposed search terms. (ECF No. 14-1). On November 28, 2012, defendant filed a letter requesting that the Court enter a protective order regarding the sixty-seven (67) electronic word searches demanded by Juster. (ECF No. 14). NHSA argues that it is entitled to a protective order because it has already produced 8000 pages of responsive documents and the additional search terms are, in its view, quite broad and vague. (Id.). Alternatively, should the Court deny its request for a protective order and require these searches, NHSA seeks an order that requires Juster to pay the costs associated with conducting these searches. (Id.). In response, Juster filed a letter in opposition to defendant s application for fee-shifting on electronic discovery. (Pl. s Letter, Nov. 28, 2012, ECF No. 15). Juster maintains that it seeks reasonable discovery that is tailored to the breach issues in this case and is limited to the timeframe of 2011 and (Id. at 1). Therefore, Juster asserts that NHSA should conduct the relevant searches, including the searches involving the sixty-seven (67) proposed terms, and bear the cost of doing so. (Id. at 1). However, if the Court decides to shift the cost of defendant s discovery to plaintiff, Juster requests that the Court order NHSA to reimburse Juster for the electronic discovery costs it has incurred already. (Id. at 10). 4

197 IV. DISCUSSION Fed. R. Civ. P. 26(c)(1) provides, in relevant part: A party or any person from whom discovery is sought may move for a protective order in the court where the action is pending or as an alternative on matters relating to a deposition, in the court for the district where the deposition will be taken. The motion must include a certification that the movant has in good faith conferred or attempted to confer with other affected parties in an effort to resolve the dispute without court action. The court may, for good cause, issue an order to protect a party or person from annoyance, embarrassment, oppression, or undue burden or expense.... Here, defendant moves for a protective order with regard to the additional sixty-seven (67) word searches requested by plaintiff. However, defendant has failed to provide any law or analysis in support of its request for a protective order. That is, defendant presents no compelling factual basis or sufficient legal background to support its motion for a protective order. It is true, as defendant argues, that several of the sixty-seven terms appear, at first blush, to be somewhat broad. See, e.g., Nov. 28, 2012, Motion for Discovery Fees, Exh. 1 (list of proposed search terms, which include SEC, fee, debt, and Tax! ). However, NHSA points out that it has complied with Juster s request to search approximately one hundred terms, including indicative, development, security, fee, and finance! Nov. 28, 2012, Letter, ECF No. 15, Exh. B (exclamation point in original). Moreover, given the nature of the dispute, the Court cannot find that the search terms are unreasonable. In addition, NHSA fails to show how it would be unreasonably cumulative or duplicative to perform the requested search discovery. Therefore, the Court cannot conclude that there is good cause for a protective order. Further, defendant has failed to comply with the plain language of Fed. R. Civ. P. 26(c)(1), which requires the inclusion of a certification that the movant has in good faith 5

198 conferred or attempted to confer with other affected parties in an effort to resolve the dispute without court action. In the absence of such a certification from defendant, the Court cannot grant its request for a protective order. Therefore, because NHSA has failed to carry its burden of demonstrating good cause, the Court denies defendant s request for a protective order. Next, defendant argues that Juster should bear the cost of the requested ESI because the scope of information sought is unduly burdensome and unreasonable. Defense counsel alleges that this matter has been one of the most document-intensive cases that the firm has ever encountered, and that NHSA has already provided Juster with over 8000 pages in hard-copy documents in response to Juster s other discovery requests. (Def. s Letter, ECF No. 14). Therefore, NHSA submits that the Court should require Juster to pay for its costs in retaining an outside vendor to provide the technical support needed to respond to Juster s ESI discovery requests. (Id.). However, for the reasons set forth below, the Court denies defendant s motion for discovery fees. There is a general presumption that the responding party must bear the expense of complying with discovery requests. Zubulake v. UBS Warburg LLC, 216 F.R.D. 280, 283 (S.D.N.Y. 2003). Therefore, the responding party has the burden of proof on a motion for costshifting. Id. To that end, cost-shifting should only be considered when electronic discovery imposes an undue burden or expense on the responding party. Id. at 287. As a preliminary matter, [c]ost-shifting is potentially appropriate only when inaccessible data is sought. Id. at 284; see Zubulake v. UBS Warburg LLC, 217 F.R.D. 309, 318 (S.D.N.Y. 2003) (Zubulake II) ( [W]hether production of documents is unduly burdensome or expensive turns primarily on whether it is kept in an accessible or inaccessible format (a distinction that corresponds closely to 6

199 the expense of production). ). [I]n the world of electronic data, thanks to search engines, any data that is retained in a machine readable format is typically accessible. Zubulake II, 217 F.R.D. at 318. Specifically, active, online data, near-line data, and offline storage/archives are typically identified as accessible electronic data. Id. at Backup tapes and erased, fragmented, or damaged data are typically identified as inaccessible electronic data. Id. at Thus, electronic data that is stored in a readily usable format is deemed accessible whereas electronic data that is not readily usable (i.e., data that must be restored, de-fragmented, or reconstructed) is considered inaccessible. Id. at 321. Here, NHSA, as the responding party, has failed to satisfy its burden of showing that the ESI sought by Juster is inaccessible. NHSA has not asserted that any of the requested data is located on backup tapes. (Def. s Mot., ECF No. 14); (see Pl. s Opp n Mem. at 6, ECF No. 15); cf. Universal Del., Inc., v. Comdata Corp., 2010 U.S. Dist. LEXIS 32158, at *22 (E.D. Pa. Mar. 31, 2010) (finding that [s]ome cost-shifting is appropriate because the data at issue in this case is kept in an inaccessible format, i.e., back-up tapes ). It has not asserted that any of the requested data is erased, fragmented, or damaged in any way. (See ECF No. 14). To the contrary, by asserting that it has hired an outside vendor to perform the word searches, NHSA has acknowledged that the ESI is accessible. It has also failed to show that the ESI sought by Juster imposes an undue burden or expense. Rather, the Court finds that NHSA seeks merely to avoid the cost associated with what it presumes to be duplicative and expensive word searches. As a result, the Court cannot find that the ESI requested by Juster falls into either category of inaccessible electronic data. Because such data is in fact accessible to NHSA, defendant must bear the attendant discovery costs. Even though defendant has failed its threshold requirement of 7

200 demonstrating that Juster seeks inaccessible ESI, this Court will nonetheless provide additional reasons for denying NHSA s fee-shifting request. The Zubulake Court set forth a seven-factor test to determine whether discovery costs should be shifted, which are weighted more-or-less in the following order: 1) the extent to which the request is specifically tailored to discover relevant information; 2) the availability of such information from other sources; 3) the total cost of production, compared to the amount in controversy; 4) the total cost of production; 5) the relative ability of each party to control costs and its incentive to do so; 6) the importance of the issues at stake in the litigation; and 7) the relative benefits to the parties of obtaining the information. 216 F.R.D. at 284. The Third Circuit has adopted that criteria for when cost-shifting might be appropriate in electronic discovery. See Wachtel v. Guardian Life Ins., 2007 U.S. Dist. LEXIS (D.N.J. June 18, 2007) (following Zubulake); see also Boeynaems v. La Fitness Int l, 285 F.R.D. 331, 336 (E.D. Pa. 2012) (stating that Zubulake is undoubtedly the leading opinion in regard to standards for requiring that the requesting party share discovery costs). Applying the Zubulake factors in light of all of the circumstances, the Court does not find that fee-shifting is warranted. First, the Court agrees with Juster that the ESI requests are sufficiently tailored to discover relevant information regarding communications between the various actors who took part in the transaction at issue. (Pl. s Opp n Mem. at 7, ECF No. 15). The requested searches are reasonable and restricted to the time period of 2011 to 2012, which is the relevant time frame during which the parties were in negotiation. Further, it is important to note that after Juster provided NHSA with the proposed discovery request, NHSA agreed in 8

201 writing that each party would bear its own costs in connection with producing discovery. (Id. at 2; ECF No. 15, Exh. A 4). Accordingly, the first factor is in favor of Juster. Second, until NHSA actually runs the requested searches, neither NHSA nor anybody else can know whether the ESI word searches will turn up information that would have been available from any other source. While it is true that the Court, on motion or on its own, must limit the frequency or extent of discovery otherwise allowed if it determines that the discovery sought is unreasonably cumulative or duplicative, or can be obtained from some other source that is more convenient, less burdensome, or less expensive, Fed. R. Civ. P. 26(b)(2), NHSA fails to show how it would be unreasonably cumulative or duplicative to perform the requested ESI discovery. As such, the Court is not compelled to impose such a limitation on plaintiff s requested ESI discovery. Further, it is irrelevant that NHSA has already turned over 8000 pages of documents to Juster because production of information in hard copy documentary form does not preclude a party from receiving that same information in computerized/electronic form. Anti-Monopoly, Inc. v. Hasbro, Inc., 1995 U.S. Dist. LEXIS 16355, at *2 (S.D.N.Y. Nov. 3, 1995). Therefore, the second Zubulake factor also weighs in favor of Juster. Factors three through five concern the financial aspect of the discovery request. The amount in controversy here is well in excess of the amount that NHSA estimates it will cost to perform the requested ESI discovery. NHSA alleges that it will cost between $6000 to $16,000 to process the word searches and eliminate duplicates. (Def. s Letter, ECF No. 14). The amount that Juster alleges it is owed is not less than $41 million. (Answer 15, ECF No. 7). Further, Juster alleges that NHSA s resources dwarf the estimated cost of the requested discovery.... The estimated discovery costs at issue here are negligible in comparison to the NHSA s 9

202 resources. (Pl. s Opp n Letter at 9, ECF No. 15). Given the amount of damages at stake in this litigation, and NHSA s ability to absorb the cost of the requested ESI discovery, the projected discovery costs are not so substantial as to support a conclusion that fee-shifting is warranted. See, e.g., Zubulake, 216 F.R.D. at ( [T]he total cost of restoring the remaining seventytwo tapes extrapolates to $165, the cost of restoration is surely not significantly disproportionate to the projected value of this case. This factor weighs against cost-shifting. ). As a result, Zubulake factors three through five weigh in favor of Juster. Factors six and seven are less critical to the analysis of whether cost-shifting is appropriate in electronic discovery. See Zubulake, 216 F.R.D. at 284 (listing the factors for determination of fee-shifting in descending order of importance). As such, factors six and seven, standing alone, are generally insufficient to support a party s motion for fee-shifting. Here, factors six and seven do not weigh in favor of plaintiff, nor do they necessarily weigh in favor of defendant. The sixth factor examines the actual allegations involved in the litigation. It is usually invoked where the litigation presents a particularly novel issue. Zubulake, 216 F.R.D. at 289. The litigation here is a private contractual dispute between fairly sophisticated parties. The issues at stake include the nature of the parties negotiations and what one party could have reasonably understood the other party to have meant during their discussions. The outcome of this litigation does not have great potential for broad public impact because it does not implicate any widely-applicable public policy. See Zubulake II, 217 F.R.D. at 321 (noting that cases involving toxic tort class actions, environmental actions, so-called impact or social reform litigation, cases involving criminal conduct, or cases implicating important legal or constitutional questions are of the ilk that may have broad public impact). Therefore, the sixth factor is not 10

203 dispositive to the instant matter. See id. at 321 (noting that the sixth factor will only rarely come into play ). The seventh factor concerns the potential benefits to either party that may result from the ESI discovery. However, any such benefits cannot be known until the word searches are performed. The last factor is thus also neutral. Having established that the Zubulake factors, on balance, weigh in favor of plaintiff, this Court finds that principles of fundamental fairness further support its denial of defendant s feeshifting request. See Boeynaems, 285 F.R.D. at 335. Here, both parties have requested ESI discovery from the other party, and both parties agreed to pay their own costs in producing discovery. (See Exh. A to ECF No. 15). The amount of discovery requested by each party is comparable Juster requested sixty-seven (67) word searches in addition to forty-nine (49) specific documents, and NHSA requested the results from one hundred (100) search terms. This is not a situation where one party is unduly burdened by the opposing party s discovery requests, which may implicate fee-shifting. See Major Tours, Inc. v. Colorel, 2009 U.S. Dist. LEXIS 97554, at *11, 19 (determining that plaintiffs and defendants should share the cost of searching defendants backup tapes, given that thousands of relevant documents were s that had been produced already to plaintiffs). Therefore, because defendant has failed to demonstrate that it has been unduly burdened by plaintiff s ESI request or that plaintiff s request is eminently unreasonable, it fails to meet its burden of showing why cost-shifting is appropriate. Because the Court finds that defendant is not entitled to a fee-shifting order, plaintiff s subsequent request that NHSA should reimburse it for the electronic discovery it has incurred during the course of this litigation is rendered moot. Therefore, the Court declines to analyze the merits of plaintiff s request for fee-shifting. 11

204 V. CONCLUSION For the reasons above, the Court denies defendant s motion for a protective order, as well as its alternative request for fee-shifting. The Court also denies as moot plaintiff s request for fee-shifting. An Order consistent with this Opinion will follow. Date: February 11, 2013 s/ Michael A. Hammer United States Magistrate Judge 12

205 Dictum The newsletter of the NJSBA Young Lawyers Division Vol. 37, No. 1 March 2013 New Jersey State Bar Association Dictum 1 Go to Index

206 Recent Developments in New Jersey Law: Social Media and More by Fernando M. Pinguelo, Esq. and Chris Borchert From MySpace and Facebook to Twitter and LinkedIn, online social networking platforms have radically changed the way we share, store and consume information. Today, users can tap social media outlets for everything from big data storage to daily deals and discounts; and, with recent advances in phone and tablet technology, accessing online content remotely has never been easier. As a result, state legislatures and courts are beginning to take notice of the far-reaching implications of social and other cyber-related media. In New Jersey jurisdictions, key cases and statutes have emerged that address these broad legal implications. In fact, in 2011, New Jersey enacted the nation s most comprehensive anti-cyber bullying law the Anti-Bullying Bill of Rights Act. It is no secret that today s social media platforms are characterized by wide-ranging functionality, and so lawyers of all practices must stay current on the laws and cases relating to social and cyber media. What follows are a few notable cases that caught our attention. Social Media and School Speech In J.S. ex rel. Snyder v. Blue Mountain School District, 1 a student created a parody MySpace profile of her principal, which contained adult language and sexually explicit content. Although the profile was created at the student s home on a computer owned by her parents, the school nevertheless suspended her for 10 days for twice violating the school s disciplinary code (false accusation about a school staff member and a copyright violation of its computer use policy for misappropriating the principal s photograph from the school website). The student s parents filed a Section 1983 action against the school district, claiming violation of free speech rights, due process rights, and state law. Takeaway: Applying the Tinker standard, the Third Circuit found that the MySpace page did not substantially disrupt the operation of the school, nor could school officials have reasonably forecasted a substantial disruption. The court also found that the speech did not make its way on campus sufficiently to classify it as school speech. Social Media and Juror Bias In U.S. v. Fumo, 2 the Third Circuit considered whether the district court had erred in its refusal to grant the defendant a new trial on the grounds of a jury s exposure to extraneous information, and the purported prejudice and partiality that may have resulted. The defendant claimed that a juror who posted comments relating to the case on his Facebook and Twitter accounts brought widespread public attention to the jury s deliberations, creating a cloud of intense and widespread media coverage...and [the] public expectation that a verdict [wa]s imminent[,] thereby violating his Sixth Amendment right to a fair and impartial trial. Takeaway: A trial court s jury instructions before and at the close of a trial should include instructions not to communicate information to anyone by any means, including social media, about the case. The Third Circuit found no plausible theory for how the defendant suffered any prejudice, let alone substantial prejudice, from the juror s Facebook and Twitter comments. Social Media and Shield Law Protections In Too Much Media v. Hale, 3 the defendant, a self-described journalist, posted Internet messages about Too Much Media, LLC (TMM), a company that produced software used in the adult entertainment industry. Specifically, the defendant alleged that a breach in TMM s software exposed the personal information of TMM customers who believed they were accessing pornographic websites anonymously. Claiming she had conducted a probe on the alleged breach, the defendant posted that TMM violated New Jersey s identity theft protection laws, threatened people who questioned its conduct, and profited from the alleged New Jersey State Bar Association Dictum 9 Go to Index

207 breach. TMM sued, alleging the defendant s posts were defamatory and made in a false light. When TMM sought to depose her during discovery to ascertain her alleged sources, the defendant moved for a protective order, asserting she was a reporter entitled to protection under New Jersey s Shield Law. Takeaway: While New Jersey s Shield Law affords broad protections to news media, including non-traditional news outlets such as blogs, a self-appointed newsperson is not necessarily a reporter entitled to Shield Law protections. Those seeking to invoke the privilege must demonstrate that the means by which they disseminate news is similar to traditional news sources, such as newspapers, magazines, etc. Social Media and Employment In Pietrylo v. Hillstone Restaurant Group, 4 the plaintiffs, who were employed as servers at a restaurant owned by the defendant, created a MySpace group to vent about any BS we deal with at work without any eyes spying in on us. The group was created as a private group, and invitations were required to join. Central to the case is the fact that an invited employee, who was an authorized user of the group, accessed the page for one of the restaurant managers and later provided the password to another manager. When managers learned that comments disparaged the restaurant and its managers and customers, they terminated the plaintiffs employment. The plaintiffs filed suit, alleging violations of federal and state stored communications statutes, wrongful termination, and invasion of privacy. Takeaway: Content found on blogs and public social networking profiles that employees generate may still enjoy the benefit of protected privacy when restrictions on access are implemented. Employers should exercise caution when asking for access to private employee information. Cloud Computing and Legal Ethics While cloud computing (i.e., the delivery of computing and storage capacity as a service to users over the Internet) can offer significant benefits to lawyers in managing their practices, particularly solo and small firm practitioners for whom flexibility and cost control are a must, attorneys must take careful steps before employing Internet or cloud-based software programs or using remote sites to store client-related information. In Advisory Committee on Professional Ethics Opinion 701 (April 2006), the New Jersey Advisory Committee on Professional Ethics addressed the ethical implications associated with cloud-based data storage. Takeaway: The ethical issues underlying the use of cloud services are the duty of confidentiality owed to clients and the duty to serve competently. Attorneys must take reasonable, affirmative steps to ensure the confidentiality of client information that travels over the Internet or other network. This obligation includes assuring to a reasonable extent that a third-party cloud services provider is aware of the lawyer s obligation of confidentiality, and of its own obligation, whether by contract, professional standards, or otherwise, to assist in preserving confidentiality. Fernando M. Pinguelo, a partner of Norris McLaughlin & Marcus, P.A. and chair of its cyber security and data protection group, is a trial lawyer who devotes his practice to complex business disputes with an emphasis on cyber, media, and employment matters in federal and state courts. Chris Borchert is a second-year student at the University of Connecticut School of Law, where he is pursuing a certificate in intellectual property in addition to his J.D. As a summer associate with the firm, he focused primarily on matters relating to e-discovery, cyber security and social media. This article was first distributed at a New Jersey Institute for Continuing Legal Education seminar. Endnotes 1. J.S. ex rel. Snyder v. Blue Mountain School Dist., 650 F. 3d 915 (3rd Cir. 2011). 2. U.S. v. Fumo, 655 F. 3d 288 (3rd Cir. 2011). 3. Too Much Media v. Shellee Hale, 206 N.J. 209 (2011). 4. Pietrylo v. Hillstone Restaurant Group (D.N.J. 2008). New Jersey State Bar Association Dictum 10 Go to Index

208 Romano v Steelcase Inc. :: September, 2010 :: New York Other Courts Decisions :: New... Page 1 of 6 Justia.com Lawyer Directory Legal Answers Law more? in Other Courts Sign In Search Romano v Steelcase Inc. Justia on Romano v Steelcase Inc NY Slip Op [30 Misc 3d 426] September 21, 2010 Spinner, J. Supreme Court, Suffolk County Published by New York State Law Reporting Bureau pursuant to Judiciary Law 431. As corrected through Wednesday, February 23, 2011 [*1] Kathleen Romano, Plaintiff, v Steelcase Inc. et al., Defendants. Supreme Court, Suffolk County, September 21, 2010 APPEARANCES OF COUNSEL Gallagher Gosseen Faller & Crowley, Garden City, for Steelcase Inc., defendant. John T. Ryan & Associates, Riverhead, for Educational & Institutional Cooperative Services Inc., defendant. Orrick Herrington & Sutcliffe LLP, New York City, for Facebook. Kelner & Kelner, New York City, for plaintiff. {**30 Misc 3d at 427} OPINION OF THE COURT Jeffrey Arlen Spinner, J. Ordered, that defendant Steelcase's motion is hereby granted as set forth herein below. Defendant Steelcase moves this court for an order granting said defendant access to plaintiff's current and historical Facebook and MySpace pages and accounts, including all deleted pages and related information upon the grounds that plaintiff has placed certain information on these social networking sites which is believed to be inconsistent with her claims in this action concerning the extent and nature of her injuries, especially her claims for loss of enjoyment of life. The present application was brought on by order to show cause. The court has reviewed the submissions both in favor of and in opposition to the relief sought, as well as the applicable federal statutory law, specifically the Stored [*2]Communications Act (18 USC 2701 et seq.), which prohibits an entity such as Facebook and MySpace from disclosing such information without the consent of the owner of the account (see 18 USC 2702 [b] [3]; Flagg v City of Detroit, 252 FRD 346, 352 [ED Mich 2008]). Scope of Permissible Discovery More Share Pursuant to CPLR 3101, there shall be full disclosure of all nonprivileged matter which is material and necessary to the defense or prosecution of an action. To this end, trial courts have broad discretion in the supervision of discovery, and in determining what is "material and necessary" (see Allen v Crowell-Collier Publ. Co., 21 NY2d 403 [1968]; Andon v Mott St. Assoc., 94 NY2d 740 [2000]; Cabellero v City of New York, 48 AD3d 727 [2d Dept 2008]). Within the context of discovery, "necessary" has been interpreted as meaning " 'needful' and not indispensable" (see Allen at 407). The "material and necessary" standard is to be interpreted liberally, requiring disclosure of "any facts bearing on the controversy which will assist preparation for trial by sharpening the issues and reducing delay and prolixity. The test is one of usefulness and reason" (see Allen, 21 NY2d at 406; Andon, supra; Hoenig v Westphal, 52 NY2d 605, 608 [1981] [pretrial discovery is to be encouraged, limited only by a test for materiality of "usefulness and reason"]).{**30 Misc 3d at 428} Like Daily Opinion Summaries Subscribe to Justia's FREE Daily Newsletter Opinion Summaries Subscribe Now Today on Verdict What s the Matter with North Dakota and Arkansas? Two State Legislatures Pass Highly Restrictive and Unconstitutional Abortion Laws Justia columnist and Hofstra law professor Joanna Grossman comments on two states decisions to pass abortion laws despite the fact that under U.S. Supreme Court precedent, it is very clear that these new laws are unconstitutional. By Joanna L. Grossman Ask a Lawyer Question: Ask Question 129,092 people like this. Sign Up to see what your friends like. About Legal Answers Connect with Justia Follow justiacom Fan Justia Justia Like 129,092 people like Justia. Facebook social plugin Find a Lawyer Search Browse Lawyers Lawyers - Get Listed Now! Get a free full directory profile listing Each discovery request is to be decided on a case-by-case basis, keeping in mind the 4/2/2013

209 Romano v Steelcase Inc. :: September, 2010 :: New York Other Courts Decisions :: New... Page 2 of 6 strong public policy in favor of open disclosure (see Andon sought is sufficiently related to the issues in litigation so as to m it in preparation for trial reasonable, then discovery should be ; Matter of Beryl, 118 AD2d 705 [2d Dept 1986]). It information sought may not be admissible at trial as "pretria only to proof that is admissible but also to matters that may l admissible proof" (see Twenty Four Hour Fuel Oil Corp. v Hunte 175, 175 [1st Dept 1996]; Polygram Holding, Inc. v Cafaro, Dept 2007] [" 'disclosure extends not only to admissible proof documents which may lead to the disclosure of admissible p which might be used in cross-examination"]). Information Sought from Internet Sites Plaintiffs who place their physical condition in disclosure material which is necessary to the Westphal, supra). Accordingly, in an action seeking damage discovery is generally permitted with respect to materials that the issue of damages and the extent of a plaintiff's injury (se York, 205 AD2d 755 [2d Dept 1994]), including a plaintiff's cla of life (see Orlando v Richmond Precast, Inc., 53 AD3d 534 action to recover damages for personal injuries, records so necessary to the defense regarding plaintiff's claim of loss of en v City of New York, 276 AD2d 789 [2d Dept 2000]; Mora v Med. Ctr. of N.Y., 8 Misc 3d 868 [Sup Ct, NY County 2005]). Thus, in Sgambelluri v Recinos (192 Misc 2d 777 [Sup Ct, Na action arising out of a motor vehicle accident, the court held video taken two years after the incident was clearly permanency of injuries. As a result of the accident, permanent injuries to her neck and back, and testified at her de longer participate in certain activities such as running or hors sought a copy of her wedding video on the basis that it might various activities such as dancing, which would be relevant objected on{**30 Misc 3d at 429} the basis of the personal n court decided in favor of disclosure, noting its relevancy to the injuries. In so finding, the court reasoned that although the tape, as contemplated by CPLR 3101 (i), the statute's "langua encompass any film, photograph or videotape... involving paragraph one of subdivision (a), i.e., a party. This is consisten of New York courts, allowing liberal disclosure. Moreover, subdivision (i) only strengthens the argument for open disclo [internal quotation marks omitted].) Like the plaintiff in Sgambelluri, plaintiff herein also claims sh injuries as a result of the incident and that she can no long activities or that these injuries have affected her enjoyment of to plaintiff's claims, Steelcase contends that a review of the pub MySpace and Facebook pages reveals that she has an to Florida and Pennsylvania during the time prohibited such activity. In light of this, defendant sought to deposition regarding her MySpace and Facebook accounts, to those depositions, served plaintiff with a notice for discovery an inter alia, "authorizations to obtain full access to and copies historical records/information on her Facebook and MySpace refused to provide the requested authorizations. Both Facebook and MySpace are social networking sites w information about their personal lives, including posting ph information about what they are doing or thinking. Indeed, Fac "it helps you share information with your friends and people "Facebook is about sharing information with others."[fn1] "social networking service that allows Members to create unique in order to find and communicate with old and new friends" an "online community" where "you can share photos, journals growing network{**30 Misc 3d at 430} of mutual 4/2/2013

210 Romano v Steelcase Inc. :: September, 2010 :: New York Other Courts Decisions :: New... Page 3 of 6 lifestyle portal that reaches millions of people around the world."[fn3] Both sites allow the user to set privacy levels to control with whom they share their information. growing network{**30 Misc 3d at 430} of mutual The information sought by defendant regarding plaintiff's Facebook and MySpace accounts is both material and necessary to the defense of this action and/or could lead to admissible evidence. In this regard, it appears that plaintiff's public profile page on Facebook shows her smiling happily in a photograph outside the confines of her home despite her claim that she has sustained permanent injuries and is largely confined to her house and bed. In light of the fact that the public portions of plaintiff's social networking sites contain material that is contrary to her claims and deposition testimony, there is a reasonable likelihood that the private portions of her sites may contain further evidence such as information with regard to her activities and enjoyment of life, all of which are material and relevant to the defense of this action. Preventing defendant from accessing plaintiff's private postings on Facebook and MySpace would be in [*3]direct contravention to the liberal disclosure policy in New York State. Although there is no New York case law directly addressing the issues raised by this application, there are instructive cases from other jurisdictions. Recently, in Ledbetter v Wal-Mart Stores, Inc. (2009 WL , 2009 US Dist LEXIS [Civ Action No. 06-cv WYD-MJW, D Colo, Apr. 21, 2009]), defendant store sought, via subpoena, production of the content of plaintiffs' social networking sites.[fn4] Information contained on the public access areas contradicted plaintiffs' allegations regarding the effect of their injuries on their daily lives. When the networking sites refused to provide the information absent plaintiffs' consent or request, defendant moved to compel production and plaintiffs moved for a protective order. Both plaintiffs had claimed physical and psychological injuries as a result of the accident which gave rise to lawsuit. By order dated April 21, 2009, Magistrate Judge Watanabe denied plaintiffs' motion and held that the information sought by the subpoenas{**30 Misc 3d at 431} was "reasonably calculated to lead to the discovery of admissible evidence as is relevant to the issues in the case." (2009 WL , *2, 2009 US Dist LEXIS , *5.) Likewise, in Leduc v Roman (2009 CarswellOnt 843 [Feb. 20, 2009]), a matter pending in the Superior Court of Justice, Ontario, Canada, defendant also requested production of the plaintiff's Facebook pages, including private pages. Plaintiff claimed that as a result of injuries allegedly sustained in a car accident, his enjoyment for life had lessened. Canadian law requires that each party disclose every document relating to any matter in the action over which he has possession or control absent a claim of privilege. Plaintiff had failed to disclose the information, which defendant only learned about following a defense psychiatric examination. After only being able to access the limited portions of plaintiff's public profile page, defendant sought an order requiring production of all site materials as well as preservation of the materials. The decision denying the request was reversed on appeal, with the appellate court disagreeing that defendant was on a fishing expedition. In this regard, Judge Brown noted that it was "beyond controversy" that a person's Facebook pages may contain relevant documents ( 23); that other Canadian cases had permitted into evidence photographs posted on a person's Facebook page showing them engaged in activities despite their claim to the contrary; and that it is reasonable to infer from the social networking purpose of Facebook that even if a person only maintains a private profile, with the public profile merely listing their name, relevant information exists on their limited-access private pages ( 36). In deciding to permit the examination into the private Facebook profile, the court set forth: "To permit a party claiming very substantial damages for loss of enjoyment of life to hide behind self-set privacy controls on a website, the primary purpose of which is to enable people to share information about how they lead their social lives, risks depriving the opposite party of access to material that may be relevant to ensuring a fair trial."( 35; see also Kent v Laverdiere, 2009 CarswellOnt 1986 [Ont Sup Ct, Apr. 14, 2009] [as plaintiff asserted that accident disfigured her and lessened her enjoyment of life, any photos on Facebook or [*4]MySpace showing her in healthy state, enjoying life, would be relevant]; Bishop v Minichiello, 2009 BCSC 358 [Apr. 7, 2009] [defendant's motion for production of{**30 Misc 3d at 432} plaintiff's computer's hard drive so it could analyze how much time plaintiff spent on Facebook granted as the information sought was relevant to the issues in the case]; Goodridge v King, 2007 CarswellOnt 7637 [Ont Sup Ct, Oct. 30, 2007] [in action in which plaintiff 4/2/2013

211 Romano v Steelcase Inc. :: September, 2010 :: New York Other Courts Decisions :: New... Page 4 of 6 claimed various injuries, including loss of enjoyment of life and disfigurement following a car accident, photos posted by plaintiff on her Facebook account were evidence to the contrary, showing her socializing and dating]; Kourtesis v Joris, 2007 CarswellOnt 5962 [Ont Sup Ct, Sept. 24, 2007] [in proceeding concerning costs, court noted that during trial, Facebook photos of plaintiff were important element of case; apparently plaintiff testified that she no longer had a social life because of her injuries, yet the photographs taken after the accident showed her at a party].)[fn5] Thus, it is reasonable to infer from the limited postings on plaintiff's public Facebook and MySpace profile pages that her private pages may contain material and information that are relevant to her claims or that may lead to the disclosure of admissible evidence. To deny defendant an opportunity to access these sites not only would go against the liberal discovery policies of New York favoring pretrial disclosure, but would condone plaintiff's attempt to hide relevant information behind self-regulated privacy settings. Plaintiff's Privacy Concerns Production of plaintiff's entries on her Facebook and MySpace accounts would not be violative of her right to privacy,[fn6] and any such concerns are outweighed by defendant's need for the information. The Fourth Amendment's right to privacy protects people, not places (see Katz v United States, 389 US 347, 351 [1967] ["What a person knowingly exposes to the public, even in his own home or office, is not a subject of Fourth Amendment protection"]). In determining whether a right to privacy exists via the Fourth Amendment, courts apply the reasonableness{**30 Misc 3d at 433} standard set forth in the concurring opinion of Justice Harlan in Katz: "first that a person have exhibited an actual (subjective) expectation of privacy and, second, that the expectation be one that society is prepared to recognize as reasonable." (Id. at 361 [internal quotation marks omitted].) New York courts have yet to address whether there exists a right to privacy regarding what one posts on their on-line social networking pages such as Facebook and MySpace. However, [*5]whether one has a reasonable expectation of privacy in Internet postings or s that have reached their recipients has been addressed by the Second Circuit, which has held that individuals may not enjoy such an expectation of privacy (see United States v Lifshitz, 369 F3d 173, 190 [2004]): " 'Users would logically lack a legitimate expectation of privacy in the materials intended for publication or public posting. They would lose a legitimate expectation of privacy in an that had already reached its recipient; at this moment, the er would be analogous to a letter-writer, whose expectation of privacy ordinarily terminates upon delivery of the letter.' " (Quoting Guest v Leis, 255 F3d 325, 333 [6th Cir 2001].) Likewise, whether one has a reasonable expectation of privacy in s and other writings that have been shared with others, including entries on Facebook and MySpace, has been addressed by the United States District Court of New Jersey, which ordered such entries produced in Beye v Horizon Blue Cross Blue Shield of N.J. (Civ Case No [D NJ, Dec. 14, 2007]). In this regard, the court stated that "[t]he privacy concerns are far less where the beneficiary herself chose to disclose the information." As to the entries which had not been shared with others, they were to be preserved. At issue in Beye were on-line journals and diary entries of minor children who had been denied health care benefits for their eating disorders (see also Moreno v Hanford Sentinel, Inc., 172 Cal App 4th 1125, 91 Cal Rptr 3d 858 [Ct App, 5th Dist 2009] [no person would have reasonable expectation of privacy where that person took affirmative act of posting own writing on MySpace, making it available to anyone with a computer and opening it up to the public eye]; Dexter v Dexter, 2007 WL , 2007 Ohio App LEXIS 2388 [Ct App, 11th Dist, Portage County 2007] [no reasonable expectation of privacy regarding MySpace writings open to public view]).{**30 Misc 3d at 434} Indeed, as neither Facebook nor MySpace guarantee complete privacy, plaintiff has no legitimate reasonable expectation of privacy. In this regard, MySpace warns users not to forget that their profiles and MySpace forums are public spaces,[fn7] and Facebook's privacy policy set forth, inter alia, that "[y]ou post User Content... on the Site at your own risk. Although we allow you to set privacy options that limit access to your pages, please be aware that no security measures are perfect or impenetrable." 4/2/2013

212 Romano v Steelcase Inc. :: September, 2010 :: New York Other Courts Decisions :: New... Page 5 of 6 Further that "[w]hen you use Facebook, certain information you post or share with third parties (e.g., a friend or someone in your network), such as personal information, comments, messages, photos, videos... may be shared with others in accordance with the privacy settings you select. All such sharing of information is done at your own risk. Please keep in mind that if [*6]you disclose personal information in your profile or when posting comments, messages, photos, videos, Marketplace listing or other items, this information may become publicly available."[fn8] Thus, when plaintiff created her Facebook and MySpace accounts, she consented to the fact that her personal information would be shared with others, notwithstanding her privacy settings. Indeed, that is the very nature and purpose of these social networking sites, else they would cease to exist. Since plaintiff knew that her information may become publicly available, she cannot now claim that she had a reasonable expectation of privacy. As recently set forth by commentators regarding privacy and social networking sites: given the millions of users, "[i]n this environment, privacy is no longer grounded in reasonable expectations, but rather in some theoretical protocol better known as wishful thinking."[fn9] Further, defendant's need for access to the information outweighs any privacy concerns that may be voiced by plaintiff. Defendant has attempted to obtain the sought-after information{**30 Misc 3d at 435} via other means: e.g., via deposition and notice for discovery; however, these have proven to be inadequate since counsel has thwarted defendant's attempt to question plaintiff in this regard or to obtain authorizations from plaintiff for the release of this information. The materials, including photographs, contained on these sites may be relevant to the issue of damages and may disprove plaintiff's claims. Without access to these sites, defendant will be at a distinct disadvantage in defending this action. Ordered, that defendant Steelcase's motion for an order granting said defendant access to plaintiff's current and historical Facebook and MySpace pages and accounts, including all deleted pages and related information, is hereby granted in all respects; and it is further ordered, that, within 30 days from the date of service of a copy of this order, as directed herein below, plaintiff shall deliver to counsel for defendant Steelcase a properly executed consent and authorization as may be required by the operators of Facebook and MySpace, permitting said defendant to gain access to plaintiff's Facebook and MySpace records, including any records previously deleted or archived by said operators. Footnotes Footnote 1: Facebook Principles, (last visited Apr. 3, 2009). Footnote 2: About Us, MySpace.com/index.dfm?fuseaction=misc.aboutus (last visited June 16, 2009). Footnote 3: MySpace Safety Highlights, frseaction=cms.viewpage & placement=safety (last visited June 18, 2009). Footnote 4: Facebook, MySpace and Meetup.com. Footnote 5: See Charles Foster, Uncovering the Truth: Social Networks are a Treasure Trove of Information, Claims Canada, Oct./Nov. 2008, (last viewed June 18, 2009). Footnote 6: In New York, there is no common-law right to privacy. (See Ava v NYP Holdings, Inc., 20 Misc 3d 1108[A], 2008 NY Slip Op 51281[U] [Sup Ct, NY County 2008].) Footnote 7: MySpace General Tips, frseaction=cms.veiwpage & placement=safety_pagetips (last visited June 18, 2009). Footnote 8: Facebook Principles, effective as of November 26, 2008, (last viewed June 18, 2009). 4/2/2013

213 Romano v Steelcase Inc. :: September, 2010 :: New York Other Courts Decisions :: New... Page 6 of 6 Footnote 9: Dana L. Fleming and Joseph M. Herlihy, What Happens When the College Rumor Mill Goes Online?, Privacy, Defamation and Online Social Networking Sites, 53 Boston BJ [No. 1] 16 (Jan./Feb. 2009). Copyright Justia :: Company :: Terms of Service :: Privacy Policy :: Contact Us 4/2/2013

214 "Private" Facebook and MySpace Postings are Discoverable : E-Discovery Law Alert: M... Page 1 of 2 E-Discovery Law Alert Posted at 1:44 PM on October 11, 2010 by Melissa DeHonney "Private" Facebook and MySpace Postings are Discoverable A New York trial court has ordered a personal injury plaintiff to produce her Facebook and MySpace postings, notwithstanding that plaintiff self-designated them as private. Justice Jeffrey Arlen Spinner, in Romano v. Steelcase Inc., 2010 N.Y. Slip Op , 2010 N.Y. Misc. LEXIS 4538 (N.Y. Sup. Ct., Suffolk Cty. Sept. 21, 2010), reasoned that New York s "liberal discovery policies" favored allowing access to posts that might undermine plaintiff's claim for loss of enjoyment of life and further that, "as neither Facebook nor MySpace guarantee complete privacy, Plaintiff has no legitimate reasonable expectation of privacy." Read the decision here. The Romano opinion suggests that, if a discovering party makes a threshold showing that content posted on social media websites is relevant, discovery may be had: "In light of the fact that the public portions of Plaintiff s social networking sites contain material that is contrary to her claims and deposition testimony, there is a reasonable likelihood that the private portions of her sites may contain further evidence such as information with regard to her activities and enjoyment of life, all of which are material and relevant to the defense of this action." As with traditional modes of discovery, it appears that "fishing expeditions" will not be allowed. Most of the cases the Romano court cited where access to social media was granted were likewise personal injury actions in which the individual plaintiffs were ordered to produce their online journals. Still, the opinion signals to businesses that they, too, might have to produce content posted on social media sites. With more and more businesses promoting themselves through this medium, the attendant preservation obligations should be considered. Posting potentially discoverable information with thirdparties can complicate compliance. For the same reason, businesses should be mindful of what their employees post on social networking sites. An employee s postings may be deemed to reflect on the company and might even bear on matters relevant to a lawsuit in which the company is a party. All too often, what people say online is as candid as what they might say to a few close co-workers behind closed doors. Yet, online posts are available for the world to see. Even seemingly innocuous posts -- for example, "I was in way over my head today at work" -- can prove damaging in litigation. The most prudent approach would include both restricting employees' access to these sites at the office and implementing a social media policy addressing employees' online postings. Additional discussions on this topic from members of the Gibbons Employment Law Department and the E-Discovery Task Force can be found here and here. 4/2/2013

215 "Private" Facebook and MySpace Postings are Discoverable : E-Discovery Law Alert: M... Page 2 of 2 Melissa DeHonney is an Associate on the Gibbons E-Discovery Task Force. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 4/2/2013

216 McCann v Harleysville Ins. Co. of N.Y. :: November, 2010 :: New York Appellate Divisi... Page 1 of 1 Justia.com Lawyer Directory Legal Answers Law more? Sign In in Appellate Division, Fourth Department Search Justia > US Law > US Case Law > New York Case Law > New York Appellate Division, Fourth Department Decisions > November, 2010 > McCann v Harleysville Ins. Co. of N.Y. NEW - Receive Justia's FREE Daily Newsletters of Opinion Summaries for the US Supreme Court, all US Federal Appellate Courts & the 50 US State Supreme Courts and Weekly Practice Area Opinion Summaries Newsletters. Subscribe Now McCann v Harleysville Ins. Co. of N.Y. Share Justia on Like 129,094 people like this. Sign Up to see what your friends like. McCann v Harleysville Ins. Co. of N.Y NY Slip Op [78 AD3d 1524] November 12, 2010 Appellate Division, Fourth Department Published by New York State Law Reporting Bureau pursuant to Judiciary Law 431. As corrected through Wednesday, January 19, 2011 Kara R. McCann, Respondent, v Harleysville Insurance Company of New York, Appellant. (Appeal No. 1.) [* 1] Chelus, Herdzik, Speyer & Monte, P.C., Buffalo (Christopher R. Poole of counsel), for defendant-appellant. Anspach Meeks Ellenberger LLP, Buffalo (David M. Stillwell of counsel), for plaintiff - respondent. Appeal from an order of the Supreme Court, Erie County (James H. Dillon, J.), entered August 19, 2009 in a personal injury action. The order denied the motion of defendant to compel disclosure. It is hereby ordered that the order so appealed from is unanimously affirmed without costs. Memorandum: Plaintiff commenced an action seeking damages for injuries she Daily Opinion Summaries Subscribe to Justia's FREE Daily Newsletter Opinion Summaries Subscribe Now Today on Verdict What s the Matter with North Dakota and Arkansas? Two State Legislatures Pass Highly Restrictive and Unconstitutional Abortion Laws Justia columnist and Hofstra law professor Joanna Grossman comments on two states decisions to pass abortion laws despite the fact that under U.S. Supreme Court precedent, it is very clear that these new laws are unconstitutional. By Joanna L. Grossman Ask a Lawyer Question: sustained when the vehicle she was operating collided with a vehicle driven by defendant's insured. Plaintiff thereafter settled that action and commenced the instant action against defendant seeking "supplementary uninsured/underinsured motorist coverage." In appeal No. 1, defendant appeals from an order denying its motion to compel disclosure of photographs and seeking "an authorization for plaintiff's Facebook account." According to defendant, the information sought was relevant with respect to the issue whether plaintiff sustained a serious injury in the accident. We conclude in appeal No. 1 that Supreme Court properly denied defendant's motion "as overly broad," without prejudice "to service of new, proper discovery demands" (see generally Slate v State of New York, 267 AD2d 839, 841 [1999]). In appeal No. 2, defendant appeals from an order denying its subsequent motion Ask Question About Legal Answers Connect with Justia Follow justiacom Fan Justia Justia Like 129,094 people like Justia. seeking to compel plaintiff to produce photographs and an authorization for plaintiff's Facebook account information and granting plaintiff's cross motion for a protective order. Although defendant specified the type of evidence sought, it failed to establish a factual predicate with respect to the relevancy of the evidence (see Crazytown Furniture v Brooklyn Union Gas Co., 150 AD2d 420, 421 [1989]). Indeed, defendant essentially sought permission to conduct "a fishing expedition" into plaintiff's Facebook account based on the mere hope of finding relevant evidence (Auerbach v Klein, 30 AD3d 451, Facebook social plugin Find a Lawyer Search Browse Lawyers 452 [2006]). Nevertheless, although we conclude that the court properly denied defendant's motion in appeal No. 2, we agree with defendant that the court erred in granting plaintiff's cross motion for a protective order. Under the circumstances Lawyers - Get Listed Now! Get a free full directory profile listing presented here, the court abused its discretion in prohibiting defendant from seeking disclosure of plaintiff's Facebook [*2]account at a future date. We therefore modify the order in appeal No. 2 accordingly. Present Martoche, J.P., Lindley, Sconiers, Pine and Gorski, JJ. Copyright Justia :: Company :: Terms of Service :: Privacy Policy :: Contact Us 4/2/2013

217 New York's Appellate Division Finds Facebook Accounts Off-Limits When Discovery D... Page 1 of 2 E-Discovery Law Alert Posted at 12:54 PM on January 31, 2011 by Robert D. Brown, Jr. New York's Appellate Division Finds Facebook Accounts Off-Limits When Discovery Demands are Non-Specific In McCann v. Harleysville Insurance Co. of New York, 910 N.Y.S.2d 614, 2010 N.Y. App. Div. LEXIS 8396 (N.Y. App. Div. Nov. 12, 2010), New York s Appellate Division, Fourth Department affirmed the trial court s refusal to compel Plaintiff to produce information regarding or provide access to her Facebook account. Plaintiff was injured in an auto accident with one of Harleysville s insured. She filed a personal injury suit against the insured, which resulted in a settlement. Plaintiff thereafter commenced a new action directly against Harleysville for certain uninsured/underinsured auto insurance benefits. During discovery, Harleysville sought disclosure of photographs from Plaintiff and an authorization for access to her Facebook account. Upon Plaintiff s refusal, Harleysville moved to compel disclosure of Plaintiff s Facebook account information. Harleysville argued that the information sought was relevant to whether McCann suffered a serious injury in the accident without specifying why. The trial court disagreed, finding that Harleysville s request was overly broad, but noted that the denial was without prejudice to service of new, proper discovery demands. Harleysville then made a second request for access to Plaintiff s Facebook account, which the Fourth Department noted specified the type of evidence Harleysville sought. Plaintiff again refused to disclose her Facebook account information and Harleysville again moved to compel. The trial court again denied the motion, finding that Harleysville failed to establish the relevancy of the evidence sought, and further granted Plaintiff s cross-motion for a protective order. On appeal by Harleysville, the Fourth Department, in a succinct opinion, held that the trial court properly denied Harleysville s two motions to compel. The Court likened Harleysville s motions to a request for permission to conduct a fishing expedition into Plaintiff s Facebook account in hopes of finding relevant evidence. It disagreed, however, with the trial court s entry of a protective order, determining that Harleysville should be allowed to seek disclosure of Plaintiff s Facebook account in the future. The McCann decision is consistent with Romano v. Steelcase Inc., 2010 N.Y. Slip Op , 2010 N.Y. Misc. LEXIS 4538 (N.Y. Sup. Ct. Sept. 21, 2010) and McMillen v. Hummingbird Speedway, Inc., Case No CD, 2010 WL (Pa. Ct. of Com. Pl. Sept. 9, 2010), which were previously discussed here and here. Implicit in all three decisions is that non-public communications and information found on social networking websites will not be protected from disclosure when it is properly demonstrated that the sought-after information is relevant to the litigation. As was previously discussed here, publicly accessible information on such sites has already been found to be discoverable. Unlike the boilerplate and catch-all discovery requests in personal injury actions, requests for access to a 4/2/2013

218 New York's Appellate Division Finds Facebook Accounts Off-Limits When Discovery D... Page 2 of 2 litigant s Facebook or other social networking account must be specifically tailored and, if possible, tied to actual facts or information found on the public portions of a litigant s account. Robert D. Brown, Jr. is an Associate on the Gibbons E-Discovery Task Force. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 4/2/2013

219 PRO I HO NOTARY FRANKLIN COIJNTY PA nail NOV 8 P L: 3 I rifl bl. P RUT HO NOTARY DL PUT Y_O OPINION AND ORDER "Facebook helps you connect and share with the people in your life." But what if the people in your life want to use your Facebook posts against you in a civil lawsuit? Whether and to what extent online social networking information is discoverable in a civil case is the issue currently before the Court. s TRUE, 1. DALLUD, PROTHONOTARY

220 Largent v. Reed No Opinion Page 2 Defendant Jessica Rosk0 2 has filed a Motion to Compel Plaintiff Jennifer Largent s Facebook Login Information. Rosko has a good faith belief that information on Jessica Largent s Facebook profile is relevant to Rosko s defense in this matter. For the following reasons, the Court holds that the information sought is discoverable, and we will grant the motion to compel. Background I. Underlying Facts This case arises out of a chain-reaction auto accident that occurred four years ago. According to the pleadings, Plaintiff Keith Largent was driving a 1986 Honda Shadow Motorcycle on Lincoln Way East in Chambersburg, with Plaintiff Jessica Largent as a passenger. Compl. 114, 19. At an intersection, Rosko collided with a minivan driven by Additional Defendant Sagrario Pena, pushing the van into Plaintiffs motorcycle. j As a result of the crash, Plaintiffs allege serious and permanent physical and mental injuries, pain, and suffering. Id. TT 11-14, 17, 21-24, 27. On April 27, 2009, Plaintiffs filed their four-count Complaint against Rosko. Plaintiffs allege two counts each of negligence and loss of consortium. On July 20, Rosko filed an Answer with New Matter, to which Plaintiffs filed a Reply on August 11. The pleadings joining Pena as an additional defendant are not relevant here, and he is not a party to the instant motion. During the deposition of Jennifer Largent, taken May 18, 2011, Defense counsel discovered that she has a Facebook profile, that she had used it regularly to play a game called FrontierVille, and that she last accessed it the night before the deposition. Def. 5 Mot. to Compel, Ex. A, Dep. of Jennifer Elaine Largent, 90-91, 94. Rosko refused, however, to disclose any 2. Ms. Rosko was formerly known as Jessica Reed. Def. s Answer with New Matter 13. The Court will refer to her by her current name.

221 Largent v. Reed No Opinion Page 3 information about the account, and Plaintiffs counsel advised that it would not voluntarily turn over such information. Id. This motion to compel followed on August 1, II. Facebook Facebook is a free social networking site. To join, a user must set up a profile, which is accessible only through the user s ID (her ) and a password. Facebook allows users to interact with, instant message, , and friend or unfriend other users; to play online games; and to upload notes, photos, and videos. Facebook users can post status updates about what they are doing or thinking. Users can post their current location to other friends, suggest restaurants, businesses, or politicians or political causes to "like," and comment or "like" other friends posts. 3 Social networking websites like Facebook, Google+, and MySpace are ubiquitous. Facebook, which is only seven years old, has more than 800 million active users, 50% of whom are active on the site at any given day. 4 Facebook has spawned a field of academic research, books, and a movie. Social networking websites also have a dark sidethey have caused criminal investigations and prosecutions and civil tort actions. See, e.g., Chapman V. Unemp t Comp. Bd. of Review, 20 A.3d 603 (Pa. Cmwlth. 2011) (employee fired for posting on Facebook while at work); United States v. Drew, 259 F.R.D. 449 (C.D. Cal. 2009) (woman criminally prosecuted for breaching MySpace s terms of use); In re Rolando 5., 129 Cal. Rptr. 3d 49 (Ct. App. 2011) (prosecution of a juvenile who hacked another child s Facebook account and posted 3. Facebook currently does not allow a person to "dislike" (or in Facebook parlance, "un-like") a friend s post, probably for good reason. 4. Facebook Statistics (last visited Oct. 25, 2011).

222 Largent v. Reed No Opinion Page 4 vulgar material therein); Finkel v. Dauber, 906 N.Y.S.2d 697 (Sup. Ct. Nassau 2010) (lawsuit concerning allegedly defamatory material posted in a Facebook group). Facebook has a detailed, ever-changing privacy policy. Only people with a user account can access Facebook. For all practical purposes, anyone with an account can set up a Facebook account. 5 Users can set their privacy settings to various levels, although a person s name, profile picture, and user ID are always publically available. At the least restrictive setting, named "public," all 800 million users can view whatever is on a certain user s profile. At an intermediate level, only a user s Facebook friends can view such information, and at the least restrictive, only the user can view his or her profile. Facebook also currently allows users to customize their privacy settings. Facebook alerts users that Facebook friends may "tag" them in any posting, such as a photograph, a note, a video, or a status update. A tag is a link to a user s profile: If someone clicks on the link, they will see your public information and anything else you let them see. Anyone can tag you in anything. Once you are tagged in a post, you and your friends will be able to see it. For example, your friends may be able to see the post in their News Feed or when they search for you. It may also appear on your profile. You can choose whether a post you ve been tagged in appears on your profile. You can either approve each post individually or approve all posts by your friends. If you approve a post and later change your mind, you can always remove it from your profile. 5. To comply with federal law, one must be 13 or older to have a Facebook account. This policy is apparently hard to enforce and is openly flouted. 6. Facebook Data Use PolicySharing and finding you on Facebook, (last visited Oct. 27, 2011).

223 Largent v. Reed No Opinion Page 5 If you do not want someone to tag you in their posts, we encourage you to reach out to them and give them that feedback. If that does not work, you can block them. This will prevent them from tagging you going forward. If you are tagged in a private space (such as a message or a group) only the people who can see the private space can see the tag. Similarly, it you are tagged in a comment, only the people who can see the comment can see the tag. 7 Therefore, users of Facebook know that their information may be shared by default, and a user must take affirmative steps to prevent the sharing of such information. Discussion Facebook also alerts users that it may reveal information pursuant to legal requests: Responding to legal requests and preventing harm We may share your information in response to a legal request (like a search warrant, court order or subpoena) if we have a good faith belief that the law requires us to do so. This may include responding to legal requests from jurisdictions outside of the United States where we have a good faith belief that the response is required by law in that jurisdiction, affects users in that jurisdiction, and is consistent with internationally recognized standards. We may also share information when we have a good faith belief it is necessary to: detect, prevent and address fraud and other illegal activity; to protect ourselves and you from violations of our Statement of Rights and Responsibilities; and to prevent death or imminent bodily harm. 8 Rosko has moved to compel Jennifer Largent to disclose her Facebook username and password. She claims that, as of January 2011, Largent s Facebook profile was public, meaning that anyone with an account could read or view her profile, posts, and photographs. Rosko says that certain posts on Largent s Facebook account contradict her claims of serious and severe 7. Facebook Data Use PolicySharing and finding you on Facebook, 8. Facebook Data Use PolicySome things you need to know, (internal hyperlinks removed).

224 Largent v. Reed No Opinion Page 6 injury. Specifically, Rosko claims that Largent had posted several photographs that show her enjoying life with her family and a status update about going to the gym. Jennifer Largent responds that the information sought is irrelevant and does not meet the prima facie threshold under Pennsylvania Rule of Civil Procedure She further argues that disclosure of her Facebook account access information would cause unreasonable embarrassment and annoyance. Finally, she claims that disclosure may violate privacy laws such as the Stored Communications Act of 1986, Pub. L. No , tit. II, 201, 100 Stat (codified as amended at 18 U.S.C ). In Pennsylvania, I. Discovery Standard a party may obtain discovery regarding any matter, not privileged, which is relevant to the subject matter involved in the pending action, whether it relates to the claim or defense of the party seeking discovery or to the claim or defense of any other party, including the existence, description, nature, content, custody, condition, and location of any books, documents, or other tangible things and the identity and location of persons having knowledge of any discoverable matter. Pa. R.C.P (a). It is no objection that the material sought will be inadmissible at trial, so long as the material "appears reasonably calculated to lead to the discovery of admissible evidence." Pa. R.C.P (b). Therefore, the material Rosko seeks must be relevant and not privileged. The Pennsylvania discovery rules are broad, and the relevancy threshold is slight. George v. Schirra, 814 A.2d 202, 204 (Pa. Super. 2002). Relevancy is not limited to the issues raised in the pleadings, and it carries a broader meaning than the admissibility standard at trial. Id. at 205; 9 Goodrich Amram 2d (a):(6).

225 Largent v. Reed No Opinion Page 7 There are no Pennsylvania appellate opinions addressing whether material contained on social networking websites is discoverable in a civil case. This is most likely because social networking is a recent phenomenon and issues are just beginning to percolate in the courts. $ Evan B. North, Comment, Facebook Isn t Your Space Anymore: Discovery of Social Networking Websites, 58 U. Kan. L. Rev. 1279, 1308 (2010) (noting social networking websites effect on discovery). Because of the lack of binding authority, the Parties have cited trial courts in this State and others. Rosko cites two cases in which the court permitted discovery of material on social network websites. Zimmerman v. Weis Mkts., Inc., No. CV , 2011 WL (Pa. C.P. Northumberland May 19,2011); McMillen v. Hummingbird Speedway, Inc., No CD, 2010 WL (Pa. C.P. Jefferson Sept. 9, 2010). In response, Plaintiffs cite two cases where courts have denied discovery of Facebook material. Piccolo v. Paterson, No (Pa. C.P. Bucks May 6, 2011); Kennedy v. Norfolk S. Corp., No (Pa. C.P. Phila. Jan 15, 20l1). Courts in other jurisdictions have also allowed discovery of social networking data in civil lawsuits. See, e.g., Offenback v. LM Bowman, Inc., No. 1:10-CV-1789, 2011 WL (M.D. Pa. June 22, 2011); EEOC v. Simply Storage Mgmt., LLC, 270 F.R.D. 430 (S.D. Ind. 2010); Romano v. Steelcase, Inc., 907 N.Y.S.2d 650 (Sup. Ct. Suffolk 2011). In Offenback, a personal injury case, the court ordered the plaintiff to turn over data contained on his Facebook page in a form mutually agreeable to the parties. Offenback, Counsel s citation of these cases (via an online Pittsburgh Post-Gazette article and an article written by an attorney from a law firm specializing in plaintiffs personal injury representation) is inadequate. Counsel is reminded that the proper way to cite an unreported case is, at minimum, to provide a docket number so that the Court does not need to conduct a wild goose chase to find the case.

226 Largent v. Reed No Opinion Page 8 WL , at *2..3. Specifically, the court ordered Offenback to turn over information that contradicted his claim of injury. j4 In Simply Storage, a Title VII sexual harassment case, the court allowed discovery of Facebook material including status updates, communications between two plaintiffs who alleged emotional distress injuries, and photographs and videos. Simply Storage, 240 F.R.D. at 436. In Romano, a personal injury action, the court ordered access - to all the plaintiffs social networking website information. Romano, 907 N.Y.S.2d at 657. As far as the threshold relevancy inquiry is concerned, it is clear that material on social networking websites is discoverable in a civil case. Pennsylvania s discovery rules are broad, and there is no prohibition against electronic discovery of relevant information. Furthermore, courts in other jurisdiction with similar rules have allowed discovery of social networking data. Rosko claims a good faith basis for seeking material contained on Jennifer Largent s Facebook account. Largent has pleaded that she suffers from, among other things, chronic physical and mental pain. Compi At her deposition, Largent testified that she suffers from depression and spasms in her legs, and uses a cane to walk. Def. s Mot. to Compel, Ex. A 65, 85. Rosko claims that Largent s formerly public Facebook account included status updates about exercising at a gym and photographs depicting her with her family that undermine her claim for damages. The information sought by Rosko is clearly relevant. The information sought by Rosko might prove that Largent s injuries do not exist, or that they are exaggerated. Therefore, Rosko satisfies the relevancy requirement. II. Privilege and Privacy Concerns Having determined that Rosko satisfies the threshold relevancy requirement, the Court must determine whether privilege or privacy rights protect against discovery. Privileged matter is

227 Largent v. Reed No Opinion Page 9 not discoverable. Pa R.C.P. 4011(c); (a). The term "privilege" refers only to those recognized by the common law, statutory law, or the Constitution. S.M. ex rel. R.M. v. Children & Youth Servs. of Del. Cnty., 686 A.2d 872, (Pa. Cmwlth. 1996). If either Pennsylvania s law of privilege or statutory law, such as the Stored Communications Act, prohibits disclosure, the relevant information Rosko seeks is not discoverable. A. Privilege Under Pennsylvania Law Pennsylvania disfavors privileges, and the law recognizes only a limited number of privileges. Joe v. Prison Health Servs., Inc., 782 A.2d 24, (Pa. Cmwlth. 2001). There is no confidential social networking privilege under existing Pennsylvania law. McMillan 2010 WL There is no reasonable expectation of privacy in material posted on Facebook. Almost all information on Facebook is shared with third parties, and there is no reasonable privacy expectation in such information. 10 Cf. Commonwealth v. Proetto, 771 A.2d 823, 828 (Pa. Super. 2001). When a user communicates on Facebook, her posts may be shared with strangers. McMillan, 2010 WL And making a Facebook page "private" does not shield it from discovery. Simply Storage Mgmt., 270 F.R.D. at 434; Patterson v. Turner Constr. Co., N.Y.S.2d ---, 2011 N.Y. Slip Op , 2011 WL (App. Div. Oct. 27, 2011). This is so because, as explained above, even "private" Facebook posts are shared with others. The Court holds that no general privacy privilege protects Jennifer Largent s Facebook material from discovery. No court has recognized such privilege, and neither will we. By 10. There may be a reasonable expectation of privacy in undelivered Facebook messages. That expectation of privacy vanishes once the reaches the intended recipient. Commonwealth v. Proetto, 771 A.2d 823, 828 (Pa. Super. 2001); accord United States v. Lifshitz, 369 F.3d 173, 190 (2d Cir. 2004). ("[Computer users have no reasonable] expectation of privacy in transmissions over the Internet or that have already arrived at the recipient.").

228 Largent v. Reed No I Opinion Page 10 definition, there can be little privacy on a social networking website. Facebook s foremost purpose is to "help you connect and share with the people in your life." That can only be accomplished by sharing information with others. Only the uninitiated or foolish could believe that Facebook is an online lockbox of secrets. B. The Stored Communications Act The Court next must determine whether the Stored Communications Act (SCA or Act) prohibits disclosure of Jennifer Largent s Facebook information." The SCA is part of the Electronic Communications Privacy Act, Pub. L. No , 100 Stat (1986). The SCA fills the gaps left by the Fourth Amendment, which weakly protects the digital and electronic worlds. Orin S. Kerr, A User s Guide to the Stored Communications Act, and a Legislature s Guide to Amending It, 72 Geo. Wash. L. Rev. 1208, (2004). The SCA does this by creating limits on the government s ability to compel Internet Service Providers (ISPs) to disclose information about their users, and it places limits on ISPs ability to voluntarily disclose information about their customers and subscribers to the government. See 18 U.S.C Crucial to the resolution of this motion, the SCA regulates only ISPs or other types of network supporters. It divides ISPs into two categories: electronic communications services (ECSs) and remote computing services (RCSs). An ECS is "any service which provides to users thereof the ability to send or receive wire or electronic communications." 18 U.S.C. 2510(15). An RCS stores data long-term for processing or storage. j4 2711(2). The terms are somewhat confusing because they reflect the state of computing technology as it existed in 1986 (a time before smartphones, Facebook, and the World Wide Web). Kerr, 72 Geo. Wash. L. Rev, at Largent argues that disclosure of her Facebook information "may violate privacy laws." Pis. Answer to Def. S Mot. to Compel 121. As she cites only the SCA, it is the only authority that the Court addresses.

229 Largent v. Reed No Opinion Page 11 To simplify greatly, RCSs store information for longer periods of time than ECSs. 12 The SCA applies differently to each but, as will be apparent below, the minutiae are irrelevant for our purposes. Only one court has addressed whether Facebook is an entity covered by the SCA. See Crispin v. Christian Audigier, Inc., 717 F. Supp. 2d 965 (C.D. Cal. 2010). In Crispin, the defendants served subpoenas upon Facebook and other social networking sites seeking information about the plaintiff s online postings. Id. at 969. The plaintiff filed a motion to quash the subpoenas arguing, among other things, that the SCA prohibited disclosure. Id. In a comprehensive opinion, the court held that Facebook is both an ECS and an RCS, depending on which function of the site is at issue. Id. at , 990. The court granted the motion to quash. In doing so, it held that civil subpoenas are never permissible under the SCA. I4 at (quoting Viacom Int l, Inc. v. YouTube, Inc., 253 F.R.D. 256, 264 (S.D.N.Y. 2008); In re Subpoena Duces Tecum to AOL, LLC, 550 F. Supp. 2d 606, 611 (E.D. Va. 2008); O Grady v. Super. Ct., 139 Cal. App. 4th 1423 (2006)). Crispin is distinguishable. In that case, the defendants sought information via subpoena to Facebook and other social networking sites. In this case, Rosko seeks the information directly from Jennifer Largent. The SCA does not apply because Largent is not an entity regulated by the SCA. She is neither an RCS nor an ECS, and accessing Facebook or the Internet via a home computer, smartphone, laptop, or other means does not render her an RCS or ECS. See Kerr, 72 Geo. Wash. L. Rev, at She cannot claim the protection of the SCA, because that Act does not apply to her. "The SCA is not a catch-all statute designed to protect the privacy of stored 12. For a much more comprehensive explanation of the SCA, see the law review article cited herein and Crispin v. Christian Audigier, Inc., 717 F. Supp. 2d 965, (C.D. Cal. 2010).

230 Largent v. Reed No Opinion Page 12 Internet communications." Id. Rather, it only applies to the enumerated entities. Largent being neither an ECS nor an RCS, the SCA does not protect her Facebook profile from discovery. III. Breadth of Discovery Request Finally, having determined that the information sought by Rosko is relevant and not privileged, the Court must consider whether her request is overbroad. No discovery is permitted if it is not relevant to the pending action or if it would cause unreasonable annoyance, embarrassment, oppression, burden, or expense. Pa. R.C.P The mere existence of some annoyance or embarrassment is insufficient to bar discovery. 9A Goodrich Amram 2d 4011(b): 1. Unreasonableness is determined on a case-by-case basis. As we noted above, Largent has no privacy rights in her Facebook postings, and there is no general Facebook social networking privilege. Furthermore, she cannot claim the protections of the Stored Communications Act. We further note that, in filing a lawsuit seeking monetary damages, Largent has placed her health at issue, which vitiates certain privacy interests. Any posts on Facebook that concern Largent s health, mental or physical, are discoverable, and any privilege concerning such information is waived. Gormley v. Edgar, 995 A.2d 1197, 1206 (Pa. Super. 2010); Kraus v. Taylor, 710 A.2d 1142 (Pa. Super. 1998), alloc. granted, 727 A.2d 1109 (Pa. 1999), and alloc. dismissed, 743 A.2d 451 (Pa. 2000). Largent complains that Rosko s motion is akin to asking her to turn over all of her private photo albums and requesting to view her personal mail. Pls. Answer to Def. s Mot. to Compel 23. But those analogies are mistaken in their characterization of material on Facebook. Photographs posted on Facebook are not private, and Facebook postings are not the same as

231 Largent v. Reed No Opinion Page 13 personal mail. Largent points to nothing specific that leads the Court to believe that discovery would cause unreasonable embarrassment. Bald assertions of embarrassment are insufficient. As the court stated in McMillan, Facebook posts are not truly private and there is little harm in disclosing that information in discovery. 13 Nor does the Court believe that allowing Rosko access to Largent s Facebook profile will cause unreasonable annoyance. The court notes that the entire cost of investigating Largent s Facebook information will be borne by Rosko. Also, Largent can still access her account while Rosko is investigating. As Rosko argues, this is one of the least burdensome ways to conduct discovery. Finally, the Court finds it significant that the only two Pennsylvania trial courts (of which we are aware) have granted discovery in identical situations. Zimmerman, 2011 WL ; McMillen 2010 WL The cases cited by Largent, though to the contrary, lack any persuasive authority because those orders are unsupported by any written opinion or memorandum. We agree with Rosko that information contained on Jennifer Largent s Facebook profile is discoverable. It is relevant and not covered by any privilege, and the request is not unreasonable. We will thus allow Rosko access to Largent s Facebook account to look for the necessary information. Plaintiff Jessica Largent must turn over her Facebook login information to Defense counsel within 14 days of the date of the attached Order. Defense counsel is allotted a 13. The Court does not hold that discovery of a party s social networking information is available as a matter of course. Rather, there must be a good faith basis that discovery will lead to relevant information. Here, that has occurred because Jennifer Largent s profile was formerly public. In other cases, it might be advisable to submit interrogatories and requests for production of documents to find out if any relevant information exists on a person s online social networking profiles.

232 Largent v. Reed No Opinion Page day window in which to inspect Largent s profile. After the window closes, Plaintiff may change her password to prevent any further access to her account by Defense counsel. An Order follows.

233 ORDER OF CO URT November Z_, 2011, the Court having reviewed Defendant Jessica Rosko Motion to Compel Plaintiff Jennifer Largent s Facebook Login Information, Plaintiffs Answer thereto, the briefs, the record, and the law, IT IS HEREBY ORDERED that Defendant s Motion to Compel be, and hereby is, GRANTED. IT IS FURTHER ORDERED that Plaintiff Jennifer Largent shall turn over to Defense counsel her Facebook username and password within 14 days of the date of this Order. Plaintiff shall not delete or otherwise erase any information on her Facebook account. After 35 days from the date of this Order, Plaintiff may change her Facebook login password to prevent further access by Defense counsel. EST TRUE PY LI DA L. BEARD PR )?H )7 1Y

234 Largent v. Reed No Order of Court Page 2 Pursuant to the requirements of Pa. R.C.P. 236 (a)(2), (b), (d), the Prothonotary shall immediately give written notice of the entry of this Order, including a copy of this Order, to each party s attorney of record, or if unrepresented, to each party; and shall note in the docket the giving of such notice and the time and manner thereof By the Court, J. The Prothonotary shall give notice to: Christopher T. Moyer, Esq., Counsel for Plaintiffs Donald L. Carmelite, Esq., Counsel for Defendant Stephen J. Magley, Esq., Counsel for Additional Defendant

235

236

237

238

239

240

241

242

243 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 1 of 12 Page ID #: DANIELLE MAILHOIT, v. Plaintiff, HOME DEPOT U.S.A., INC., et al., UNITED STATES DISTRICT COURT CENTRAL DISTRICT OF CALIFORNIA Defendants. ) ) ) ) ) ) ) ) ) ) ) ) No. CV DOC (SSx) ORDER GRANTING IN PART AND DENYING IN PART DEFENDANT S MOTION TO COMPEL FURTHER RESPONSES TO DEFENDANT S REQUEST FOR PRODUCTION REGARDING SOCIAL NETWORKING SITE MATERIAL (SET ONE) (Dkt. No. 105) I. INTRODUCTION On August 7, 2012, Defendant filed a Motion to Compel Further Responses to Defendant s Request for Production of Documents (Set One). (Dkt. No. 105). The parties filed a Joint Stipulation concurrently with the Motion pursuant to Local Rule 37, ( Jt. Stip. ), including the declarations of Elizabeth A. Falcone in support of the Motion, (Dkt. No. 107), and Kenneth Helmer in opposition to the Motion. (Dkt. No. 110).

244 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 2 of 12 Page ID #: The Court held a hearing on the Motion on August 28, For the reasons stated below, the Motion is GRANTED IN PART and DENIED IN PART. II. THE PARTIES CONTENTIONS Defendant requests an Order compelling Plaintiff to produce documents responsive to Requests for Production Nos , which collectively seek: (1) Any profiles, postings or messages (including status updates, wall comments, causes joined, groups joined, activity streams, blog entries) from social networking sites from October 2005 (the approximate date Plaintiff claims she first was discriminated against by Home Depot), through the present, that reveal, refer, or relate to any emotion, feeling, or mental state of Plaintiff, as well as communications by or from Plaintiff that reveal, refer, or relate to events that could reasonably be expected to produce a significant emotion, feeling, or mental state; (2) Third-party communications to Plaintiff that place her own communications in context; (3) All social networking communications between Plaintiff and any current or former Home Depot employees, or which in any way refer [or] pertain to her employment at Home Depot or this lawsuit; or 2

245 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 3 of 12 Page ID #: (4) Any pictures of Plaintiff taken during the relevant time (Jt. Stip. at 2). 2 1 period and posted on Plaintiff s profile or tagged or otherwise linked to her profile. Defendant argues that it is entitled to Plaintiff s communications posted on social networking sites ( SNS ) such as Facebook and LinkedIn to test Plaintiff s claims about her mental and emotional state. (Id. at 1). According to Defendant, Plaintiff testified at her deposition that she suffers from post traumatic stress disorder, depression and isolation, and has cut herself off from communication with friends because of Defendant s alleged wrongdoing. (Id.). Defendant argues that SNS communications are particularly likely to contain relevant information because in this day and age, many communications between friends and/or about an individual s emotional state are communicated via social media. (Id.). Defendant states that it has evidence suggesting that Plaintiff maintains Facebook and LinkedIn accounts and 1 Tagging is the process by which a third party posts a picture and links people in the picture to their profiles so that the picture will appear in the profiles of the person who tagged the people in the picture, as well as on the profiles of the people who were identified in the picture. EEOC v. Simply Storage Mgmt., LLC, 270 F.R.D. 430, 436 n.3 (S.D. Ind. 2010). 2 As written, Requests for Production Nos appear to require production of the entire contents of Plaintiff s SNS accounts and are overbroad, as Defendant appears to have recognized. (See Jt. Stip. at 4, 15-16, 26 & 37). The instant Motion is limited to a request for an Order compelling production of only the four categories of documents described above, which arguably overlap several different requests for production as originally written. (See Jt. Stip. at 2, 9). The Court will therefore address these four categories instead of the four original production requests that they supersede. 3

246 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 4 of 12 Page ID #: that publicly available information from those sites undermines Plaintiff s claims of isolation and loss of friendship. (Id. at 8). Plaintiff acknowledges that social media is discoverable to the extent it is adequately tailored to satisfy the relevance standard, but argues that Plaintiff s requests are impermissibly overbroad. (Id. at 11). According to Plaintiff, rather than tailor its requests, Defendant seeks to rummage through the entirety of [Plaintiff s] social media profiles and communications in the hope of concocting some inference about her state of mind. (Id. at 3). Plaintiff further argues that the requested discovery is unduly burdensome because she has already testified about her emotional distress, as well as produced or agreed to produce documents and communications pertaining to her emotional distress damages going as far back as 2004, (id.), which Plaintiff maintains constitute sufficiently relevant responses. (Id. at 14). In particular, Plaintiff asserts that she has already responded to requests for her communications with sixteen different current or former Home Depot employees, which Plaintiff contends presumably include her communications via social media. (Id.). 4

247 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 5 of 12 Page ID #: III. DISCUSSION A. Discovery Requests For Social Networking Site Content Must Be Reasonably Calculated To Lead To The Discovery Of Admissible Evidence And Describe The Information To Be Produced With Reasonable Particularity A party may obtain discovery regarding any nonprivileged matter that is relevant to any party s claim or defense -- including the existence, description, nature, custody, condition, and location of any documents or other tangible things. Fed. R. Civ. P. 26(b)(1). Relevancy is construed broadly to encompass any matter that bears on, or that reasonably could lead to other matter[s] that could bear on any issue that is or may be in the case. Chavez v. DaimlerChrysler Corp., 206 F.R.D. 615, 619 (S.D. Ind. 2002) (internal quotations omitted). The Supreme Court has instructed that the limitation on discovery to relevant materials must be firmly applied, as the discovery provisions, like all of the Federal Rules of Civil Procedure, are subject to the injunction of Rule 1 that they be construed to secure the just, speedy, and inexpensive determination of every action. Herbert v. Lando, 441 U.S. 153, 177, 99 S. Ct. 1635, 60 L. Ed. 2d 115 (1979) (quoting Fed. R. Civ. P. 1) (emphasis in original). Pursuant to Federal Rule of Civil Procedure 34(a), a party may request documents in the responding party s possession, custody, or control. Rule 34(b) requires the requesting party to describe the items to be produced with reasonable particularity and specify a 5

248 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 6 of 12 Page ID #: reasonable time, place, and manner for the inspection. Fed. R. Civ. P. 34(b)(1-2). The test for reasonable particularity is whether the request places a party upon reasonable notice of what is called for and what is not. Bruggeman ex rel. Bruggeman v. Blagojevich, 219 F.R.D. 430, 436 (N.D. Ill. 2004) (quoting Parsons v. Jefferson Pilot Corp., 141 F.R.D. 408, 412 (M.D. N.C. 1992)); see also Regan-Touhy v. Walgreen Co., 526 F.3d 641, (10th Cir. 2008) ( Though what qualifies as reasonabl[y] particular surely depends at least in part on the circumstances of each case, a discovery request should be sufficiently definite and limited in scope that it can be said to apprise a person of ordinary intelligence what documents are required and [to enable] the court... to ascertain whether the requested documents have been produced. ) (quoting Wright & Miller, 8A Federal Practice and Procedure 2211, at 415). All-encompassing demands that do not allow a reasonable person to ascertain which documents are required do not meet the particularity standard of Rule 34(b)(1)(A). In re Asbestos Products Liability Litigation (No. VI), 256 F.R.D. 151, 157 (E.D. Pa. 2009). The Court recognizes that social networking site content may be subject to discovery under Rule 34. Generally, SNS content is neither privileged nor protected by any right of privacy. Davenport v. State Farm Mut. Auto. Ins. Co., 2012 WL at *1 (M.D. Fla. Feb. 21, 2012). However, [d]iscovery of SNS requires the application of basic discovery principles in a novel context. Simply Storage Mgmt., 270 F.R.D. at 434. In particular, several courts have found that even though certain SNS content may be available for public view, the Federal Rules do not grant a requesting party a generalized right to rummage at 6

249 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 7 of 12 Page ID #: will through information that [the responding party] has limited from public view but instead require a threshold showing that the requested information is reasonably calculated to lead to the discovery of admissible evidence. Tompkins v. Detroit Metropolitain Airport, 278 F.R.D. 387, 388 (E.D. Mich. 2012); see also Davenport, 2012 WL at *1 ( A request for discovery [of SNS content] must still be tailored... so that it appears reasonably calculated to lead to the discovery of admissible evidence. ) (quoting Fed. R. Civ. P. 26(b)(1)); Mackelprang v. Fidelity Nat l Title Agency of Nevada, Inc., 2007 WL at *7 (D. Nev. Jan 9, 2007) ( Ordering... release of all of the private messages on Plaintiff s Myspace.com internet account would allow Defendants to cast too wide a net for any information that might be relevant and discoverable. ). Where discovery requests seek SNS communications in connection with claims involving the responding party s mental or emotional health, several courts have also found that the simple fact that a claimant has had social communication is not necessarily probative of the particular mental and emotional health issues in the case. Rather, it must be the substance of the communication that determines relevance. Simply Storage Mgmt., 270 F.R.D. at 435; Holter v. Wells Fargo and Co., 281 F.R.D. 340, 344 (D. Minn. 2011). As one court reasoned, To be sure, anything that a person says or does might in some theoretical sense be reflective of her emotional state. But that is hardly justification for requiring the production of every thought she may have reduced to writing, or, indeed, the deposition of everyone she may have talked to. Rozell v. Ross-Holst, 2006 WL at *3-4 (S.D. N.Y. Jan. 20, 2006). Thus, while a party may conduct discovery concerning another party s 7

250 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 8 of 12 Page ID #: emotional state, the discovery itself must still comply with the general principles underlying the Federal Rules of Civil Procedure that govern discovery. A court can limit discovery if it determines, among other things, that the discovery is: (1) unreasonably cumulative or duplicative; (2) obtainable from another source that is more convenient, less burdensome, or less expensive; or (3) the burden or expense of the proposed discovery outweighs its likely benefit. Favale v. Roman Catholic Diocese of Bridgeport, 235 F.R.D. 553, 558 (D. Conn. 2006) (internal citations and quotation marks omitted). The district court enjoys broad discretion when resolving discovery disputes, which should be exercised by determining the relevance of discovery requests, assessing oppressiveness, and weighing these factors in deciding whether discovery should be compelled. Id. (internal citations and quotation marks omitted). B. The Majority Of Defendant s Social Media Requests Fail Rule 34(b)(1)(A) s Reasonable Particularity Requirement And Therefore Are Not Reasonably Calculated To Lead To The Discovery Of Admissible Evidence The Court finds that three of the four categories of SNS communications sought by Defendant fail Rule 34(b)(1)(A) s reasonble particularity requirement, and, as such, are not reasonably calculated to lead to the discovery of admissible evidence. Consequently, the Court DENIES Defendant s Motion with respect to Categories 1, 2 and 4 of the revised requests. (See Jt. Stip. at 2). 8

251 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 9 of 12 Page ID #: Category 1 seeks any profiles, postings or messages (including status updates, wall comments, causes joined, groups joined, activity streams, blog entries) from any social networking site from October 2005 through the present that reveal, refer, or relate to any emotion, feeling, or mental state of Plaintiff, as well as communications by or from Plaintiff that reveal, refer, or relate to events that could reasonably be expected to produce a significant emotion, feeling, or mental state. (Jt. Stip. at 2). Plaintiff has placed her emotional state at issue in this action and it is conceivable that some SNS communications may support or undermine her claims of emotional distress. Nonetheless, the extremely broad description of the material sought by this category fails to put a reasonable person of ordinary intelligence on notice of which specific documents or information would be responsive to the request, and therefore fails to satisfy Rule 34(b)(1)(A) s requirement that production requests be stated with reasonable particularity. Even if the first part of this category, which seeks communications relating to any emotion, could be understood to encompass only communications containing specific emotive words (which the request does not identify), the category would still arguably require the production of many materials of doubtful relevance, such as a posting with the statement I hate it when my cable goes out. The second part of the category, which seeks communications relating to events that could reasonably be expected to produce a significant emotion, is similarly vague and overbroad. Arguably, watching a football game or a movie on television is an event that may produce some sort of significant emotion, but it is unclear whether Plaintiff would be required to 9

252 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 10 of 12 Page ID #: produce messages relating to such activities. Without more specific guidance, Category 1 is not reasonably particular. The language of the request does not provide sufficient notice to the responding party of what should be considered responsive material. Defendant fails to make the threshold showing that the request at issue is reasonably calculated to lead to the discovery of admissible evidence. Category 2, which requests third-party communications to Plaintiff that place her own communications in context, also fails. To the extent that the reference to Plaintiff s own communications means communications regarding emotions produced in response to Category 1, Category 2 is entirely predicated on Category 1 and fails for the same vagueness concerns discussed above. Apart from these deficiencies, even if the universe of documents referred to as Plaintiff s own communications could be reasonably circumscribed and understood, the phrase in context is vague and also fails to provide notice to Plaintiff of which specific third party communications are and are not called for by the request. Finally, Category 4, which requests any pictures of Plaintiff taken during the relevant time period and posted on Plaintiff s profile or tagged or otherwise linked to her profile, is impermissibly overbroad. Defendant fails to make the threshold showing that every picture of Plaintiff taken over a seven-year period and posted on her profile by her or tagged to her profile by other people would be considered relevant under Rule 26(b)(1) or would lead to admissible evidence. See Simply Storage Mgmt., 270 F.R.D. at 436 ( [A] picture posted on a third party s profile in which a claimant is merely 10

253 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 11 of 12 Page ID #: tagged [] is less likely to be relevant. ) (footnote omitted). All encompassing production requests do not meet Rule 34(b)(1)(A) s reasonably particularity requirement, In re Asbestos Products Liability Litigation (No. VI), 256 F.R.D. at 157, and discovery rules do not allow a requesting party to engage in the proverbial fishing expedition, in the hope that there might be something of relevance in [the producing party s] Facebook account. Tompkins, 278 F.R.D. at 388 (emphasis in original). 3 In contrast, Category 3, which requests all SNS communications between Plaintiff and any current or former Home Depot employees, or which in any way refer... to her employment at Home Depot or this lawsuit, adequately places Plaintiff on notice of the materials to be produced and is reasonably calculated to lead to the discovery of admissible evidence. Plaintiff notes that she has already responded to requests for communications between Plaintiff and sixteen different current or former Home Depot employees, which would presumably include 3 The Court acknowledges that Categories 1, 2 and 4 are closely modeled after three categories of SNS communications that the court in Simply Storage Mgmt. ordered produced. See Simply Storage Mgmt., 270 F.R.D. at 436. That court recognized, however, that the categories were not drawn... with the precision litigants and their counsel typically seek. Id. The court admonished counsel to make judgment calls in good faith pursuant to the guidelines articulated by the court in carrying out the order and stated that the requesting party could challenge the production if it believed it fell short of those guidelines. Id. These admonishments suggest that the court itself was concerned about the parties' ability to carry out the order. As noted in the discussion above, this Court finds that the requests suggested in Simply Storage Mgmt. are overbroad and vague. These requests fail to provide enough direction to the responding party to comply with Rule 34(b)(1)(A). Accordingly, this Court declines to compel responses to the discovery modeled on requests described in the Simply Storage Mgmt. case. 11

254 Case 2:11-cv DOC-SS Document 136 Filed 09/07/12 Page 12 of 12 Page ID #: communications via social media. (Jt. Stip. at 36). Plaintiff s responses to those requests indicate that a search for the communications described in Category 3 is both technically feasible and not overly burdensome. (See, e.g., Helmer Decl., Exh. A at 8-9). Plaintiff did not provide argument or evidence to the contrary in opposition to the current motion. Defendant s Motion with respect to Category 3. IV. CONCLUSION Consequently, the Court GRANTS For the foregoing reasons, Defendant s Motion to Compel Further Responses to Defendant s Request for Production of Documents (Set One) is GRANTED IN PART and DENIED IN PART. Defendant s Motion is DENIED with respect to Categories 1, 2 and 4. Defendant s Motion is GRANTED with respect to Category 3. Plaintiff is ORDERED to serve a written response and to produce documents responsive to Category 3, if any exist, within fourteen (14) days of the date of this Order. IT IS SO ORDERED. DATED: September 7, 2012 /S/ SUZANNE H. SEGAL UNITED STATES MAGISTRATE JUDGE 12

255 No Fishing Expeditions Allowed When It Comes to Discovery of Social Media : E-Disco... Page 1 of 2 E-Discovery Law Alert Posted at 9:55 AM on October 12, 2012 by Gibbons P.C. No Fishing Expeditions Allowed When It Comes to Discovery of Social Media A recent decision in California, Mailhoit v. Home Depot, U.S.A., Inc. et al., Civ. No (D.E. 105, C.D. Cal. Sept. 7, 2012) reiterates the limits to which social media information is discoverable. Consistent with Fed. R. Civ. P. 26(b)(1) and 34, the Court made clear in the context of a motion to compel that discovery requests for social networking site content must be reasonably calculated to lead to the discovery of admissible evidence and describe the information to be produced with reasonable particularity. At issue in Mailhoit were four categories of discovery sought by the defendant in connection with the plaintiff s claims of discrimination and emotional distress: 1. Any profiles, postings or messages... from social networking sites from October 2005 (the approximate date Plaintiff first claimed she was discriminated against by Home Depot), through the present, that reveal, refer, or relate to any emotion, feeling, or mental state of Plaintiff; 2. Third-party communications to Plaintiff that place her own communications in context; 3. All social networking communications between Plaintiff and any current or former Home Depot employees that pertain to her employment at Home Depot or this lawsuit; and 4. Any pictures of Plaintiff taken during the relevant time period and posted on Plaintiff s profile. In analyzing categories 1, 2 and 4, the Court determined that although some communications could support or refute Plaintiff s claims of emotional distress, Defendant had failed to put a reasonable person of ordinary intelligence on notice of which specific documents or information would be responsive to the request, thereby failing Rule 34(b)(1)(A) s reasonable particularity requirement. As a result, Defendant failed to make the threshold showing that the request at issue is reasonably calculated to lead to the discovery of admissible evidence. Indeed, although evidence from social networking sites is neither privileged nor protected for privacy reasons, the requesting party does not have a generalized right to rummage at will through information that [the responding party] has limited from public view. Rather, consistent with Fed. R. Civ. P. 26(c), information requests must be tailored so that it appears reasonably calculated to lead to the discovery of admissible evidence. Simply because a communication may, for example, reflect one s state of mind, that communication is not necessarily discoverable under Fed. R. Civ. P. 26(c) if it does not relate to the cause of action or damages at issue. In contrast, Category 3 was deemed a permissible request because it placed Plaintiff on adequate notice of the materials sought and was reasonably calculated to lead to the discovery of admissible evidence. 4/2/2013

256 No Fishing Expeditions Allowed When It Comes to Discovery of Social Media : E-Disco... Page 2 of 2 As a practical matter, when seeking social media discovery, attorneys should ensure that the requests are tailored to the claims at issue. What does that mean? Based upon the reasoning contained in this order, if you are dealing with a personal injury plaintiff who claims disability and loss of enjoyment, you should request, for example, any profiles, posts or messages that reflect the plaintiff s participation in any travel, sports, recreation or related activities. Simply making a blanket request for all posts will not suffice under the standard set forth in Mailhoit. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 4/2/2013

257 NOT FOR PUBLICATION UNITED STATES DISTRICT COURT DISTRICT OF NEW JERSEY ) THE KATIROLL COMPANY, INC., ) ) Plaintiff, ) ) v. ) Civil Action No (GEB) ) KATI ROLL AND PLATTERS, INC. AND ) NIRAJ JIVANI, ) MEMORANDUM OPINION ) Defendants. ) ) BROWN, Chief Judge This matter comes before the Court on a motion by Plaintiff for spoliation sanctions and to compel discovery (Doc. No. 108). The Court has considered the parties submissions and decided the motion without oral argument pursuant to Federal Rule of Civil Procedure 78. For the reasons set forth below, that motion is denied in part and granted in part. I. BACKGROUND This is a trademark infringement case involving two restaurants that sell a similar type of food called katirolls. Plaintiff originally brought this action on March 3, 2010, in the Southern District of New York alleging, among other things, infringement of their distinctive trade dress. (Compl.; Doc. No. 1). The Southern District of New York transferred the case to New Jersey based on a lack of personal jurisdiction. (Memorandum Order dated July 9, 2010; Doc. No. 31). Shortly after the transfer, this Court granted a motion for a preliminary injunction after a two-day evidentiary hearing. After setting forth its findings of fact and conclusions of law, this 1

258 Court found that Plaintiff was entitled to a preliminary injunction on its trade dress claims but found that it had not carried its burden on infringement of the trademark. (Doc. Nos. 72, 73). This motion focuses on the discovery positions of the parties. Particularly, this motion focuses on whether Defendants fact discovery has been late forthcoming and whether Defendants purposely and deliberately destroyed important discoverable information. Fact discovery in this case is ongoing and does not close until September 30, (Doc. No. 110). II. SPOILATION SANCTIONS AND MOTION TO COMPEL A. Standard of Review Parties to litigation often are required to preserve litigation evidence. This duty arises when the party in possession of the evidence knows that litigation by the party seeking the evidence is pending or probable and the party in possession of the evidence can foresee the harm or prejudice that would be caused to the party seeking the evidence if the evidence were to be discarded. Kounelis v. Sherrer, 529 F. Supp. 2d 503, 518 (D.N.J. 2008). If the party does not so preserve the evidence, they are said to have spoliated that evidence, which can give rise to sanctions. In determining whether spoliation sanctions are appropriate, the two key considerations are the degree of fault of the party who altered or destroyed the evidence and the degree of prejudice suffered by the opposing party. Id. (quoting Schmidt v. Milwaukee Elec. Tool Corp., 13 F.3d 76, 79 (3d Cir. 1994)). Sanctions for spoliation include: dismissal of a claim or granting judgment in favor of a prejudiced party; suppression of evidence; an adverse inference, referred to as the spoliation inference; fines; and attorneys fees and costs. Id. A spoliation inference is the mildest sanction and permits a jury to draw an adverse inference that the spoliated evidence might or would have been unfavorable to the position of the offending party. Veloso v. Western Bedding Supply Co., 2

259 Inc., 281 F.Supp.2d 743, 746 (D.N.J. 2003) (internal quotation marks omitted). To qualify for a spoliation inference, the movant must show at least four things: First, it is essential that the evidence in question be within the party s control. Second, it must appear that there has been actual suppression or withholding of the evidence. Third, the evidence destroyed or withheld was relevant to claims or defenses. And fourth, it was reasonably foreseeable that the evidence would later be discoverable. Mosaid Tech., Inc. v. Samsung Elec. Co., Ltd., 348 F.Supp.2d 332, 336 (D.N.J. 2004). There is a split in this district s jurisprudence as to the fault required to support the second factor the suppression or withholding of the evidence. Some courts have required that spoliation to be the result of intentional conduct before giving an adverse inference instruction. Veloso v. Western Bedding Supply Co., 281 F.Supp.2d, 743, (D.N.J. 2003); Costello v. City of Brigantine, No , 2001 WL , at *26 (D.N.J. Jun. 28, 2001). Others have held that the inference is justified where the party possessing the evidence destroyed it negligently. Kounelis, 529 F. Supp. 2d at 518 (appropriate where either negligently or intentionally destroyed, but balancing the degree of fault with the prejudice produced); Mosaid Techn., Inc. v. Samsung, 348 F.Supp.2d 332, (D.N.J. 2004); Scott v. IBM Corp., 196 F.R.D. 233, 249 (D.N.J. 2000) (finding that spoliations might be appropriate where there was nothing to indicate documents were intentionally destroyed). This Court concludes that the best rule is to use the amount of prejudice to the opposing party to help to determine the degree of fault required: Where there is substantial prejudice to the opposing party, negligence may be sufficient to warrant a spoliation inference. Where there is minimal prejudice to the opposing party, intentional conduct is required. 3

260 B. Application The Plaintiff complains to this Court about a series of discovery abuses that allegedly require a spoliation inference. However, the Court concludes in its discretion that, as to some conduct, the prejudice to Plaintiff of the alleged destruction of evidence is minimal and that it was partially at fault. Thus, no spoliation inference is appropriate. In other instances, the Court lacks sufficient information to make a determination but is troubled by aspects of Defendants failed disclosures. While the Court finds it is premature to enter a spoliation inference in these instances, the Court is open to spoliation sanctions in the future and grants the motion to compel in some instances. In still other instances, the Court determines that a spoliation instruction is warranted. 1. The Appearance of the Restaurant This Court s February 1, 2011 order gave Defendants twenty days to change the infringing color of the restaurant from orange to another color. (Doc. No. 73). Thereafter, Defendants counsel agreed that Defendants would leave the restaurant in its current condition until February 4, 2011, so that Plaintiff could take video and pictures of the restaurant in its infringing form. However, Plaintiff complains that the Defendants painted a few test patches of new colors on the wall prior to the pictures. (Pl. s Br. at 11). There is little question that the change in the appearance of the restaurant fits the first, third and fourth criterion for the spoliation instruction: the appearance was within the Defendants control, it was ultimately changed before evidence could be taken, and Defendants knew that Plaintiff wanted to take pictures of the restaurant. The only real question is about the intentionality of the conduct and the prejudice to the Plaintiff. 4

261 Plaintiff has suffered little prejudice from this action, and Defendants are minimally at fault for the issue. See Kounelis, 529 F. Supp. 2d at 519. First, Plaintiff has suffered minimal if any prejudice. Defendants counsel represented that the test patches of a gold color were 2 feet by 1 foot, and were small compared to the size of the restaurant; Plaintiff has submitted no evidence to contradict this. As such, the jury will not have difficulty imagining the entire restaurant was orange rather than orange with two small test patches of gold on either the video or the pictures taken on the day of inspection. This is particularly true in light of Plaintiff s prior assertion before this Court that Mr. Jivani s use of gold is so similar to orange that for certain lightings and on certain computer screens, it will appear as orange. Further, Plaintiff is in possession of pictures of the restaurant prior to the change, so the evidence is somewhat cumulative. (Dolich Decl. at Ex. E). Second, while Mr. Jivani was at fault for painting prior to the video and pictures, such fault was minimal because the painting was undertaken to comply with this Court s preliminary injunction order. Such conduct was not intentional. Thus, the fault and prejudice do not weigh in favor of a spoliation sanction and the Court will not order one. After Plaintiff realized that Mr. Jivani had painted the test patches, Plaintiff requested the security surveillance video taken prior to the repainting, which the Defendants agreed to provide. Defendants claim that they were unable to save the video but cite no support for that statement. (Defs. Br. at 11). As the Court acknowledged above, there is minimal prejudice to Plaintiff because the video that Plaintiff could have taken during the inspection would have been sufficient. However, the Court is not satisfied with Defendants explanation and finds it highly suspect. Indeed, after having represented that they had the video, Defendants somewhat lamely state that they were unable to save the video. (Id.). As such, the Court orders Defendants to 5

262 provide it with a full, sworn explanation of what happened to the surveillance video from each person involved in its preservation. If the Court is not satisfied with that explanation, it will conclude that the noncompliance is intentional and grant a spoliation inference. 2. Mr. Jivani s Old Facebook Pages Plaintiff asks for a spoliation sanction as a result of Mr. Jivani s failure to preserve his Facebook pages in their original state in other words, Plaintiff wanted PDFs of these pages prior to their being taken down. (Pl. s Br. at 5-6). However, Plaintiff does not dispute that Facebook took these pages down because of its own take-down request. (See Dolich Decl. at Ex. B). To hold Defendants responsible for this subsection of pages would be unjust. 1 Plaintiff also requests that the Court create an inference of spoliation based upon Mr. Jivani changing his profile picture on Facebook from a picture displaying the infringing trade dress without preserving that evidence. Defendants, for their part, claim that Plaintiff is in possession of all relevant posts prior to the change. Both parties agree that changing the user s Facebook profile picture changes the picture associated with each and every post that user has made in the past. There is little question that the evidence fulfills the third requirement, but the other requirements are at issue. Defendants argue that the Court should not impose a spoliation inference because the website was public and Plaintiff could have printed the website itself. This seems to be directed at the first requirement, that the discovery be in Defendants control. However, Courts in this district have still found public websites to be within the control of parties who own them. Arteria Property Pty Ltd. v. Universal Funding V.T.O., Inc., 2008 WL at *5 (D.N.J. 1 The fact that it was not legally improper for Plaintiff to serve this notice (as acknowledged in this Court s prior opinion), does not undermine the fact that it was the primary and but-for cause of the spoliation. 6

263 2008). Further, this is an attempt to pass the buck to Plaintiff to print websites that Defendants are obliged to produce. Given that Defendants have a discovery obligation to produce them and that only Defendants knew when the website would be changed, it is more appropriate for Defendants to have that burden. Thus, the Court concludes that the first factor, Defendants control, is present. The presence of remaining two factors is less than clear. As to the forseeability of the discoverability of the evidence, the Court notes that the change of a profile picture on Facebook is a common occurrence. Active users often change their profile pictures weekly. 2 Therefore, it is hardly surprising that Mr. Jivani has changed his profile picture during the pendency of this litigation. Further, while Mr. Jivani was on notice that he had to preserve evidence, it would not have been immediately clear that changing his profile picture would undermine discoverable evidence. As such, the Court determines that this spoliation was unintentional; however, it nonetheless concludes that the loss of this evidence is somewhat prejudicial to Plaintiff. See Mosaid Tech., Inc. v. Samsung Elec. Co., Ltd., 348 F.Supp.2d at 336. Given these considerations of fault, the Court determines that a less imposing alternative is the best solution. Mr. Jivani must to coordinate with Plaintiff s counsel to change the picture back to the allegedly infringing picture 3 for a brief time so that Plaintiff may print whatever posts it thinks are relevant such action shall not be considered an additional act of infringement. The Court however, determines that it is incumbent on Plaintiff to, during the appointed time, print 2 Further, the Court notes that it may even be somewhat misleading to have printouts of Defendant s posts with the infringing picture, as if they always appeared that way. Indeed, it is likely that many posts existed for a substantial portion of their lifetime without the allegedly infringing photograph. This is an issue that can be resolved in the proper questioning of witnesses. 3 These photographs have not been destroyed, they been attached in several PDFs to this Court. It would be a simple process to recover the photograph from those PDFs. Then, Plaintiff may print the posts it thinks are relevant to the case. 7

264 such posts as it thinks are necessary to make its case. Thereafter, Mr. Jivani must immediately change his profile picture back to his noninfringing picture. 3. Documents from Defendants Website Plaintiff complains that it does not have a copy of the Defendants website showing what the restaurant looked like prior to its repainting. Defendants respond that Plaintiff is in possession of pictures showing the restaurant prior to repainting, which are sufficient for Plaintiff to make its case. Again, the first, third, and fourth elements of the test are met. The question revolves around the second element, whether such evidence was destroyed intentionally or negligently, and what prejudice accrued to Plaintiff as a result. Essentially, this is Plaintiff s complaint about Defendants repainting prior to the inspection again. As discussed, Plaintiff had the opportunity to take pictures and video of the restaurant with a minor change. The Court does not know whether Plaintiff took advantage of that opportunity but, as discussed, the minor change in appearance was not very prejudicial to Plaintiff (particularly in light of the fact that Plaintiff has maintained that the gold color of the test patches is similar to the original orange). Consequently, Plaintiff suffered minimal prejudice. However, the Court concludes that it is implausible that Mr. Jivani has no other electronic copies of these pictures on his hard drive or in s to his web developer, etc. As such, Mr. Jivani is directed to search his home computer and any other computers he frequently uses and disclose any pictures he has of the restaurant prior to its repainting. The Court directs Defense counsel to remind Mr. Jivani of the consequences of destroying evidence before a tribunal. 8

265 The Court also determines that, without the printout of the website, the evidence will be insufficient to show that the website itself embodied pictures of the infringing restaurant and substantial prejudice has accrued to Plaintiff s ability to show that website itself infringed its trade dress. As such, on this limited issue, the Court will grant a spoliation instruction unless Defendants produce an image of the portion of the website that included the photographs within one week of the order accompanying this opinion. 4. Annual Report Plaintiff claims that a spoliation inference should result from Defendants failure to provide its 2010 annual report. Plaintiff requested the report to confirm Defendants representation that Rasik Jivani was no longer a shareholder of the corporation because he had transferred the stock to Niraj Jivani. Defendants claimed that they were unable to produce the report because it was no longer in their possession, despite the fact that they filed it during the pendency of this litigation. (See Doc. No. 59, Ex. A, p. 1). Defendants did provide documents related to the transfer of stock and stock certificates and corporate bylaws. (Defs. Br. at 12). It does appear that Defendants were at the very least grossly negligent in failing to keep a copy of a report that they agreed to provide to opposing counsel and the Court. Mosaid Tech., 348 F.Supp.2d at 336 ; (See Doc. No. 57-1, 3). However, apparently this is a public document available from the New Jersey Secretary of State. As a result, the Court cannot see any prejudice that accrues from the failure to produce it it is still available to both parties. Indeed, the Court observes that it would have been easier for Plaintiff to request the filing than to draft this portion of the motion. The Court further observes that it would have been easier for Defendants to request the filing than to respond to this portion of the motion. Given the lack of prejudice to Plaintiff, the Court finds that a spoliation sanction is inappropriate, 9

266 despite Defendants fault in not preserving their copy of the report. See Kounelis, 529 F. Supp. 2d at 519. However, counsel for Defendants is directed to request the report from the Secretary of State and provide it to Plaintiff if it has not already so provided. 5. s Between Mr. Jivani and the Web Developer According to Mr. Jivani s testimony, in October of 2011, Mr. Jivani hired a web developer for his website. Mr. Jivani, in failing to comply with this Court s preliminary injunction order, alleged that he was unable to contact the developer to put the disclaimer on the website and change the color. (4/4/11 tr. at 12-13). Plaintiff requested the s between Mr. Jivani and his web developer to confirm that he did indeed hire the developer as of October. (Pl s. Br. at 14). Plaintiff asserts that Defendants never produced the s. Defendants claim that all of the s in their possession were produced and that all other communication took place in person or by telephone. The s produced consist of one exchange, where, on February 8, 2011, Mr. Jivani s web developer sent him a copy of the image of his home page prior to adding the disclaimer: Hi Niraj, As per your phone request, Please see your website s home page s html file, in the attachment. (Before adding the disclaimer part in the Home Page ). Best Regards, Dr. Hema, Phd. Nadiyam Technologies, Inc. (Dolich Decl. at Ex. G) (emphasis added). Mr. Jivani did not reply to the substance of this , but replied on a related topic on March 16, 2011 (and immediately forwarded the reply to counsel): 10

267 Hema I have called you on March 3 rd and 4 th to change the colors[.] I am on a court order [as] I have told you before... you MUST change the color[.] [T]his is horrible service being a few weeks that have gone by and no action has been taken to change the colors! (Id.) (ellipsis in original, emphasis added). There is no doubt that Dr. Hema Latha s and Niraj Jivani s reflect that telephone conversations had taken place between the two of them. As such, if the s have not been altered or somehow fabricated, it is clear that some communication took place by telephone. Therefore, there may be no s to be had. However, there may be s that were not produced or deleted. It seems suspicious that a person with as active an online persona as Mr. Jivani only ever sent one to his web developer, an happens to cast him in a positive light. Indeed, the Court wonders how Dr. Latha came into possession of the photographs that were allegedly posted on the website if there were no s or other electronic communication between the two. 4 Further, there are relatively easy ways to confirm these questions that do not require a spoliation sanction at this juncture. follows: In order to confirm whether Mr. Jivani sent other s, the Court directs Defendants as (1) Mr. Jivani shall allow his counsel to conduct a search of each of his accounts, particularly for s from his web developer; defense counsel shall conduct such a search and turn over any s procured; 5 (2) Mr. Jivani will request, via a signed request communicated through counsel, that Dr. Hema Latha forward to defense counsel directly, any s that Mr. Jivani and Dr. Latha have exchanged, assuming he has not deleted them; 4 It would have been possible to transfer such things in person. It is also possible that the pictures were on the website prior to Dr. Latha starting his work with the website. However, both of these possibilities seem less likely than an . 5 See Local Civil Rule 26.1(d)(1). The Court also notes that archive is the primary management function on Gmail. As such, it is unusual though not unheard of, for users to actually delete s. 11

268 (3) Mr. Jivani shall have no contact, except through counsel, with Dr. Latha until such time as Dr. Latha has responded to the request for s; (4) Mr. Jivani, with the assistance of counsel, is to request phone records for each phone on which he made these calls for the months of October, 2010, and February and March of 2011 to confirm Mr. Jivani s alleged calls to Dr. Latha, to be delivered, if possible, directly to his counsel; such records shall be produced to the Court with appropriate redactions of calls not to Dr. Latha. Counsel for the Defendants shall report every week to Judge Bongiovanni by a brief letter submission on the progress of these investigations and file, as an attachment, any resulting s or phone records, appropriately redacted or under seal. Counsel may choose the day that he will consistently place such a letter on the docket. If no s are recovered and Mr. Jivani is unable to produce records of phone calls or text messages to Dr. Latha (or alleges that such phone calls were made on the phones of any third parties or pay phones), the Court will consider sanctions against him. As of now, the existence of such s is too hypothetical to impose sanctions. 7. Defendants New Facebook Page Plaintiff argues that Defendants made a new Facebook page and did not disclose its existence in a discovery response. The Court agrees that Defendants should have disclosed its existence, but because Plaintiff has found the page independently and Defendants created the page in order to comply with this Court s preliminary injunction, sanctions are not appropriate. See Kounelis, 529 F. Supp. 2d at 519 (prejudice to discovering party considered when imposing sanctions). 8. Financial Documents Plaintiff requested that Defendants produce restaurant sales reports as well as other financial data. Plaintiff does not complain about the production of many of these items. 12

269 However, Plaintiff does complain that it has not received the point of sale reports, hand logs for cash transactions, that the corporation s tax returns were unsigned, the corporation s bank account statements redacted several ATM transactions (which may be relevant for piercing analysis), and that Defendants did not produce a copy of their insurance policy. Defendants respond that the proprietary point of sale software has created problems for such production, that counsel is investigating the redaction to the bank account statements, and that they did produce the copy of the insurance policy. Defendants make no representations about the hand logs or tax returns. Nor do Defendants argue that the requests are in any way overbroad. The Court finds that these are run-of-the-mill discovery issues, on which the parties should confer to resolve. The Court does not believe a sanction or order compelling production is appropriate at this juncture. There has yet to be a litigation in which every document was produced in a timely fashion. The Court also sees no reason why Defendants must produce electronic records of their bank accounts to Plaintiff or need bear the cost of putting the data into a spreadsheet. Defendants counsel shall address each of these discovery issues in his weekly report until they are resolved. 9. Deposition of Niraj Jivani While both parties agree that Plaintiff will have another opportunity to depose Niraj Jivani, Plaintiff argues that it should be compensated for the costs of the additional deposition. Plaintiff suggests that this is justified because Defendants were late in the production of several items, including their insurance policy, prior to Mr. Jivani s deposition. The Court disagrees. As suggested by Defense counsel, the need for a supplemental deposition was also necessitated by Plaintiff s insistence on taking Mr. Jivani s deposition only a few weeks after 13

270 new counsel arrived on the case, without the opportunity for that counsel to come up to speed with the discovery requested. Further, it appears that Plaintiff has four more hours of questions, not all of which can be related to the insurance policy. As such, Plaintiff would have had to schedule an additional day of deposition anyway. Thus, the Court awards no fees for the additional deposition. 10. Counsel Are Directed to Conduct Themselves with the Collegiality and Good Faith that is Befitting of New Jersey Attorneys The Court reminds counsel that attorneys in this jurisdiction are expected to conduct themselves with professionalism, to creatively find solutions to common problems, and to cooperate with each other. The and other correspondence that has appeared before the Court do not reflect these high standards. s accusing opposing counsel of ethical violations, resistance to simple requests, attempts to pass responsibility to the other party for simple discovery requests, and inflexibility are reflected in many recent communications in this case. (See, e.g., Dolich Ex. I; Doc. No. 114; Doc. No. 117; Doc No ; Doc. No. 118; Doc. 129 Ex. 2). The Court also reminds the parties that their attempts to pass the burden of discovery to each other often result in more work in s than the work it would take to obtain the document themselves. The Court understands that there is animosity between the parties. The testimony at the preliminary injunction hearing was enough to make that abundantly clear. However, despite the animosity between their clients, attorneys before this Court are expected to confer in good faith prior to bringing issues before the Court. 14

271 III. CONCLUSION For the foregoing reasons, the Court grants in part and denies in part the request for spoliation sanctions and grants certain requests for the production of documents. The Court files an appropriate order concurrent with this opinion. Dated: August 3, 2011 /s/ Garrett E. Brown, Jr. GARRETT E. BROWN, JR., U.S.D.J. 15

272 Chief Judge Finds That Alteration of Facebook Page Can Lead to Spoliation Inference : E... Page 1 of 2 E-Discovery Law Alert Posted at 9:40 AM on December 16, 2011 by Gibbons P.C. Chief Judge Finds That Alteration of Facebook Page Can Lead to Spoliation Inference In a trademark infringement case involving two restaurants, Katiroll Company, Inc. v. Kati Roll and Platters, Inc. et al., Plaintiff sought a spoliation inference, alleging various discovery abuses involving several types of evidence including social media. Specifically, Plaintiff requested sanctions for the individual Defendant s failure to preserve his Facebook pages in two different ways. Recognizing that Facebook users change their pages frequently given the nature of the media at issue, Chief Judge Brown of the District of New Jersey crafted a creative remedy, which was based in large part on the level of prejudice to Plaintiff. Regarding the first Facebook issue, Plaintiff sought PDF versions of Defendant's Facebook pages before they were taken down pursuant to Plaintiff s take-down request. The Court declined to sanction Defendants for actions taken at Plaintiff s request because it would be "unjust." Plaintiff also sought a spoliation inference because the individual Defendant altered his profile picture on Facebook. The prior picture reflected the infringing trade dress of the restaurant at issue but was not preserved. The Court recited the four requirements for a spoliation inference, which the Chief Judge described as the mildest of sanctions: (1) whether the evidence was in the party s control; (2) whether the evidence was actually suppressed or withheld; (3) whether the evidence was relevant vis-à-vis the claims or defenses at issue; and (4) whether it was reasonably foreseeable that the evidence at issue would subsequently be discoverable. The Court concluded that the most important consideration in determining what level of fault is required to support the second factor (an issue that is disputed within the District) is the degree of prejudice to the movant. Specifically, the Court concluded that a negligence standard may be appropriate if there was substantial prejudice whereas intentional conduct would be required if minimal prejudice resulted. The Court had little difficulty finding that Plaintiff met the third requirement. It similarly found that the first requirement was met because "defendants have a discovery obligation to produce [websites] and only defendants knew whether the website would be changed." As for the second and fourth requirements, the Court found that, although the individual knew he had to preserve evidence, it would not have been immediately clear that changing his profile would undermine discoverable evidence" given that a change of a profile picture "changes the picture associated with each and every post that user has made in the past. Although the Court found that the spoliation was not intentional, the Court also determined that the loss of information was somewhat prejudicial. But instead of imposing an adverse inference instruction (as Plaintiff had requested), the Court directed Defendant to post the image of the allegedly infringing picture for a short period of time so that Plaintiff could then print relevant posts; thereafter, Defendant was permitted to re-post the non-infringing picture. 4/1/2013

273 Chief Judge Finds That Alteration of Facebook Page Can Lead to Spoliation Inference : E... Page 2 of 2 Katiroll demonstrates the Court s understanding that the nature of Facebook invites changes to postings and pictures, and it teaches that the prejudice to the movant is a significant factor in determining the level of fault required before an adverse inference instruction will be imposed. Katiroll also demonstrates that courts will fashion novel remedies for spoliation in the social media context. Mara E. Zazzali-Hogan is a Director on the Gibbons E -Discovery Task Force. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 4/1/2013

274 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 1 of 20 PageID: 997 NOT FOR PUBLICATION UNITED STATES DISTRICT COURT DISTRICT OF NEW JERSEY UNITED STATES OF AMERICA, Criminal Action No.: (JLL) v. Plaintiff, OPINION ANTHONY SUAREZ and VINCENT TABBACHINO, Defendants. LINARES, District Judge. Currently before this Court is a Motion filed by Defendants requesting, inter alia, that as a result of the Government's failure to produce certain electronic text messages related to its investigation of Defendants the Court either suppress related evidence or issue an adverse inference instruction. The Court conducted hearings and heard counsel s arguments regarding the missing text messages on July 27, 2010, September 29, 2010, and October 8, 12, and 15, I. Background In 2008, federal law enforcement authorities began using Solomon Dwek as a cooperating witness to investigate public corruption within the state of New Jersey. The cooperating witness held himself out to be real estate developer David Essenbach and claimed to be interested in developing the greater Hudson and Bergen County, New Jersey, areas. Agents within the Federal Page 1 of 20

275 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 2 of 20 PageID: 998 Bureau of Investigation (F.B.I.) instructed the cooperating witness to express interest in such development to local public officials and offer these officials payments in exchange for expediting his projects and providing other official assistance. During the period from March to July, 2009, the cooperating witness, posing as developer David Essenbach, spoke and met with the defendants herein on multiple occasions. During the relevant time period of the investigation pertaining to the defendants herein and during meetings with said defendants, the cooperating witness exchanged numerous Short Message Service electronic communications ( text messages ) with F.B.I. Agents William Waldie, Donald Russ, and Sean McCarthy. In December 2009, Defendants made their first request for discovery of the text messages exchanged between the cooperating witness and the F.B.I. agents. Receiving no response, Defendants moved on June 16, 2010 for the Government to produce the text messages. The Court held oral argument on July 27, At oral argument, the Government indicated that it would produce the text messages. (Oral Argument Tr. 26:22-27:4, July 27, 2010.) Subsequently, in a letter to Defendants dated August 31, 2010, the Government stated that it was unable to produce any of the text messages sent by the cooperating witness to the agents because the witness had used a personal cellular telephone, and his wireless carrier retained text messages for a period of only 3 to 5 days from the date of creation. Defs. Mot. in Limine Ex. D. The Government further stated that it could not produce text messages sent by Agents Waldie, Russ, or McCarthy to the cooperating witness during the time period of March 1, 2009 through March 16, 2009, nor could it produce any text messages sent by Agents Russ or McCarthy during the entire month of July Id. The Government explained that this was because the FBI retained text messages of its agents only for so long as limited storage space on Page 2 of 20

276 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 3 of 20 PageID: 999 its servers allowed. Id. The following month, however, the Government did in fact produce text messages from Agent Waldie for the time period from March 1 through 16, The Government still did not produce any text messages from Agents Russ or McCarthy for either of the relevant time periods. On September 21, 2010, Defendants made the present Motion, and oral argument was held on September 29. At oral argument, the Government confirmed that it had produced text messages from other time periods, including May 2009, but represented to the Court that no text messages existed for Agents Russ or McCarthy for the March and July time periods. The Government offered the following explanation: THE GOVERNMENT: We do not believe -- have no reason to believe that any text messages are missing. We made a misstatement in our view to counsel that in fact the period of time in our review of the [text messages] show that there were no [text messages] in March for Agents Waldie, Russ and McCarthy. We have since found Agent Waldie s and produced to the defendants all of Agent Waldie s texts for March. We also stated that in July, there was a period of July where we thought there were no [text messages] for McCarthy and Donny Russ, and we learned at least for some periods during that time, both were on -- THE COURT: McCarthy and Russ were on vacation. THE GOVERNMENT: -- correct. * * * THE COURT: Well, what is your representation to this Court regarding the March text messages? THE GOVERNMENT: We do not believe that any text messages are missing. We have no reason to believe that. (Oral Argument Tr. 94:22-96:4, Sept. 29, 2010.) After oral argument, the Government notified the Court that these statements were made Page 3 of 20

277 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 4 of 20 PageID: 1000 in error. The Government now maintains that Agent Russ and Agent McCarthy were not on vacation during July 2009 and that text messages were in fact sent by those agents to the cooperating witness during the relevant time periods. II. Text Message Hearings As a result of the lack of information regarding the exact cause and circumstances surrounding the retention and deletion of the aforesaid messages, the Court ordered a hearing. Said hearing was held on October 8, 12, and 15, The three F.B.I. agents who communicated with the cooperating witness over text message, Agents Waldie, Russ, and McCarthy, each testified. The agents were asked to describe the contents of the text messages and to indicate whether they had personally attempted to preserve the messages. Each agent testified that he had deleted the messages from his handheld Blackberry device. One of the reasons given was a fear of overloading the device s local persistent memory. (Trial Tr. 6.12:2-16, 6.52:2-12, 6.100: :17, Oct. 8, 2010 [hereinafter Hr g. Tr. ].) Each agent testified that he was never instructed to preserve text messages. (Hr g. Tr. 6.16:3-6, 6.45: :5, 6.102:11-20.) The agents also testified that on occasion they did give instructions to the cooperating witness via text message and that the witness did relay to them the witness s interpretation of what was said during the meetings. (Hr g. Tr. 6.25: :3.) The agents, however, testified that said interpretation was irrelevant because the meetings were audio and video recorded. (Hr g. Tr. 6:14:12-22, 6.50: :22.) The first witness, Agent Waldie, testified as to the conditions under which he would document events or conversations that occurred during a meeting that was being recorded by a cooperating witness: If it was something that couldn't be heard or seen, yes. But if it was Page 4 of 20

278 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 5 of 20 PageID: 1001 something that was said, no. The tape would capture that. (Hr g. Tr. 6.14:17-22.) As Agent Waldie finished his testimony, an Assistant United States Attorney left the courtroom and spoke with Agent Russ, who had been sequestered, as he had not yet testified, and was waiting in hallway. An attorney for Defendants followed the Government attorney into the hallway. When questioned by the Court about the substance of the conversation, both the Assistant United States Attorney and Agent Russ explained that the attorney was merely attempting to remind Agent Russ of what they had previously discussed during witness preparation, specifically that the meetings between the cooperating witness and Defendants had been recorded and that any conversations described in any text messages would be preserved in the recording. (Hr g. Tr. 6.42:22-24, 6.64:7-20.) This was in violation of a Sequestration Order entered by this Court. The Court takes violations of its Orders very seriously, and although the conversation with Agent Russ appears to have pertained to a matter of small importance in the context of the hearing, the Court nevertheless admonished the attorney in question and made it clear that the Court was troubled by his actions. After the F.B.I. agents testified, the Court heard testimony from three F.B.I. information technology specialists regarding the procedures that the agency uses to preserve and retrieve data generated by the agents handheld devices. Rodrick Summers, the F.B.I. s Information Technology Unit Chief, testified as to how the F.B.I. preserves text messages sent by its agents. He stated that when an agent sends a text message from his government-issued handheld device, the message is first transmitted to a commercial wireless carrier via the carrier s cellular tower infrastructure. (Trial Tr. 7.12: :19, Oct. 12, 2010 [hereinafter Hr g. Tr. ].) Then, the commercial carrier routes one copy of the message to the designated recipient (again, via the Page 5 of 20

279 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 6 of 20 PageID: 1002 commercial cellular tower infrastructure) and routes another copy of the message to a server maintained by Research in Motion Limited (RIM). (Hr g. Tr. 7.8:14-23, 7.15:2-7.16:23.) The RIM server then transmits the message to one of a number of Blackberry Enterprise Servers (BES) maintained by the F.B.I. at its Clarksburg, West Virginia, facility. (Id. at 7.9:10-12.) During the time period relevant to this case, the F.B.I. maintained fourteen BES servers. (Id. at 7.9:7-9.) After a text message leaves an agent s handheld device and until it reaches a BES server, the message is handled entirely by non-government entities. (Id. at 7.15: :2.) Mr. Summers testified that the BES servers contain only a few days of text message data at any given time, but they are backed up to a log server on a daily basis. (Id. at 7.23:3-4.) The BES server data is also backed up daily to a series of backup tapes. (Id. at 7.25:1-9.) The log server and backup tapes are all located in Clarksburg, West Virginia. (Id. at 7.11:13-15.) Mr. Summers testified that absent a litigation hold, F.B.I. policy permits that after a period of 90 days the backup tapes may be overwritten or destroyed, and log server data may be deleted. (Id. at 7.25:1-19.) No litigation hold was in place during the time period in which the missing text messages were sent, nor was a litigation hold in place during the 90 day period following that time. The Government first ordered a litigation hold on or about January 11, (Id. at 7.39:9-13.) At the hearing, the Government produced an F.B.I. Corporate Policy Directive that provided both the data retention policy of the F.B.I. and its policy regarding litigation holds. (Text Message Hr g. Ex. 3 [hereinafter Hr g. Ex. ].) This document contains the 90 day retention period described in Mr. Summers testimony and further indicates that [l]itigation holds or other special inquiries are the only exceptions to this data backup policy. In these Page 6 of 20

280 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 7 of 20 PageID: 1003 instances, backups must be maintained until resolution of the litigation or other inquiry. (Id.) The document further indicates that this exception extends to reasonably anticipated or pending litigation. (Id.) Brian Modest, a security engineer contracted by the F.B.I., described his efforts during October through December 2009 to retrieve from the F.B.I. servers the text messages from the missing periods described above. (Hr g. Tr :7-11.) Mr. Modest testified that he used a computer tool to transfer data from the log servers onto his own database, from which he conducted keyword searches. (Id. at 6.124: :23.) Mr. Modest testified that he was unable to locate the missing text messages, even through a manual search. (Id. at 6.141:3-14.) Justin Grover, a information security engineer with the F.B.I. s Enterprise Security Operations Center (ESOC), testified about an audit performed by ESOC staff in an effort to locate the missing text messages. (Id. at 6.149:16-25.) Mr. Grover indicated that he personally performed the same audit again on October 1, (Id. at 6.150:15-21.) Mr. Grover testified that ESOC maintains a separate database of text message data from the ones described by the other witnesses. On a daily basis, ESOC transfers text message data from the BES servers through an intermediary server and onto ESOC s classified analytical network. (Id. at 6.147: :1.) The classified analytical network maintains text message data back to November (Id. at 6.147:7-18.) Mr. Grover testified that if information never resided on the BES servers, it would not be found on the classified analytical network maintained by ESOC. (Id. at 6.125:5-8.) None of the witnesses were able to explain why some text messages from the same time period were preserved while others were apparently deleted. Likewise no reasonable explanation was offered as to why there was a need to place a litigation hold on text message data Page 7 of 20

281 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 8 of 20 PageID: 1004 in January of 2010, yet no need to do so prior to that time. III. Discussion The Government has no constitutional duty routinely to deliver his entire file to defense counsel. United States v. Agurs, 427 U.S. 97, 111 (1976). In contrast to the breadth of discovery permitted in civil cases, Rule 16 of the Federal Rules of Criminal Procedure delineates the categories of information to which defendants are entitled in pretrial discovery in criminal cases, with some additional material being discoverable in accordance with statutory pronouncements and the due process clause of the Constitution. United States. v. Ramos, 27 F.3d 65, 68 (3d Cir. 1994). In general, this additional material is limited to material discoverable under the Jencks Act and under Brady v. Maryland, 373 U.S. 83 (1963). Id. Therefore, before deciding whether sanctions are warranted for the Government s failure to produce the missing text messages, the Court must determine whether those text messages fall under Rule 16 or whether they constitute Jencks or Brady material. A. Whether the Text Messages are Discoverable Federal Rule of Criminal Procedure 16(a) outlines the Government s basic disclosure requirements. Most pertinent to the facts of this case is Rule 16(a)(1)(E), which provides that [u]pon a defendant s request, the government must permit the defendant to inspect and to copy or photograph books, papers, documents, data..., if the item is within the government s possession, custody, or control and (i) the item is material to preparing the defense; (ii) the government intends to use the item in its case-in-chief at trial; or (iii) the item was obtained from or belongs to the defendant. Fed. R. Crim. P. 16(a)(1)(E). However, Rule 16(a)(2) clarifies that this rule does not authorize the discovery or inspection of statements made by prospective Page 8 of 20

282 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 9 of 20 PageID: 1005 government witnesses except as provided in 18 U.S.C [the Jencks Act]. Fed. R. Crim. P. 16(a)(2). Since the text messages at issue in this case were sent between F.B.I. agents and the cooperating witness, they were statements made by prospective government witnesses. The Court must therefore determine whether production of the text messages is required by the Jencks Act. 1 The Jencks Act and Federal Rule of Criminal Procedure 26.2 require the Government to disclose prior recorded statements of its witnesses after they testify. 18 U.S.C. 3500(b); Fed. R. Crim. P The purpose of the Jencks Act and Rule 26.2 is to permit defense counsel to have the materials necessary to impeach a Government witness. United States v. Rosa, 891 F.2d 1074, (3d Cir.1989) (citing Jencks v. United States, 353 U.S. 657, 667, 77 (1957)). Rule 26.2 provides in relevant part: After a witness other than the defendant has testified on direct examination, the court, on motion of a party who did not call the witness, must order an attorney for the government or the defendant and the defendant's attorney to produce, for the examination and use of the moving party, any statement of the witness that is in their possession and that relates to the subject matter of the witness's testimony. Fed. R. Crim. P. 26.2(a). A statement by a witness includes any written statement that the witness makes and signs, or otherwise adopts or approves. Fed. R. Crim. P. 26.2(f)(1). In the context of the destruction of evidence by F.B.I. agents, the Third Circuit has held that the rough interview notes of F.B.I. agents should be kept and produced so that the trial court can determine whether the notes should be made available to the appellant under the rule of 1 The Jencks Act and Federal Rule of Criminal Procedure 26.2 can be used interchangeably because Rule 26.2 incorporates the relevant provisions of the Jencks Act without any substantive change. United States v. Heilman, 377 Fed. Appx. 157, 195 (3d Cir. 2010) (internal quotation omitted). Page 9 of 20

283 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 10 of 20 PageID: 1006 [Brady], or the Jencks Act. United States v. Vella, 562 F.2d 275, 276 (3d Cir. 1977). The appellate court held that while the F.B.I. s destruction of the interview notes amounted to only harmless error, the Government nonetheless had a duty to preserve the documents in case they did in fact constitute Jencks material. Id. The Third Circuit has affirmed this rule and extended it to cover draft reports by agents. See United States v. Ammar, 714 F.2d 238, 259 (3d Cir. 1983) ( the government must retain and, upon motion, make available to the district court both the rough notes and the drafts of reports of its agents to facilitate the district court's determination whether they should be produced ). Here the Court is presented with a similar quandary as in Vella and Ammar. When a document is destroyed or otherwise unavailable, the court is left to speculate whether its contents constituted a statement relevant to the subject matter of the witness's testimony and whether the document was ever even in the possession of the Government. But, regardless of the content of the text messages, the Court can at least be certain here that the agents and the cooperating witness each adopted and approved each message they sent when they affirmatively chose to transmit it. Thus, the Court deems the text messages statements under the Jencks Act. As to the content of the text messages sent between the agents and the cooperating witness, Agent Waldie testified that while his communications typically concerned the logistics of the cooperating witness s meetings with Defendants, he also gave instructions to the witness. (Hr g. Tr. 6.25: :3.) The agent testified that he encouraged the witness to be blatant and to dirty up the money, meaning that the agent encouraged the witness to make it clear to Defendants that he intended to offer them a bribe. (Id. at 6.25:20, 627:21.) While each agent maintained that none of the cooperating witness s text messages to them contained any Page 10 of 20

284 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 11 of 20 PageID: 1007 information that he believed to be exculpatory, what the cooperating witness may have said to the agents during the crucial meetings and whether he abided by the agents instructions are fertile ground for cross examination. (Id. at 6.11:19-23, 6.49: :3, 6.99:22-24.) For example, the cooperating witness testified at trial that [i]t was very infrequent, if at all, that [he] actually texted an agent and that he would typically only send text messages to agents when safety issues arose or before the payment went down. (Trial Tr : :20.) This testimony is belied by telephone records produced at the text message hearing, and the cooperating witness could have been further impeached on cross examination by recitation of the cooperating witness s actual text messages, had such messages been preserved. (Hr g. Ex. 1, 2.) The need for effective cross examination of Government witnesses lies at the heart of the Jencks Act and echoes the concerns expressed in Vella and Ammar. Thus, the content of the text messages sent by the cooperating witness to the agents, and the agents response or instructions thereto, relate to the subject matter of the witness's testimony. Informational technology specialists from the F.B.I. testified regarding two theories as to why the missing text messages between the agents and the cooperating witness were not recovered. Mr. Summers, the Information Technology Unit Chief, offered the following two theories: 1) that the messages were deleted from the F.B.I. servers after 90 days pursuant to the agency s destruction policy, or 2) that the messages never reached the F.B.I. servers. (Hr g. Tr. 7.37: :21.) Thus, under the first theory, the F.B.I. was in possession of the messages both while they existed on the agents handheld devices and for the approximately 90 days that they resided on the F.B.I. servers. Under the second theory, the F.B.I. was only in possession of the messages while they existed on the agents handheld devices. In both scenarios, the Page 11 of 20

285 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 12 of 20 PageID: 1008 messages existed on the F.B.I. handheld devices until the agents manually deleted them. Each agent testified that he intentionally deleted the messages in order to free space in the phone s memory and not pursuant to any destruction policy. Indeed, neither the agents nor the specialists in charge of maintaining the F.B.I. servers were under any sort of directive to maintain the text messages in anticipation that criminal charges would be filed against Defendants. The F.B.I. s own Corporate Policy Directive in fact anticipates this sort of litigation hold on its document destruction policy. It states that the F.B.I. policy that a full data backup will be retained for 90 days does not apply in cases of reasonably anticipated or pending litigation. (Hr g. Ex. 3.) Thus, the F.B.I. appears to have been well-equipped to preserve documents relevant to this litigation had the United States Attorneys Office requested it do so. This is especially important in the context of the case at bar, where the investigation was undertaken as a joint operation by the New Jersey United States Attorneys Office and the F.B.I., and the United States Attorney was aware of the importance of preserving documents relevant to the litigation and could have requested a litigation hold on the text messages from the inception of the investigation. That request, however, did not come until January of 2010 seven months after the investigation concluded and three months after the F.B.I. began searching its servers for the missing text messages. Without an appropriately timed litigation hold, the Court is left to speculate as to the contents of the missing text messages and to entertain various theories as to the cause of their disappearance. However, it is without dispute that each missing text message at one time resided on an F.B.I. agent s handheld device. It is also clear that the United States Attorneys supervising the investigation were aware that text messaging was being used by the Page 12 of 20

286 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 13 of 20 PageID: 1009 cooperating witness, as such use was evident on the videos. These text messages were indisputably in the Government s possession at one time. Thus, given the F.B.I. s analogous preservation duty under Vella and Ammar and the failure of the Government to preserve relevant data in the midst of an ongoing investigation specifically aimed at prosecution, and thus where litigation was reasonably anticipated, the Government had a duty to preserve the Jencks material contained in the text messages. B. Appropriate Sanction for the Destruction of Jencks Material Rule 26.2(e) provides that [i]f a party who called a witness disobeys an order to produce or deliver [Jencks material], the court must strike the witness s testimony from the record, and further [i]f an attorney for the government disobeys the order, the court must declare a mistrial if justice so requires. Fed. R. Crim. P. 26.2(e). The Rule thus specifies potentially severe sanctions for disobedience of a court order to produce Jencks material. The Rule does not, however, mandate any particular sanction against parties who destroy Jencks material outside the presence of a court order. Defendants first request that the testimony of the cooperating witness and all tape recordings in which he is a party be suppressed. In United States v. Ramos, the Third Circuit addressed whether the destruction of rough interview notes, as in Vella, warranted suppression of related evidence. The Court reasoned that the mere fact that Vella and Ammar each established rules for the government to follow does not suggest that we intended the automatic suppression of evidence when those rules are violated. 27 F.3d at 69. Instead, the Court applied a good faith test to determine whether suppression is warranted under the Jencks Act. Id. (citing Arizona v. Youngblood, 488 U.S. 51, 109 (1988)). Here, Defendants have presented little Page 13 of 20

287 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 14 of 20 PageID: 1010 evidence, and the Court does not find, that the Government lost or destroyed the text messages in bad faith or that the text messages clearly contained exculpatory evidence. Accordingly, suppression of evidence related to the cooperating witness is not warranted, and to the extent that Defendants move for such sanctions such request is denied. Defendants alternatively request that the Court give an adverse inference instruction to the jury regarding the missing text messages. Since the facts of this case lie outside of the mandates of Rule 26.2(e), the Court has discretion to apply alternative sanctions. See United States v. Jackson, 649 F.2d 967, 972 n.6 (3d Cir. 1981) ( it has been held that the statutory alternatives of striking the witness testimony or declaring a mistrial are not mandatory, and whether to follow a different course rests with the discretion of the trial judge ); United States v. Ferber, 1991 WL 16751, at *6 (E.D. Pa. 1991) ( where the Jencks Act is violated through oversight or negligence not amounting to a conscious election, the Court is entitled to apply such a remedy as the justice of the case might require under the circumstances ) (internal quotations omitted). Given the relatively little criminal case law in the Third Circuit regarding the 2 application of adverse inference instructions based on spoliation of evidence, the Court consults the more thoroughly developed civil case law on the subject. See United States v. Bunty, 617 F. Supp. 2d 359 (E.D. Pa. 2008) (using the civil standard to select a spoliation sanction in a criminal case). The key considerations for determining the appropriate spoliation sanction (e.g., 2 The most factually similar criminal case that the Court has found is United States v. Fleming, 818 F. Supp. 845, 851 (E.D. Pa. 1993), in which the district court refused to give the jury an adverse inference instruction where two witnesses had destroyed evidence before they began cooperating with the Government. Since the destruction occurred before cooperation began, Fleming provides little guidance here. Page 14 of 20

288 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 15 of 20 PageID: 1011 dismissal, suppression, fines, or an adverse inference instruction) are: (1) The degree of fault of the party who altered or destroyed the evidence; (2) The degree of prejudice suffered by the opposing party; and (3) Whether there is a lesser sanction that will avoid substantial unfairness to the opposing party and, where the offending party is seriously at fault, will serve to deter such conduct by others in the future. Schmid v. Milwaukee Elec. Tool Corp., 13 F.3d 76, 79 (3d Cir. 1994). In this case there is little evidence to suggest that the Government acted in bad faith in destroying the text messages. There is, however, evidence that indicates that the defense is in fact prejudiced by the loss of this evidence. The credibility of the cooperating witness and his interpretation of what was transpiring and being said at the meetings with the defendants herein is of paramount importance, and the text messages would have enabled the defense to conduct a more effective and thorough cross examination of the cooperating witness. The Court, having already concluded that the relatively severe sanction of suppression is not warranted in this case, agrees with Defendants that the lesser sanction of an adverse inference instruction is appropriate, considering the Schmid factors. The Court finds the adverse inference instruction appropriate because: (1) The text messages were within the Government s control; (2) The text messages were intentionally deleted by the agents, and the U.S. Attorneys Office failed to take steps to preserve them; (3) The text messages that were deleted or not preserved were relevant to claims or defenses; and (4) It was reasonably foreseeable by the Government that in the context of this investigation and in light of the actions of the cooperating witness the text messages would later have been discoverable. Page 15 of 20

289 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 16 of 20 PageID: 1012 These findings by the Court fall squarely within the four elements set forth in Mosaid Technologies Inc. v. Samsung Electronics Co., 348 F. Supp. 2d 332 (D.N.J. 2004). Under Mosaid, the Court may only give an adverse inference instruction based on spoliation if the following elements are satisfied: (1) The evidence in question must be within the party's control; (2) It must appear that there has been actual suppression or withholding of the evidence; (3) The evidence destroyed or withheld was relevant to claims or defenses; and (4) It was reasonably foreseeable that the evidence would later be discoverable. 348 F. Supp. 2d at 336. As to the first element, the Court has already addressed the extent to which the text messages were within the Government s control. Even if the messages never found their way to the F.B.I. servers, they persisted on the agents handheld devices until the agents deleted them. Thus, the text messages were within the Government s control. As to the second element, the key inquiry is the meaning of actual suppression. The Third Circuit has explained that [n]o unfavorable inference arises when the circumstances indicate that the document or article in question has been lost or accidentally destroyed, or where the failure to produce it is otherwise properly accounted for. Brewer v. Quaker State Oil Ref. Corp., 72 F.3d 326, 334 (3d Cir. 1995). In Mosaid, the district court directly addressed whether the actual suppression requirement implies a requirement of bad faith. 348 F. Supp. 2d 332. After a thorough analysis of Third Circuit case law in light of the spoliation inference s remedial function of leveling the playing field after a party has destroyed or withheld relevant Page 16 of 20

290 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 17 of 20 PageID: 1013 evidence, the court in Mosaid adopted a flexible approach, which takes into account both the culpability of the offending party and the prejudice to the innocent party. Id. at It explained that cases will arise in which the offending party s culpability is largely irrelevant as it cannot be denied that the opposing party has been prejudiced. Id. The Court finds this flexible approach particularly persuasive in light of the complementary flexibility that the Court maintains to tailor jury instructions to the needs of specific cases. See Pension Comm. v. Banc of Am. Sec., 685 F. Supp. 2d 456, (S.D.N.Y. 2010) (categorizing different forms of adverse inference instructions based on the severity of the offending party s misconduct). In this case there is little evidence to suggest that the Government acted in bad faith in destroying the text messages. The most that can be said is that the Government failed to instruct the F.B.I. and the F.B.I. failed independently to recognize the need to preserve potential Jencks material in anticipation of bringing criminal charges against Defendants. Nonetheless, the prejudicial effect of this failure cannot be ignored. Defendants are now without cross examination material that could be used to thoroughly test the credibility of the cooperating witness and determine whether the cooperating witness actually did or did not follow the instructions given in the F.B.I. agents text messages, or whether the interpretation that the witness gave in the text messages as to what was said at the meetings was or was not at odds with the interpretation that the witness gave at trial. Thus, while the culpability of the Government at worst falls within the realm of gross negligence, the prejudice to Defendants is clear. The Court therefore finds actual suppression or withholding of the text messages by the Government under the flexible approach set forth in Mosaid. As to the third element, the Court must determine whether the destroyed evidence was Page 17 of 20

291 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 18 of 20 PageID: 1014 relevant to the claims and defenses in this case. Thorough cross examination of government cooperating witnesses is vital to a defendant s presentation of his or her case. Evidence relevant to cross examination is therefore likewise relevant to the defendant s case. Thus, the destroyed evidence here is relevant to claims or defenses in this case. Finally, as to the fourth element, the Court must address whether it was reasonably foreseeable that the evidence would later be discoverable. Mosaid, 348 F. Supp. 2d at 336. Here, the Government used the cooperating witness in a joint investigation with the F.B.I. to gather evidence of alleged public corruption throughout New Jersey. Indeed, the cooperating witness, who is often seen text messaging on the videos of his meetings with Defendants, was the prosecution s key trial witness and was cooperating in the investigation by virtue of a plea and cooperation agreement. Thus, the Government must have foreseen that any defense attorney viewing the same videos that the Government was privy to would seek discovery of the text messages and that the messages were therefore a potential source of Jencks material. The Court thus concludes that it is at least reasonably foreseeable that communications between the cooperating witness and the Government agents instructing him during the investigation would be discoverable in this case. Having concluded that an adverse inference instruction is warranted in this case, the Court now addresses how the instruction should be phrased. In Pension Committee v. Banc of America Securities, the court delineated three levels of adverse inference instruction, differing in severity based on the nature of the spoliating party s conduct. 685 F. Supp. 2d at 470. The court explained: In its most harsh form, when a spoliating party has acted willfully or in bad faith, a jury Page 18 of 20

292 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 19 of 20 PageID: 1015 can be instructed that certain facts are deemed admitted and must be accepted as true. At the next level, when a spoliating party has acted willfully or recklessly, a court may impose a mandatory presumption. Even a mandatory presumption, however, is considered to be rebuttable. The least harsh instruction permits (but does not require) a jury to presume that the lost evidence is both relevant and favorable to the innocent party. If it makes this presumption, the spoliating party s rebuttal evidence must then be considered by the jury, which must then decide whether to draw an adverse inference against the spoliating party. This sanction still benefits the innocent party in that it allows the jury to consider both the misconduct of the spoliating party as well as proof of prejudice to the innocent party. Such a charge should be termed a spoliation charge to distinguish it from a charge where the a jury is directed to presume, albeit still subject to rebuttal, that the missing evidence would have been favorable to the innocent party, and from a charge where the jury is directed to deem certain facts admitted. Id. at (emphasis in original). As already discussed, the Banc of America approach complements the flexible analysis advocated in Mosaid. Not only is the culpability of the spoliating party relevant to selecting an appropriate adverse inference instruction, so is the resulting prejudice to the innocent party. Given the lack of evidence suggesting bad faith on the part of the Government and in light of the particular circumstances of this case, the Court deems the least harsh, spoliation charge most appropriate. At the close of evidence, the Court will issue the following charge to the jury: During the course of this trial you have heard evidence by way of stipulation and testimony that during the Government s investigation of Defendants, the cooperating witness, Solomon Dwek, exchanged numerous text messages with F.B.I. agents supervising the investigation. The Government was obligated to preserve all of these text messages, but they were either deleted by the agents themselves or not preserved by the Government. Specifically, although some text messages were in fact preserved, the Government failed to preserve other text messages, which pertained to Agent Russ and Agent McCarthy, from two key time periods: March 1 through March 16, 2009 and the entire month of July You may infer from the Government s failure to preserve these messages, or the fact that they were deleted by agents, that the missing text messages were relevant to this case and that they were favorable to both Defendant Suarez and Defendant Tabbachino. You are not required to make this inference, however, and you must consider any rebuttal evidence that has been offered by the Page 19 of 20

293 Case 2:09-cr JLL Document 61-1 Filed 10/21/10 Page 20 of 20 PageID: 1016 Government with regard to this issue. Whether you ultimately choose to make the inference is your decision as the finder of fact. IV. Conclusion For the foregoing reasons, the Court grants in part and denies in part Defendant s Motion. An appropriate Order accompanies this Opinion. DATED: October 19, 2010 /s/ Jose L. Linares JOSE L. LINARES UNITED STATES DISTRICT JUDGE Page 20 of 20

294 Federal Judge Rules Government Failed to Preserve Text Messages and Orders Adverse I... Page 1 of 2 E-Discovery Law Alert Posted at 10:08 PM on November 2, 2010 by Jennifer Marino Thibodaux Federal Judge Rules Government Failed to Preserve Text Messages and Orders Adverse Inference Instruction in Criminal Case On October 21, 2010, in the highly publicized New Jersey government corruption case U.S. v. Suarez, et ano., No , 2010 U.S. Dist. LEXIS (D.N.J.), the Honorable Jose L. Linares, U.S.D.J., held that the FBI had a duty to preserve Short Message Service electronic communications (i.e., text messages) exchanged between its agents and their cooperating witness, Solomon Dwek, during the course of the investigation of defendants Anthony Suarez (mayor of Ridgefield, NJ) and Vincent Tabbachino (former Guttenberg, NJ councilman and police officer). Despite the lack of evidence of bad faith on the part of the government, because the text messages were not preserved, the Court found clear prejudice to defendants and ordered that the appropriate sanction was a permissive adverse inference jury instruction. From March to July 2009, Dwek assumed the identity of a real estate developer to assist the FBI with its corruption investigation of local public officials. Specifically, the FBI instructed Dwek to meet with the officials to express interest in real estate development projects and to offer bribes to them in exchange for expediting the projects and providing other official assistance. In substance, the text messages exchanged between Dwek and the agents during this period concerned investigation logistics, Dwek s impressions of what was transpiring during the investigation and the agents instructions to Dwek in carrying out the investigation (e.g., encouraging Dwek to be blatant and to dirty up the money ). Shortly after defendants moved to compel production of the text messages, the government agreed to produce them but later advised that it could not do so. The government explained that Dwek used a personal cell phone to transmit text messages, which his carrier retained for only 3 to 5 days. As to the text messages originating from or received on the agents Blackberry handheld devices, the government ultimately admitted that it failed to preserve many of these messages, notwithstanding its document retention policy concerning reasonably anticipated or pending litigation. The government did not issue a litigation hold until on or about January 11, 2010, long after the subject data would have been destroyed in the normal course of business. The Court found that the text messages were discoverable under the Jencks Act, 18 U.S.C. 3500, and Federal Rule of Criminal Procedure 26.2 and should have been preserved and produced. The Court declined to impose the harsh sanction of suppressing Dwek s testimony and all tape recordings in which he participated, but carefully analyzed the propriety of giving the jury one of various forms of adverse inference instructions. Consulting the Honorable Shira A. Scheindlin s decision in Pension Committee of the University of Montreal Pension Plan v. Banc of America Securities, LLC, 685 F. Supp. 2d 456 (S.D.N.Y. 2010) (corrected in part by Order dated May 28, 2010), the Court ordered the most flexible and least harsh adverse inference instruction: permitting, but not requiring, the jury to presume that the 4/1/2013

295 Federal Judge Rules Government Failed to Preserve Text Messages and Orders Adverse I... Page 2 of 2 lost text messages were both relevant and favorable to defendants, but mandating that the jury consider rebuttal evidence on these issues. The Court acknowledged that, although defendants were clearly prejudiced in their ability to impeach the government s cooperating witness, there was little evidence to suggest that the government acted in bad faith. On October 27, 2010, the jury acquitted defendant Suarez on all counts (conspiracy to commit extortion, attempted extortion and bribery) and convicted defendant Tabbachino of attempted extortion and bribery. As with all decisions imposing sanctions for spoliation of evidence, U.S. v. Suarez underscores the critical importance of promptly issuing a litigation hold and properly preserving evidence. Failure to discharge these now well established duties, even absent bad faith, can result in severe sanctions that may negatively impact the prosecution of a case, both in the criminal and the civil context. Mara E. Zazzali-Hogan is a Director in the Gibbons Business & Commercial Litigation Department. Jennifer Marino Thibodaux, an Associate in the Gibbons Business & Commercial Litigation Department, co-authored this post. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 4/1/2013

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331 How a "Stink Bomb" and Its Proof That Facebook Pictures Were Deleted Might... Page 1 of 2 E-Discovery Law Alert Posted at 12:55 PM on August 8, 2011 by Gibbons P.C. How a "Stink Bomb" and Its Proof That Facebook Pictures Were Deleted Might Have Blown Up a $10.6 Million Verdict Parties in all types of cases often post pictures and messages on Facebook that might be detrimental to their cases. After his wife died tragically in an automobile accident, and he brought a wrongful death case, Isaiah Lester did just that when he posted a photo of himself wearing an I [love] hot moms t-shirt and garter belt on his head while he had a beer in hand. That was his first bad choice. The defense attorneys, who represented the driver of the vehicle and his employer, saw the picture and requested similar pictures and screen shots from Lester s Facebook account. The next day, Lester s attorney, Matthew B. Murray (who is the Virginia Trial Lawyers Association s immediate pastpresident), allegedly instructed his paralegal to advise Lester to delete some Facebook pictures. Lester, however, purportedly informed Murray that he had deleted his Facebook account. Clearly, these actions reflect additional unwise decisions. Defense attorneys are now challenging the $10.6 million verdict, which may be the state s largest wrongful death case. What was even more egregious according to defense counsel, however, was that Murray denied that Lester had a Facebook account on two specific dates on which he in fact did. They further contend that to add insult to injury, Murray denied up until the time of trial that he dictated any alteration of the Facebook account. Defense attorneys also contend that if the misrepresentations made to the court and opposing counsel were not sufficiently disconcerting, that Murray then deliberately withheld evidence, namely, a stink bomb reflecting that indeed, Murray had essentially directed his client to delete pictures from his Facebook page. Lester and Murray were also accused of other improprieties. Ultimately, defense counsel proposed to the court there possible solutions: (1) dismiss Lester s claim and enter a verdict in favor of the defendants; (2) throw out Lester s claim, set aside the verdict and allow a new trial, which would limit Lester s recovery of damages and prohibit Murray or his firm from representing Lester; or (3) reduce the verdict to $2.2 million or $1.1 million and preclude Lester s attorney or the firm from collecting any contingent fees. A hearing regarding this issue will be held on September 23, Murray has reportedly since retired from the practice of law and resigned from his firm on July 6, It is unclear if any ethical or disciplinary proceedings will follow. But suffice it to say that while attorneys should advise their clients not to post potentially damning photographs or messages to social media websites, they should be wary of the consequences of advising their clients to delete or alter such information -- particularly when it is the subject of a discovery request. (Click here for information on how to obtain Facebook discovery or preserve postings/pages through Facebook s Download Your 4/3/2013

332 How a "Stink Bomb" and Its Proof That Facebook Pictures Were Deleted Might... Page 2 of 2 Information feature.) Indeed, that may be an obvious and prudent course of action for most attorneys. At least one attorney, however, failed to answer the clue phone on that issue. Mara E. Zazzali-Hogan is a Director on the Gibbons E -Discovery Task Force. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 4/3/2013

333 Lester v. Allied Part 2: "Clean Up" of Compromising Social Media Evidence Can Result i... Page 1 of 2 E-Discovery Law Alert Posted at 11:30 AM on February 21, 2012 by Gibbons P.C. Lester v. Allied Part 2: "Clean Up" of Compromising Social Media Evidence Can Result in Severe Sanctions Though some practitioners might be in denial, the follow-up sanctions orders in Lester v. Allied Concrete Co. et al. dated May 27, 2011 and September 23, 2011 should leave no room for doubt that preservation of social media is as important as any other electronic data or discovery. Similarly, the penalty for intentionally destroying such evidence may reach beyond the purse strings. As reported late last year in a blog posting regarding Lester Part I, defense counsel moved for sanctions against plaintiff Isaiah Lester and his attorney, Matthew B. Murray, for spoliation of Facebook evidence. Ultimately, although defense counsel s request for a new trial was denied after the jury had reached a $10.6 million verdict related to the tragic death of Lester s wife in an automobile accident, Lester and Murray were ordered to pay defense counsel respectively, $180,000 and $542,000. But the sanctions did not end there; Murray s misconduct was referred to the Virginia State Bar and allegations of Lester s perjury were referred to the local prosecutor. During discovery, defendants sought production of the contents of Lester s account because they believed it would help their case on the issue of damages. Defendants were inspired to pursue this evidence after discovering a photo on Facebook depicting Lester clutching a beer can, wearing a T-shirt emblazoned with I [heart] hot moms. Specifically, they served discovery requests that sought information relating to Lester s Facebook account including screen-prints and attached the compromising picture. After reviewing Lester s Facebook page in conjunction with the discovery requests, Murray s paralegal (at Murray s direction) advised Lester via to clean up the Facebook page because we do NOT want blow ups of other pics at trial so please, please clean up your facebook and myspace. She also advised there were other pics that should be deleted. Thereafter, Murray concocted a scheme to deactivate Lester s Facebook account and to advise defense counsel that on the date the answer to the discovery was signed that Lester had no Facebook page. Shortly thereafter, Lester deleted 16 photos (which were ultimately available at trial) to ensure he complied with Murray s initial clean up request, although Murray denied knowledge of deletion of these photos in discovery responses. Thereafter, Murray and Lester attempted -- unsuccessfully -- to cover up these communications and related acts. (Defendants internet technology expert was ultimately able to confirm deletion of the photos). The amount and scope of sanctions in Lester make clear that social media evidence is as important as 4/3/2013

334 Lester v. Allied Part 2: "Clean Up" of Compromising Social Media Evidence Can Result i... Page 2 of 2 other electronic discovery, and that the penalties for its spoliation can be as severe as those for destroying other evidence. While the egregious malfeasance of Lester and Murray (and Murray s paralegal, who was under his supervision) did not result in an overturned verdict (though it was reduced by around $4 million for other reasons), both counsel and client paid a high price for their cavalier treatment and destruction of Facebook evidence. Mara E. Zazzali-Hogan is a Director on the Gibbons E -Discovery Task Force. Comments (0) Gibbons P.C. gibbonslaw.com Newark New York Philadelphia Trenton Wilmington 4/3/2013

335 Ex-Juror Who "Friended" Defendant Faces Jail for Bragging on Facebook About Dismiss... Page 1 of 2 E-Discovery Law Alert Posted at 9:02 AM on March 22, 2012 by Jennifer Marino Thibodaux Ex-Juror Who "Friended" Defendant Faces Jail for Bragging on Facebook About Dismissal From Jury Duty By now, attorneys should know to advise their clients to watch out for Friend requests from jurors during a trial. The latest debacle concerning jurors use of social media involves a juror friending a party and then bragging about his resulting dismissal from the panel. For that juror, his Facebook antics landed him a three-day jail sentence. Click here and here for additional coverage regarding this incident. Back in December, a Florida man serving jury duty sent a Friend request to the defendant. The defendant told her attorney about the request, and Circuit Judge Nancy Donnellan dismissed the juror and admonished him for his actions. But, the story doesn t end there. After his dismissal, the ex-juror took to his Facebook page again. This time, he posted a comment: Score... I got dismissed!! [A]pparently they frown upon sending a friend request to the defendant... haha. Unfortunately for the ex-juror, court officials became aware of the post, leading Judge Donnellan to call the ex-juror back into Court to face criminal contempt charges. At the end of a two-hour hearing, Judge Donnellan commented, I cannot think of a more insidious threat to the erosion of democracy than citizens who do not care. She sentenced the ex-juror to three days in jail, and he was led from the courtroom in handcuffs. This is just another example of social media impacting judicial proceedings. (For recent blog posts on this issue, click here.) As the attorney for the plaintiff in this case commented, it was a very big deal to his client who had $48,000 in medical bills on the line, and if the female defendant had, for example, responded to the Friend request and went on a date with the juror, perhaps the juror would have found in her favor. In light of these missteps, attorneys must be keen to clients Facebook page and activity and should consider monitoring the Facebook pages of the defendant and any jurors -- especially any juror who has been dismissed for the improper use of social media. Jennifer Marino Thibodaux is an Associate on the Gibbons E-Discovery Task Force. Comments (0) Gibbons P.C. gibbonslaw.com 4/1/2013

336 Ex-Juror Who "Friended" Defendant Faces Jail for Bragging on Facebook About Dismiss... Page 2 of 2 Newark New York Philadelphia Trenton Wilmington 4/1/2013

337 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK x MONIQUE DA SILVA MOORE, et al., Plaintiffs, -against- PUBLICIS GROUPE & MSL GROUP, Defendants : : : : : x 11 Civ (ALC) (AJP) OPINION AND ORDER ANDREW J. PECK, United States Magistrate Judge: In my article Search, Forward: Will manual document review and keyword searches be replaced by computer-assisted coding?, I wrote: To my knowledge, no reported case (federal or state) has ruled on the use of computer-assisted coding. While anecdotally it appears that some lawyers are using predictive coding technology, it also appears that many lawyers (and their clients) are waiting for a judicial decision approving of computer-assisted review. Perhaps they are looking for an opinion concluding that: "It is the opinion of this court that the use of predictive coding is a proper and acceptable means of conducting searches under the Federal Rules of Civil Procedure, and furthermore that the software provided for this purpose by [insert name of your favorite vendor] is the software of choice in this court." If so, it will be a long wait..... Until there is a judicial opinion approving (or even critiquing) the use of predictive coding, counsel will just have to rely on this article as a sign of judicial approval. In my opinion, computer-assisted coding should be used in those cases where it will help "secure the just, speedy, and inexpensive" (Fed. R. Civ. P. 1) determination of cases in our e-discovery world. G:\AJP\DA SILVA MOORE - ESI

338 2 Andrew Peck, Search, Forward, L. Tech. News, Oct. 2011, at 25, 29. This judicial opinion now recognizes that computer-assisted review is an acceptable way to search for relevant ESI in appropriate cases. 1/ CASE BACKGROUND In this action, five female named plaintiffs are suing defendant Publicis Groupe, "one of the world's 'big four' advertising conglomerates," and its United States public relations subsidiary, defendant MSL Group. (See Dkt. No. 4: Am. Compl. 1, 5, ) Plaintiffs allege that defendants have a "glass ceiling" that limits women to entry level positions, and that there is "systemic, company-wide gender discrimination against female PR employees like Plaintiffs." (Am. Compl. 4-6, 8.) Plaintiffs allege that the gender discrimination includes (Am. Compl. 8.) (a) paying Plaintiffs and other female PR employees less than similarly-situated male employees; (b) failing to promote or advance Plaintiffs and other female PR employees at the same rate as similarly-situated male employees; and (c) carrying out discriminatory terminations, demotions and/or job reassignments of female PR employees when the company reorganized its PR practice beginning in Plaintiffs assert claims for gender discrimination under Title VII (and under similar New York State and New York City laws) (Am. Compl ), pregnancy discrimination under Title VII and related violations of the Family and Medical Leave Act (Am. Compl ), as 1/ To correct the many blogs about this case, initiated by a press release from plaintiffs' vendor the Court did not order the parties to use predictive coding. The parties had agreed to defendants' use of it, but had disputes over the scope and implementation, which the Court ruled on, thus accepting the use of computer-assisted review in this lawsuit. G:\AJP\DA SILVA MOORE - ESI

339 3 well as violations of the Equal Pay Act and Fair Labor Standards Act (and the similar New York Labor Law) (Am. Compl ). The complaint seeks to bring the Equal Pay Act/FLSA claims as a "collective action" (i.e., opt-in) on behalf of all "current, former, and future female PR employees" employed by defendants in the United States "at any time during the applicable liability period" (Am. Compl , ), and as a class action on the gender and pregnancy discrimination claims and on the New York Labor Law pay claim (Am. Compl ). Plaintiffs, however, have not yet moved for collective action or class certification at this time. Defendant MSL denies the allegations in the complaint and has asserted various affirmative defenses. (See generally Dkt. No. 19: MSL Answer.) Defendant Publicis is challenging the Court's jurisdiction over it, and the parties have until March 12, 2012 to conduct jurisdictional discovery. (See Dkt. No. 44: 10/12/11 Order.) as follows: COMPUTER-ASSISTED REVIEW EXPLAINED My Search, Forward article explained my understanding of computer-assisted review, By computer-assisted coding, I mean tools (different vendors use different names) that use sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e., training by) a human reviewer. Unlike manual review, where the review is done by the most junior staff, computer-assisted coding involves a senior partner (or [small] team) who review and code a "seed set" of documents. The computer identifies properties of those documents that it uses to code other documents. As the senior reviewer continues to code more sample documents, the computer predicts the reviewer's coding. (Or, the computer codes some documents and asks the senior reviewer for feedback.) G:\AJP\DA SILVA MOORE - ESI

340 4 When the system's predictions and the reviewer's coding sufficiently coincide, the system has learned enough to make confident predictions for the remaining documents. Typically, the senior lawyer (or team) needs to review only a few thousand documents to train the computer. Some systems produce a simple yes/no as to relevance, while others give a relevance score (say, on a 0 to 100 basis) that counsel can use to prioritize review. For example, a score above 50 may produce 97% of the relevant documents, but constitutes only 20% of the entire document set. Counsel may decide, after sampling and quality control tests, that documents with a score of below 15 are so highly likely to be irrelevant that no further human review is necessary. Counsel can also decide the cost-benefit of manual review of the documents with scores of Andrew Peck, Search, Forward, L. Tech. News, Oct. 2011, at 25, 29. 2/ My article further explained my belief that Daubert would not apply to the results of using predictive coding, but that in any challenge to its use, this Judge would be interested in both the process used and the results: [I]f the use of predictive coding is challenged in a case before me, I will want to know what was done and why that produced defensible results. I may be less interested in the science behind the "black box" of the vendor's software than in whether it produced responsive documents with reasonably high recall and high precision. That may mean allowing the requesting party to see the documents that were used to train the computer-assisted coding system. (Counsel would not be required to explain why they coded documents as responsive or non-responsive, just what the coding was.) Proof of a valid "process," including quality control testing, also will be important / From a different perspective, every person who uses uses predictive coding, even if they do not realize it. The "spam filter" is an example of predictive coding. G:\AJP\DA SILVA MOORE - ESI

341 5 Of course, the best approach to the use of computer-assisted coding is to follow the Sedona Cooperation Proclamation model. Advise opposing counsel that you plan to use computer-assisted coding and seek agreement; if you cannot, consider whether to abandon predictive coding for that case or go to the court for advance approval. Id. THE ESI DISPUTES IN THIS CASE AND THEIR RESOLUTION After several discovery conferences and rulings by Judge Sullivan (the then-assigned District Judge), he referred the case to me for general pretrial supervision. (Dkt. No. 48: 11/28/11 Referral Order.) At my first discovery conference with the parties, both parties' counsel mentioned that they had been discussing an "electronic discovery protocol," and MSL's counsel stated that an open issue was "plaintiff's reluctance to utilize predictive coding to try to cull down the" approximately three million electronic documents from the agreed-upon custodians. (Dkt. No. 51: 3/ 12/2/11 Conf. Tr. at 7-8.) Plaintiffs' counsel clarified that MSL had "over simplified [plaintiffs'] stance on predictive coding," i.e., that it was not opposed but had "multiple concerns... on the way in which [MSL] plan to employ predictive coding" and plaintiffs wanted "clarification." (12/2/11 Conf. Tr. at 21.) The Court did not rule but offered the parties the following advice: Now, if you want any more advice, for better or for worse on the ESI plan and whether predictive coding should be used,... I will say right now, what should 3/ When defense counsel mentioned the disagreement about predictive coding, I stated that: "You must have thought you died and went to Heaven when this was referred to me," to which MSL's counsel responded: "Yes, your Honor. Well, I'm just thankful that, you know, we have a person familiar with the predictive coding concept." (12/2/11 Conf. Tr. at 8-9.) G:\AJP\DA SILVA MOORE - ESI

342 6 not be a surprise, I wrote an article in the October Law Technology News called Search Forward, which says predictive coding should be used in the appropriate case. Is this the appropriate case for it? You all talk about it some more. And if you can't figure it out, you are going to get back in front of me. Key words, certainly unless they are well done and tested, are not overly useful. Key words along with predictive coding and other methodology, can be very instructive. I'm also saying to the defendants who may, from the comment before, have read my article. If you do predictive coding, you are going to have to give your seed set, including the seed documents marked as nonresponsive to the plaintiff's counsel so they can say, well, of course you are not getting any [relevant] documents, you're not appropriately training the computer. (12/2/11 Conf. Tr. at ) The December 2, 2011 conference adjourned with the parties agreeing to further discuss the ESI protocol. (12/2/11 Conf. Tr. at ) The ESI issue was next discussed at a conference on January 4, (Dkt. No. 71: 1/4/12 Conf. Tr.) Plaintiffs' ESI consultant conceded that plaintiffs "have not taken issue with the use of predictive coding or, frankly, with the confidence levels that they [MSL] have proposed...." (1/4/12 Conf. Tr. at 51.) Rather, plaintiffs took issue with MSL's proposal that after the computer was fully trained and the results generated, MSL wanted to only review and produce the top 40,000 documents, which it estimated would cost $200,000 (at $5 per document). (1/4/12 Conf. Tr. at 47-48, 51.) The Court rejected MSL's 40,000 documents proposal as a "pig in a poke." (1/4/12 Conf. Tr. at ) The Court explained that "where [the] line will be drawn [as to review and production] is going to depend on what the statistics show for the results," since "[p]roportionality requires consideration of results as well as costs. And if stopping at 40,000 is going to leave a tremendous number of likely highly responsive documents unproduced, [MSL's proposed cutoff] doesn't work." (1/4/12 Conf. Tr. at 51-52; see also id. at 57-58; Dkt. No. 88: 2/8/12 Conf. Tr. at 84.) G:\AJP\DA SILVA MOORE - ESI

343 7 The parties agreed to further discuss and finalize the ESI protocol by late January 2012, with a conference held on February 8, (1/4/12 Conf. Tr. at 60-66; see 2/8/12 Conf. Tr.) Custodians The first issue regarding the ESI protocol involved the selection of which custodians' s would be searched. MSL agreed to thirty custodians for a "first phase." (Dkt. No. 88: 2/8/12 Conf. Tr. at ) MSL's custodian list included the president and other members of MSL's "executive team," most of its HR staff and a number of managing directors. (2/8/12 Conf. Tr. at 24.) Plaintiffs sought to include as additional custodians seven male "comparators," explaining that the comparators' s were needed in order to find information about their job duties and how their duties compared to plaintiffs' job duties. (2/8/12 Conf. Tr. at ) Plaintiffs gave an example of the men being given greater "client contact" or having better job assignments. (2/8/12 Conf. Tr. at ) The Court held that the search of the comparators' s would be so different from that of the other custodians that the comparators should not be included in the s subjected to predictive coding review. (2/8/12 Conf. Tr. at 28, 30.) As a fallback position, plaintiffs proposed to "treat the comparators as a separate search," but the Court found that plaintiffs could not describe in any meaningful way how they would search the comparators' s, even as a separate search. (2/8/12 Conf. Tr. at ) Since the plaintiffs likely could develop the information needed through depositions of the comparators, the Court ruled that the comparators' s would not be included in phase one. (2/8/12 Conf. Tr. at 31.) Plaintiffs also sought to include MSL's CEO, Olivier Fleuriot, located in France and whose s were mostly written in French. (2/8/12 Conf. Tr. at ) The Court concluded that G:\AJP\DA SILVA MOORE - ESI

344 8 because his s with the New York based executive staff would be gathered from those custodians, and Fleuriot's s stored in France likely would be covered by the French privacy and 4/ blocking laws, Fleuriot should not be included as a first-phase custodian. (2/8/12 Conf. Tr. at 35.) Plaintiffs sought to include certain managing directors from MSL offices at which no named plaintiff worked. (2/8/12 Conf. Tr. at ) The Court ruled that since plaintiffs had not yet moved for collective action status or class certification, until the motions were made and granted, discovery would be limited to offices (and managing directors) where the named plaintiffs had worked. (2/8/12 Conf. Tr. at ) The final issue raised by plaintiffs related to the phasing of custodians and the discovery cutoff dates. MSL proposed finishing phase-one discovery completely before considering what to do about a second phase. (See 2/8/12 Conf. Tr. at 36.) Plaintiffs expressed concern that there would not be time for two separate phases, essentially seeking to move the phase-two custodians back into phase one. (2/8/12 Conf. Tr. at ) The Court found MSL's separate phase approach to be more sensible and noted that if necessary, the Court would extend the discovery cutoff to allow the parties to pursue discovery in phases. (2/8/12 Conf. Tr. at 36, 50.) 4/ See, e.g., Societe Nationale Industrielle Aerospatiale v. U.S. Dist. Ct. for the S.D. of Iowa, 482 U.S. 522, 107 S. Ct (1987); see also The Sedona Conference, International Principles on Discovery, Disclosure & Data Protection (2011), available at G:\AJP\DA SILVA MOORE - ESI

345 9 Sources of ESI The parties agreed on certain ESI sources, including the "EMC SourceOne [ ] Archive," the "PeopleSoft" human resources information management system and certain other sources including certain HR "shared" folders. (See Dkt. No. 88: 2/8/12 Conf. Tr. at 44-45, ) As to other "shared" folders, neither side was able to explain whether the folders merely contained forms and templates or collaborative working documents; the Court therefore left those shared folders for phase two unless the parties promptly provided information about likely contents. (2/8/12 Conf. Tr. at ) The Court noted that because the named plaintiffs worked for MSL, plaintiffs should have some idea what additional ESI sources, if any, likely had relevant information; since the Court needed to consider proportionality pursuant to Rule 26(b)(2)(C), plaintiffs needed to provide more information to the Court than they were doing if they wanted to add additional data sources into phase one. (2/8/12 Conf. Tr. at ) The Court also noted that where plaintiffs were getting factual information from one source (e.g., pay information, promotions, etc.), "there has to be a limit to redundancy" to comply with Rule 26(b)(2)(C). (2/8/12 Conf. Tr. at 54.) 5/ The Predictive Coding Protocol The parties agreed to use a 95% confidence level (plus or minus two percent) to create a random sample of the entire collection; that sample of 2,399 documents will be 5/ The Court also suggested that the best way to resolve issues about what information might be found in a certain source is for MSL to show plaintiffs a sample printout from that source. (2/8/12 Conf. Tr. at ) G:\AJP\DA SILVA MOORE - ESI

346 10 reviewed to determine relevant (and not relevant) documents for a "seed set" to use to train the predictive coding software. (Dkt. No. 88: 2/8/12 Conf. Tr. at ) An area of disagreement was that MSL reviewed the 2,399 documents before the parties agreed to add two additional concept groups (i.e., issue tags). (2/8/12 Conf. Tr. at 62.) MSL suggested that since it had agreed to provide all 2,399 documents (and MSL's coding of them) to plaintiffs for their review, plaintiffs can code them for the new issue tags, and MSL will incorporate that coding into the system. (2/8/12 Conf. Tr. at 64.) Plaintiffs' vendor agreed to that approach. (2/8/12 Conf. Tr. at 64.) To further create the seed set to train the predictive coding software, MSL coded certain documents through "judgmental sampling." (2/8/12 Conf. Tr. at 64.) The remainder of the seed set was created by MSL reviewing "keyword" searches with Boolean connectors (such as "training and Da Silva Moore," or "promotion and Da Silva Moore") and coding the top fifty hits from those searches. (2/8/12 Conf. Tr. at 64-66, 72.) MSL agreed to provide all those documents (except privileged ones) to plaintiffs for plaintiffs to review MSL's relevance coding. (2/8/12 Conf. Tr. at 66.) In addition, plaintiffs provided MSL with certain other keywords, and MSL used the same process with plaintiffs' keywords as with the MSL keywords, reviewing and coding an additional 4,000 documents. (2/8/12 Conf. Tr. at 68-69, 71.) All of this review to create the seed set was done by senior attorneys (not paralegals, staff attorneys or junior associates). (2/8/12 Conf. Tr. at ) MSL reconfirmed that "[a]ll of the documents that are reviewed as a function of the seed set, whether [they] are ultimately coded relevant or irrelevant, aside from privilege, will be turned over to" plaintiffs. (2/8/12 Conf. Tr. at 73.) G:\AJP\DA SILVA MOORE - ESI

347 11 The next area of discussion was the iterative rounds to stabilize the training of the software. MSL's vendor's predictive coding software ranks documents on a score of 100 to zero, i.e., from most likely relevant to least likely relevant. (2/8/12 Conf. Tr. at 70.) MSL proposed using seven iterative rounds; in each round they would review at least 500 documents from different concept clusters to see if the computer is returning new relevant documents. (2/8/12 Conf. Tr. at ) After the seventh round, to determine if the computer is well trained and stable, MSL would review a random sample (of 2,399 documents) from the discards (i.e., documents coded as nonrelevant) to make sure the documents determined by the software to not be relevant do not, in fact, contain highly-relevant documents. (2/8/12 Conf. Tr. at ) For each of the seven rounds and the final quality-check random sample, MSL agreed that it would show plaintiffs all the documents it looked at including those deemed not relevant (except for privileged documents). (2/8/12 Conf. Tr. at 76.) Plaintiffs' vendor noted that "we don't at this point agree that this is going to work. This is new technology and it has to be proven out." (2/8/12 Conf. Tr. at 75.) Plaintiffs' vendor agreed, in general, that computer-assisted review works, and works better than most alternatives. (2/8/12 Conf. Tr. at 76.) Indeed, plaintiffs' vendor noted that "it is fair to say [that] we are big proponents of it." (2/8/12 Conf. Tr. at 76.) The Court reminded the parties that computer-assisted review "works better than most of the alternatives, if not all of the [present] alternatives. So the idea is not to make this perfect, it's not going to be perfect. The idea is to make it significantly better than the alternatives without nearly as much cost." (2/8/12 Conf. Tr. at 76.) G:\AJP\DA SILVA MOORE - ESI

348 12 following caveat: The Court accepted MSL's proposal for the seven iterative reviews, but with the But if you get to the seventh round and [plaintiffs] are saying that the computer is still doing weird things, it's not stabilized, etc., we need to do another round or two, either you will agree to that or you will both come in with the appropriate QC information and everything else and [may be ordered to] do another round or two or five or 500 or whatever it takes to stabilize the system. (2/8/12 Conf. Tr. at 76-77; see also id. at 83-84, 88.) On February 17, 2012, the parties submitted their "final" ESI Protocol which the 6/ Court "so ordered." (Dkt. No. 92: 2/17/12 ESI Protocol & Order.) Because this is the first Opinion dealing with predictive coding, the Court annexes hereto as an Exhibit the provisions of the ESI Protocol dealing with the predictive coding search methodology. OBSERVATIONS ON PLAINTIFF'S OBJECTIONS TO THE COURT'S RULINGS On February 22, 2012, plaintiffs filed objections to the Court's February 8, 2012 rulings. (Dkt. No. 93: Pls. Rule 72(a) Objections; see also Dkt. No. 94: Nurhussein Aff.; Dkt. No. 95: Neale Aff.) While those objections are before District Judge Carter, a few comments are in order. 6/ Plaintiffs included a paragraph noting its objection to the ESI Protocol, as follows: Plaintiffs object to this ESI Protocol in its entirety. Plaintiffs submitted their own proposed ESI Protocol to the Court, but it was largely rejected. The Court then ordered the parties to submit a joint ESI Protocol reflecting the Court's rulings. Accordingly, Plaintiffs jointly submit this ESI Protocol with MSL, but reserve the right to object to its use in this case. (ESI Protocol J.1 at p. 22.) G:\AJP\DA SILVA MOORE - ESI

349 13 Plaintiffs' Reliance on Rule 26(g)(1)(A) is Erroneous Plaintiffs' objections to my February 8, 2012 rulings assert that my acceptance of MSL's predictive coding approach "provides unlawful 'cover' for MSL's counsel, who has a duty under FRCP 26(g) to 'certify' that their client's document production is 'complete' and 'correct' as of the time it was made. FRCP 26(g)(1)(A)." (Dkt. No. 93: Pls. Rule 72(a) Objections at 8 n.7; accord, id. at 2.) In large-data cases like this, involving over three million s, no lawyer using any search method could honestly certify that its production is "complete" but more importantly, Rule 26(g)(1) does not require that. Plaintiffs simply misread Rule 26(g)(1). The certification required by Rule 26(g)(1) applies "with respect to a disclosure." Fed. R. Civ. P. 26(g)(1)(A) (emphasis added). That is a term of art, referring to the mandatory initial disclosures required by Rule 26(a)(1). Since the Rule 26(a)(1) disclosure is information (witnesses, exhibits) that "the disclosing party may use to support its claims or defenses," and failure to provide such information leads to virtually automatic preclusion, see Fed. R Civ. P. 37(c)(1), it is appropriate for the Rule 26(g)(1)(A) certification to require disclosures be "complete and correct." Rule 26(g)(1)(B) is the provision that applies to discovery responses. It does not call for certification that the discovery response is "complete," but rather incorporates the Rule 26(b)(2)(C) proportionality principle. Thus, Rule 26(g)(1)(A) has absolutely nothing to do with MSL's obligations to respond to plaintiffs' discovery requests. Plaintiffs' argument is based on a misunderstanding of Rule 26(g)(1). 7/ 7/ Rule 26(g)(1) provides: (continued...) G:\AJP\DA SILVA MOORE - ESI

350 14 Rule 702 and Daubert Are Not Applicable to Discovery Search Methods Plaintiffs' objections also argue that my acceptance of MSL's predictive coding protocol "is contrary to Federal Rule of Evidence 702" and "violates the gatekeeping function underlying Rule 702." (Dkt. No. 93: Pls. Rule 72(a) Objections at 2-3; accord, id. at ) 8/ 7/ (...continued) (g) Signing Disclosures and Discovery Requests, Responses, and Objections. (1) Signature Required; Effect of Signature. Every disclosure under Rule 26(a)(1) or (a)(3) and every discovery request, response, or objection must be signed by at least one attorney of record in the attorney's own name.... By signing, an attorney or party certifies that to the best of the person's knowledge, information, and belief formed after a reasonable inquiry: (A) with respect to a disclosure, it is complete and correct as of the time it is made; and (B) with respect to a discovery request, response, or objection, it is: Fed. R. Civ. P. 26(g)(1) (emphasis added). (i) consistent with these rules and warranted by existing law or by a nonfrivolous argument for extending, modifying, or reversing existing law, or for establishing new law; (ii) not interposed for any improper purpose, such as to harass, cause unnecessary delay, or needlessly increase the cost of litigation; and (iii) neither unreasonable nor unduly burdensome or expensive, considering the needs of the case, prior discovery in the case, the amount in controversy, and the importance of the issues at stake in the action. 8/ As part of this argument, plaintiffs complain that although both parties' experts (i.e., vendors) spoke at the discovery conferences, they were not sworn in. (Pls. Rule 72(a) Objections at 12: "To his credit, the Magistrate [Judge] did ask the parties to bring [to the (continued...) G:\AJP\DA SILVA MOORE - ESI

351 15 9/ Federal Rule of Evidence 702 and the Supreme Court's Daubert decision deal with the trial court's role as gatekeeper to exclude unreliable expert testimony from being submitted to the jury at trial. See also Advisory Comm. Notes to Fed. R. Evid It is a rule for admissibility of evidence at trial. If MSL sought to have its expert testify at trial and introduce the results of its ESI protocol into evidence, Daubert and Rule 702 would apply. Here, in contrast, the tens of thousands of s that will be produced in discovery are not being offered into evidence at trial as the result of a scientific process or otherwise. The admissibility of specific s at trial will depend upon each itself (for example, whether it is hearsay, or a business record or party admission), not how it was found during discovery. and found in discovery. Rule 702 and Daubert simply are not applicable to how documents are searched for Plaintiffs' Reliability Concerns Are, At Best, Premature Finally, plaintiffs' objections assert that "MSL's method lacks the necessary standards for assessing whether its results are accurate; in other words, there is no way to be certain if MSL's method is reliable." (Dkt. No. 93: Pls. Rule 72(a) Objections at ) Plaintiffs' concerns may be appropriate for resolution during or after the process (which the Court will be closely 8/ 9/ (...continued) conference] the ESI experts they had hired to advise them regarding the creation of an ESI protocol. These experts, however, were never sworn in, and thus the statements they made in court at the hearings were not sworn testimony made under penalty of perjury.") Plaintiffs never asked the Court to have the experts testify to their qualifications or be sworn in. Daubert v. Merrell Dow Pharms., Inc., 509 U.S. 579, 113 S. Ct (1993). G:\AJP\DA SILVA MOORE - ESI

352 16 supervising), but are premature now. For example, plaintiffs complain that "MSL's method fails to include an agreed-upon standard of relevance that is transparent and accessible to all parties.... Without this standard, there is a high-likelihood of delay as the parties resolve disputes with regard to individual documents on a case-by-case basis." (Id. at 14.) Relevance is determined by plaintiffs' document demands. As statistics show, perhaps only 5% of the disagreement among reviewers comes from close questions of relevance, as opposed to reviewer error. (See page 18 n.11 below.) The issue regarding relevance standards might be significant if MSL's proposal was not totally transparent. Here, however, plaintiffs will see how MSL has coded every used in the seed set (both relevant and not relevant), and the Court is available to quickly resolve any issues. Plaintiffs complain they cannot determine if "MSL's method actually works" because MSL does not describe how many relevant documents are permitted to be located in the final random sample of documents the software deemed irrelevant. (Pls. Rule 72(a) Objections at ) Plaintiffs argue that "without any decision about this made in advance, the Court is simply kicking the can down the road." (Id. at 16.) In order to determine proportionality, it is necessary to have more information than the parties (or the Court) now has, including how many relevant documents will be produced and at what cost to MSL. Will the case remain limited to the named plaintiffs, or will plaintiffs seek and obtain collective action and/or class action certification? In the final sample of documents deemed irrelevant, are any relevant documents found that are "hot," "smoking gun" documents (i.e., highly relevant)? Or are the only relevant documents more of the same thing? One hot document may require the software to be re-trained (or some other search method employed), while several documents that really do not add anything to the case might not matter. These types G:\AJP\DA SILVA MOORE - ESI

353 17 of questions are better decided "down the road," when real information is available to the parties and the Court. FURTHER ANALYSIS AND LESSONS FOR THE FUTURE The decision to allow computer-assisted review in this case was relatively easy the parties agreed to its use (although disagreed about how best to implement such review). The Court recognizes that computer-assisted review is not a magic, Staples-Easy-Button, solution appropriate for all cases. The technology exists and should be used where appropriate, but it is not a case of machine replacing humans: it is the process used and the interaction of man and machine that the courts needs to examine. The objective of review in ediscovery is to identify as many relevant documents as possible, while reviewing as few non-relevant documents as possible. Recall is the fraction of relevant documents identified during a review; precision is the fraction of identified documents that are relevant. Thus, recall is a measure of completeness, while precision is a measure of accuracy or correctness. The goal is for the review method to result in higher recall and higher precision than another review method, at a cost proportionate to the "value" of the case. See, e.g., Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, Rich. J.L.& Tech., Spring 2011, at 8-9, available at G:\AJP\DA SILVA MOORE - ESI

354 18 The slightly more difficult case would be where the producing party wants to use 10/ computer-assisted review and the requesting party objects. The question to ask in that situation is what methodology would the requesting party suggest instead? Linear manual review is simply too expensive where, as here, there are over three million s to review. Moreover, while some lawyers still consider manual review to be the "gold standard," that is a myth, as statistics clearly show that computerized searches are at least as accurate, if not more so, than manual review. Herb Roitblatt, Anne Kershaw, and Patrick Oot of the Electronic Discovery Institute conducted an empirical assessment to "answer the question of whether there was a benefit to engaging in a traditional human review or whether computer systems could be relied on to produce comparable results," and concluded that "[o]n every measure, the performance of the two computer systems was at least as accurate (measured against the original review) as that of human re-review." Herbert L. Roitblatt, Anne Kershaw & Patrick Oot, Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, 61 J. Am. Soc'y for Info. Sci. & Tech. 70, 79 (2010). 11/ 10/ 11/ The tougher question, raised in Klein Prods. LLC v. Packaging Corp. of Am. before Magistrate Judge Nan Nolan in Chicago, is whether the Court, at plaintiffs' request, should order the defendant to use computer-assisted review to respond to plaintiffs' document requests. The Roitblatt, Kershaw, Oot article noted that "[t]he level of agreement among human reviewers is not strikingly high," around 70-75%. They identify two sources for this variability: fatigue ("A document that they [the reviewers] might have categorized as responsive when they were more attentive might then be categorized [when the reviewer is distracted or fatigued] as non-responsive or vice versa."), and differences in "strategic judgment." Id. at Another study found that responsiveness "is fairly well defined, and that disagreements among assessors are largely attributable to human error," with only 5% of reviewer disagreement attributable to borderline or questionable issues as to relevance. Maura R. Grossman & Gordon V. Cormack, Inconsistent Assessment of (continued...) G:\AJP\DA SILVA MOORE - ESI

355 19 Likewise, Wachtell, Lipton, Rosen & Katz litigation counsel Maura Grossman and University of Waterloo professor Gordon Cormack, studied data from the Text Retrieval Conference Legal Track (TREC) and concluded that: "[T]he myth that exhaustive manual review is the most effective and therefore the most defensible approach to document review is strongly refuted. Technology-assisted review can (and does) yield more accurate results than exhaustive manual review, with much lower effort." Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual 12/ Review, Rich. J.L.& Tech., Spring 2011, at 48. The technology-assisted reviews in the Grossman- Cormack article also demonstrated significant cost savings over manual review: "The technologyassisted reviews require, on average, human review of only 1.9% of the documents, a fifty-fold savings over exhaustive manual review." Id. at 43. Because of the volume of ESI, lawyers frequently have turned to keyword searches to cull (or other ESI) down to a more manageable volume for further manual review. Keywords have a place in production of ESI indeed, the parties here used keyword searches (with Boolean connectors) to find documents for the expanded seed set to train the predictive coding software. In too many cases, however, the way lawyers choose keywords is the equivalent of the 11/ 12/ (...continued) Responsiveness in E-Discovery: Difference of Opinion or Human Error? 9 (DESI IV: 2011 ICAIL Workshop on Setting Standards for Searching Elec. Stored Info. in Discovery, Research Paper), available at grossman3.pdf. Grossman and Cormack also note that "not all technology-assisted reviews... are created equal" and that future studies will be needed to "address which technology-assisted review process(es) will improve most on manual review." Id. G:\AJP\DA SILVA MOORE - ESI

356 20 13/ child's game of "Go Fish." The requesting party guesses which keywords might produce evidence to support its case without having much, if any, knowledge of the responding party's "cards" (i.e., the terminology used by the responding party's custodians). Indeed, the responding party's counsel often does not know what is in its own client's "cards." Another problem with keywords is that they often are over-inclusive, that is, they find responsive documents but also large numbers of irrelevant documents. In this case, for example, a keyword search for "training" resulted in 165,208 hits; Da Silva Moore's name resulted in 201,179 hits; "bonus" resulted in 40,756 hits; "compensation" resulted in 55,602 hits; and "diversity" resulted in 38,315 hits. (Dkt. No. 92: 2/17/12 ESI Protocol Ex. A.) If MSL had to manually review all of the keyword hits, many of which would not be relevant (i.e., would be false positives), it would be quite costly. Moreover, keyword searches usually are not very effective. In 1985, scholars David Blair and M. Maron collected 40,000 documents from a Bay Area Rapid Transit accident, and instructed experienced attorney and paralegal searchers to use keywords and other review techniques to retrieve at least 75% of the documents relevant to 51 document requests. David L. Blair & M. E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System, 28 Comm. ACM 289 (1985). Searchers believed they met the goals, but their average recall was just 20%. Id. This result has been replicated in the TREC Legal Track studies over the past few years. 13/ See Ralph C. Losey, "Child's Game of 'Go Fish' is a Poor Model for e-discovery Search," in Adventures in Electronic Discovery (2011). G:\AJP\DA SILVA MOORE - ESI

357 21 Judicial decisions have criticized specific keyword searches. Important early decisions in this area came from two of the leading judicial scholars in ediscovery, Magistrate Judges John Facciola (District of Columbia) and Paul Grimm (Maryland). See United States v. O'Keefe, 37 F. Supp. 2d 14, 24 (D.D.C. 2008) (Facciola, M.J.); Equity Analytics, LLC v. Lundin, 248 F.R.D. 331, 333 (D.D.C. 2008) (Facciola, M.J.); Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 260, 262 (D. Md. 2008) (Grimm, M.J.). I followed their lead with Willaim A. Gross Construction Associates, Inc., when I wrote: This Opinion should serve as a wake-up call to the Bar in this District about the need for careful thought, quality control, testing, and cooperation with opposing counsel in designing search terms or "keywords" to be used to produce s or other electronically stored information ("ESI")..... Electronic discovery requires cooperation between opposing counsel and transparency in all aspects of preservation and production of ESI. Moreover, where counsel are using keyword searches for retrieval of ESI, they at a minimum must carefully craft the appropriate keywords, with input from the ESI's custodians as to the words and abbreviations they use, and the proposed methodology must be quality control tested to assure accuracy in retrieval and elimination of "false positives." It is time that the Bar even those lawyers who did not come of age in the computer era understand this. William A. Gross Constr. Assocs., Inc. v. Am. Mfrs. Mut. Ins. Co., 256 F.R.D. 134, 134, 136 (S.D.N.Y. 2009) (Peck, M.J.). Computer-assisted review appears to be better than the available alternatives, and thus should be used in appropriate cases. While this Court recognizes that computer-assisted review is not perfect, the Federal Rules of Civil Procedure do not require perfection. See, e.g., Pension Comm. of Univ. of Montreal Pension Plan v. Banc of Am. Sec., 685 F. Supp. 2d 456, 461 (S.D.N.Y. G:\AJP\DA SILVA MOORE - ESI

358 ). Courts and litigants must be cognizant of the aim of Rule 1, to "secure the just, speedy, and inexpensive determination" of lawsuits. Fed. R. Civ. P. 1. That goal is further reinforced by the proportionality doctrine set forth in Rule 26(b)(2)(C), which provides that: Fed. R. Civ. P. 26(b)(2)(C). On motion or on its own, the court must limit the frequency or extent of discovery otherwise allowed by these rules or by local rule if it determines that: (i) the discovery sought is unreasonably cumulative or duplicative, or can be obtained from some other source that is more convenient, less burdensome, or less expensive; (ii) the party seeking discovery has had ample opportunity to obtain the information by discovery in the action; or (iii) the burden or expense of the proposed discovery outweighs its likely benefit, considering the needs of the case, the amount in controversy, the parties' resources, the importance of the issues at stake in the action, and the importance of the discovery in resolving the issues. In this case, the Court determined that the use of predictive coding was appropriate considering: (1) the parties' agreement, (2) the vast amount of ESI to be reviewed (over three million documents), (3) the superiority of computer-assisted review to the available alternatives (i.e., linear manual review or keyword searches), (4) the need for cost effectiveness and proportionality under Rule 26(b)(2)(C), and (5) the transparent process proposed by MSL. This Court was one of the early signatories to The Sedona Conference Cooperation Proclamation, and has stated that "the best solution in the entire area of electronic discovery is cooperation among counsel. This Court strongly endorses The Sedona Conference Proclamation (available at William A. Gross Constr. Assocs., Inc. v. Am. G:\AJP\DA SILVA MOORE - ESI

359 23 Mfrs. Mut. Ins. Co., 256 F.R.D. at 136. An important aspect of cooperation is transparency in the discovery process. MSL's transparency in its proposed ESI search protocol made it easier for the Court to approve the use of predictive coding. As discussed above on page 10, MSL confirmed that "[a]ll of the documents that are reviewed as a function of the seed set, whether [they] are ultimately coded relevant or irrelevant, aside from privilege, will be turned over to" plaintiffs. (Dkt. No. 88: 2/8/12 Conf. Tr. at 73; see also 2/17/12 ESI Protocol at 14: "MSL will provide Plaintiffs' counsel with all of the non-privileged documents and will provide, to the extent applicable, the issue tag(s) coded for each document.... If necessary, counsel will meet and confer to attempt to resolve any disagreements regarding the coding applied to the documents in the seed set.") While not all experienced ESI counsel believe it necessary to be as transparent as MSL was willing to be, such transparency allows the opposing counsel (and the Court) to be more comfortable with computer- 14/ assisted review, reducing fears about the so-called "black box" of the technology. This Court highly recommends that counsel in future cases be willing to at least discuss, if not agree to, such transparency in the computer-assisted review process. Several other lessons for the future can be derived from the Court's resolution of the ESI discovery disputes in this case. First, it is unlikely that courts will be able to determine or approve a party's proposal as to when review and production can stop until the computer-assisted review software has been trained and the results are quality control verified. Only at that point can the parties and the Court 14/ It also avoids the GIGO problem, i.e., garbage in, garbage out. G:\AJP\DA SILVA MOORE - ESI

360 24 see where there is a clear drop off from highly relevant to marginally relevant to not likely to be relevant documents. While cost is a factor under Rule 26(b)(2)(C), it cannot be considered in isolation from the results of the predictive coding process and the amount at issue in the litigation. Second, staging of discovery by starting with the most likely to be relevant sources (including custodians), without prejudice to the requesting party seeking more after conclusion of that first stage review, is a way to control discovery costs. If staging requires a longer discovery period, most judges should be willing to grant such an extension. (This Judge runs a self-proclaimed "rocket docket," but informed the parties here of the Court's willingness to extend the discovery cutoff if necessary to allow the staging of custodians and other ESI sources.) Third, in many cases requesting counsel's client has knowledge of the producing party's records, either because of an employment relationship as here or because of other dealings between the parties (e.g., contractual or other business relationships). It is surprising that in many cases counsel do not appear to have sought and utilized their client's knowledge about the opposing party's custodians and document sources. Similarly, counsel for the producing party often is not sufficiently knowledgeable about their own client's custodians and business terminology. Another way to phrase cooperation is "strategic proactive disclosure of information," i.e., if you are knowledgeable about and tell the other side who your key custodians are and how you propose to search for the requested documents, opposing counsel and the Court are more apt to agree to your approach (at least as phase one without prejudice). Fourth, the Court found it very helpful that the parties' ediscovery vendors were present and spoke at the court hearings where the ESI Protocol was discussed. (At ediscovery G:\AJP\DA SILVA MOORE - ESI

361 25 programs, this is sometimes jokingly referred to as "bring your geek to court day.") Even where as here counsel is very familiar with ESI issues, it is very helpful to have the parties' ediscovery vendors (or in-house IT personnel or in-house ediscovery counsel) present at court conferences where ESI issues are being discussed. It also is important for the vendors and/or knowledgeable counsel to be able to explain complicated ediscovery concepts in ways that make it easily understandable to judges who may not be tech-savvy. CONCLUSION This Opinion appears to be the first in which a Court has approved of the use of computer-assisted review. That does not mean computer-assisted review must be used in all cases, or that the exact ESI protocol approved here will be appropriate in all future cases that utilize computer-assisted review. Nor does this Opinion endorse any vendor (the Court was very careful not to mention the names of the parties' vendors in the body of this Opinion, although it is revealed in the attached ESI Protocol), nor any particular computer-assisted review tool. What the Bar should take away from this Opinion is that computer-assisted review is an available tool and should be seriously considered for use in large-data-volume cases where it may save the producing party (or both parties) significant amounts of legal fees in document review. Counsel no longer have to worry about being the "first" or "guinea pig" for judicial acceptance of computer-assisted review. As with keywords or any other technological solution to ediscovery, counsel must design an appropriate process, including use of available technology, with appropriate quality control testing, to review G:\AJP\DA SILVA MOORE - ESI

362 26 and produce relevant ESI while adhering to Rule 1 and Rule 26(b )(2)(C) proportionality. Computerassisted review now can be considered judicially-approved for use in appropriate cases. SO ORDERED. Dated: New York, New York February 24,2012 Andrew J. Peck United States Magistrate Judge Copies by ECF to: All Counsel Judge Andrew L. Carter, Jr. G:WP\DA SILVA MOORE ESI

363 EXHIBIT

364 UNITED STATES DISTRICT COURT SOUTHERN DISTRICT OF NEW YORK MONIQUE DA SILVA MOORE, MARYELLEN O'DONOHUE, LAURIE MAYERS, HEATHER PIERCE, and K.ATHERINE WILKINSON, on behalf of themselves and all others similarly situated, vs. Plaintiffs, PUBLICIS GROUPE SA and MSLGROUP, Defendants. Case No. 11-cv-1279 (ALC) (AJP) PARTIES' PROPOSED PROTOCOL RELATING TO THE PRODUCTION OF ELECTRONICALLY STORED INFORMATION ("ESI") cf'~--r. A. Scope 1. General. The procedures and protocols outlined herein govern the production of electronically stored information ("ESI") by MSLGROUP Americas, Inc. ("MSL") during the pendency ofthis litigation. The parties to this protocol will take reasonable steps to comply with this agreed-upon protocol for the production of documents and information existing in electronic format. Nothing in this protocol will be interpreted to require disclosure of documents or information protected from disclosure by the attorney-client privilege, work-product product doctrine or any other applicable privilege or immunity. It is Plaintiffs' position that nothing in this protocol will be interpreted to waive Plaintiffs' right to object to this protocol as portions of it were mandated by the Court over Plaintiffs' objections, including Plaintiffs' objections to the predictive coding methodology proposed by MSL. 2. Limitations and No-Waiver. This protocol provides a general framework for the production of ESI on a going forward basis. The Parties and their attorneys do not intend by this protocol to waive their rights to the attorney work-product privilege, except as specifically required herein, and any such waiver shall be strictly and narrowly construed and shall not

365 extend to other matters or information not specifically described herein. All Parties preserve their attorney client privileges and other privileges and there is no intent by the protocol, or the production of documents pursuant to the protocol, to in any way waive or weaken these privileges. All documents produced hereunder are fully protected and covered by the Parties' confidentiality and clawback agreements and orders ofthe Court effectuating same. 3. Relevant Time Period. January 1, 2008 through February 24, 2011 for all non ESI relating to topics besides pay discrimination and for all s. January 1, 2005 through February 24, 2011 for all non- ESI relating to pay discrimination for New York Plaintiffs. B. ESI Preservation 1. MSL has issued litigation notices to designated employees on February 10, 2010, March 14,2011 and June 9, C. Sources 1. The Parties have identified the following sources of potentially discoverable ESI at MSL. Phase I sources will be addressed first, and Phase II sources will be addressed after Phase I source searches are complete. Sources marked as "NIA" will not be searched by the Parties. Active corporate system that provides mail communication and functions. c GroupWise E- Legacy corporate system that provided mail communication and functions. d IBM Sametime Lotus Notes Instant Messaging and collaboration N/A N/A N/A Page 2

366 dedicated to individual users. (With the exception of2 horne directories for which MSL will collect and analyze the data to determine the level ofduplication as compared to the EMC SourceOne Archive. The parties will meet and confer regarding the selection of the two I custodians. ) f Shared Folders Shared network storage locations on the file server(s) II I that are accessible by individual users, groups ofusers or entire departments. (With the exception ofthe following Human Resources shared folders which will be in Phase I: Corporate HR, North America HR and New York HR.)! g i Database Servers Backend databases (e.g. Oracle, SQL, MySQL) used to N/A store information for front end applications or other purposes. h Halogen Software Performance management program provided by I Halogen to conduct performance evaluations. I 1 I Noovoo Corporate Intranet site.. II j Corporate addresses that employees may utilize to provide I Feedback the company with comments, suggestions and overall feedback. k Hyperion Oracle application that offers global financial N/A Financial consolidation, reporting and analysis. Management ("HFM") I VurvlTaleo Talent recruitment software. II m ServiceNow Help Desk application used to track employee computer N/A related requests. n PeopleSoft Human resources information management system. I 0 PRISM PeopleSoft component used for time and billing I management. p Portal A project based portal provided through Oracle/BEA II Systems. q Desktops/Laptops Fixed and portable computers provided to employees to II perform work related activities. (With the exception of 2 desktop/laptop hard drives for which MSL will collect and analyze the data to determine the level of duplication as compared to the EMC SourceOne Archive. The parties will meet and confer regarding the selection ofthe two custodians.) r Publicis Benefits Web based site that maintains information about II Connection employee benefits and related information. s GEARS Employee expense reporting system. II t MS&LCity Former corporate Intranet. N/A RAdium Appliclltion which aggregates instant messages. N/A Pidgin Applicationyvhich aggregates instant message. N/A IBM Lotus Mobile device synchronization and security system. N/A I Page 3

367 i Traveler and Mobilelron i y Mobile Communication I i Devices z Yammer aa SalesForce.com bb Removable Storage Devices Portable PDAs, smart phones, tablets used for communication. Social media and collaboration Eortal. Web-based customer relationship management application. Portable storage media, external hard drives, thumb drives, etc. used to sto~e copies ofwork related ESI. N/A N/A N/A N/A i a. EMC SourceOne - MSL uses SourceOne, an EMC archiving system that captures and stores all messages that pass through the corporate system. In addition, if a user chooses to save an instant messaging chat conversation from IBM Sametime (referenced below), that too would be archived in SourceOne. Defendant MSL also acknowledges that calendar items are regularly ingested into the SourceOne system. The parties have agreed that this data source will be handled as outlined in section E below. b. Lotus Notes - MSL currently maintains multiple Lotus Notes Domino servers in various data centers around the world. All communication and calendar items are joumaled in real time to the EMC SourceOne archive. The parties have agreed to not collect any information from this data source at this time. c. GroupWise Prior to the implementation of the Lotus Notes environment, GroupWise was used for all and calendar functionality. Before the decommissioning of the Group Wise servers, MSL created backup tapes of all servers that housed the GroupWise databases. The parties have agreed to not collect any information from this data source at this time. d. IBM Sametime MSL provides custodians with the ability to have real time chat conversations via the IBM Sametime application that is part ofthe Lotus Notes suite of products. Page 4

368 e. Home Directories Custodians with corporate network access at MSL also have a dedicated and secured network storage location where they are able to save files. MSL will collect the home directory data for 2 custodians and analyze the data to determine the level of duplication of documents in this data source against the data contained in the EMC SourceOne archive for the same custodians. (The parties will meet and confer regarding the selection of the two custodians.) The results of the analysis will be provided to Plaintiffs so that a determination can be made by the parties as to whether MSL will include this data source in its production of ESI to Plaintiffs. If so, the parties will attempt to reach an agreement as to the approach used to collect, review and produce responsive and non-privileged documents. f. Shared Folders - Individual employees, groups of employees and entire departments at MSL are given access to shared network storage locations to save and share files. As it relates to the Human Resources related shared folders (i.e., North America HR Drive (10.2 OB), Corporate HR Drive (440 MB), NY HR Drive (1.9 OB), Chicago HR Drive (1.16 OB), Boston HR Drive (43.3 MB), and Atlanta HR Drive (6.64 OB)), MSL will judgment ally review and produce responsive and non-privileged documents from the North America HR Drive, Corporate HR Drive, and NY HR Drive. MSL will produce to Plaintiffs general information regarding the content of other Shared Folders. The parties will meet and confer regarding the information gathered concerning the other Shared Folders and discuss whether any additional Shared Folders should be moved to Phase I. g. Database Servers -MSL has indicated that it does not utilize any database servers, other than those that pertain to the sources outlined above in C, which are likely to contain information relevant to Plaintiffs' claims. h. Halogen Software MSL utilizes a third party product, Halogen, for performance management and employee evaluations. The parties will meet and confer in order to Page 5

369 exchange additional information and attempt to reach an agreement as to the scope of data and the approach used to collect, review and produce responsive and non-privileged documents. 1. Noovoo - MSL maintains a corporate Intranet site called "Noovoo" where employees are able to access Company-related information. MSL will provide Plaintiffs with any employment-related policies maintained within Noovoo. J. Corporate Feedback - MSL has maintained various addresses that employees may utilize to provide the company with comments, suggestions and overall feedback. These and The parties have agreed that all responsive and nonprivileged ESI will be produced from these accounts and any other accounts that fall under this category of information. At present, MSL intends to manually review the contents of each of these accounts. However, if after collecting the contents of each of the accounts MSL determines that a manual review would be impractical, the parties will meet and confer as to the approach used to collect, review and produce responsive and non-privileged documents. k. Hyperion Financial Management ("HFM") MS L uses an Oracle application called HFM that offers global financial consolidation, reporting and analysis capabilities. 1. Vurv/Taleo - Since approximately 2006, MSL used an application known as Vurv as its talent recruitment software. As of August 31, 2011, as a result of Vurv being purchased by Taleo, MSL has been using a similar application by Taleo as its talent recruitment software. The application, which is accessed through MSL's public website, allows users to search for open positions as well as input information about themselves. To the extent Plaintiffs Page 6

370 contend they were denied any specific positions, they will identify same and the Parties will meet and confer to discuss what, if any, information exists within Vurv/Taleo regarding the identified position. If information exists in Vurv/Taleo or another source regarding these positions, MSL will produce this information, to the extent such information is discoverable. m. ServiceNow MSL utilizes ServiceNow as its Help Desk application. This system covers a wide variety ofrequests by employees for computer-related assistance (e.g., troubleshoot incidents, install software, etc.). n. PeopleSoft - MSL utilizes PeopleSoft, an Oracle-based software product, to record employee data such as date of hire, date of termination, promotions, salary increases, transfers, etc. MSL has produced data from this source and will consider producing additional data in response to a specific inquiry from Plaintiffs. o. PRISM MSL utilizes PRISM for tracking time and billing. It is used primarily to track an employee's billable time. MSL will consider producing additional data in response to a specific inquiry from Plaintiffs. p. Portal - MSL maintains a portal provided through Oracle/BEA Systems. The portal is web-based and is used for light workflow activities (such as reviewing draft documents). q. Desktops/Laptops MSL provided employees with desktop and/or laptop computers to assist in work related activities. MSL will collect the desktop/laptop hard drive data for 2 custodians and analyze the data to determine the level of duplication of documents in this data source against the data contained in the EMC SourceOne archive for the same custodians. (The parties will meet and confer regarding the selection of the two custodians.) The results of the analysis will be provided to Plaintiffs so that a determination can be made by the parties as to whether MSL will include this data source in its production of ESI to Plaintiffs. Page 7

371 If so, the Parties will attempt to reach an agreement as to the approach used to collect, review and produce responsive and non-privileged documents. r. Publicis Benefits Connection Plaintiffs understand that MSL provides employees with access to a centralized web based site that provides access to corporate benefits information and other related content. s. GEARS - MSL maintains a centralized web-based expense tracking and reporting system called "GEARS" where users are able to enter expenses and generate reports. 1. MS&L City - MSL maintained a corporate web-based Intranet prior to migrating to Noovoo. u. Adium - This is a free and open source instant messaging client for Mac OS X users. v. Pidgin - Pidgin is a chat program which lets users log into accounts on multiple chat networks simultaneously. However, the data resides with a third party messaging provider (e.g. AIM, Yahoo!, Google Talk, MSN Messenger, etc.). w. IBM Lotus Traveler and Mobilelron MSL maintains these systems for e mail device sync and security features for employees' mobile devices, including Blackberry devices, iphones, ipads, Android phones, and Android tablets. x. Mobile Communication Devices - MSL provides mobile devices and/or connectivity including Blackberry devices, iphones, ipads, Android phones, and Android tablets to designated employees. y. Yammer This is an instant messaging application hosted externally, used for approximately one year in or around 2008 through z. SalesForce.com This is a web-based customer relationship management application but it was not widely used. Page 8

372 aa. Removable Storage Devices - MSL does not restrict authorized employees from using removable storage devices. D. Custodians 1. The Parties agree that MSL will search the accounts of the following individuals as they exist on MSL's EMC SourceOne archive. (Except where a date range is noted, the custodian's entire account was collected from the archive.) Page 9

373 30. I Hass, Mark CEO (former) 31. I Morsman, Michael Managing Director, Ann Arbor (former) E. Search Methodologyl 1. General. The Parties have discussed the methodologies or protocols for the search and review of ESI collected from the EMC SourceOne archive and the following is a summary of the Parties' agreement on the use of Predictive Coding. This section relates solely to the EMC SourceOne data source (hereinafter referred to as the.. collection"). 2. General Overview of Predictive Coding Process. MSL will utilize the Axcelerate software by Recommind to search and review the collection for production in this case. The process begins with Jackson Lewis attorneys developing an understanding of the entire collection while identifying a small number of documents, the initial seed set, that is representative of the categories to be reviewed and coded (relevance, privilege, issue-relation). It is the step when the first seed sets are generated which is done by use of search and analytical tools, including keyword, Boolean and concept search, concept grouping, and, as needed, up to 40 other automatically populated filters available within the Axcelerate system. This assists in the attorneys' identification ofprobative documents for each category to be reviewed and coded. Plaintiffs' counsel will be provided with preliminary results of MSL's hit counts using keyword searches to create a high priority relevant seed set, and will be invited to contribute their own proposed keywords. Thereafter, Plaintiffs' counsel will be provided with the nonprivileged keyword hits - both from MSL's keyword list and Plaintiffs' keyword list which were reviewed and coded by MSL. Plaintiffs' counsel will review the documents produced and promptly provide defense counsel with their own evaluation of the initial coding applied to the documents, including identification of any documents it believes were incorrectly coded. To the 1 As noted in Paragraphs A(l) and J ofthis Protocol, Plaintiffs object to the predictive coding methodology proposed by MSL. Page 10

374 extent the parties disagree regarding the coding of a particular document, they will meet and confer in an effort to resolve the dispute prior to contacting the Court for resolution. The irrelevant documents so produced shall be promptly returned after review and analysis by Plaintiffs' counsel and/or resolution of any disputes by the Court. The seed sets are then used to begin the Predictive Coding process. Each seed set of documents is applied to its relevant category and starts the software ''training'' process. The software uses each seed set to identify and prioritize all substantively similar documents over the complete corpus of the collection. The attorneys then review and code a judgmental sample of at least 500 of the "computer suggested" documents to ensure their proper categorization and to further calibrate the system by recoding documents into their proper categories. Axcelerate learns from the new corrected coding and the Predictive Coding process is repeated. Attorneys representing MSL will have access to the entire collection to be searched and wi11lead the computer training, but they will obtain input from Plaintiffs' counsel during the iterative seed selection and quality control processes and will share the information used to craft the search protocol as further described herein. All non-privileged documents reviewed by MSL during each round of the iterative process (i.e., both documents coded as relevant and irrelevant) will be produced to Plaintiffs' counsel during the iterative seed set selection process. Plaintiffs' counsel will review the documents produced and promptly provide defense counsel with its own evaluation of the initial coding applied to the documents, including identification of any documents it believes were incorrectly coded. To the extent the Parties disagree regarding the coding of a particular document, they will meet and confer in an effort to resolve the dispute prior to contacting the Court for resolution. Again, the irrelevant documents Page 11

375 so produced shall be promptly returned after review and analysis by Plaintiffs' counsel andlor resolution of any disputes by the Court. At the conclusion of the iterative review process, all document predicted by Axcelerate to be relevant will be manually reviewed for production. However, depending on the number of documents returned, the relevancy rating of those documents, and the costs incurred during the development of the seed set and iterative reviews, MSL reserves the right to seek appropriate relief from the Court prior to commencing the final manual review. The accuracy of the search processes, both the systems' functions and the attorney judgments to train the computer, will be tested and quality controlled by both judgmental and statistical sampling. In statistical sampling, a small set of documents is randomly selected from the total corpus of the documents to be tested. The small set is then reviewed and an error rate calculated therefrom. The error rates can then be reliably projected on the total corpus, having a margin of error directly related to the sample size. 3. Issue Tags. The parties agree that, to the extent applicable, as part of the seed set training described above, as well as during the iterative review process, all documents categorized as relevant and not privileged, to the extent applicable, also shall be coded with one or more of the following agreed-upon issue tags: a. Reorganization. b. Promotion! Assignments. c. Work/Life Balance. d. Termination. e. Compensation. f. MatemitylPregnancy. g. ComplaintslHR. Page 12

376 h. Publicis Groupe/Jurisdiction. This issue coding will take place during the initial random sample, creation of the seed set and initial and iterative training (see paragraphs 4, 5 and 6 below). This input shall be provided to Plaintiffs' counsel along with the initial document productions. Plaintiffs' counsel shall promptly report any disagreements on classification, and the parties shall discuss these issues in good faith, so that the seed set training may be improved accordingly. This issue-tagging and disclosure shall take place during the described collaborative seed set training process. The disclosures here made by MSL on its issue coding are not required in the final production set. 4. Initial Random Sample. Using the Axcelerate software to generate a random sample ofthe entire corpus of documents uploaded to the Axcelerate search and review platform, MSL's attorneys will conduct a review of the random sample for relevance and to develop a baseline for calculating recall and precision. To the extent applicable, any relevant documents also will be coded with one or more of the issue tags referenced in paragraph E.3 above. The random sample consists of 2,399 documents, which represents a 95% confidence level with a confidence estimation of plus or minus 2%. The Parties agree to utilize the random sample generated prior to the finalization of this protocol. However, during Plaintiffs' counsel's review of the random sample, they may advise as to whether they believe any of the documents should be coded with one or more of the subsequently added issue codes (i.e., ComplaintslHR and Publicis Groupe/Jurisdiction) and will, as discussed above, indicate any disagreement with MSL's classifications. 5. Seed Set. a. Defendant MSL. To create the initial seed set of documents that will be used to "train" the Axcelerate software as described generally above, MSL primarily utilized keywords listed on Exhibits A and B to this protocol, but also utilized other judgmental analysis Page 13

377 and search techniques designed to locate highly relevant documents, including the Boolean, concept search and other features ofaxcelerate. Given the volume of hits for each keyword (Exhibit A), MSL reviewed a sampling of the hits and coded them for relevance as well as for the following eight preliminary issues: (i) Reorganization; (ii) Promotion; (iii) Work/Life Balance; (iv) Termination; (v) Compensation; and (vi) Maternity. Specifically, except for key words that were proper names, MSL performed several searches within each set of key word hits and reviewed a sample of the hits. The Axcelerate software ranked the hits in order of relevance based on the software's analytical capabilities and the documents were reviewed in decreasing order of relevance (i.e., each review of the sample of supplemental searches started with the highest ranked documents). Exhibit B identifies the supplemental searches conducted, the number of hits, the number of documents reviewed, the number of documents coded as potentially responsive and general comments regarding the results. In addition, to the extent applicable, documents coded as responsive also were coded with one or more issue tags. MSL will repeat the process outlined above and will include the newly defined issues and newly added custodians. MSL will provide Plaintiffs' counsel with all of the non-privileged documents and will provide, to the extent applicable, the issue tag(s) coded for each document, as described above. Plaintiffs' counsel shall promptly review and provide notice as to any documents with which they disagree where they do not understand the coding. If necessary, counsel will meet and confer to attempt to resolve any disagreements regarding the coding applied to the documents in this seed set. b. Plaintiffs. To help create the initial seed set of documents that will be used to "train" the Axcelerate software, Plaintiffs provided a list of potential key words to MSL. MSL provided Plaintiffs with a hit list for their proposed key words. This process was repeated twice with the hit list for Plaintiffs' most recent set of keywords attached as Exhibit C. MSL Page 14

378 will review 4,000 randomly sampled documents from Plaintiffs' supplemental list ofkey words to be coded for relevance and issue tags. MSL will provide Plaintiffs' counsel with all nonprivileged documents and will provide, to the extent applicable, the issue tag(s) coded for each document. Plaintiffs' counsel shall promptly review and provide notice as to any documents with which they disagree with or where they do not understand the coding. If necessary, the Parties' counsel will meet and confer to attempt to resolve any disagreements regarding the coding applied to the documents in this seed set. c. Judgmental Sampling. In addition to the above, a number of targeted searches were conducted by MSL in an effort to locate documents responsive to several of Plaintiffs' specific discovery requests. Approximately 578 documents have already been coded as responsive and produced to Plaintiffs. In addition, several judgmental searches were conducted which resulted in approximately 300 documents initially being coded as responsive and several thousand additional documents coded as irrelevant. The documents coded as relevant and non-privileged also will be reviewed by Plaintiffs' counsel and, subject to their feedback, included in the seed set. An explanation shall be provided by MSL's attorneys for the basis of the bulk tagging of irrelevant documents (primarily electronic periodicals and newsletters that were excluded in the same manner as spam junk mail is excluded). The explanation shall include the types of documents bulk tagged as irrelevant as well as the process used to identify those types of documents and other similar documents that were bulk tagged as irrelevant. 6. Initial And Iterative Training. Following the creation of the first seed set, the Axcelerate software will review the entire data set to identify other potentially relevant documents. MSL will then review and tag a judgmental based sample, consisting of a minimum of 500 documents, including all documents ranked as highly relevant or hot, of the new Page 15

379 "Computer Suggested" documents, which were suggested by the Axcelerate software. MSL's attorneys shall act in consultation with the Axcelerate software experts to make a reasonable, good faith effort to select documents in the judgmental sample that will serve to enhance and increase the accuracy of the predictive coding functions. The results of this first iteration, both the documents newly coded as relevant and not relevant for particular issue code or codes, will be provided to Plaintiffs' counsel for review and comment. (All documents produced by the parties herein to each other, including, without limitation, these small seed set development productions, shall be made under the Confidentiality Stipulation in this matter as well as any clawback agreement that shall be reduced to an order acceptable to the Court. Any documents marked as irrelevant shall be returned to counsel for MSL at the conclusion of the iterative training phase, unless the relevancy of any documents are disputed, in which case they may be submitted to the Court for review.) Upon completion of the initial review, and any related meet and confer sessions and agreed upon coding corrections, the Axcelerate software will be run again over the entire data set for suggestions on other potentially relevant documents following the same procedures as the first iteration. The purpose of this second and any subsequent iterations of the Predictive Coding process will be to further refine and improve the accuracy of the predictions on relevance and various other codes. The results of the second iteration shall be reviewed and new coding shared with Plaintiffs' counsel as described for the first iteration. This process shall be repeated five more times, for a total of seven iterations, unless the change in the total number of relevant documents predicted by the system as a result of a new iteration, as compared to the last iteration, is less than five percent (5%), and no new documents are found that are predicted to be hot (aka highly relevant), at which point MSL shall have the discretion to stop the iterative process and begin the final review as next described. If more than 40,000 documents are Page 16

380 returned in the final iteration, then MSL reserves the right to apply to the Court for relief and limitations in its review obligations hereunder. Plaintiffs reserve the right, at all times, to challenge the accuracy and reliability of the predictive coding process and the right to apply to the Court for a review ofthe process. 7. Final Search and Production. All ofthe documents predicted to be relevant in the final iteration described in paragraph six above will be reviewed by MSL, unless it applies to the court for relief hereunder. All documents found by MSL's review to be relevant and nonprivileged documents will be promptly produced to Plaintiffs. If more than 40,000 documents are included in the final iteration, then MSL reserves its right to seek payment from Plaintiffs for all reasonable costs and fees MSL incurred related to the attorney review and production of more 40,000 documents. ~is-,:m:rvtsjtm:r;s-im1:-mterrde{tmnrwr'ivernv-f';lfsi7ttnrlmrsee:lm:lfl-t~11l.i of all discovery costs conclusion of this action under 28 U.S.C. at the (4) and Rule 54(d)(l) Federal Rules of Civil Procedure. [P~ to the inclusion of MSL's proposed cost-shifting language as prem~~--;~d argumentative, in contravention of the Court's January 4, 2012 and / ~i Uft", 8, 2012 cn:ders Plailltiffs belieye costs should be subjed to a separate bearing J 8. Quality Control by Random Sample of Irrelevant Documents. In addition, at the conclusion of this search protocol development process described above, and before the final search and production described in Paragraph 7 above, MSL will review a random sample of 2,399 documents contained in the remainder of the database that were excluded as irrelevant. The results of this review, both the documents coded as relevant and not relevant, but not privileged, will be provided to Plaintiffs' counsel for review. (Any documents initially coded as "not relevant" will be provided subject to the Confidentiality Stipulation and any clawback agreements entered in this matter will be returned to counsel for MSL within 60 days of their Page 17

381 production.) The purpose for this review is to allow calculation of the approximate degree of recall and precision of the search and review process used. If Plaintiffs object to the proposed review based on the random sample quality control results, or any other valid objection, they shall provide MSL with written notice thereof within five days of the receipt of the random sample. The parties shall then meet and confer in good faith to resolve any difficulties, and failing that shall apply to the Court for relief. MSL shall not be required to proceed with the final search and review described in Paragraph 7 above unless and until objections raised by Plaintiffs have been adjudicated by the Court or resolved by written agreement ofthe Parties. F. Costs I. MSL proposes to limit the costs of its final review and production of responsive ESI from the MSL collection to an additional $200,000, above and beyond the approximately $350,000 it has already paid or is anticipated to pay in e-discovery related activities as previously described and disclosed to Plaintiffs. -~;pp;~"!1itv-...,..;h+wtrm1"f'"'"4~i;;!l coding process described in paragraphs 6 and right to seek relief from the ourt (e.g., a cost shifting rd and/or ruling that MSL need to ",,/ review more than a specified number 0 ocume~~rsuant to the principles ofproportionality. See Rule 1, Rule 26(b)(2)(C), Ru!y46(b)(2J /' /' Rule 26(g), Federal Rules of Civil /~ Procedure; Commentary o~pfoportionality in Electronic. covery, 11 SEDONA CONF. J. 289 /' (201 0); Oot, et a~dating Reasonableness in a Reasonable Inq. Review, 8J~i:/; (2010); Also see Rule 403 of the Federal Evidence Co./' (inadmissibility of.cumulative evidence). [Plaintiffs object to the inclusion of MSL's proposed cos ifting language as premature and argumentative, in contraven Ion Page 18

382 and- FelJrual y 8, 201% orders. "Plahdiffs beheve costs should be subject to a separate ~-- 2. Plaintiffs agree to bear all of the costs associated with their compliance with the terms of this protocol and with the receipt and review of ESI produced hereunder including the costs associated with its ESI experts at DOAR Litigation Consulting who will be involved with Plaintiffs in all aspects of this ESI protocol. Plaintiffs propose that MSL bear all of the costs associated with its obligations under the terms of this protocol and do not agree to limit the amount ofinformation subject to the review and production of ESI by MSL. G. Format of Production For Documents Produced From Axcelerate 1. TIFFlNative File Format Production. Documents will be produced as singlepage TIFF images with corresponding multi-page text and necessary load files. The load files will include an image load file as well as a metadata (.DAT) file with the metadata fields identified on Exhibit D. Defendant MSL will produce spreadsheets (.xls files) and PowerPoint presentations (.ppt files) in native form as well as any documents that cannot be converted to TIFF format (e.g., audio or video files, such as mp3s, wavs, megs, etc.). In addition, for any redacted documents that are produced, the documents' metadata fields will be redacted where required. For the production of ESI from non- sources, the parties will meet and confer to attempt to reach an agreement ofthe format of production. 2. Appearance. Subject to appropriate redaction, each document's electronic image will convey the same information and image as the original document. Documents that present imaging or formatting problems will be promptly identified and the parties will meet and confer in an attempt to resolve the problems. 3. Document Numbering. Each page of a produced document will have a legible, unique page identifier "Bates Number" electronically "burned" onto the image at a location Page 19

383 that does not obliterate, conceal or interfere with any infonnation from the source document. The Bates Number for each page of each document will be created so as to identify the producing party and the document number. In the case of materials redacted in accordance with applicable law or confidential materials contemplated in any Confidentiality Stipulation entered into by the parties, a designation may be "burned" onto the document's image at a location that does not obliterate or obscure any infonnation from the source document. 4. Production Media. The producing party will produce documents on readily accessible, computer or electronic media as the parties may hereafter agree upon, including CD-ROM, DVD, external hard drive (with standard PC compatible interface), (the "Production Media"). Each piece of Production Media will be assigned a production number or other unique identifying label corresponding to the date of the production of documents on the Production Media (e.g., "Defendant MSL Production April 1,2012") as well as the sequence of the material in that production (e.g. "-001", "-002"). For example, if the production comprises document images on three DVDs, the producing party may label each DVD in the following manner "Defendant MSL Production April 1,2012", "Defendant MSL Production April 1, ", "Defendant MSL Production April 1, " Additional infonnation that will be identified on the physical Production Media includes: (1) text referencing that it was produced in da Silva Moore v. Publicis Groupe SA, et al.; and (2) the Bates Number range of the materials contained on the Production Media. Further, any replacement Production Media will cross-reference the original Production Media and clearly identify that it is a replacement and cross-reference the Bates Number range that is being replaced. 5. Write Protection and Preservation. All computer media that is capable of writeprotection should be write-protected before production. Page 20

384 6. Inadvertent Disclosures. The tenus of the Parties' Clawback Agreement and Court Order shall apply to this protocol 7. Duplicate Production Not Required. A party producing data in electronic form need not produce the same document in paper format. H. Timing. 1. To the extent a timeframe is not specifically outlined herein, the parties will use their reasonable efforts to produce ESI in a timely manner consistent with the Court's discovery schedule. 2. The parties will produce ESI on a rolling basis. I. General Provisions. 1. Any practice or procedure set forth herein may be varied by agreement of the parties, and first will be confirmed in writing, where such variance is deemed appropriate to facilitate the timely and economical exchange of electronic data. 2. Should any party subsequently determine it cannot in good faith proceed as required by this protocol, the parties will meet and confer to resolve any dispute before seeking Court intervention. 3. The Parties agree that e-discovery will be conducted in phases and, at the conclusion of the search process described in Section E above, the Parties will meet and confer regarding whether further searches of additional custodians and/or the Phase II sources is warranted and/or reasonable. If agreement cannot be reached, either party may seek relief from the Court. Page 21

385 J. Plaintiffs' Objection 1. Plaintiffs object to this ESI Protocol in its entirety. Plaintiffs submitted their own proposed ESI Protocol to the Court, but it was largely rejected. The Court then ordered the parties to submit a joint ESI Protocol reflecting the Court's rulings. Accordingly, Plaintiffs jointly submit this ESI Protocol with MSL, but reserve the right to object to its use in this case. This protocol may be executed in counterparts. Each counterpart, when so executed, will be deemed and original, and will constitute the same instrument. By: JANETTE WIPPER, ESQ. DEEPIKA BAINS, ESQ. SIHAM NURHUSSEIN, ESQ. SANFORD WITTELS & HEISLER, LLP Attorneys for Plaintiffs and Class 555 Montgomery Street, Ste San Francisco, CA Telephone: (415) By: BRETT M. ANDERS, ESQ. VICTORIA WOODIN CHA VEY, ESQ. JEFFREY W. BRECHER, ESQ. JACKSON LEWIS LLP Attorneys for Defendant MSLGROUP 58 South Service Road, Suite 410 Melville, NY Telephone: (631) Date:, 2012 Date: :, 2012 Page 22

386 Taking the Plunge: Judge Peck Issues First Decision Endorsing Computer-Assisted ESI :... Page 1 of 3 E-Discovery Law Alert Posted at 12:40 PM on March 2, 2012 by Phillip J. Duffy Taking the Plunge: Judge Peck Issues First Decision Endorsing Computer-Assisted ESI Late last year, Magistrate Judge Andrew Peck of the U.S. District Court for the Southern District of New York, one of the most prominent judicial thought leaders in e-discovery, wrote an article entitled Search, Forward in which he opined that computer-assisted ESI review "should be used where it will help 'secure the just, speedy and inexpensive' (Fed. R. Civ. P. 1) determination of cases", but he forecast that lawyers awaiting a judicial opinion endorsing predictive coding might have "a long wait." As it turns out, the wait wasn't very long at all; on Friday, February 24, 2012, less than 6 months after the publication of his article, Judge Peck himself issued the first judicial opinion approving the use of predictive coding "in appropriate cases." Judge Peck issued his opinion in Monique Da Silva Moore, et al., v. Publicis Groupe & MSL Group, Civ. No (ALC)(AJP) (S.D.N.Y. February 24, 2012), a gender discrimination suit brought by five female plaintiffs against Publicis Groupe, "one of the world's 'big four advertising conglomerates,'" and MSL Group, its U.S. public relations subsidiary. Faced with more than 3 million documents to be reviewed, the parties agreed to use predictive coding -- a process involving senior attorney review and coding of a "seed set" of documents that are then used to train a computer to search the entire data set for relevant documents and cull them -- but they disagreed regarding the methodology. Defendants' ESI proposal contemplated a series of "iterative rounds" to test and refine the searches and stabilize the training of the search software, with production to plaintiffs of both relevant and irrelevant documents returned by the refined searches. Plaintiffs would then be invited to provide feedback to allow further refinement of the searches. Plaintiffs objected to defendants' protocol and submitted their own, which the Court apparently rejected. In addressing plaintiffs' objections, Judge Peck underscored the cost-savings rationale expressed in his article and carried over to his decision: "computer-assisted review 'works better than most of the alternatives, if not all of the [present] alternatives. So the idea is not to make this perfect, it's not going to be perfect. The idea is to make it significantly better than the alternatives without nearly as much cost.'" Consistent with Judge Peck's directive, the parties submitted a Joint ESI Protocol, which Judge Peck "so ordered " despite plaintiffs' continuing objections. On February 22, 2012, plaintiffs filed objections to Judge Peck's rulings with District Judge Carter. In sum, plaintiffs contended Judge Peck's acceptance of defendants' protocol allowed defendants to violate FRCP 26(g)'s requirement to certify defendants' document production is complete and further is not sufficiently reliable to pass muster under Federal Rule of Evidence 702 and the U.S. Supreme Court's 3/27/2013

387 Taking the Plunge: Judge Peck Issues First Decision Endorsing Computer-Assisted ESI :... Page 2 of 3 decision in Daubert. In the opinion, Judge Peck addressed and ultimately dismissed plaintiffs' objections. The Court reasoned that plaintiffs misunderstood Rule 26(g), which does not require such certification and observed that it would be impossible to certify the completeness of a document production of this magnitude. Moreover, Judge Peck stated that Rule 702 and Daubert are inapplicable to the manner in which parties search for documents in discovery. Judge Peck also explained, in detail, the reasons for his decision to endorse predictive coding in appropriate cases and took the time to offer some "lessons for the future." In sum, the Court found the use of predictive coding appropriate in Da Silva Moore based on the parties' agreement with the concept, the vast data set of more than 3 million documents, the superiority of computer-assisted review to the alternatives (manual review or keyword searches) the need for cost effectiveness and proportionality under FRCP 26(b)(2)(C) and the transparent process defendants proposed. In this regard, Judge Peck stressed the importance of cooperation among counsel and reiterated the Court's endorsement of The Sedona Conference Cooperation Proclamation. Moreover, and notwithstanding plaintiffs' expressed concerns, Judge Peck praised defendants' transparency and credited their agreement to provide to plaintiffs all of the documents used to create the seed set, both relevant and non-relevant, as essentially paving the way for the Court's decision. So, what does all of this mean? Is predictive coding the only way to proceed with large scale document review in the future? Will it eliminate the need for attorney involvement in the document review process? Is computer assisted review the panacea that frustrated and cash strapped litigants have been searching for to solve the incredible cost and manpower burdens of e-discovery. Well, the short answer to these questions is "no." As Judge Peck made clear in his opinion, the Court did not order the use of predictive coding, the parties agreed to it (at least some form of it); there is no requirement that computer-assisted review be used in all cases. Nor, the Court was careful to note, is Da Silva Moore a pronouncement that the ESI Protocol used in that case (which is attached to the opinion will be appropriate in other cases that use predictive coding. Finally, the Court avoided endorsing any particular e-discovery service provider or technology. As for the continued need for attorney involvement in the review process, as noted in this blog's prior June 2011 posting, while e-discovery software may make lawyers more efficient, "human knowledge, reaction and intuition as to facts, issues and nuances of legal theories make the role of the live attorney indispensable." Indeed, Judge Peck acknowledged as much in his opinion: "[t]he Court recognizes that computer-assisted review is not a magic, Staples-Easy-Button, solution appropriate for all cases. The technology exists and should be used where appropriate, but it is not a case of machine replacing humans: it is the process used and the interaction of man and machine that the courts need to examine." Da Silva Moore is most significant in that it opens the door for parties to consider the use of computer assisted review or predictive coding in appropriate cases without the fear that a party or judge will resist on the basis that no court has accepted the use of this modern technology in the e-discovery process. The courts have now officially taken the plunge, and we predict that the pool will get crowded quickly. Phillip J. Duffy is a Director on the Gibbons E-Discovery Task Force. Comments (0) Gibbons P.C. gibbonslaw.com 3/27/2013

388 Taking the Plunge: Judge Peck Issues First Decision Endorsing Computer-Assisted ESI :... Page 3 of 3 Newark New York Philadelphia Trenton Wilmington 3/27/2013

389 11 Predictive Coding Lessons from Judge Peck s Opinion in Da Silva Moore By Matthew F. Knouff This article was extracted from a four-part series published online on the Lawyer to Lawyer blog in March The ideas and opinions expressed in this article are solely the opinions of the author and do not necessarily reflect the position of Magistrate Judge Andrew J. Peck, CDS Legal, or any other person or entity. One would be hard-pressed to find a legal topic hotter than predictive coding at the moment. Whether your buzz term of choice is computer-assisted review, technology-assisted review, predictive coding, or computer-assisted coding, the debate over the use of this new technology has revolved around misconceptions of its role in the discovery process and fears about its efficacy in comparison to other methods of search and retrieval. In Monique Da Silva Moore, et. al. v. Publicis Groupe & MSL Group No. 11 Civ (ALC) (AJP) (S.D.N.Y. Feb 24, 2012), the Honorable Andrew J. Peck, United States Magistrate Judge for the Southern District of New York, gives us the first opinion approving the use of computer-assisted review, which in his article Search, Forward from the October 2011 edition of Law Technology News he defines as, tools that use sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e. training by) a human reviewer. Those who looked to Judge Peck s commentary in Search, Forward as a sign of judicial approval now have a ruling which can begin to assuage the fears associated with breaking from the costly and oft-ineffective traditions of exhaustive human review and search methodologies solely built on keywords. In Da Silva Moore, a gender discrimination case, the parties discussed an electronic discovery protocol which included the use of predictive coding. A dispute arose not over the use of the actual technology, but rather over the protocol surrounding its use. In brief, the ESI protocol proposed by the defendant (MSL), and objected to in its entirety by the plaintiffs, included the following steps: I. MSL would use technology to generate a random sample set from the document universe (2,399 documents out of approximately three million). This sample would be reviewed and coded according to certain categories (e.g. relevance, privilege, and issue-relation), beginning the process of creating the seed sets used to train the computer. In addition, defense counsel would use judgmental sampling techniques that leveraged the use of keywords (including those Matthew F. Knouff is the General Counsel & ediscovery Counsel for Complete Discovery Source. For more information visit proposed by plaintiffs), Boolean operators, conceptual searches, and similar methods, to identify documents to further develop the seed sets. II. The non-privileged results from this initial phase would be reviewed by plaintiffs counsel with any disputes regarding coding being subject to conference. The resultant sets would be the initial seed sets used to code the full document universe. III. The initial seed sets would be run against the entire corpus to identify similar documents. MSL would then review sample sets of at least 500 computer-coded documents to evaluate coding decisions, make any changes, and further train the system. At least seven iterations of refinement would be performed. Plaintiffs would receive all documents reviewed during each iteration, allowing them the opportunity to participate in calibrating the predictive process. IV. The entire set of relevant documents following the final iteration would be reviewed by defense counsel. MSL proposed cost-shifting for the review and production of documents exceeding the 40,000 mark. V. MSL would perform quality control by reviewing a random sample of documents excluded during the computer-assisted review process as being non-responsive. These documents would be provided to plaintiffs for review to determine if the levels of recall and precision are acceptable. In evaluating the ESI protocol and plaintiffs objections to the Court s acceptance thereof, Judge Peck debunked myths, reiterated some old lessons, and provided counsel with much needed guidance. In an age where increasing volumes of ESI highlight the shortcomings of traditional search and retrieval methods, the need to effectuate the just, speedy, and inexpensive determination requires that legal practitioners adapt accordingly. Computer-assisted review can provide counsel with a means to tackle the realities of modern litigation discovery head on. Whether you decide to take the leap into computer assisted review technology now that you no longer have to worry about being the first or a guinea pig as Judge Peck comments, or decide to resist the Matrix with all your might, it behooves you to consider eleven key lessons that can be gleaned from Peck s opinion: 1. Technology-assisted review is as effective, if not more so, than manual review. The long-held belief that manual review is the most effective method for finding all the relevant documents during discovery is a myth. Although humans are very continued next page Peace of Mind for Serious ediscovery (855)

390 smart, and have big brains that rival the reasoning power of any computer, they get fatigued, they make mistakes, and they can produce wildly divergent decisions even when evaluating the same documents. Judge Peck sites to various scholarly works to support his opinion that statistics clearly show that computerized searches are at least as accurate, if not more so, than manual review. 2. The standard for review is not perfection; rather the process should be better than the alternatives at a lower cost. A perfect methodology for returning 100% of all the relevant information in a case has never existed, and probably will never exist. Judge Peck explicitly concedes that computerassisted review is not perfect, and reiterates Judge Shira Scheindlin s Pension Committee decision that parties are not expected to meet such a standard. Rather, the aim is to secure the just, speedy, and inexpensive determination of lawsuits reinforced by the FRCP 26(b)(2)(C) proportionality doctrine. In comparing predictive coding to keyword searching, Judge Peck illustrates numerous shortcomings associated with keywords including the inherent problems in selecting terms without having the requisite knowledge about a dataset, over-inclusiveness, and their usual lack of effectiveness. He goes on to state, Computer assisted review appears to be better than the available alternatives. 3. Proportionality plays a key role in developing a protocol around computer-assisted review, but is evaluated in light of the results. Judge Peck rejected the part of Defendant s proposed ESI protocol that limited review and production to the top 40,000 documents returned by the ESI protocol. He stated, [p]roportionality requires consideration of results as well as costs. And if stopping at 40,000 is going to leave a tremendous number of highly likely responsive documents unproduced, [MSL s proposed cutoff] doesn t work. The threshold for where a producing party can rely on technology to corral an unnecessarily high review and production burden depends on an analysis of the results rather than setting an arbitrary cap. This does not mean that early agreement on such a cap is not possible or even desirable, but costs cannot be considered in isolation from the results of the predictive coding process and the amount at issue in the litigation. A court s ability to make proportionality determinations is also dependent upon developments within the case. Such issues may include the number of documents ultimately produced, the costs involved, the discovery of additional hot documents, which may require refinement of the seed set and additional training iterations, and the expansion of the scope of discovery. 4. FRE 702 and Daubert are not applicable to search methodology during discovery. In holding that Defendant s ESI protocol did not run afoul of FRE 702, Judge Peck highlighted that FRE 702 and Daubert apply to the admissibility of expert testimony at trial, not to search and retrieval methodology during the discovery process. 5. Humans are not being replaced by technology. A common, though cartoonish, concern involves a veritable doomsday scenario where attorneys are replaced by machines and the legal profession as we know it ceases to exist. Counsel can rest easy. Predictive coding is meant to help humans practice law. Actual people are necessary to train the system by selecting a seed set, reviewing resultant sets, and refining the protocol. Keep in mind that counsel still has to use the information to build a case. We are a long way from robots making courtroom appearances. Think of technology-assisted review technology as another tool in your arsenal; albeit a tool that only works effectively within a defined process. As Judge Peck stated, The technology exists and should be used where appropriate, but it is not a case of machine replacing humans: it is the process used and the interaction of man and machine that the court needs to examine. 6. FRCP 26(g)(1) does not require counsel to certify that a production is complete. In objecting to Judge Peck s acceptance of defendant s ESI protocol, plaintiff argued that the court provided unlawful cover for defense counsel who has a duty to certify the completeness of document productions under FRCP 26(g)(A)(1). First, Judge Peck highlighted plaintiffs erroneous reliance on this subsection which applies to initial disclosures. FRCP 26(g)(1)(B) applies to discovery and incorporates proportionality under FRCP 26(b)(2)(C) as opposed to a requirement for completeness. 7. Cooperation and transparency are important considerations for gaining acceptance from the bench. Judge Peck noted his strong endorsement of the Sedona Conference Cooperation Proclamation and stated, the best solution in the entire area of electronic discovery is cooperation among counsel. The defendant MSL was extremely transparent with its ESI protocol in that plaintiff [could] see how [defendant] has coded every used in the seed set (both relevant and non-relevant). Judge Peck went on to state that defendant s transparency in its proposed ESI search protocol made it easier for the Court to approve the use of predictive coding. continued next page Peace of Mind for Serious ediscovery (855)

391 8. Leverage your client s knowledge to help develop the search protocol. Parties often have the best source of information about another party s documents right under their noses: their client. Judge Peck expressed surprise that in many cases counsel do not appear to have sought and utilized their client s knowledge about the opposing party s custodians and document resources. He went on to make a comment that, hopefully, will shock most practitioners: [C] ounsel for the producing party often is not sufficiently knowledgeable about their own client s custodians and business terminology. Putting aside the ethical obligations that may be implicated by such a deficiency, when handling ediscovery in modern litigation you must know your client s data, key custodians, business terminology, and internal processes. Having this type of knowledge in advance will not only provide many strategic advantages, but when you openly share information about target custodians and your proposed search methodology opposing counsel and the Court are more apt to agree to your approach (at least as phase one without prejudice). 9. Phased discovery can help control costs. The defendant proposed a phased approach to discovery where 30 custodians would be reviewed during an initial phase, after which a second phase would be considered. The Court agreed with this approach and addressed plaintiffs concern over having enough time to complete both phases by extending the timeframe for discovery if necessary. With this approach the defendant was able to avoid the up-front costs of searching seven additional custodians. In addition, Judge Peck held that the Defendant company s CEO, whose s were in France and likely protected by French privacy laws should not be reviewed initially because those s could be obtained through various custodians located in New York. This also helped reduce the costs associated with complex crossborder ediscovery efforts. Those of you that just said, Wait! Phased discovery is not necessarily a lesson learned about the use of computer assisted review! are arguably correct. However, I present it to you because it can be a vital part of any effective technology-assisted review protocol. 10. Predictive coding is not necessarily appropriate for every case. Judge Peck stated several times that he approved of computer-assisted review in appropriate cases. In the same manner that a chainsaw is not the best tool for all types of yard work, technology-assisted review is not the ideal tool in all cases. In Da Silva Moore it was deemed appropriate, in part, because of the large document universe. Some may argue that every case involving millions of records does not necessarily require computer-assisted review since there are several other factors that can weigh on such a decision. However, I submit that it is important to at least evaluate every case to determine if it would be an appropriate solution. If it can help cut costs, defensibly streamline the discovery process, reduce burden on all parties (including the court), and identify relevant information more effectively than any alternative method, there would appear to me to be a strong case for using computer-assisted review. 11. Rely on a trusted vendor to lend their expertise. Judge Peck explicitly stated that his opinion is not an endorsement for any particular vendor or technology. However, he does note that it was very helpful that the parties ediscovery vendors were present and spoke at the court hearings where the ESI Protocol was discussed. He went on to say that vendors can be helpful even when counsel is knowledgeable about ESI issues. All vendors are not created equal unfortunately, and it is important to vet providers of computer-assisted review for their knowledge of the technology as well as their ability to implement processes and protocols to ensure its successful deployment. Just because someone has a band saw in their garage does not mean they can build an armoire that will last. Judge Peck states that, it is important for the vendor to be able to explain complicated ediscovery concepts in ways that make it easily understandable to judges who may not be tech-savvy. Make sure you ask your vendor lots of questions so that you are comfortable that they can successfully fill this role. The Da Silva Moore opinion significantly advances the case for predictive coding. Judge Peck ruled that its use was warranted given: (1) the parties agreement, (2) the vast amount of ESI to be reviewed (over three million documents), (3) the superiority of computer-assisted review to the available alternatives (i.e., linear manual review or keyword searches), (4) the need for cost effectiveness and proportionality under Rule 26(b)(2)(C), and (5) the transparent process proposed by the [defendant]. While it appears that we have a long way to go before computer-assisted review supplants the traditional, and more costly, manual review methods, the new technology and associated processes present a means to address the problems associated with expanding amounts of ESI and the associated ediscovery burdens that may arise. The legal industry does not exist in isolation from developments in technology, but rather, is intimately intertwined with such advancement. As a result, legal practitioners must adapt. With his opinion in Da Silva Moore, Judge Peck gives all legal professionals a means to jut ahead a few positions on the evolutionary chart. Peace of Mind for Serious ediscovery (855)

392 The CAR Keys to Defensibility ( Computer Assisted Review ) Complete Discovery Source Share 20 The CAR Keys to Defensibility ( Computer Assisted Review By Dino Medina Complete Discovery Source As published in the New York County Lawyer, September 2012 The purpose of pretrial discovery in the American legal system is to promote the full disclosure of facts relevant to a given dispute. To this end, the Federal Rules permit a litigating party to seek all information that is reasonably calculated to lead to the discovery of admissible evidence.[1] There was a time when discovery requests would yield only boxes of documents from which to locate relevant material. The societal shift from paper documents to electronically stored information (ESI) over the past ten years, however, has led to sharp increases in discovery data volume, review time and cost. The following statistics illustrate this phenomenon: Statisticians estimate that only 5% to 7% of all data is created outside of a computer system.[2] 73% of all production costs are attributable to review of ESI.[3] The cost to review one gigabyte of ESI is between $5,000 and $30,000.[4] As discovery in the legal industry continues to evolve in this electronic age, defensible methods to manage increased data volume, speed-up review time and curb review cost have become essential. Technology s latest solution to these discovery issues is computer-assisted review (CAR). CAR is a process by which analytics technology leverages the subject matter proficiency of human document reviewers to make conceptually-based coding decisions on all documents within a particular universe or corpus. Attorneys with subject matter expertise must first generate a seed or training set of documents (e.g., 500 docs), code it consistently using a specified protocol and run an analytics engine against the seed set to extrapolate its coding to the remaining documents in the corpus. Next, the analytics software generates a random sample from the corpus, which is then re-coded by the attorneys with subject matter expertise. Once re-coded, the analytics tool will again extrapolate the coding to the corpus. Third, an overturn report is run which, among other things, allows one to see the percentage of computer coding decisions overturned by the attorneys with subject matter expertise. Several iterations of this process are implemented until the computer s coding is overturned by the attorneys with subject matter expertise at a rate of less than 5%. This process is proven to be significantly less costly and time consuming than traditional manual document review.[5] Given all of the recent hype surrounding CAR and the judicial acceptance of this black box technology in the Da Silva Moore, Global Aerospace and Kleen Products cases, one might wonder: can CAR really be as reliable (and therefore, legally defensible) as traditional manual review in identifying relevant data? This question has become one of primordial importance to large-scale litigators tasked with sorting through huge volumes of ESI to find responsive and privileged material in an efficient and cost-effective manner. As will become evident, CAR technology combined with a robust audit trail can provide a defensible mechanism to meet one s e-discovery obligations. How does one achieve defensibility in the context of CAR? The producing party must build a detailed audit trail that satisfies a two-pronged test. The first prong speaks to the reasonableness of the CAR process; the selected method must accurately capture a reasonably sufficient number of the responsive, non-privileged ESI pursuant to a given document request, and any remaining unproduced ESI is predominantly not relevant or privileged.[6] An audit trail satisfying the first CAR defensibility prong entails reaching agreement with one s adversary as to the degree of recall and precision the CAR software will utilize in gathering, analyzing and coding discovery documents. Industry News Technology Update: CDS Nytrix is Moving More Than Review Into the Cloud Judge Carter Upholds Peck s Decision in Da Silva Moore Cloud Could Cut $12 Billion from US Government Annual Deficit Technology Assisted Review Is it defensible? Inadvertent Production of Two Privileged Pages Among Over Two Million May Waive the Attorney-Client Privilege Predictive Coding Based Legal Methods for Search and Review New York Appellate Court: Disintegration of Business Relationship Requires Preservation of ESI Third Circuit: Winners Still Have to Shoulder Most E-Discovery Costs Allergan, Inc. v. Merz Pharma: Central District of CA Hammers Merz for Failure to Timely Produce Documents Recovery of E-Discovery Costs 28 USC 1920(4) Tools The Basics: What is ediscovery? 52 e-discovery Terms You Should Know CDS and Relativity Assisted Review Case Study Recent e-discovery Case Law 10 Tips: Planning for ediscovery ediscovery Best Practices Guide 11 Predictive Coding Lessons from Judge Peck s Opinion in Da Silva Moore, March 2011 Articles/Case Studies The "CAR" Keys to Defensibility - Computer Assisted Review Predictive Coding: Rise of the Machines? Managing ediscovery Strategy and Implementation IPM Case Study: CDS Tames Big Data with Big Storage UNC JOLT: US Legal Holds Across Borders: A Conundrum? Richmond JOLT: Admissibility of Non-US Evidence Subscribe to our YouTube Channel

393 The CAR Keys to Defensibility ( Computer Assisted Review ) Complete Discovery Source Recall is the number of potentially relevant documents the system retrieves relative to the number of documents in the entire collection[7] (e.g., as a result of a keyword search or concept-based search). Precision is the number of relevant documents identified in relation to the number of potentially relevant documents previously retrieved. Recall is an indicator of the system s effectiveness in finding potentially relevant data, and precision is an indicator of the accuracy of the document coding process. Recall is a measure of completeness or quantity, whereas precision is a measure of exactness or quality. With too little recall, potentially relevant information will be overlooked; with too little precision, one s relevant document set becomes riddled with non-relevant documents. The key to defensibility is a memorialized negotiation and agreement on these issues; it would be difficult for one s adversary to claim production deficiencies in the face of a written recall and precision rate agreement. Additionally important in satisfying the first defensibility prong as it relates to CAR, is the ability to document the process of coding seed documents and subsequent system training documents. It is generally understood that the technology used to identify relevant data is only as good as those training it. Because most difficulties in the CAR process stem from human review errors, the seed and subsequent training phases must be completed accurately and consistently by attorneys with expertise in the particular subject matter. Without accurate and consistent human input, the CAR analytics engine is unable to produce reliable results. Attorneys with subject matter expertise should draft a protocol detailing the criteria for accurately and consistently categorizing documents for defensibility purposes. Finally, the statistical components underlying the CAR process must be scientifically sound and documented to satisfy the first defensibility prong. These statistical components include random sampling, confidence levels and confidence intervals. Random sampling is a means to obtain a truly representative sample of documents, by enabling all documents within a corpus to have an equal chance of being selected. Random sampling can be used to select the initial seed set and must be used to draw samples for each of the iterative training rounds. A confidence level expresses the level of certainty one can expect that a sample set of documents is drawn randomly from the system and reflective of the types of documents in a given population. A confidence interval is the margin of error expressed as both positive and negative numbers that a selected sample is truly random. Most CAR experts use the 95% confidence level, with an accompanying margin of error of +/-2.5%. The key here for prong one defensibility is that (a) both the confidence level and confidence interval be agreed to by the parties and (b) a report is generated at each phase of the CAR analytics process to document the manner in which samples are drawn and coded by the software. The second defensibility prong is aimed at satisfaction of Rules 1 and 26 of the Federal Rules of Civil Procedure. That is, the burden or expense of [the] discovery [process must] not outweigh its likely benefit [8] and assist in obtaining the just, speedy and inexpensive resolution of a case.[9] Document intensive cases in which traditional manual review is prohibitively costly, time consuming and error prone make prime candidates for passing muster under this second defensibility prong. Consider the following data from a 2012 study conducted by the RAND Corporation: Computer-assisted review enables case teams to identify at least as many relevant documents as manual review with almost the same level of accuracy. Computer-assisted review can cut attorney review time by as much as 75%. Computer-assisted review can achieve an 80% cost savings over traditional manual review.[10] While these CAR statistics are impressive, the key to defensibility under this second prong is compiling detailed CAR vs. manual review cost comparison data, relevancy effectiveness data and estimated review time data. In conclusion, the volume, complexity and variety of today s data is challenging the legal industry to find new ways to defensibly manage discovery. It is important to approach CAR defensibility in terms of method reasonableness combined with a robust audit trail. Such a framework will enable parties to rapidly and accurately identify relevant documents, completely eliminate irrelevant documents, prioritize the most relevant documents for review, achieve substantial cost savings and pass judicial scrutiny. [1] See Fed. R. Civ. P. 26(b)(1) [2] Craig Ball, The DNA of Data, Law Technology News, April, 2005.

394 The CAR Keys to Defensibility ( Computer Assisted Review ) Complete Discovery Source [3] Nicholas M. Pace & Laura Zakaras. Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery. Santa Monica, CA: RAND Corporation, [4] Dean Gonsowski, E-Discovery Costs: Pay Now or Pay Later, Inside Counsel, May 23, [5] Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E- Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech. 11 (2011). [6] The Sedona Conference, The Sedona Conference Commentary on Achieving Quality in the E-Discovery Process, 10 SEDONA CONF. J. 299, 320 (2009). [7] Alex Hoover, Top Ten Points on Computer Assisted Review, Association of Corporate Counsel, September 1, [8] FED. R. CIV. P. 1. [9] FED. R. CIV. P. 26(b)(2)(C)(iii). [10] See Pace & Zakaras, supra note 3. Legal Notice and Privacy Policy - Contact

395 Richmond Journal of Law and Technology Vol. XVII, Issue 3 TECHNOLOGY-ASSISTED REVIEW IN E-DISCOVERY CAN BE MORE EFFECTIVE AND MORE EFFICIENT THAN EXHAUSTIVE MANUAL REVIEW By Maura R. Grossman * & Gordon V. Cormack ** Cite as: Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII RICH. J.L. & TECH. 11 (2011), * Maura R. Grossman is counsel at Wachtell, Lipton, Rosen & Katz. She is co-chair of the E-Discovery Working Group advising the New York State Unified Court System, and a member of the Discovery Subcommittee of the Attorney Advisory Group to the Judicial Improvements Committee of the U.S. District Court for the Southern District of New York. Ms. Grossman is a coordinator of the Legal Track of the National Institute of Standards and Technology s Text Retrieval Conference ( TREC ), and an adjunct faculty member at Rutgers School of Law Newark and Pace Law School. Ms. Grossman holds a J.D. from Georgetown University Law Center, and an M.A. and Ph.D. in Clinical/School Psychology from Adelphi University. The views expressed herein are solely those of the Author and should not be attributed to her firm or its clients. Gordon V. Cormack is a Professor at the David R. Cheriton School of Computer Science, and co-director of the Information Retrieval Group, at the University of Waterloo. He is a coordinator of the TREC Legal Track, and Program Committee member of TREC at large. Professor Cormack is the co-author of Information Retrieval: Implementing and Evaluating Search Engines (MIT Press, 2010), as well as more than 100 scholarly articles. Professor Cormack holds a B.Sc., M.Sc., and Ph.D. in Computer Science from the University of Manitoba. ** The Authors would like to thank Ellen Voorhees and Ian Soboroff at NIST for providing access to the raw TREC 2009 data. The Authors gratefully acknowledge the helpful comments received from Hon. John M. Facciola (D.D.C.), Hon. Paul W. Grimm (D. Md.), and Hon. Andrew J. Peck (S.D.N.Y.) on an earlier draft of this paper. 1

396 Richmond Journal of Law and Technology Vol. XVII, Issue 3 ABSTRACT E-discovery processes that use automated tools to prioritize and select documents for review are typically regarded as potential cost-savers but inferior alternatives to exhaustive manual review, in which a cadre of reviewers assesses every document for responsiveness to a production request, and for privilege. This Article offers evidence that such technology-assisted processes, while indeed more efficient, can also yield results superior to those of exhaustive manual review, as measured by recall and precision, as well as F 1, a summary measure combining both recall and precision. The evidence derives from an analysis of data collected from the TREC 2009 Legal Track Interactive Task, and shows that, at TREC 2009, technology-assisted review processes enabled two participating teams to achieve results superior to those that could have been achieved through a manual review of the entire document collection by the official TREC assessors. 2

397 Richmond Journal of Law and Technology Vol. XVII, Issue 3 I. INTRODUCTION [1] The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery cautions that: [T]here appears to be a myth that manual review by humans of large amounts of information is as accurate and complete as possible perhaps even perfect and constitutes the gold standard by which all searches should be measured. Even assuming that the profession had the time and resources to continue to conduct manual review of massive sets of electronic data sets (which it does not), the relative efficacy of that approach versus utilizing newly developed automated methods of review remains very much open to debate. 1 While the word myth suggests disbelief, literature on the subject contains little scientific evidence to support or refute the notion that automated methods, while improving on the efficiency of manual review, yield inferior results. 2 This Article presents evidence supporting the position that a technology-assisted process, in which humans examine only a small fraction of the document collection, can yield higher recall and/or precision than an exhaustive manual review process, in which humans code and examine the entire document collection. [2] A technology-assisted review process involves the interplay of humans and computers to identify the documents in a collection that are responsive to a production request, or to identify those documents that should be withheld on the basis of privilege. 3 A human examines and 1 The Sedona Conference, The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery, 8 SEDONA CONF. J. 189, 199 (2007) [hereinafter Sedona Search Commentary]. 2 Id. at 194 ( The comparative efficacy of the results of manual review versus the results of alternative forms of automated methods of review remains very much an open matter of debate. ). 3 See Douglas W. Oard et al., Evaluation of information retrieval for E-discovery, 18:4 ARTIFICIAL INTELLIGENCE & LAW 347, 365 (2010) ( In some cases... the end user will interact directly with the system, specifying the query, reviewing results, modifying the 3

398 Richmond Journal of Law and Technology Vol. XVII, Issue 3 codes only those documents the computer identifies a tiny fraction of the entire collection. 4 Using the results of this human review, the computer codes the remaining documents in the collection for responsiveness (or privilege). 5 A technology-assisted review process may involve, in whole or in part, the use of one or more approaches including, but not limited to, keyword search, Boolean search, conceptual search, clustering, machine learning, relevance ranking, and sampling. 6 In contrast, exhaustive manual review requires one or more humans to examine each and every document in the collection, and to code them as responsive (or privileged) or not. 7 [3] Relevant literature suggests that manual review is far from perfect. 8 Moreover, recent results from the Text Retrieval Conference ( TREC ), sponsored by the National Institute of Standards and Technology ( NIST ), show that technology-assisted processes can achieve high levels of recall and precision. 9 By analyzing data collected query, and so on. In other cases, the end user s interaction with the system will be more indirect.... ). 4 See Sedona Search Commentary supra note 1, at See Maura R. Grossman & Terry Sweeney, What Lawyers Need to Know About Search Tools, THE NAT L L.J. (Aug. 23, 2010), available at lawtechnologynews/pubarticleltn.jsp?id= &slreturn=1&hbxlogin=1 ( machine learning tools, use seed sets of documents previously identified as responsive or unresponsive to rank the remaining documents from most to least likely to be relevant, or to classify the documents as responsive or nonresponsive. ). 6 See, e.g., Sedona Search Commentary, supra note 1, at ; CORNELIS JOOST VAN RIJSBERGEN, INFORMATION RETRIEVAL (2d ed. 1979). The specific technologies employed in the processes that are the subjects of this study are detailed infra Parts III.A. III.B. 7 See, e.g., Herbert L. Roitblat et al., Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, 61 J. AM. SOC Y. FOR INFO. SCI. AND TECH. 70, 70 (2010). 8 See, e.g., Sedona Search Commentary, supra note 1. 9 Bruce Hedin et al., Overview of the TREC 2009 Legal Track, in NIST SPECIAL PUBLICATION: SP , THE EIGHTEENTH TEXT RETRIEVAL CONFERENCE (TREC 2009) PROCEEDINGS 16 & tbl.5 (2009), available at 4

399 Richmond Journal of Law and Technology Vol. XVII, Issue 3 during the course of the TREC 2009 Legal Track Interactive Task, 10 the Authors demonstrate that the levels of performance achieved by two technology-assisted processes exceed those that would have been achieved by the official TREC assessors law students and lawyers employed by professional document-review companies had they conducted a manual review of the entire document collection. [4] Part II of this Article describes document review and production in the context of civil litigation, defines commonly used terms in the field of information retrieval, and provides an overview of recent studies. Part III details the TREC 2009 Legal Track Interactive Task, including the H5 and Waterloo efforts, as well as the TREC process for assessment and goldstandard creation. Part IV uses statistical inference to compare the recall, precision, and F 1 scores that H5 and Waterloo achieved to those the TREC assessors would have achieved had they reviewed all of the documents in the collection. Part V presents a qualitative analysis of the nature of manual review errors. Parts VI, VII, and VIII, respectively, discuss the results, limitations, and conclusions associated with this study. Ultimately, this Article addresses a fundamental uncertainty that arises in determining what is reasonable and proportional: Is it true that if a human examines every document from a particular source, that human will, as nearly as possible, correctly identify all and only the documents that should be produced? That is, does exhaustive manual review guarantee that production will be as complete and correct as possible? Or can technology-assisted review, in which a human examines only a fraction of the documents, do better? II. CONTEXT [5] Under Federal Rule of Civil Procedure 26(g)(1) ( Rule 26(g)(1) ), an attorney of record must certify to the best of [his or her] knowledge, legal.umiacs.umd.edu/legaloverview09.pdf; see also Douglas W. Oard et al., Overview of the TREC 2008 Legal Track, in NIST SPECIAL PUBLICATION: SP , THE SEVENTEENTH TEXT RETRIEVAL CONFERENCE (TREC 2008) PROCEEDINGS 8 (2008), available at 10 See Hedin et al., supra note 9, at 2. 5

400 Richmond Journal of Law and Technology Vol. XVII, Issue 3 information, and belief formed after a reasonable inquiry, that every discovery request, response, or objection is consistent with [the Federal Rules of Civil Procedure]... not interposed for any improper purpose, such as to harass, cause unnecessary delay, or needlessly increase the cost of litigation[, and is] neither unreasonable nor unduly burdensome or expensive, considering the needs of the case, prior discovery in the case, the amount in controversy, and the importance of the issues at stake in the action. 11 Similarly, Federal Rule of Civil Procedure 26(b)(2)(C)(iii) ( Rule 26(b)(2)(C)(iii) ) requires a court to limit discovery when it determines that the burden or expense of the proposed discovery outweighs its likely benefit, considering the needs of the case, the amount in controversy, the parties resources, the importance of the issues at stake in the action, and the importance of the discovery in resolving the issues. 12 Thus, Rules 26(g)(1) and 26(b)(2)(C)(iii) require that discovery requests and responses be proportional. 13 However, Federal Rule of Civil Procedure 37(a)(4) ( Rule 37(a)(4) ) provides that an evasive or incomplete disclosure, answer or response must be treated as a failure to disclose, answer, or respond[,] and therefore requires that discovery responses be complete. 14 Together, Rules 26(g)(1), 26(b)(2)(C)(iii), and 37(a)(4) reflect the tension between completeness on one hand, and burden and cost on the other that exists in all electronic discovery ( e-discovery ) processes. 15 In 11 FED. R. CIV. P. 26(g)(1). 12 FED. R. CIV. P. 26(b)(2)(C)(iii). 13 The Sedona Conference, The Sedona Conference Commentary on Proportionality in Electronic Discovery, 11 SEDONA CONF. J. 289, 294 (2010) [hereinafter Sedona Proportionality Commentary]. 14 FED. R. CIV. P. 37(a)(4). 15 Typically, a responding party will not only seek to produce all responsive documents, but to identify only the responsive documents, in order to guard against overproduction or waiver of privilege. See, e.g., Mt. Hawley Ins. Co. v. Felman Prod., Inc., 271 F.R.D. 125, 136 (S.D.W. Va. 2010) (finding that plaintiff s over-production of documents by more than 30% was a factor in waiver of privilege). 6

401 Richmond Journal of Law and Technology Vol. XVII, Issue 3 assessing what is reasonable and proportional with respect to e-discovery, parties and courts must balance these competing considerations. 16 [6] One of the greatest challenges facing legal stakeholders is determining whether or not the cost and burden of identifying and producing electronically stored information ( ESI ) is commensurate with its importance in resolving the issues in dispute. 17 In current practice, the problem of identifying responsive (or privileged) ESI, once it has been collected, is almost always addressed, at least in part, by a manual review process, the cost of which dominates the e-discovery process. 18 A natural question to ask, then, is whether this manual review process is the most effective and efficient one for identifying and producing the ESI most likely to resolve a dispute. A. Information Retrieval [7] The task of finding all, and only, the documents that meet some requirement is one of information retrieval ( IR ), a subject of scholarly 16 See Harkabi v. Sandisk Corp., No. 08 Civ (WHP), 2010 WL , at *1 (S.D.N.Y Aug. 23, 2010) ( Electronic discovery requires litigants to scour disparate data storage mediums and formats for potentially relevant documents. That undertaking involves dueling considerations: thoroughness and cost. ). 17 See id. at *8 ( Integral to a court s inherent power is the power to ensure that the game is worth the candle that commercial litigation makes economic sense. Electronic discovery in this case has already put that principle in jeopardy. ); Hopson v. Mayor of Balt., 232 F.R.D. 228, 232 (D. Md. 2005) ( This case vividly illustrates one of the most challenging aspects of discovery of electronically stored information how properly to conduct Rule 34 discovery within a reasonable pretrial schedule, while concomitantly insuring that requesting parties receive appropriate discovery, and that producing parties are not subjected to production timetables that create unreasonable burden, expense, and risk of waiver of attorney-client privilege and work product protection ). See generally Sedona Proportionality Commentary, supra note Marisa Peacock, The True Cost of ediscovery, CMSWIRE, cms/enterprise-cms/the-true-cost-of-ediscovery php (2009) (citing Sedona Search Commentary, supra note 1, at 192); Ashish Prasad et al., Cutting to the Document Review Chase: Managing a Document Review in Litigation and Investigations, 18 BUS. LAW TODAY, 2, Nov. Dec

402 Richmond Journal of Law and Technology Vol. XVII, Issue 3 research for at least a century. 19 In IR terms, some requirement is referred to as an information need, and relevance is the property of whether or not a particular document meets the information need. 20 For e-discovery, the information need is typically specified by a production request (or by the rules governing privilege), and the definition of relevance follows. 21 Cast in IR terms, the objective of review in e-discovery is to identify as many relevant documents as possible, while simultaneously identifying as few nonrelevant documents as possible. 22 The fraction of relevant documents identified during a review is known as recall, while the fraction of identified documents that are relevant is known as precision. 23 That is, recall is a measure of completeness, while precision is a measure of accuracy, or correctness. 24 [8] The notion of relevance, although central to information science, and the subject of much philosophical and scientific investigation, remains elusive. 25 While it is easy enough to write a document describing an 19 The concepts and terminology outlined in Part II.A may be found in many information retrieval textbooks. For a historical perspective, see GERARD SALTON & MICHAEL J. MCGILL, INTRODUCTION TO MODERN INFORMATION RETRIEVAL (1983); VAN RIJSBERGEN, supra note 6. For a more modern treatment, see STEFAN BÜTTCHER ET AL., INFORMATION RETRIEVAL: IMPLEMENTING AND EVALUATING SEARCH ENGINES (2010). 20 See BÜTTCHER ET AL., supra note 19, at 5-6, See Hedin et al., supra note 9, at See VAN RIJSBERGEN, supra note 6, at See David C. Blair & M. E. Maron, An Evaluation of Retrieval Effectiveness for a Full- Text Document-Retrieval System, 28 COMMC NS ACM 289, 290 (1985) ( Recall measures how well a system retrieves all the relevant documents; and Precision, how well the system retrieves only the relevant documents. ); VAN RIJSBERGEN, supra note 6, at See VAN RIJSBERGEN, supra note 6, at See Tefko Saracevic, Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II: Nature and Manifestations of Relevance, 58 J. AM. SOC Y FOR INFO. SCI. & TECH (2007); Tefko Saracevic, Relevance: A Review of the Literature and a Framework for Thinking on the Notion in 8

403 Richmond Journal of Law and Technology Vol. XVII, Issue 3 information need and hence relevance, determining the relevance of any particular document requires human interpretation. 26 It is well established that human assessors will disagree in a substantial number of cases as to whether a document is relevant, regardless of the information need or the assessors expertise and diligence. 27 [9] A review resulting in higher recall and higher precision than another review is more nearly complete and correct, and therefore superior, 28 while a review with lower recall and lower precision is inferior. 29 If one result has higher recall while the other has higher precision, it is not immediately obvious which should be considered superior. To calculate a review s effectiveness, researchers often employ F 1 the harmonic mean of recall and precision 30 a commonly used summary measure that rewards results achieving both high recall and high precision, while penalizing those that have either low recall or low precision. 31 The value of F 1 is always intermediate between recall and precision, but is generally closer to the lesser of the two. 32 For example, a result with 40% recall and 60% precision has F 1 = 48%. Following Information Science. Part III: Behavior and Effects of Relevance, 58:13 J. AM. SOC Y FOR INFO. SCI. & TECH (2007). 26 See Peter Bailey et al., Relevance Assessment: Are Judges Exchangeable and Does It Matter?, in SIGIR 08 PROCEEDINGS OF THE 31ST ANNUAL INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL 667 (2008); see also VAN RIJSBERGEN, supra note 6, at See Bailey et al., supra note 26, at See Blair & Maron, supra note See id. 30 =. 31 See BÜTTCHER ET AL., supra note 19, at See id. 9

404 Richmond Journal of Law and Technology Vol. XVII, Issue 3 TREC, this Article reports recall and precision, along with F 1 summary measure of overall review effectiveness. 33 as a B. Assessor Overlap [10] The level of agreement between independent assessors may be quantified by overlap also known as the Jaccard index the number of documents identified as relevant by two independent assessors, divided by the number identified as relevant by either or both assessors. 34 For example, suppose assessor A identifies documents {W,X,Y,Z} as relevant, while assessor B identifies documents {V,W,X}. Both assessors have identified two documents {W,X} as relevant, while either or both have identified five documents {V,W,X,Y,Z} as relevant. So the overlap is 2/5, or forty percent. Informally, overlap of less than fifty percent indicates that the assessors disagree on whether or not a document is relevant more often than when they agree that a document is relevant. 35 [11] In her study, Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, Ellen Voorhees measured overlap between primary, secondary, and tertiary reviewers who each made 14,968 assessments of relevance for 13,435 documents, 36 with respect to See Hedin et al., supra note 9, at Ellen M. Voorhees, Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 36 INFO. PROCESSING & MGMT 697, 700 (2000), available at ( Overlap is defined as the size of the intersection of the relevant document sets divided by the size of the union of the relevant document sets. ); see CHRISTOPHER D. MANNING ET AL., AN INTRODUCTION TO INFORMATION RETRIEVAL 61 (2009) (draft), available at nlp.stanford.edu/ir-book/pdf/irbookonlinereading.pdf; see also Raimundo Real & Juan M. Vargas, The Probabilistic Basis of Jaccard s Index of Similarity, 45 SYSTEMATIC BIOLOGY 380, 381 (1996). 35 See Ellen M. Voorhees, The Philosophy of Information Retrieval Evaluation, in EVALUATION OF CROSS-LANGUAGE INFORMATION RETRIEVAL SYSTEMS SECOND WORKSHOP OF THE CROSS-LANGUAGE EVALUATION FORUM, CLEF 2001 DARMSTADT, GERMANY, SEPTEMBER 3-4, 2001 REVISED PAPERS 355, 364 (Carol Peters et al. eds., 2002). 36 from Ellen M. Voorhees to Gordon V. Cormack (Jul. 31, :34 EDT) (on file with authors). The numbers in the text are derived from the file, 10

405 Richmond Journal of Law and Technology Vol. XVII, Issue 3 information needs (or topics, in TREC parlance), in connection with Ad Hoc Task of the Fourth Text Retrieval Conference ( TREC 4 ). 37 As illustrated in Table 1, the overlap between primary and secondary assessors was 42.1%; 38 the overlap between primary and tertiary assessors was 49.4%; 39 and the overlap between secondary and tertiary assessors was 42.6%. 40 [12] Perhaps due to the assessors expertise, 41 Voorhees overlap results are among the highest reported for pairs of human assessors. Her findings demonstrate that assessors disagree at least as often as they agree that a document is relevant. 42 Voorhees concluded: The scores for the [secondary and tertiary] judgments imply a practical upper bound on retrieval system performance is 65% precision at 65% recall since that is the level at which humans agree with one another. 43 threewayjudgments, attached to Voorhees . Some of the documents were assessed for relevance to more than one topic. 37 Voorhees, supra note 34, at 708; see also Donna Harman, Overview of the Fourth Text REtrieval Conference (TREC-4), in NIST SPECIAL PUBLICATION : THE FOURTH TEXT RETRIEVAL CONFERENCE (TREC-4) 2 (2004), available at (follow the first link under PAPERS ). 38 See infra Table 1; see also Voorhees, supra note 34, at 701 tbl See infra Table 1; see also Voorhees, supra note 34, at 701 tbl See infra Table 1; see also Voorhees, supra note 34, at 701 tbl All assessors were professional information retrieval experts. Voorhees, supra note 34, at See id. 43 Id. 11

406 Richmond Journal of Law and Technology Vol. XVII, Issue 3 [13] It is not widely accepted that these findings apply to e-discovery. 44 This legal exceptionalism appears to arise from common assumptions within the legal community: 1. that the information need (responsiveness or privilege) is more precisely defined for e-discovery than for classical information retrieval; that lawyers are better able to assess relevance and privilege than the non-lawyers typically employed for information retrieval tasks; 46 and 3. that the most defensible way to ensure that a production is accurate is to have a lawyer examine each and every document See Sedona Search Commentary, supra note 1 (noting the widespread perception that manual review is nearly perfect). If that perception were correct, manual reviewers would have close to 100% overlap, contrary to Voorhees findings. Vorhees, supra note 34, at 701 tbl Oard et al., supra note 3, at 362 ( It is important to recognize that the notion of relevance that is operative in E-discovery is, naturally, somewhat more focused than what has been studied in information seeking behavior studies generally.... ). 46 Cf. Alejandra P. Perez, Assigning Non-Attorneys to First-Line Document Reviews Requires Safeguards, THE E-DISCOVERY (LeClairRyan), Jan. 2011, at 1, available at pdf (opining that non-attorney document reviewers typically require additional training, particularly regarding the legal concept of privilege). 47 See Sedona Search Commentary, supra note 1, at 203 ( Some litigators continue to primarily rely upon manual review of information as part of their review process. Principal rationales [include]... the perception that there is a lack of scientific validity of search technologies necessary to defend against a court challenge.... ); see also Thomas E. Stevens & Wayne C. Matus, A Comparative Advantage To Cut E-Discovery Costs, NAT L L.J. (Sept. 4, 2008), NLJ.jsp?id= (describing a general reluctance by counsel to rely on anything but what they perceive to be the most defensible positions in electronic discovery, even if those solutions do not hold up any sort of honest analysis of cost or quality ). 12

407 Richmond Journal of Law and Technology Vol. XVII, Issue 3 Assumptions (1) and (2) are amenable to scientific evaluation, as is the overarching question of whether technology-assisted review can improve upon exhaustive manual review. Assumption (3) a legal opinion should be informed by scientific evaluation of the first two assumptions. Assessment Primary Secondary Tertiary Primary 100% Secondary 42.1% 100% Tertiary 49.4% 42.6% 100% Table 1: Overlap in relevance assessments by primary, secondary, and tertiary assessors for the TREC 4 Ad Hoc Task. 48 [14] Recently, Herbert Roitblat, Anne Kershaw, and Patrick Oot studied the level of agreement among review teams using data produced to the Department of Justice ( DOJ ) in response to a Second Request that stemmed from MCI s acquisition of Verizon. 49 In their study, two independent teams of professional assessors, Teams A and B, reviewed a random sample of 5,000 documents. 50 Roitblat and his colleagues reported the level of agreement and disagreement between the original production, Team A, and Team B, as a contingency matrix, 51 from which the Authors calculated overlap, as shown in Table The overlap between Team A and the original production was 16.3%; 53 the overlap between Team B and the original production was 15.8%; 54 and the overlap between Teams A and B was 28.1%. 55 These and other studies of overlap 48 Voorhees, supra note 34, at 701 tbl See Roitblat et al., supra note 7, at See id. at Id. at 74 tbl See infra Table Id. 54 Id. 55 Id. 13

408 Richmond Journal of Law and Technology Vol. XVII, Issue 3 indicate that relevance is not a concept that can be applied consistently by independent assessors, even if the information need is specified by a production request and the assessors are lawyers. 56 Assessment Production Team A Team B Production 100% Team A 16.3% 100% Team B 15.8% 28.1% 100% Table 2: Overlap in relevance assessments between original production in a Second Request, and two subsequent manual reviews. 57 C. Assessor Accuracy [15] Measurements of overlap provide little information regarding the accuracy of particular assessors because there is no gold standard against which to compare them. 58 One way to resolve this problem is to deem one assessor s judgments correct by definition, and to use those judgments as the gold standard for the purpose of evaluating the other assessor(s). 59 [16] In the Voorhees study, the primary assessor composed the information need specification for each topic. 60 It may therefore be reasonable to take the primary assessor s coding decisions to be the gold standard. In the Roitblat, Kershaw, and Oot study, a senior attorney familiar with the case adjudicated all instances of disagreement between Teams A and B. 61 Although Roitblat and his colleagues sought to 56 See Roitblat et al., supra note 7, at 73; Voorhees, supra note The Authors derived the information in Table 2 from the Roitblat, Kershaw, and Oot study. Roitblat et al., supra note 7, at 74; see supra para Roitblat et al., supra note 7, at See Voorhees, supra note 34, at Id. 61 Roitblat et al., supra note 7, at

409 Richmond Journal of Law and Technology Vol. XVII, Issue 3 measure agreement, 62 it may be reasonable to use their adjudicated results as the gold standard. These adjudicated results deemed the senior attorney s opinion correct in cases where Teams A and B disagreed, and deemed the consensus correct in cases where Teams A and B agreed. 63 Assuming these gold standards, Table 3 shows the effectiveness of the various assessors in terms of recall, precision, and F Note that recall ranges from 52.8% to 83.6%, while precision ranges from 55.5% to 81.9%, and F 1 ranges from 64.0% to 70.4%. 65 All in all, these results appear to be reasonable, but hardly perfect. Can technology-assisted review improve on them? D. Technology-Assisted Review Accuracy [17] In addition to the two manual review groups, Roitblat, Kershaw, and Oot had two service providers (Teams C and D) use technologyassisted review processes to classify each document in the dataset as 62 Id. at 72 ( Formally, the present study is intended to examine the hypothesis: The rate of agreement between two independent reviewers of the same documents will be equal to or less than the agreement between a computer-aided system and the original review. ). 63 Id. at 74. Id. The 1,487 documents on which Teams A and B disagreed were submitted to a senior Verizon litigator (P. Oot), who adjudicated between the two teams, again without knowledge of the specific decisions made about each document during the first review. This reviewer had knowledge of the specifics of the matter under review, but had not participated in the original review. This authoritative reviewer was charged with determining which of the two teams had made the correct decision. 64 See infra Table 3. Recall and precision for the secondary and tertiary assessors, using the primary assessor as the gold standard, are provided by Voorhees, supra note 34, at 701 tbl.2; recall and precision for Teams A and B, using the adjudicated results as the gold standard, were derived from Roitblat et al., supra note 7, at 74 tbl.1; F 1 was calculated from recall and precision using the formula at supra note See infra Table 3. 15

410 Richmond Journal of Law and Technology Vol. XVII, Issue 3 relevant or not. 66 Unfortunately, the adjudicated results described in Part II.C. were made available to one of the two service providers, and therefore, cannot be used as a gold standard to evaluate the accuracy of the providers efforts. 67 Study Review Recall Precision F 1 Voorhees Secondary 52.8% 81.3% 64.0% Voorhees Tertiary 61.8% 81.9% 70.4% Roitblat et al. Team A 77.1% 60.9% 68.0% Roitblat et al. Team B 83.6% 55.5% 66.7% Table 3: Recall, precision, and F 1 of manual assessments in studies by Voorhees, and Roitblat et al. Voorhees evaluated secondary and tertiary assessors with respect to a primary assessor, who was deemed correct. The Authors computed recall, precision, and F 1 from the results reported by Roitblat et al. for Teams A and B, using their adjudicated results as the gold standard. 68 [18] Instead, Roitblat and his colleagues reported recall, precision, and F 1 using, as an alternate gold standard, the set of documents originally produced to, and accepted by, the DOJ. 69 There is little reason to believe that this original production, and hence the alternate gold standard, was perfect. 70 The first two rows of Table 4 show the recall and precision of manual review Teams A and B when evaluated with respect to this 66 Roitblat et al., supra note 7, at Id. at 74 ( One of these systems based its classifications in part on the adjudicated results of Teams A and B, but without any knowledge of how those teams decisions were related to the decisions made by [the] original review team. As a result, it is not reasonable to compare the classifications of these two systems to the classifications of the two re-review teams, but it is reasonable to compare them to the classifications of the original review. ). 68 Voorhees, supra note 34, at 701 tbl.2; Roitblat et al. supra note 7, at 74 tbl Roitblat et al., supra note 7, at Id. at 76 ( The use of precision and recall implies the availability of a stable ground truth against which to compare the assessments. Given the known variability of human judgments, we do not believe that we have a solid enough foundation to claim that we know which documents are truly relevant and which are not. ). 16

411 Richmond Journal of Law and Technology Vol. XVII, Issue 3 alternate gold standard. 71 These results are much worse than those in Table Team A achieved 48.8% recall and 19.7% precision, while Team B achieved 52.9% recall and 18.3% precision. 73 The corresponding F 1 scores were 28.1% and 27.2%, respectively less than half of the F 1 scores achieved with respect to the gold standard derived using the senior attorney s opinion. 74 [19] The recall and precision Roitblat, Kershaw, and Oot reported were computed using the original production as the gold standard, and are dramatically different from those shown in Table 3, which were computed using their adjudicated results as the gold standard. 75 Nevertheless, both sets of results appear to suggest the relative accuracy between Teams A and B: Team B has higher recall, while Team A has higher precision and higher F 1, regardless of which gold standard is applied. 76 [20] The last two rows of Table 4 show the effectiveness of the technology-assisted reviews conducted by teams C and D, as reported by Roitblat, Kershaw, and Oot using the original production as the gold standard. 77 The results suggest that technology-assisted review Teams C and D achieved about the same recall as manual review Teams A and B, and somewhat better precision and F However, due to the use of the alternate gold standard, the result is inconclusive. 79 Because the 71 See id. at 76 tbl.2; infra Table Compare supra Table 3, with infra Table See infra Table 4; see also Roitblat et al., supra note 7, at Compare supra Table 3, with infra Table Compare supra Table 3, with infra Table 4. See generally Roitblat et al., supra note 7, at 76 tbl See supra Table 3; infra Table 4; Roitblat et al., supra note 7, at 76 tbl See infra Table 4; see also Roitblat et al., supra note 7, at See infra Table See Roitblat et al., supra note 7, at 76 ( The use of precision and recall implies the availability of a stable ground truth against which to compare the assessments. Given the 17

412 Richmond Journal of Law and Technology Vol. XVII, Issue 3 improvement from using technology-assisted review, as reported by Roitblat and his colleagues, is small compared to the difference between the results observed using the two different gold standards, it is difficult to determine whether the improvement represents a real difference in effectiveness as compared to manual review. Study Review Method Recall Precision F 1 Roitblat et al. Team A Manual 48.8% 19.7% 28.1% Roitblat et al. Team B Manual 52.9% 18.3% 27.2% Roitblat et al. Team C Tech. Asst. 45.8% 27.1% 34.1% Roitblat et al. Team D Tech. Asst. 52.7% 29.5% 37.8% Table 4: Recall, precision, and F 1 of manual and technology-assisted review teams, evaluated with respect to the original production to the DOJ. The first two rows of this table differ from the last two rows of Table 3 only in the gold standard used for evaluation. 80 [21] In a heavily cited study by David C. Blair and M.E. Maron, skilled paralegal searchers were instructed to retrieve at least 75% of all documents relevant to 51 requests for information pertaining to a legal matter. 81 For each request, the searchers composed keyword searches using an interactive search system, retrieving and printing documents for further review. 82 This process was repeated until the searcher was satisfied that 75% of the relevant documents had been retrieved. 83 Although the searchers believed they had found 75% of the relevant documents, their average recall was only 20.0%. 84 Despite this low rate of known variability of human judgments, we do not believe that we have a solid enough foundation to claim that we know which documents are truly relevant and which are not. ). 80 Id. at See Blair & Maron, supra note 23, at Id. 83 Id. 84 Id. at 293; see also Maureen Dostert & Diane Kelly, Users Stopping Behaviors and Estimates of Recall, in SIGIR 09 PROCEEDINGS OF THE 32ND ANNUAL INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL (2009) (showing that most subjects in an interactive information 18

413 Richmond Journal of Law and Technology Vol. XVII, Issue 3 recall, the searchers achieved a high average precision of 79.0%. 85 From the published data, 86 the Authors calculated the average F 1 score to be 28.0% remarkably similar to that observed by Roitblat and his colleagues for manual review. 87 [22] Blair and Maron argue that the searchers would have been unable to achieve higher recall even if they had known there were many relevant documents that were not retrieved. 88 Researcher Gerald Salton disagrees. 89 He claims that it would have been possible for the searchers to achieve higher recall at the expense of lower precision, either by broadening their queries or by taking advantage of the relevance ranking capability of the search system. 90 [23] Overall, the literature offers little reason to believe that manual review is perfect. But is it as complete and accurate as possible, or can it be improved upon by technology-assisted approaches invented since Blair and Maron s study? [24] As previously noted, recent results from TREC suggest that technology-assisted approaches may indeed be able to improve on manual review. 91 In the TREC 2008 Legal Track Interactive Task, H5, a San retrieval experiment reported they had found about 51-60% of the relevant documents when, on average, recall was only 7%). 85 See Blair & Maron, supra note 23, at Id. 87 See Roitblat et al., supra note 7 at See Blair & Maron, supra note 23, at See Gerard Salton, Another Look at Automatic Text-Retrieval Systems, 29:7 COMMC NS ACM 648, 650 (1986). 90 Id. at See generally Hedin et al., supra note 9; Oard et al., supra note 9. 19

414 Richmond Journal of Law and Technology Vol. XVII, Issue 3 Francisco-based legal information retrieval firm, 92 employed a usermodeling approach 93 to achieve recall, precision, and F 1 of 62.4%, 81.0%, and 70.5%, respectively, in response to a mock request to produce documents from a 6,910,192-document collection released under the tobacco Master Settlement Agreement. 94 In the course of this effort, H5 examined only 7,992 documents 95 roughly 860 times fewer than the 6,910,192 it would have been necessary to examine in an exhaustive manual review. Yet the results compare favorably with those previously reported for manual review or keyword search, exceeding what Voorhees characterizes as a practical upper bound on what may be achieved, given uncertainties in assessment See Contact Us, H5, (last visited Mar. 22, 2011); Who We Are, H5, (last visited Apr. 11, 2011). 93 Christopher Hogan et al., H5 at TREC 2008 Legal Interactive: User Modeling, Assessment & Measurement, in NIST SPECIAL PUBLICATION: SP , THE SEVENTEENTH TEXT RETRIEVAL CONFERENCE (TREC 2008) PROCEEDINGS (2008), available at h5.legal.rev.pdf (last visited Mar. 23, 2011). 94 Oard et al., supra note 9, at 30 tbl.15; see also Complex Document Image Processing (CDIP), ILL. INST. TECH., CDIP.html (last visited Apr. 11, 2011); Master Settlement Agreement, NAT L ASS N ATTORNEYS GEN. (Nov. 1998), available at Pages%20and%20Exhibits.pdf; TREC 2008, Complaint for Violation of the Federal Securities Laws, Mellon v. Echinoderm Cigarettes, Inc., (2008), available at 95 Hogan et al., supra note 92, at Voorhees, supra note 34, at

415 Richmond Journal of Law and Technology Vol. XVII, Issue 3 Topic Production Request 201 All documents or communications that describe, discuss, refer to, report on, or relate to the Company s engagement in structured commodity transactions known as prepay transactions. 202 All documents or communications that describe, discuss, refer to, report on, or relate to the Company s engagement in transactions that the Company characterized as compliant with FAS 140 (or its predecessor FAS 125). 203 All documents or communications that describe, discuss, refer to, report on, or relate to whether the Company had met, or could, would, or might meet its financial forecasts, models, projections, or plans at any time after January 1, All documents or communications that describe, discuss, refer to, report on, or relate to any intentions, plans, efforts, or activities involving the alteration, destruction, retention, lack of retention, deletion, or shredding of documents or other evidence, whether in hard-copy or electronic form. 205 All documents or communications that describe, discuss, refer to, report on, or relate to energy schedules and bids, including but not limited to, estimates, forecasts, descriptions, characterizations, analyses, evaluations, projections, plans, and reports on the volume(s) or geographic location(s) of energy loads. 206 All documents or communications that describe, discuss, refer to, report on, or relate to any discussion(s), communication(s), or contact(s) with financial analyst(s), or with the firm(s) that employ them, regarding (i) the Company s financial condition, (ii) analysts coverage of the Company and/or its financial condition, (iii) analysts rating of the Company s stock, or (iv) the impact of an analyst s coverage of the Company on the business relationship between the Company and the firm that employs the analyst. 207 All documents or communications that describe, discuss, refer to, report on, or relate to fantasy football, gambling on football, and related activities, including but not limited to, football teams, football players, football games, football statistics, and football performance. Table 5: Mock production requests ( topics ) composed for the TREC 2009 Legal Track Interactive Task TREC 2009, Complaint, Grumby v. Volteron Corp., 14 (2009) available at _J_final.pdf; see also Hedin et al., supra note 9, at

416 Richmond Journal of Law and Technology Vol. XVII, Issue 3 [25] One of the Authors was inspired to try to reproduce these results at TREC 2009 using an entirely different approach: statistical active learning, originally developed for spam filtering. 98 At the same time, H5 reprised its approach for TREC The TREC 2009 Legal Track Interactive Task used the same design as TREC 2008, but employed a different complaint 100 and seven new mock requests to produce documents (see Table 5) from a new collection of 836,165 messages and attachments captured from Enron at the time of its collapse. 101 Each participating team was permitted to request as many topics as they wished, however, due to resource constraints, the most topics that any team was assigned was four of the seven See generally Gordon V. Cormack & Mona Mojdeh, Machine Learning for Information Retrieval: TREC 2009 Web, Relevance Feedback and Legal Tracks, in NIST SPECIAL PUBLICATION: SP , THE EIGHTEENTH TEXT RETRIEVAL CONFERENCE (TREC 2009) PROCEEDINGS (2009), available at trec18/papers/uwaterloo-cormack.web.rf.legal.pdf. 99 Hedin et al., supra note 9, at See generally TREC 2009, Complaint, supra note Hedin et al., supra note 9, at 4; see Information Released in Enron Investigation, FED. ENERGY REG. COMM N, info-release.asp (last visited Apr. 11, 2011) [hereinafter FERC]; from Bruce Hedin to Gordon V. Cormack (Aug. 31, :33 EDT) (on file with authors) ( I have attached full list of the 836,165 document-level IDs.... ). The collection is available at Practice Topic and Assessments for TREC 2010 Legal Learning Task, U. WATERLOO, (follow The TREC 2009 dataset ) (last visited Apr. 18, 2011). 102 Hedin et al., supra note 9, at 7; from Bruce Hedin to Gordon V. Cormack & Maura R. Grossman (Mar. 24, :46 EDT) (on file with authors). 22

417 Richmond Journal of Law and Technology Vol. XVII, Issue 3 Team Topic Reviewed Produced Recall Precision F 1 Waterloo 201 6,145 2, % 91.2% 84.0% Waterloo ,646 8, % 88.4% 76.4% Waterloo 203 4,369 2, % 69.2% 76.9% H ,000 2, % 84.4% 80.1% Waterloo ,446 23, % 90.7% 82.8% Average: 15,521 7, % 84.7% 80.0% Table 6: Effectiveness of H5 and Waterloo submissions to the TREC 2009 Legal Track Interactive Task. 103 [26] Together, H5 and Waterloo produced documents for five distinct TREC 2009 topics; 104 the results of their efforts are summarized in Table 6. The five efforts employed technology-assisted processes, with the number of manually reviewed documents for each topic ranging from 4,369 to 34, (or 0.5% to 4.1% of the collection). That is, the total human effort for the technology-assisted processes measured by the number of documents reviewed was between 0.5% and 4.1% of that which would have been necessary for an exhaustive manual review of all 836,165 documents in the collection. 106 The number of documents produced for each topic ranged from 2,154 to 23, (or 0.3% to 2.8% of the collection; about half the number of documents reviewed). Over the five efforts, the average recall and precision were 76.7% and 84.7%, 103 See infra, para See Hedin et al., supra note 9, at Cormack & Mojdeh, supra note 98, at 6 tbl.2 (showing that Waterloo reviewed between 4,369 documents (for Topic 203) and 34,446 documents (for Topic 207); see E- mail from Dan Brassil to Maura R. Grossman (Dec. 17, :21 EST) (on file with authors) ( [H5] sampled and reviewed 20,000 documents ). 106 See sources cited supra note NIST Special Publication : The Seventeenth Text REtrieval Conference Proceedings (TREC 2008) Appendix: Per Topic Scores: TREC 2009 Legal Track, Interactive Task, 3 tbl.4, 4 tbl.8, 5 tbl.12, 6 tbl.16, 9 tbl.26 app09int2.pdf. 23

418 Richmond Journal of Law and Technology Vol. XVII, Issue 3 respectively; no recall was lower than 67.3%, and no precision was lower than 69.2%, 108 placing all five efforts above what Voorhees characterized as a practical upper bound on what may be achieved, given uncertainties in assessment. 109 [27] Although it appears that the TREC results are better than those previously reported in the literature, either for manual or technologyassisted review, they do not include any direct comparison between manual and technology-assisted review. 110 To draw any firm conclusion that one is superior to the other, one must compare manual and technology-assisted review efforts using the same information needs, the same dataset, and the same evaluation standard. 111 The Roitblat, Kershaw, and Oot study is the only peer-reviewed study known to the Authors suggesting that technology-assisted review may be superior to manual review if only in terms of precision, and only by a small amount based on a common information need, a common dataset, and a common gold standard, albeit one of questionable accuracy. 112 [28] This Article shows conclusively that the H5 and Waterloo efforts are superior to manual reviews conducted contemporaneously by TREC assessors, using the same topics, the same datasets, and the same gold standard. The manual reviews considered for this Article were the First- Pass Assessments undertaken at the request of the TREC coordinators for 108 See Hedin et al, supra note 9, at Voorhees, supra note 34, at See e.g., Oard et al., supra note 9, at See Voorhees, supra note 35, at 356 ( The [Cranfield] experimental design called for the same set of documents and same set of information needs to be used for each [search method], and for the use of both precision and recall to evaluate the effectiveness of the search. ). 112 See Roitblat et al., supra note 7, at 76 ( The use of precision and recall implies the availability of a stable ground truth against which to compare the assessments. Given the known variability of human judgments, we do not believe that we have a solid enough foundation to claim that we know which documents are truly relevant and which are not. ). 24

419 Richmond Journal of Law and Technology Vol. XVII, Issue 3 the purpose of evaluating the participating teams submissions. 113 In comparing the manual and technology-assisted reviews, the Authors used exactly the same adjudicated gold standard as TREC. 114 III. TREC Legal Track Interactive Task [29] TREC is an annual event hosted by NIST, with the following objectives: to encourage research in information retrieval based on large test collections; to increase communication among industry, academia, and government by creating an open forum for the exchange of research ideas; to speed the transfer of technology from research labs into commercial products by demonstrating substantial improvements in retrieval methodologies on real-world problems; and to increase the availability of appropriate evaluation techniques for use by industry and academia, including development of new evaluation techniques more applicable to current systems. 115 Since its inception in 2006, 116 the TREC Legal Track has had the goal to develop search technology that meets the needs of lawyers to engage in effective discovery in digital document collections Hedin et al., supra note 9, at 3 (describing the First-Pass Assessment process). 114 See id. at Text REtrieval Conference (TREC), Overview, NAT L INST. STANDARDS & TECH., (last updated Aug. 10, 2010). 116 See Jason R. Baron, The TREC Legal Track: Origins and Reflections on the First Year, 8 SEDONA CONF. J. 251, 253 (2007); see also Jason R. Baron et al., TREC-2006 Legal Track Overview, in NIST SPECIAL PUBLICATION: SP , THE FIFTEENTH TEXT RETRIEVAL CONFERENCE (TREC 2006) PROCEEDINGS 1-2 (2006), available at 25

420 Richmond Journal of Law and Technology Vol. XVII, Issue 3 [30] Within the TREC Legal Track, the Interactive Task simulates the process of review of a large population of documents for responsiveness to one or more discovery requests in a civil litigation. 118 In 2008, the first year of the Interactive Task, 119 the population of documents used was the Illinois Institute of Technology Complex Document Information Processing Test Collection, version 1.0 ( IIT CDIP ), 120 consisting of about seven million documents that were released in connection with various lawsuits filed against certain U.S. tobacco companies and affiliated research institutes. 121 A mock complaint and three associated requests for production (or topics) were composed for the purposes of the Interactive Task. 122 Participating teams were required to produce the responsive documents for one or more of the three requests. 123 [31] The population of documents used for TREC 2009 consisted of messages and attachments that Enron produced in response to requests by FERC. 124 A mock complaint and seven associated requests for production were composed for the purposes of TREC Participating teams requested as many topics as they desired to undertake, but time and cost constraints limited the number of topics that any team was assigned to a maximum of four Text Retrieval Conference (TREC), TREC Tracks, NAT L INST. STANDARDS & TECH., (last updated Feb. 24, 2011). 118 See Oard et al., supra note 9, at See id. at Id. at 3; see Complex Document Image Processing (CDIP), supra note See Oard et al., supra note 9, at 3; Complex Document Image Processing (CDIP), supra note See Oard et al., supra note 9 at 3, Id. at See Hedin et al., supra note 9, at 4; see also FERC, supra note See Hedin et al., supra note 9, at See id. at 7 tbl.1. 26

421 Richmond Journal of Law and Technology Vol. XVII, Issue 3 [32] Aside from the document collections, the mock complaints, and the production requests, the conduct of the 2008 and 2009 Interactive Tasks was identical. 127 Participating teams were given the document collection, the complaint, and the production requests several weeks before production was due. 128 Teams were allowed to use any combination of technology and human input; the exact combination differed from team to team. 129 However, the size of the document population, along with time and cost constraints, rendered it infeasible for any team to conduct an exhaustive review of every document. 130 To the Authors knowledge, no team examined more than a small percentage of the document population; H5 and Waterloo, in particular, used various combinations of computer search, knowledge engineering, machine learning, and sampling to select documents for manual review. 131 [33] To aid the teams in their efforts, as well as to render an authoritative interpretation of responsiveness (or relevance, within the context of TREC), a volunteer Topic Authority ( TA ) a senior attorney familiar with the subject matter was assigned for each topic. 132 The TA played three critical roles: to consult with the participating teams to clarify the notion of relevance, in a manner chosen by the teams; 127 See id. at See Text Retrieval Conference (TREC), TREC-2008 Legal Track Interactive Task: Guidelines, 8, 17 (2008), trec-legal.umiacs.umd.edu/2008interactiveguidelines.pdf [hereinafter TREC-2008 Guidelines]; see also from Dan Brassil to Maura R. Grossman, supra note TREC-2008 Guidelines, supra note 128, at 4, 7; see also from Bruce Hedin to Gordon V. Cormack (Apr. 07, :56 EDT) (confirming that teams were permitted to use any combination of technology and human input). 130 See TREC-2008 Legal Track Interactive Task: Guidelines, supra note 128, at See Hogan et al., supra note 9, at 5; Cormack & Mojdeh, supra note 98, at See Hedin et al., supra note 9, at 2. 27

422 Richmond Journal of Law and Technology Vol. XVII, Issue 3 to prepare a set of written guidelines used by the human reviewers to evaluate, after the fact, the relevance of documents produced by the teams; and to act as a final arbiter of relevance in the adjudication process. 133 [34] The TREC coordinators evaluated the various participant efforts using estimates of recall, precision, and F 1 based on a two-pass human assessment process. 134 In the first pass, human reviewers assessed a stratified sample of about 7,000 documents for relevance. 135 For some topics (Topics 201, 202, 205, and 206), the reviewers were primarily volunteer law students supervised by the TREC coordinators; for others (Topics 203, 204, and 207), the reviewers were lawyers employed and supervised by professional document-review companies, who volunteered their services. 136 [35] The TREC coordinators released the first-pass assessments to participating teams, which were invited to appeal relevance determinations with which they disagreed. 137 For each topic, the TA adjudicated the appeals, and the TA s opinion was deemed to be correct and final. 138 The gold standard of relevance for the documents in each sample was therefore: The same as the first-pass assessment, for any document that participants did not appeal; or 133 Id. at 2-3; see Oard et al., supra note 9, at Hedin et al., supra note 9, at See id. at Id. at Id. at Id. 28

423 Richmond Journal of Law and Technology Vol. XVII, Issue 3 The TA s opinion, for any document that participants did appeal. The TREC coordinators used statistical inference to estimate recall, precision, and F 1 for the results each participating team produced. 139 [36] Assuming participants diligently appealed the first-pass assessments with which they disagreed, it is reasonable to conclude that TREC s two-pass assessment process yields a reasonably accurate gold standard. Moreover, that same gold standard is suitable to evaluate not only the participants submissions, but also the first-pass assessments of the human reviewers. 140 [37] Parts III.A and III.B briefly describe the processes employed by the two participants whose results this Article compares to manual review. Notably, the methods the two participants used differ substantially from those typically described in the industry as clustering or concept search. 141 A. H5 Participation [38] At TREC 2009, H5 completed one topic (Topic 204). 142 According to Dan Brassil of H5, the H5 process involves three steps: (i) definition of relevance, (ii) partly-automated design of deterministic queries, and (iii) measurement of precision and recall. 143 Once relevance is defined, the two remaining processes of (1) sampling and query design and (2) measurement of precision and recall are conducted 139 Id. at 3, See Hedin et al., supra note 9, at 13 (describing the construction of the gold standard). 141 Sedona Search Commentary, supra note 1, at Hedin et al., supra note 9, at from Dan Brassil to Maura R. Grossman, supra note

424 Richmond Journal of Law and Technology Vol. XVII, Issue 3 iteratively allowing for query refinement and correction until the clients accuracy requirements are met. 144 [39] H5 describes how its approach differs from other information retrieval methods as follows: It utilizes an iterative issue-focusing and datafocusing methodology that defines relevancy in detail; most alternative processes provide a reductionist view of relevance (e.g.: a traditional coding manual), or assume that different individuals share a common understanding of relevance. [H5 s approach] is deterministic: each document is assessed against the relevance criteria and a relevant / not relevant determination is made.... [The approach] is built on precision: whereas many alternative approaches start with a small number [of] keywords intended to be broad so as to capture a lot of relevant data (with the consequence of many false positives), H5 s approach is focused on developing in an automated or semi-automated fashion large numbers of deterministic queries that are very precise: each string may capture just a few documents, but nearly all documents so captured will be relevant; and all the strings together will capture most relevant documents in the collection. 145 In the course of its TREC 2009 effort, H5 sampled and reviewed a total of 20,000 documents. 146 H5 declined to quantify the number of person-hours 144 Id. 145 Id. (citing Dan Brassil et al., The Centrality of User Modeling to High Recall with High Precision Search, in 2009 IEEE Int l Conf. on Systems, Man, and Cybernetics, 91, Id. 30

425 Richmond Journal of Law and Technology Vol. XVII, Issue 3 it expended during the seven to eight week time period between the assignment of the topic and the final submission date. 147 B. Waterloo Participation [40] The University of Waterloo ( Waterloo ) completed four topics (Topics 201, 202, 203, and 207). 148 Waterloo s approach consisted of three phases: (i) interactive search and judging, (ii) active learning, and (iii) recall estimation. 149 The interactive search and judging phase used essentially the same tools and approach [Waterloo] used in TREC Waterloo coupled the Wumpus search engine 151 to a custom web interface that provided document excerpts and permitted assessments to be coded with a single mouse click. 152 Over the four topics, roughly 12,500 documents were retrieved and reviewed, at an average rate of about 3 documents per minute (about 22 seconds per document; 76 hours in 147 Id.; from Dan Brassil to Maura R. Grossman (Feb. 16, :58 EST) (on file with authors). 148 Cormack & Mojdeh, supra 98, at Id. at Id. at 2. See generally, Gordon V. Cormack et al., Efficient Construction of Large Test Collections, in SIGIR 98 PROCEEDINGS OF THE 21ST ANNUAL INTERNATIONAL ACM SIGIR CONFERENCE ON RESEARCH AND DEVELOPMENT IN INFORMATION RETRIEVAL 282, 284 (1998). 151 Welcome to the Wumpus Search Engine!, WUMPUS, (last visited Apr. 11, 2011). 152 See Cormack & Mojdeh, supra note 98, at 3 & fig.2; see also infra Figure 1. We used the Wumpus search engine and a custom html interface that showed hits-in-context and radio buttons for adjudication.... Available for reference were links to the full text of the document and to the full message containing the document, including attachments in their native format. Cormack & Mojdeh, supra note 98, at 3. 31

426 Richmond Journal of Law and Technology Vol. XVII, Issue 3 total). 153 Waterloo used the resulting assessments to train an on-line active learning system, previously developed for spam filtering. 154 [41] The active learning system yields an estimate of the [probability] that each document is relevant. 155 Waterloo developed an efficient user interface to review documents selected by this relevance score (see Figure 2). 156 The primary approach was to examine unjudged documents in decreasing order of score, skipping previously adjudicated documents. 157 The process displayed each document as text and, using a single keystroke, coded each document as relevant or not relevant. 158 Among the four topics, [a]bout 50,000 documents were reviewed, at an average rate of 20 documents per minute (3 seconds per document) or 42 hours in total. 159 From time to time, [Waterloo] revisited the interactive search and judging system, to augment or correct the relevance assessments as new information came to light from Gordon V. Cormack to K. Krasnow Waterman (Feb. 24, :25 EST) (on file with authors) (indicating that 12,508 documents were reviewed at a rate of 22 seconds per document, i.e., hours in total). 154 Cormack & Mojdeh, supra note 98, at Id. at Id. 157 Id. 158 Id. 159 Cormack & Mojdeh, supra note 98, at Id. 32

427 Richmond Journal of Law and Technology Vol. XVII, Issue 3 Figure 1: Waterloo s interactive search and judging interface. 161 [42] The third and final phase estimated the density of relevant documents as a function of the score assigned by the active learning system, based on the assessments rendered during the active learning phase. 162 Waterloo used this estimate to gauge the tradeoff between recall and precision, and to determine the number of documents to produce so as to optimize F 1, as required by the task guidelines Id. at 3 & fig See id. at Id. at 3, 6; see Hedin et al., supra note 9, at 3. 33

428 Richmond Journal of Law and Technology Vol. XVII, Issue 3 [43] For Waterloo s TREC 2009 effort, the end result was that a human reviewed every document produced; 164 however, the number of documents reviewed was a small fraction of the entire document population (14,396 of the 836,165 documents were reviewed, on average, per topic). 165 Total review time for all phases was about 118 hours; 30 hours per topic, on average. 166 Figure 2: Waterloo s minimalist review interface See Cormack & Mojdeh supra note 98, at 6 ( the optimal strategy was to include no unassessed documents ). 165 Id., at 6 tbl.2; from Bruce Hedin to Gordon V. Cormack, supra note 101 ( I have attached full list of the 836,165 document-level IDs ) hours is the sum of 76 hours for the interactive search and judging phase (supra para. 39) and 42 hours for the active learning phase (supra para. 41). Since Waterloo did four topics, the average effort per topic was 29.5 hours. 167 Cormack & Mojdeh, supra note 98, at 4 fig.3. 34

429 Richmond Journal of Law and Technology Vol. XVII, Issue 3 IV. QUANTITATIVE ANALYSIS [44] This Article s purpose is to refute the hypothesis that manual review is the best approach by showing that technology-assisted review can yield results that are more nearly complete and more accurate than exhaustive manual review, as measured by recall, precision, and F 1. To compare technology-assisted to manual review, the study required: 1. The results of one or more technology-assisted reviews. For this purpose, the Authors used the H5 review and the four Waterloo reviews conducted during the course of their participation in the TREC 2009 Legal Track Interactive Task The results of manual reviews for the same topics and datasets as the technology-assisted reviews. For this purpose, the Authors used the manual reviews that TREC conducted on document samples for the purpose of evaluating the results that the participating teams submitted A gold standard determination of relevance or nonrelevance. For this purpose, the Authors used the TREC final adjudicated assessments, for which the TA was the ultimate arbiter. 170 [45] The Authors evaluated the results of the technology-assisted reviews and the manual reviews in exactly the same manner, using the 168 The TREC results are available online, but use, dissemination and publication of the material is limited. Text REtrieval Conference (TREC), Past Results, NAT L INST. STANDARDS & TECH., (last visited Apr. 11, 2011) ( Individuals may request access to the protected area containing the raw results by contacting the TREC Program Manager. Before receiving access, individuals will be asked to sign an agreement that acknowledges the limited uses for which the data can be used. ). 169 Text REtrieval Conference (TREC), Relevance Judgments and Evaluation Tools for the Interactive Task, NAT L INST. STANDARDS & TECH., (last visited Apr. 11, 2011). 170 Id.; see Hedin et al., supra note 9, at

430 Richmond Journal of Law and Technology Vol. XVII, Issue 3 TREC methodology and the TREC gold standard. 171 To compare the effectiveness of the reviews, this Article reports, for each topic: 1. Recall, precision, and F 1 for both the technology-assisted and manual reviews The difference in recall, the difference in precision, and the difference in F 1 between the technology-assisted and manual reviews The significance of the difference for each measure, expressed as P. 174 Traditionally, P < 0.05 is interpreted to mean that the difference is statistically significant; P > 0.1 is interpreted to mean that the measured difference is not statistically significant. Smaller values of P imply stronger significance; P < indicates overwhelming significance. 175 The Authors used 100 bootstrap samples of paired differences to estimate the standard error of measurement, assuming a two-tailed normal distribution, to compute P. 176 Table 7 shows recall, precision, and F 1 for the technology-assisted and manual reviews for each of the five topics, as well as the overall average for the five technology-assisted reviews and the five manual reviews. For brevity, the difference in each measure is not shown, but is easily 171 See Hedin et al., supra note 9, at See id. at 3 (reporting recall, precision, and F 1 for TREC participants); infra Table 7 (reporting recall, precision, and F 1 for the TREC manual reviews). 173 See infra Table 7. A positive difference in some measure indicates that the technology-assisted review is superior in that measure, while a negative difference indicates that it is inferior. 174 BÜTTCHER ET AL., supra note 19, at See id. 176 See id. at The bootstrap... is a method for simulating an empirical distribution modeling f (S) by sampling the sample s. ). Id. at

431 Richmond Journal of Law and Technology Vol. XVII, Issue 3 computed from the table. For example, for Topic 201, the difference in recall between Waterloo and TREC is 77.8% 75.6% = +2.2%. Topic Team Recall Precision F Waterloo TREC (Law Students) ( ) 77.8% 75.6% (*) 91.2% 5.0% (*) 84.0% 9.5% 202 Waterloo TREC (Law Students) 67.3% ( ) 79.9% (*) 88.4% 26.7% (*) 76.4% 40.0% 203 Waterloo TREC (Professionals) (*) 86.5% 25.2% (*) 69.2% 12.5% (*) 76.9% 16.7% 204 H5 TREC (Professionals) (*) 76.2% 36.9% (*) 84.4% 25.5% (*) 80.1% 30.2% 207 Waterloo TREC (Professionals) 76.1% ( ) 79.0% ( ) 90.7% 89.0% 82.8% ( ) 83.7% Avg. H5/Waterloo TREC ( ) 76.7% 59.3% (*) 84.7% 31.7% (*) 80.0% 36.0% Table 7: Effectiveness of TREC 2009 Legal Track technology-assisted approaches (H5 and Waterloo) compared to exhaustive manual reviews (TREC). Results marked (*) are superior and overwhelmingly significant (P < 0.001). Results marked ( ) are superior but not statistically significant (P > 0.1). 177 [46] For each topic and each measure, the larger value is marked with either (*) or ( ); (*) indicates that the measured difference is overwhelmingly significant (P < 0.001), while ( ) indicates that it is not statistically significant (P > 0.1). As Table 7 illustrates, all of the measured differences are either overwhelmingly significant or not statistically significant. 178 V. QUALITATIVE ANALYSIS [47] The quantitative results show that the recall of the manual reviews varies from about 25% (Topic 203) to about 80% (Topic 202). That is, human assessors missed between 20% and 75% of all relevant documents. 179 Is this shortfall the result of clerical error, a 177 For the information contained in this table, see Past Results, supra note 168; Relevance Judgments and Evaluation Tools for the Interactive Task, supra note 169. For details on the calculation and meaning of P, see supra para Supra Table See supra Table 7. 37

432 Richmond Journal of Law and Technology Vol. XVII, Issue 3 misinterpretation of relevance, or disagreement over marginal documents whose responsiveness is debatable? If the missed documents are marginal, the shortfall may be of little consequence; but if the missed documents are clearly responsive, production may be inadequate, and under Rule 37(a)(4), such a production could constitute a failure to respond. 180 [48] To address this question, the Authors examined the documents that the TREC assessors coded as nonresponsive to Topics 204 and 207, but H5 or Waterloo coded as responsive, and the TA adjudicated as responsive. Recall from Table 5 that Topic 204 concerned shredding and destruction of documents, while Topic 207 concerned football and gambling. The Authors chose these topics because they were more likely to be easily accessible to the reader, as opposed to other topics, which were more technical in nature. In addition, lawyers employed by professional review companies assessed these two topics using accepted practices for manual review. 181 [49] For Topic 204, 160 of the assessed documents were coded as nonresponsive by the manual reviewers and responsive by H5 and the TA; 182 Topic 207, 51 documents met these same criteria except that Waterloo and the TA made the responsiveness determinations. 183 From these numbers, the Authors extrapolated that the manual reviewers would 180 See FED. R. CIV. P. 37(a)(4). 181 See Hedin et al., supra note 9, at 8 ( The review of the samples for three of the seven Interactive topics (203, 204, and 207) was carried out by two firms that include professional document-review services among their offerings. ). 182 The Authors identified these documents by comparing the submitted results, see Past Results, supra note 168 (file input.h52009.gz), the first-pass assessments, see Relevance Judgments and Evaluation Tools for the Interactive Task, supra note 169 (file qrels_doc_pre_all.txt), and the final adjudicated results, see id. (file qrels_doc_post_all.txt). 183 The Authors identified these documents by comparing the submitted results, see Past Results, supra note 168 (file input.watlint.gz), the first-pass assessments, see Relevance Judgments and Evaluation Tools for the Interactive Task, supra note 169 (file qrels_doc_pre_all.txt), and the final adjudicated results, see id. (file qrels_doc_post_all.txt). 38

433 Richmond Journal of Law and Technology Vol. XVII, Issue 3 have missed 1,918 and 1,273 responsive documents (for Topics 204 and 207, respectively), had they reviewed the entire document collection. [50] For each of these documents, the Authors used their judgment to assess whether the document had been miscoded due to: Inarguable error: Under any reasonable interpretation of relevance, the reviewer should have coded the document as responsive, but did not. Possible reasons for such error include fatigue or inattention, overlooking part of the document, poor comprehension, or data entry mistakes in coding the document. 184 For example, a document about shredding (see Figure 3) is responsive on its face to Topic 204; similarly Fantasy Football (see Figure 4) is responsive on its face to Topic 207. Date: Tuesday, January 22, :31:39 GMT Subject: I m in. I ll be shredding till 11am so I should haveplenty of time to make it. Figure 3: Topic 204 Inarguable error. A professional reviewer coded this document as nonresponsive, although it clearly pertains to document shredding, as specified in the production request Cf. Jeremy M. Wolfe et al., Low Target Prevalence Is a Stubborn Source of Errors in Visual Search Tasks, 136 J. EXPERIMENTAL PSYCH. 623, (2007) (showing that in visual search tasks, humans have much higher error rates when the prevalence of target items is low). 185 See supra Table 5. Figure 3 is an excerpt from document in the TREC 2009 dataset, supra note

434 Richmond Journal of Law and Technology Vol. XVII, Issue 3 From: Bass, Eric Sent: Thursday, January 17, :19 AM To: Lenhart, Matthew Subject: FFL Dues You owe $80 for fantasy football. When can you pay? Figure 4: Topic 207 Inarguable error. A professional reviewer coded this document as nonresponsive, although it clearly pertains to fantasy football, as specified in the production request. 186 Interpretive error: Under some reasonable interpretation of relevance but not the TA s interpretation as provided in the topic guidelines an assessor might consider the document as nonresponsive. For example, a reviewer might have construed an automated message stating, your mailbox is nearly full; please delete unwanted messages (see Figure 5) as nonresponsive to Topic 204, although the TA defined it as responsive. Similarly, an assessor might have construed a message concerning children s football (see Figure 6) as nonresponsive to Topic 207, although the TA defined it as responsive. 186 See supra Table 5. Figure 4 is an excerpt from document from the TREC 2009 dataset, supra note

435 Richmond Journal of Law and Technology Vol. XVII, Issue 3 WARNING: Your mailbox is approaching the size limit This warning is sent automatically to inform you that your mailbox is approaching the maximum size limit. Your mailbox size is currently KB. Mailbox size limits: When your mailbox reaches KB you will receive this message.to check the size of your mailbox: Right-click the mailbox (Outlook Today), Select Properties and click the Folder Size button. This method can be used on individual folders as well. To make more space available, delete any items that are no longer needed such as Sent Items and Journal entries. Figure 5: Topic 204 Interpretive error. A professional reviewer coded this automated message as nonresponsive, although the TA construed such messages to be responsive to Topic Subject: RE: Meet w/ Belden I need to leave at 3:30 today to go to my stepson s football game. Unfortunately, I have a 2:00 and 3:00 meeting already. Is this just a general catch-up discussion? Figure 6: Topic 207 Interpretive error. The reviewer may have construed a children s league football game to be outside of the scope of gambling on football. The TA deemed otherwise. 188 Arguable error: Reasonable, informed assessors might disagree or find it difficult to determine whether or not the document met the TA s conception of responsiveness (e.g., Figures 7 and 8). 187 See supra Table 5. Figure 5 is an excerpt from document in the TREC 2009 dataset, supra note See supra Table 5. Figure 6 is an excerpt from document in the TREC 2009 dataset, supra note

436 Richmond Journal of Law and Technology Vol. XVII, Issue 3 Subject: Original Guarantees Just a followup note: We are still unclear as to whether we should continue to send original incoming and outgoing guarantees to Global Contracts (which is what we have been doing for about 4 years, since the Corp. Secretary kicked us out of using their vault on 48 for originals because we had too many documents). I think it would be good practice if Legal and Credit sent the originals to the same place, so we will be able to find them when we want them. So my question to y all is, do you think we should send them to Global Contracts, to you, or directly the the 48th floor vault (if they let us!). Figure 7: Topic 204 Arguable error. This message concerns where to store particular documents, not specifically their destruction or retention. Applying the TA s conception of relevance, reasonable, informed assessors might disagree as to its responsiveness. 189 Subject: RE: How good is Temptation Island 2 They have some cute guy lawyers this year-but I bet you probably watch that manly Monday night Football. Figure 8: Topic 207 Arguable error. This message mentions football, but not a specific football team, player, or game. Reasonable, informed reviewers might disagree about whether or not it is responsive according to the TA s conception of relevance. 190 [51] When rendering assessments for the qualitative analysis, the Authors considered the mock complaint, 191 the topics, 192 and the topicspecific assessment guidelines memorializing the TA s conception of relevance, which were given to the human reviewers for reference 189 See supra Table 5. Figure 7 is an excerpt from document in the TREC 2009 dataset, supra note See supra Table 5. Figure 8 shows an excerpt from document in the TREC 2009 dataset, supra note See generally Complaint, Grumby v. Volteron Corp., supra note Id. at 14; Hedin et al, supra note 9, at

437 Richmond Journal of Law and Technology Vol. XVII, Issue 3 purposes. 193 Table 8 summarizes the findings: The vast majority of missed documents are attributable either to inarguable error or to misinterpretation of the definition of relevance (interpretive error). Remarkably, the findings identify only 4% of all errors as arguable. Error Type Topic Inarguable Interpretive Arguable Total Total Fraction 65% 31% 4% 100% Table 8: Number of responsive documents that human reviewers missed, categorized by the nature of the error. 65% of missed documents are relevant on their face. 31% of missed documents are clearly relevant, when the topicspecific guidelines are considered. Only 4% of missed documents, in the opinion of the Authors, have debatable responsiveness, according to the topicspecific guidelines. 194 VI. RESULTS AND DISCUSSION [52] Tables 6 and 7 show that, by all measures, the average efficiency and effectiveness of the five technology-assisted reviews surpasses that of the five manual reviews. The technology-assisted reviews require, on average, human review of only 1.9% of the documents, a fifty-fold savings over exhaustive manual review. For F 1 and precision, the measured difference is overwhelmingly statistically significant (P < 0.001); 195 for recall the measured difference is not significant (P > 0.1). 196 These measurements provide strong evidence that the technology-assisted 193 Text REtrieval Conference (TREC), TREC-2009 Legal Track Interactive Task, Topic-Specific Guidelines Topic 204, U. WATERLOO, (last updated Oct. 22, 2009); Text REtrieval Conference (TREC), TREC-2009 Legal Track Interactive Task, Topic-Specific Guidelines Topic 207, U. WATERLOO, (last updated Oct. 22, 2009). 194 See sources cited supra note See supra Tables 6, Id. 43

438 Richmond Journal of Law and Technology Vol. XVII, Issue 3 processes studied here yield better overall results, and better precision, in particular, than the TREC manual review process. The measurements also suggest that the technology-assisted processes may yield better recall, but the statistical evidence is insufficiently strong to support a firm conclusion to this effect. [53] It should be noted that the objective of TREC participants was to maximize F 1, not recall or precision, per se. 197 It happens that they achieved, on average, higher precision. 198 Had the participants considered recall to be more important, they might have traded off precision (and possibly F 1 ) for recall, by using a broader interpretation of relevance, or by adjusting a sensitivity parameter in their software. [54] Table 7 shows that, for four of the five topics, the technologyassisted processes achieve substantially higher F 1 scores, largely due to their high precision. Nonetheless, for a majority of the topics, the technology-assisted processes achieve higher recall as well; for two topics, substantially higher. 199 For Topic 207, there is no meaningful difference in effectiveness between the technology-assisted and manual reviews, for any of the three measures. There is not one single measure for which manual review is significantly better than technology-assisted review. [55] For three of the five topics (Topics 201, 202, and 207) the results show no significant difference in recall between the technology-assisted and manual reviews. This result is perhaps not surprising, since the recall scores are all on the order of 70% the best that might be reasonably achieved, given the level of agreement among human assessors. As such, the results support the conclusion that technology-assisted review can achieve at least as high recall as manual review, and higher precision, at a fraction of the review effort, and hence, a fraction of the cost. 197 See Hedin et al., supra note 9, at See supra Tables 6, See supra Table 7. 44

439 Richmond Journal of Law and Technology Vol. XVII, Issue 3 VII. LIMITATIONS [56] The 2009 TREC effort used a mock complaint and production requests composed by lawyers to be as realistic as possible. 200 Furthermore, the role of the TA was intended to simulate that of a senior attorney overseeing a real document review. 201 Finally, the dataset consisted of real messages captured within the context of an actual investigation. 202 These components of the study are perhaps as realistic as might reasonably be achieved outside of an actual legal setting. 203 One possible limitation is that the Enron story, and the Enron dataset, are both well known, particularly since the Enron documents are frequently used in vendor product demonstrations. 204 Both participants and TAs may have had prior knowledge of both the story and dataset, affecting their strategies and assessments. In addition, there is a tremendous body of extrinsic information that may have influenced participants and assessors alike, including the results of the actual proceedings, commentaries, 205 books, Hedin et al., supra note 9, at See id.; see also Oard et al., supra note 9, at See Hedin et al., supra note 9, at See id. 204 See, e.g., John Markoff, Armies of Expensive Lawyers Replaced by Cheaper Software, N.Y. TIMES, Mar. 5, 2011, A1, available at see also from Jonathan Nystrom to Maura R. Grossman (Apr. 5, :12 EDT) (on file with authors) (confirming use of the Enron data set for product demonstrations); from Jim Renehan to Maura R. Grossman (Apr. 5, :06 EDT) (on file with authors) (confirming use of the Enron data set for product demonstrations); from Lisa Schofield to Maura R. Grossman (Apr. 5, :27 EDT) (on file with authors) (confirming use of the Enron data set for product demonstrations); from Edward Stroz to Maura R. Grossman (Apr. 5, :32 EDT) (on file with authors) (confirming use of the Enron data set for product demonstrations). 205 See, e.g., John C. Coffee Jr., What Caused Enron?: A Capsule Social and Economic History of the 1990 s, 89 CORNELL L. REV. 269 (2004); Paul M. Healy & Krishna G. Palepu, The Fall of Enron, 17 J. ECON. PERSP. 3 (2003). 45

440 Richmond Journal of Law and Technology Vol. XVII, Issue 3 and even a popular movie. 207 may have had on the results. It is unclear what effect, if any, these factors [57] In general, the TREC teams were privy to less detailed guidance than the manual reviewers, placing the technology-assisted processes at a disadvantage. For example, Topic 202 required the production of documents related to transactions that the Company characterized as compliant with FAS Participating teams were required to undertake research to identify the relevant transactions, as well as the names of the parties, counterparties, and entities involved. 209 Manual reviewers, on the other hand, were given detailed guidelines specifying these elements. 210 [58] Moreover, TREC conducted manual review on a stratified sample containing a higher proportion of relevant documents than the collection as a whole, 211 and used statistical inference to evaluate the result of reviewing every document in the collection. 212 Beyond the statistical uncertainty, there also is uncertainty as to whether manual reviewers would have had the same error rate had they reviewed the entire collection. It is not unreasonable to think that, because the proportion of relevant documents would have been lower in the collection than it was in the sample, reviewer recall and precision might have been even lower, because reviewers would have tended to miss the needles in the haystacks due to fatigue, inattention, boredom, and related human factors. This 206 See, e.g., LOREN FOX, ENRON: THE RISE AND FALL (2002); BETHANY MCLEAN AND PETER ELKIND, THE SMARTEST GUYS IN THE ROOM: THE AMAZING RISE AND SCANDALOUS FALL OF ENRON (2003). 207 ENRON: THE SMARTEST GUYS IN THE ROOM (Magnolia Pictures 2005). 208 Hedin et al., supra note 9, at See id. at See id. at See id. at 12, tbl See generally id. 46

441 Richmond Journal of Law and Technology Vol. XVII, Issue 3 sampling effect, combined with the greater guidance provided to the human reviewers, may have resulted in an overestimate of the effectiveness of manual review, and thus understated the results of the study. [59] Of note is the fact that the appeals process involved reconsideration and potential reversal only of manual coding decisions that one or more participating teams appealed, presumably because their results disagreed with the manual reviewers decisions. 213 The appeals process depended on participants exercising due diligence in identifying the assessments with which they disagreed. 214 And while it appears that H5 and Waterloo exercised such diligence, it became apparent to the Authors during the course of their analysis that a few assessor errors were overlooked. 215 These erroneous assessments were deemed correct under the gold standard, with the net effect of overstating the effectiveness of manual reviews, while understating the effectiveness of technologyassisted review. 216 It is also likely that the manual review and technologyassisted processes incorrectly coded some documents that were not appealed. 217 The impact of the resulting errors on the gold standard would be to overstate both recall and precision for manual review, as well as for technology-assisted review, with no net advantage to either. 213 See Hedin et al., supra note 9 at 3, There is no benefit, and therefore no incentive, for participating teams to appeal coding decisions with which they agree. 214 See id. If participating teams do not appeal the manual reviewers incorrect decisions, those incorrect decisions will be incorporated into the gold standard, compromising its accuracy and usefulness. 215 Hedin et al., supra note 9 at 14, tbl.4 (showing that for every topic, H5 and Waterloo appealed the majority of disagreements between their results and the manual assessments). 216 See supra note 214. If the manual review is incorrect, and the technology-assisted review is correct, the results will overstate the effectiveness of manual review at the expense of technology-assisted review. 217 Given that neither the manual reviewers nor the technology-assisted processes are infallible, it stands to reason that they may occasionally agree on coding decisions that are incorrect. 47

442 Richmond Journal of Law and Technology Vol. XVII, Issue 3 [60] In designing this study, the Authors considered only the results of two of the eleven teams participating in TREC 2009, because they were considered most likely to demonstrate that technology-assisted review can improve upon exhaustive manual review. The study considered all submissions by these two teams, which happened to be the most effective submissions for five of the seven topics. The study did not consider Topics 205 and 206, because neither H5 nor Waterloo submitted results for them. Furthermore, due to a dearth of appeals, there was no reliable gold standard for Topic The Authors were aware before conducting their analysis that the H5 and Waterloo submissions were the most effective for their respective topics. To show that the results are significant in spite of this prior knowledge, the Authors applied Bonferroni correction, 219 which multiplies P by 11, the number of participating teams. Even under Bonferroni correction, the results are overwhelmingly significant. VIII. CONCLUSION [61] Overall, the myth that exhaustive manual review is the most effective and therefore, the most defensible approach to document review is strongly refuted. Technology-assisted review can (and does) yield more accurate results than exhaustive manual review, with much lower effort. Of course, not all technology-assisted reviews (and not all manual reviews) are created equal. The particular processes found to be superior in this study are both interactive, employing a combination of computer and human input. While these processes require the review of orders of magnitude fewer documents than exhaustive manual review, neither entails the naïve application of technology absent human judgment. Future work may address which technology-assisted review process(es) will improve most on manual review, not whether technologyassisted review can improve on manual review. 218 Hedin et al., supra note 9, at ( Topic 206 represents the one topic, out of the seven featured in the 2009 exercise, for which we believe the post-adjudication results are not reliable.... We do not believe, therefore, that any valid conclusions can be drawn from the scores recorded for this topic.... ). 219 See BÜTTCHER ET AL., supra note 19, at

443 FEDERAL COURTS LAW REVIEW Volume 7, Issue THE GROSSMAN-CORMACK GLOSSARY OF TECHNOLOGY-ASSISTED REVIEW with Foreword by John M. Facciola, U.S. Magistrate Judge FOREWORD When I use a word, Humpty Dumpty said, in rather a scornful tone, it means just what I choose it to mean neither more nor less. The question is, said Alice, whether you can make words mean so many different things. Lewis Carroll, Through the Looking Glass, What Alice Found There 1 A word is not a crystal, transparent and unchanged, it is the skin of a living thought and may vary greatly in color and content according to the circumstances and the time in which it is used. Justice Oliver Wendell Holmes Jr., Towne v. Eisner, 245 U.S. 418, 425 (1918) In the heels of the higgling lawyers, Bob, Too many slippery ifs and buts and howevers, Too much hereinbefore provided whereas, Too many doors to go in and out of. When the lawyers are through What is there left, Bob? Can a mouse nibble at it And find enough to fasten a tooth in? Carl Sandburg, The Lawyers Know Too Much 2 1. THE COLLECTED STORIES OF LEWIS CARROLL 238 (Citadel Press 1994). 2. THE COMPLETE POEMS OF CARL SANDBURG 189 (Harcourt, Brace and Co., rev. ed. 1970).

444 2 FEDERAL COURTS LAW REVIEW [Vol. 7 It is always the words. Humpty Dumpty, Holmes, and Sandburg, who sensed the power of language instinctively, knew how quickly their meanings can slip away. What seemed clear when the contract or statute was drafted is now recondite. When the parties contracted to buy and sell a horse, did they mean a mare or a stallion? When the statute required that a lawyer be disbarred upon conviction of a crime of moral turpitude, did the legislature mean a lawyer who gets convicted of assault after a brawl in a bar? Lawyers and judges are mocked for their continued use of Latin but they know that it is so much easier to say res judicata and take advantage of the encrusted meaning of those words than to start fresh and try to improve on what they convey. Law books come and go but Black s Law Dictionary will always be around. The words, as Paul Simon might put it, keep slip, sliding away. 3 It is hard enough when the world in which the words are used remains static, like the farm on which the horse, be it mare or stallion, lives. But, what happens when the movement of technology radically transforms what a word might have once meant? What is the original of an ? Is another a copy of it when the visible text is the same but the metadata created in its production by a computer, rather than a human being, is entirely different? What happens when the meaning of the words in a statute applied to a process that was in existence when the statute was enacted but now no longer exists? Some of the definitions in the Stored Communications Act, 4 enacted in 1986, may drive judges to distraction since they were premised on technology used in 1986 but is no longer and must be applied to new processes that no one knew would exist when the statute was enacted. What kind of words can be used in a statute or a rule that are capacious enough to hold their meaning despite unknown technological change but precise enough to convey a definite meaning? It may be hard to believe, but there was not even a clear indication in the Federal Rules of Civil Procedure until 2006 that electronically stored information was within the scope of what a party had to produce in 3. PAUL SIMON, Slip Slidin Away, on THE ESSENTIAL PAUL SIMON (Warner Bros. 2007). 4. See, e.g., The definition of remote computing service in 18 U.S.C. 2711(2) ( the provision to the public of computer storage or processing services by means of electronic communications systems ). As Professor Orin Kerr explains, the statute freez[es] into law the understanding of computer network use as of Orin S. Kerr, A User s Guide to the Stored Communications Act, and a Legislator s Guide to Amending It, 72 GEO. WASH. L. REV. 1208, 1214 (2004). In 1986, users would use remote computing services to outsource computing tasks such as storing extra files or processing data when doing so was beyond the capacity of their computers. Id. Given the storage and processing capacities of new computers and tablets, this kind of distant processing capacity has disappeared. In its wake, however, are the difficult questions of the application of the definitions in the Stored Communications Act to cloud computing. See William Jeremy Robison, Free at What Cost? Cloud Computing Privacy under the Stored Communications Act, 98 GEO. L.J. 1195, 1210, (2010).

445 2013] Glossary of Technology-Assisted Review 3 discovery. 5 Indeed, until 2006, the word phonorecords appeared in Federal Rules of Civil Procedure 34(a) 6 which must have, at the time, 7 mystified anyone under 35. The pace of technological change makes the situation worse. That pace is astonishing. What was thought to be the impossible crossing of the Atlantic Ocean in a plane and the landing of a man on the moon occurred in my father s lifetime. Yet, George Washington s troops moved no faster than Caesar s, and it took thousands of years before human beings discovered how to transmit messages by the telegraph using electricity. 8 Until then, a messenger, whether from Marathon to Athens, or through the colonial towns of Massachusetts, had to deliver them by hand. The legal system deals in words and the pace of technological process is creating billions of them on a nearly daily basis, 9 creating a set of problems that were unimaginable a few years ago. The first problem is that this explosion of words has been matched by the ever-increasing capacity of machines to capture and preserve them. It is a simple fact that an ipod has much more memory, i.e., capability to store information indefinitely, than the first computer 10 which took up an entire room. Additionally, the cost of storage is diminishing. A one-terabyte drive can be purchased for about $ and can hold what would otherwise be hundreds of thousands of pages of paper. Indeed, now the information can be kept on a distant server with the space rented from a vendor for that purpose and retrieved by use of the Internet ( cloud computing ). For the first time, it is cheaper for human beings to buy a new file cabinet and keep more paper than to clean the useless clutter out of the old file cabinet to make room for the new information. 5. Compare FED. R. CIV. P. 34(a)(1)(A) (West 2006 rev. ed.) (describing any designated documents or electronically stored information including writings, drawings, graphs, charts, photographs, sound recordings, images, and other data or data compilations stored in any medium from which information can be obtained either directly or, if necessary, after translation by the responding party into a reasonably usable form as discoverable) with FED. R. CIV. P. 34(a)(1)(A) (West 2006) (mentioning only any designated documents (including writings, drawings, graphs, charts, photographs, phonorecords, and other data compilations from which information can be obtained, translated, if necessary, by the respondent through detection devices into reasonably usable form ). 6. FED. R. CIV. P. 34(a)(1)(A) (West 2006). 7. Vinyl records are making a comeback; eight track tapes will not. See, e.g., Brian Passey, Vinyl Records spin back into vogue, USATODAY.COM (Feb. 26, 2011). 8. See TOM STANDAGE, THE VICTORIAN INTERNET (Walker & Co. 1998). 9. See George L. Paul & Jason R. Baron, Information Inflation: Can the Legal System Adapt?, 13 RICH. J.L. & TECH. 10, (2007). 10. See, e.g., What is a Mainframe Operating System?, WISEGEEK, what-is-a-mainframe-operating-system.htm (last visited Nov. 19, 2012). 11. See AMAZON, (listing numerous options under $100 for a one-terabyte external hard drive).

446 4 FEDERAL COURTS LAW REVIEW [Vol. 7 This phenomenon has led to the consequence that litigants are confronted with the often horrifying costs of searching through immense amounts of data to find what they need. The Federal Rules of Civil Procedure contemplate a system of demand and production; plaintiff asks for all documents pertaining to the merger of companies A and B and defendant either objects or produces them. But, when there are now hundreds of thousands of documents that may meet the definition of pertinent, how can defendant find them and not go bankrupt in the process? Of course, defendant may be the victim of its own failure to maintain a responsible record-keeping process in which a principled decision-making process guides what will be kept and what will be thrown out. And, as anyone knows who has ever cleaned out a closet or an old hard drive, keeping everything is no solution. It only increases the expense and cost of finding what you want or need. Nevertheless, the affordability of cheap storage has led too many entities in our society to be quickly overwhelmed by their inability to search for what they need, whether because they need it to run their business or because they must produce it in discovery. They may find that the cost of searching and producing is so great that settling the lawsuit may be the only way out of an otherwise impossible situation. It is understandable that, nature and technology abhorring a vacuum, a new scientific methodology has emerged to aid in the collection and searching process. Technology-Assisted Review, called by its nickname Predictive Coding, describes a process whereby computers are programmed to search a large amount of data to find quickly and efficiently the data that meet a particular requirement. Computer science and the sciences of statistics and psychology inform its use. While it bruises the human ego, scientists, including the authors of this glossary we are publishing, have determined that machines are better at the task of making such discoveries than humans. 12 Lawyers love to think that there is no substitute for their reviewing each document page by page. Not only is there a substitute, but an improvement. It is now indubitable that technology-assisted review is an appreciably better and more accurate means of searching a set of data. 13 That is hardly surprising news to those judges and lawyers who have experienced the mind-numbing tedium of reviewing large data sets only to find that one is seeing the same 12. See Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and Efficient Than Exhaustive and Manual Review, XVII RICH. J.L. & TECH. 11 (2011). 13. Id.

447 2013] Glossary of Technology-Assisted Review 5 chain again and again, or worse, wading through mountains of data and finding nothing of any pertinence to the case being litigated. The great benefits of technology-assisted review, however, bring in new concerns and questions for judges and lawyers. In a paper universe, the manner in which a party searched through a file cabinet hardly raises any significant issues. If what was produced appeared to be what was demanded and there were no inexplicable gaps, that was that and the court and parties moved on to other things. Now, the methodology of the use of technology-assisted review may itself be in dispute, with the parties controverted to each other s use of a particular method or tool. Those controversies have already lead to judicial decisions that have to grapple with a wholly new way of searching and with scientific principles derived from the science of statistics or other disciplines. 14 Lawyers and judges once again have to learn a whole new vocabulary to resolve the emerging and inevitable battle of the experts. To aid in the creation of that vocabulary, we publish with pride the glossary created by Maura R. Grossman and Gordon V. Cormack who are two of the most respected and acknowledged experts in this field. We agree with them that the creation of a clear and common vocabulary is essential to a comprehension of the legal issues at stake. We are particularly gratified that the authors announced that they intend the glossary to be interactive so that others can suggest additional clarifications, revisions, and additions. We are certain that, if the experience of the courts in the first few years of the information technology revolution is any guide, the learning curve will be steep and that it must be climbed quickly if courts are going to be able to resolve promptly the controversies before them at the least expense. We are equally certain that the bench and bar will find this glossary useful as this new science develops and grows. The Editors of the Federal Courts Law Review By John M. Facciola, U.S. Magistrate Judge 14. See, e.g., EORHB, Inc. v. HOA Holdings, Civ. Ac. No VCL (Del. Ch. Oct. 19, 2012); Kleen Prods. LLC v. Packaging Corp., Civ. No. 10C 5711, 2012 WL at *84-85 (N.D. Ill. Sept. 28, 2012); In re Actos (Pioglitazone) Prods. Liab. Litig., MDL No. 6:11-md-2299 (W.D. La. July 27, 2012); Global Aerospace Inc. v. Landow Aviation, L.P., No. CL (Va. Cir. Ct. Apr. 23, 2012); Moore v. Publicis Groupe & MSL Group, No. 11 Civ (ACL) (AJP), 2012 WL (S.D.N.Y. Feb. 24, 2012).

448 6 FEDERAL COURTS LAW REVIEW [Vol. 7 PREAMBLE Disruptive technology is a term that was coined by Harvard Business School Professor Clayton M. Christensen, in his 1997 book, The Innovator s Dilemma, to describe a new technology that unexpectedly displaces an established technology. The term is used in business and technology literature to describe innovations that improve a product or service in ways that the market did not expect, typically by designing for a different set of consumers in the new market and later, by lowering prices in the existing market. Products based on disruptive technologies are typically cheaper to produce, simpler, smaller, better performing, more reliable, and often more convenient to use. Technology-Assisted Review (TAR) is such a disruptive technology. Because disruptive technologies differ from sustaining technologies ones that rely on incremental improvements to established technologies they bring with them new features, new vernaculars, and other challenges. The introduction of TAR into the legal community has brought with it much confusion because different terms are being used to refer to the same thing (e.g., technology-assisted review, computer-assisted review, computer-aided review, predictive coding, and content-based advanced analytics, to name but a few), and the same terms are also being used to refer to different things (e.g., seed set and control sample ). Moreover, the introduction of complex statistical concepts and terms of art from the science of information retrieval have resulted in widespread misunderstanding and sometimes perversion of their intended meanings. This glossary is written in an effort to bring order to chaos by introducing a common framework and set of definitions for use by the bench, the bar, and service providers. This glossary endeavors to be comprehensive, but its definitions are necessarily brief. Interested readers may look elsewhere for detailed information concerning any of these topics. The terms in this glossary are presented in alphabetical order, with defined terms in capital letters. We envision this glossary to be a living, breathing work that will evolve over time. Towards that end, we invite our colleagues in the industry to send us comments on our definitions, as well as any additional terms they would like to see included in the glossary, so that we can reach a consensus on a consistent, common language relating to TAR. Comments can be sent to us at and Subsequent versions of this glossary will be available online at The authors would like to acknowledge the helpful comments provided by Craig Ball, Michael Levine, Ralph Losey, Amir Milo, and

449 2013] Glossary of Technology-Assisted Review 7 Keith Roland on an earlier draft of this work. We are very grateful to Magistrate Judge John M. Facciola for his enthusiastic support. We hope that you will find this glossary useful. Maura R. Grossman* Wachtell, Lipton, Rosen & Katz New York, New York Gordon V. Cormack University of Waterloo Waterloo, Ontario January 2013 * The views expressed herein are solely those of the author and should not be attributed to her firm or its clients.

450 8 FEDERAL COURTS LAW REVIEW [Vol. 7 THE GLOSSARY Accept on Zero Error: A technique in which the training of a Machine Learning method is gauged by taking a Sample after each training step, and deeming the training process complete when the learning method codes a Sample with 0% Error (i.e., 100% Accuracy). Accuracy: The fraction of Documents that are correctly coded by a search or review effort. Note that Accuracy + Error = 100%, and that Accuracy = 100% Error. While high Accuracy is commonly advanced as evidence of an effective search or review effort, its use can be misleading because it is heavily influenced by Prevalence. Consider, for example, a Document Population containing one million Documents, of which ten thousand (or 1%) are Relevant. A search or review effort that identified 100% of the Documents as Not Relevant, and, therefore, found none of the Relevant Documents, would have 99% Accuracy, belying the failure of that search or review effort. Active Learning: An Iterative Training regimen in which the Training Set is repeatedly augmented by additional Documents chosen by the Machine Learning Algorithm, and coded by one or more Subject Matter Expert(s). Actos: See In Re: Actos. Agreement: The fraction of all Documents that two reviewers code the same way. While high Agreement is commonly advanced as evidence of an effective review effort, its use can be misleading, for the same reason that the use of Accuracy can be misleading. When the vast majority of Documents in a Population are Not Relevant, a high level of Agreement will be achieved when the reviewers agree that these Documents are Not Relevant, irrespective of whether or not they agree that any of the Relevant Documents are Relevant. Algorithm: A formally specified series of computations that, when executed, accomplishes a particular goal. The Algorithms used in E- Discovery are implemented as computer software. Area Under the ROC Curve (AUC): From Signal Detection Theory, a summary measure used to assess the quality of Prioritization. AUC is the Probability that a randomly chosen Relevant Document is given a higher priority than a randomly chosen Non-Relevant Document. An AUC score of 100% indicates a perfect ranking, in which all Relevant Documents have

451 2013] Glossary of Technology-Assisted Review 9 higher priority than all Non-Relevant Documents. An AUC score of 50% means the Prioritization is no better than chance. Artificial Intelligence: An umbrella term for computer methods that emulate human judgment. These include Machine Learning and Knowledge Engineering, as well as Pattern Matching (e.g., voice, face, and handwriting recognition), robotics, and game playing. Bag of Words: A Feature Engineering method in which the Features of each Document comprise the set of words contained in that Document. Documents are determined to be Relevant or Not Relevant depending on what words they contain. Elementary Keyword Search and Boolean Search methods, as well as some Machine Learning methods, use the Bag of Words model. Bayes / Bayesian / Bayes Theorem: A general term used to describe Algorithms and other methods that estimate the overall Probability of some eventuality (e.g., that a Document is Relevant), based on the combination of evidence gleaned from separate observations. In Electronic Discovery, the most common evidence that is combined is the occurrence of particular words in a Document. For example, a Bayesian Algorithm might combine the evidence gleaned from the fact that a Document contains the words credit, default, and swap to indicate that there is a 99% Probability that the Document concerns financial derivatives, but only a 40% Probability if the words credit and default, but not swap, are present. The most elementary Bayesian Algorithm is Naïve Bayes; however, most Algorithms dubbed Bayesian are more complex. Bayesian Algorithms are named after Bayes Theorem, coined by the 18th century mathematician, Thomas Bayes. Bayes Theorem derives the Probability of an outcome, given the evidence, from: (i) the probability of the outcome, independent of the evidence; (ii) the probability of the evidence, given the outcome; and (iii) the probability of the evidence, independent of the outcome. Bayesian Classifier / Bayesian Filter / Bayesian Learning: A colloquial term used to describe a Machine Learning Algorithm that uses a Bayesian Algorithm resembling Naïve Bayes. Bigram: An N-Gram where N = 2 (i.e., a 2-gram). Binomial Calculator / Binomial Estimation: A statistical method used to calculate Confidence Intervals, based on the Binomial Distribution, that

452 10 FEDERAL COURTS LAW REVIEW [Vol. 7 models the random selection of Documents from a large Population. Binomial Estimation is generally more accurate, but less well known, than Gaussian Estimation. A Binomial Estimate is substantially better than a Gaussian Estimate (which, in contrast, relies on the Gaussian or Normal Distribution) when there are few (or no) Relevant Documents in the Sample. When there are many Relevant and many Non-Relevant Documents in the Sample, Binomial and Gaussian Estimates are nearly identical. Binomial Distribution: The Probability that a Random Sample from a large Population will contain any particular number of Relevant Documents, given the Prevalence of Relevant Documents in the Population. Used as the basis for Binomial Estimation. Binomial Estimate: A Statistical Estimate of a Population characteristic using Binomial Estimation. It is generally expressed as a Point Estimate accompanied by a Margin of Error and a Confidence Level, or as a Confidence Interval accompanied by a Confidence Level. Blair and Maron: Authors of an influential 1985 study (David C. Blair & M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System, 28 COMMC NS ACM 289 (1985)), showing that attorneys supervising skilled paralegals believed they had found at least 75% of the Relevant Documents from a Document Collection, using search terms and iterative search, when they had in fact found only 20%. That is, the searchers believed they had achieved 75% Recall, but had achieved only 20% Recall. In the Blair and Maron study, the attorneys and paralegals used an iterative approach, examining the retrieved Documents and refining their search terms until they believed they were done. Many current commentators incorrectly distinguish the Blair and Maron study from current iterative approaches, failing to note that the Blair and Maron searchers did in fact refine their search terms based on their review of the Documents that were returned in response to their queries. Boolean Search: A Keyword Search in which the Keywords are combined using operators such as AND, OR, and [BUT] NOT. The result of a Boolean Search is precisely determined by the words contained in the Documents. (See also Bag of Words.) Bulk Coding: The process of Coding all members of a group of Documents (identified, for example, by Deduplication, Near-Deduplication,

453 2013] Glossary of Technology-Assisted Review 11 Threading, or Clustering) based on the review of only one or a few members of the group. Also referred to as Bulk Tagging. Bulk Tagging: See Bulk Coding. Classical, Gaussian, or Normal Calculator / Classical, Gaussian, or Normal Estimation: A method of calculating Confidence Intervals based on the assumption that the quantities to be measured follow a Gaussian (Normal) Distribution. This method is most commonly taught in introductory statistics courses, but yields inaccurate Confidence Intervals when the Prevalence of items with the characteristic being measured is low. (Cf. Binomial Calculator / Binomial Estimation.) Classifier / Classification / Classified / Classify: An Algorithm that Labels items as to whether or not they have a particular property; the act of Labeling items as to whether or not they have a particular property. In Technology-Assisted Review, Classifiers are commonly used to Label Documents as Responsive or Non-Responsive. Clustering: An Unsupervised Learning method in which Documents are segregated into categories or groups so that the Documents in any group are more similar to one another than to those in other groups. Clustering involves no human intervention, and the resulting categories may or may not reflect distinctions that are valuable for the purpose of a search or review effort. Code / Coded / Coding: The action of Labeling a Document as Relevant or Non-Relevant, or the set of Labels resulting from that action. Sometimes interpreted narrowly to include only the result(s) of a Manual Review effort; sometimes interpreted more broadly to include automated or semiautomated Labeling efforts. Coding is generally the term used in the legal industry; Labeling is the equivalent term in Information Retrieval. Collection: See Document Collection. Computer-Aided Review: See Technology-Assisted Review. Computer-Assisted Review (CAR): See Technology-Assisted Review. Concept Search: An industry-specific term generally used to describe Keyword Expansion techniques, which allow search methods to return Documents beyond those that would be returned by a simple Keyword or

454 12 FEDERAL COURTS LAW REVIEW [Vol. 7 Boolean Search. Methods range from simple techniques such as Stemming, Thesaurus Expansion, and Ontology search, through statistical Algorithms such as Latent Semantic Indexing. Confidence Interval: As part of a Statistical Estimate, a range of values estimated to contain the true value, with a particular Confidence Level. Confidence Level: As part of a Statistical Estimate, the chance that a Confidence Interval derived from a Random Sample will include the true value. For example, 95% Confidence means that if one were to draw 100 independent Random Samples of the same size, and compute the Confidence Interval from each Sample, about 95 of the 100 Confidence Intervals would contain the true value. It is important to note that the Confidence Level is not the Probability that the true value is contained in any particular Confidence Interval; it is the Probability that the method of estimation will yield a Confidence Interval that contains the true value. Confusion Matrix: A two-by-two table listing values for the number of True Negatives (TN), False Negatives (FN), True Positives (TP), and False Positives (FP) resulting from a search or review effort. As shown below, all of the standard evaluation measures are algebraic combinations of the four values in the Confusion Matrix. Also referred to as a Contingency Table. An example of a Confusion Matrix (or Contingency Table) is provided immediately below. Coded Relevant Coded Non-Relevant Truly Relevant True Positives (TP) False Negatives (FN) Truly Non-Relevant False Positives (FP) True Negatives (TN) Accuracy = 100% Error = (TP + TN) / (TP + TN + FP + FN) Elusion = 100% Negative Predictive Value = FN / (FN + TN) Error = 100% Accuracy = (FP + FN) / (TP + TN + FP + FN) Fallout = False Positive Rate = 100% True Negative Rate = FP / (FP+TN) False Negative Rate = 100% True Positive Rate = FN / (FN+TP) Negative Predictive Value = 100% Elusion = TN / (TN + FN) Precision = Positive Predictive Value = TP / (TP + FP) Prevalence = Yield = Richness = (TP + FN) / (TP + TN + FP + FN) Recall = True Positive Rate = Sensitivity = TP / (TP+FN) True Negative Rate = Specificity = TN / (TN + FP)

455 2013] Glossary of Technology-Assisted Review 13 Content-Based Advanced Analytics (CBAA): See Technology-Assisted Review. Contingency Table: See Confusion Matrix. Control Set: A Random Sample of Documents coded at the outset of a search or review process that is separate from and independent of the Training Set. Control Sets are used in some Technology-Assisted Review processes. They are typically used to measure the effectiveness of the Machine Learning Algorithm at various stages of training, and to determine when training may cease. Crossover Trial: An Experimental Design for comparing two search or review processes using the same Document Collection and Information Need, in which one process is applied first, followed by the second, and then the results of the two efforts are compared. (Cf. Parallel Trial.) Culling: The practice of narrowing a larger data set to a smaller data set for the purposes of review, based on objective criteria (such as file types or date restrictors), or subjective criteria (such as Keyword Search Terms). Documents that do not match the criteria are excluded from the search and from further review. Cutoff: A given score or rank in a Prioritized list, resulting from a Relevance Ranking search or Machine Learning Algorithm, such that the Documents above the Cutoff are deemed to be Relevant and Documents below the Cutoff are deemed to be Non-Relevant. In general, a higher Cutoff will yield higher Precision and lower Recall, while a lower Cutoff will yield lower Precision and higher Recall. Also referred to as a Threshold. Da Silva Moore: Da Silva Moore v. Publicis Groupe, Case No. 11 Civ (ALC) (AJP), 2012 WL (S.D.N.Y. Feb. 24, 2012), aff d, 2012 WL (S.D.N.Y. Apr. 26, 2012). The first federal case to recognize Computer-Assisted Review as an acceptable way to search for relevant ESI in appropriate cases. The opinion was written by Magistrate Judge Andrew J. Peck and affirmed by District Judge Andrew L. Carter. Decision Tree: A step-by-step method of distinguishing between Relevant and Non-Relevant Documents, depending on what combination of words (or other Features) they contain. A Decision Tree to identify Documents pertaining to financial derivatives might first determine whether or not a

456 14 FEDERAL COURTS LAW REVIEW [Vol. 7 Document contained the word swap. If it did, the Decision Tree might then determine whether or not the Document contained the word credit, and so on. A Decision Tree may be created through Knowledge Engineering or Machine Learning. Deduplication: A method of replacing multiple identical copies of a Document by a single instance of that Document. Deduplication can occur within the data of a single custodian (also referred to as Vertical Deduplication), or across all custodians (also referred to as Horizontal Deduplication). Dimensionality Reduction: A Feature Engineering method used to reduce the total number of Features considered by a Machine Learning Algorithm. Simple Dimensionality Reduction methods include Stemming and Stop Word elimination. More complex Dimensionality Reduction methods include Latent Semantic Indexing and Hashing. Document: In the context of Electronic Discovery, a discrete item of Electronically Stored Information that may be the subject or result of a search or review effort. Document Collection: The process of gathering Electronically Stored Information for search, review, and production; the set of Documents resulting from such a process. In many cases, the Document Collection and Document Population are the same; however, it is important to note that Document Population refers to the set of Documents over which a particular Statistical Estimate is calculated, which may be the entire Document Collection, a subset of the Document Collection (e.g., the documents with a particular file type or matching particular Search Terms), a superset of the Document Collection (e.g., the universe from which the Document Collection was gathered), or any combination thereof. Document Population: The set of Electronically Stored Information or Documents about which a Statistical Estimate may be made. Early Case Assessment (ECA): An industry-specific term generally used to describe a variety of tools or methods for investigating and quickly learning about a Document Collection for the purposes of estimating the risk(s) and cost(s) of pursuing a particular legal course of action. EDI Study: See JASIST Study.

457 2013] Glossary of Technology-Assisted Review 15 EDI-Oracle Study: An ongoing initiative (as of January 2013) of the Electronic Discovery Institute to evaluate participating vendors search and document review efforts using a Document Collection contributed by Oracle America, Inc. Electronic Discovery / E-Discovery: The process of identifying, preserving, collecting, processing, searching, reviewing, and producing Electronically Stored Information that may be Relevant to a civil, criminal, or regulatory matter. Electronically Stored Information (ESI): Used in Federal Rule of Civil Procedure 34(a)(1)(A) to refer to discoverable information stored in any medium from which the information can be obtained either directly or, if necessary, after translation by the responding party into a reasonably usable form. Although Rule 34(a)(1)(A) references Documents or Electronically Stored Information, individual units of review and production are commonly referred to as Documents, regardless of the medium. Elusion: The fraction of Documents identified as Non-Relevant by a search or review effort that are in fact Relevant. Elusion is estimated by taking a Random Sample from the Null Set and determining how many (or what Proportion of) Documents are actually Relevant. A low Elusion value has commonly been advanced as evidence of an effective search or review effort (see, e.g., Kleen), but that can be misleading because it quantifies only those Relevant Documents that have been missed by the search or review effort; it does not quantify the Relevant Documents found by the search or review effort (i.e., Recall). Consider, for example, a Document Population containing one million Documents, of which ten thousand (or 1%) are Relevant. A search or review effort that returned 1,000 Documents, none of which were Relevant, would have 1.001% Elusion, belying the failure of the search. Elusion = 100% Negative Predictive Value. Threading: Grouping together messages that are part of the same discourse, so that they may be understood, reviewed, and coded consistently as a unit. EORHB: EORHB v. HOA Holdings LLC, Civ. Action No VCL, tr. and slip op. (Del. Ch. Oct. 19, 2012). The first case in which a court sua sponte directed the parties to use Predictive Coding as a replacement for Manual Review (or to show cause why this was not an appropriate case for Predictive Coding), absent either party s request to employ Predictive

458 16 FEDERAL COURTS LAW REVIEW [Vol. 7 Coding. Vice Chancellor J. Travis Laster also ordered the parties to use the same E-Discovery vendor and to share a Document repository. Error / Error Rate: The fraction of all Documents that are incorrectly coded by a search or review effort. Note that Accuracy + Error = 100%, and that 100% Accuracy = Error. While a low Error Rate is commonly advanced as evidence of an effective search or review effort, its use can be misleading because it is heavily influenced by Prevalence. Consider, for example, a Document Population containing one million Documents, of which ten thousand (or 1%) are relevant. A search or review effort that found none of the relevant Documents would have 1% Error, belying the failure of the search or review effort. ESI: See Electronically Stored Information. Experimental Design: A standard procedure accepted in the scientific community for the evaluation of competing hypotheses. There are many valid experimental designs. Some that can be appropriate for evaluating Technology-Assisted Review processes include Crossover Trials and Parallel Trials. F 1 : The Harmonic Mean of Recall and Precision, often used in Information Retrieval studies as a measure of the effectiveness of a search or review effort, which accounts for the tradeoff between Recall and Precision. In order to achieve a high F 1 score, a search or review effort must achieve both high Recall and high Precision. Fallout: See False Positive Rate. False Negative (FN): A Relevant Document that is missed (i.e., incorrectly identified as Non-Relevant) by a search or review effort. Also known as a Miss. False Negative Rate (FNR): The fraction (or Proportion) of Relevant Documents that are Missed (i.e., incorrectly identified as Non-Relevant) by a search or review effort. Note that False Negative Rate + Recall = 100%, and that 100% Recall = False Negative Rate. False Positive (FP): A Non-Relevant Document that is incorrectly identified as Relevant by a search or review effort.

459 2013] Glossary of Technology-Assisted Review 17 False Positive Rate (FPR): The fraction (or Proportion) of Non-Relevant Documents that are incorrectly identified as Relevant by a search or review effort. Note that False Positive Rate + True Negative Rate = 100%, and that 100% True Negative Rate = False Positive Rate. In Information Retrieval, also known as Fallout. Feature Engineering: The process of identifying Features of a Document that are used as input to a Machine Learning Algorithm. Typical Features include words and phrases, as well as metadata such as subjects, dates, and file types. One of the simplest and most common Feature Engineering techniques is Bag of Words. More complex Feature Engineering techniques include the use of Ontologies and Latent Semantic Indexing. Features: The units of information used by a Machine Learning Algorithm to Classify or Prioritize Documents. Typical Features include text fragments, such as words or phrases, and metadata such as sender, recipient, and sent date. See also Feature Engineering. Find Similar: A search method that identifies Documents that are similar to a particular exemplar. Find Similar is commonly misconstrued to be the mechanism behind Technology-Assisted Review. Gain Curve: A graph that shows the Recall that would be achieved for a particular Cutoff. The Gain Curve directly relates the Recall that can be achieved to the effort that must be expended to achieve it, as measured by the number of Documents that must be reviewed and Coded. Gaussian Calculator / Gaussian Estimation: See Classical, Gaussian, or Normal Calculator / Classical, Gaussian, or Normal Estimation. Gaussian Distribution: See Normal Distribution. Gaussian Estimate: A Statistical Estimate of a Population characteristic using Gaussian Estimation. It is generally expressed as a Point Estimate accompanied by a Margin of Error and a Confidence Level, or as a Confidence Interval accompanied by a Confidence Level. Global Aerospace: Global Aerospace Inc. v. Landow Aviation, Consol. Case No. CL 61040, 2012 WL (Va. Cir. Ct. Apr. 23, 2012). The first State Court Order approving the use of Predictive Coding by the producing party, over the objection of the requesting party, without prejudice to the requesting party raising an issue with the Court as to the

460 18 FEDERAL COURTS LAW REVIEW [Vol. 7 completeness or the contents of the production, or the ongoing use of Predictive Coding. The order was issued by Loudoun County Circuit Court Judge James H. Chamblin. Global Deduplication: Deduplication of Documents across multiple custodians. Also referred to as Horizontal Deduplication. (Cf. Vertical Deduplication.) Gold Standard: The best available determination of the Relevance or Non- Relevance of all (or a sample) of a Document Population, used as benchmark to evaluate the effectiveness of a search and review effort. Also referred to as Ground Truth. Goodhart s Law: An observation made in 1975 by Charles Goodhart, Chief Adviser to the Bank of England, that statistical economic indicators, when used for regulation, become unreliable. Restated and generalized in 1997 by University of Cambridge Professor Marilyn Strathern as When a measure becomes a target, it ceases to be a good measure. Within the context of Electronic Discovery, Goodhart s Law suggests that the value of Information Retrieval measures such as Recall and Precision may be compromised if they are prescribed as the definition of the reasonableness of a search or review effort. Grossman and Cormack: Authors of the JOLT (a.k.a. Richmond Journal) Study. Ground Truth: See Gold Standard. Harmonic Mean: The reciprocal of the average of the reciprocals of two or more quantities. If the quantities are named a and b, their Harmonic Mean 2 is. In Information Retrieval, F 1 is the Harmonic Mean of Recall and 1 a + 1 b Precision. The Harmonic Mean, unlike the more common arithmetic mean (i.e., average), falls closer to the lower of the two quantities. As a summary measure, a Harmonic Mean may be preferable to an arithmetic mean because a high Harmonic Mean depends on both high Recall and high Precision, whereas a high arithmetic mean can be achieved with high Recall at the expense of low Precision, or high Precision at the expense of low Recall. Hashing / Hash / Hash Value: A statistical method used to reduce the contents of a Document to a single, fixed-size, alphanumeric value, which

461 2013] Glossary of Technology-Assisted Review 19 is, for all intents and purposes, unique to a particular Document; the single, fixed-size alphanumeric value resulting from Hashing a particular Document. Common Hashing Algorithms include, but are not limited to, MD5, SHA-1, and SHA-2. Hashing and Hash Values are typically used for Document identification, Deduplication, or ensuring that Documents have not been altered. Horizontal Deduplication: See Global Deduplication. (Cf. Vertical Deduplication.) Index: A list of Keywords in which each Keyword is accompanied by a list of the Documents (and sometimes the positions within the Documents) where it occurs. Manual indices have been used in books for centuries; automatic indices are used in Information Retrieval systems to identify the Documents that contain particular Search Terms. Indexing: The manual or automatic process of creating an Index. In Electronic Discovery, Indexing typically refers to the automatic construction of an electronic Index for use in an Information Retrieval system. Information Need: In Information Retrieval, the information being sought in a search or review effort. In E-Discovery, the Information Need is typically to identify Documents responsive to a request for production, or to identify Documents that are subject to privilege or work-product protection. Information Retrieval: The science of how to find information to meet an Information Need. While modern Information Retrieval relies heavily on computers, the discipline predates the invention of computers. In Re: Actos: In Re: Actos (Pioglitazone) Products Liability Litigation, MDL No. 6:11-md-2299 (W.D. La. July 27, 2012). A product liability action with a Case Management Order (CMO) that memorializes the parties agreement on a search methodology proof of concept to evaluate the potential utility of advanced analytics as a Document identification mechanism for the review and production of Electronically Stored Information. The search protocol provides for the use of a Technology- Assisted Review tool on the of four key custodians. The CMO was issued by District Judge Rebecca F. Doherty. Internal Response Curve: From Signal Detection Theory, a tool for estimating the number of Relevant and Non-Relevant Documents in a Population, or the number of Documents that fall above and below a particular Cutoff. The use of Internal Response Curves for this purpose

462 20 FEDERAL COURTS LAW REVIEW [Vol. 7 assumes that the scores yielded by a Machine Learning Algorithm for Relevant Documents obey a Gaussian Distribution, and the scores for Non- Relevant Documents obey a different Gaussian Distribution. These distributions are then used to predict the number of Relevant and Non- Relevant Documents in any given range of scores. Interval Sample / Interval Sampling: See Systematic Sample / Systematic Sampling. Issue Code(s) / Issue Coding: One or more subcategories of the overall Information Need to be identified in a search or review effort; the act of generating such subcategories of the overall Information Need. Examples include specification of the reason(s) for a determination of Relevance or Non-Relevance, Coding of particular subcategories of interest, and Coding of privileged, confidential, or significant ( hot ) Documents. Iterative Training: The process of repeatedly augmenting the Training Set with additional examples of Coded Documents until the effectiveness of the Machine Learning Algorithm reaches an acceptable level. The additional examples may be identified through Judgmental Sampling, Random Sampling, or by the Machine Learning Algorithm, as in Active Learning. Jaccard Index: A measure of the consistency between two sets (e.g., Documents Coded as Relevant by two different reviewers). Defined mathematically as the size of the intersection of the two sets, divided by the size of the union (e.g., the number of Documents coded as Relevant by both reviewers, divided by the number of Documents identified as Relevant by one or the other, or both reviewers). It is typically used as a measure of consistency among review efforts, but also may be used as a measure of similarity between two Documents represented as two Bags of Words. Jaccard Index is also referred to as Overlap or Mutual F 1. Empirical studies have shown that expert reviewers commonly achieve Jaccard Index scores of about 50%, and that scores exceeding 60% are rare. JASIST Study: A 2009 study (Herbert L. Roitblat, Anne Kershaw & Patrick Oot, Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, 61 J. AM. SOC Y. FOR INFO. SCI. & TECH. 70 (2010)), showing that the Positive Agreement between each of two Technology-Assisted Review methods, and a prior production to the Department of Justice, exceeded the Positive Agreement between each of two Manual Review processes and the same production. Also referred to as the EDI Study.

463 2013] Glossary of Technology-Assisted Review 21 JOLT Study: A 2011 study (Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII RICH. J.L. & TECH. 11 (2011)), available at that used data from TREC 2009 to show that two Technology-Assisted Review processes (one using Machine Learning and one using a Rule Base) generally achieved better Recall, better Precision, and greater efficiency than the TREC Manual Review process. Also known as the Richmond Journal Study, or the Richmond Study. Judgmental Sample / Judgmental Sampling: A method in which a Sample of the Document Population is drawn, based at least in part on subjective factors, so as to include the most interesting Documents by some criterion; the Sample resulting from such method. Unlike a Random Sample, the statistical properties of a Judgmental Sample may not be extrapolated to the entire Population. However, an individual (such as a quality assurance auditor or an adversary) may use Judgmental Sampling to attempt to uncover defects. The failure to identify defects may be taken as evidence (albeit not statistical evidence, and certainly not proof) of the absence of defects. Keyword: A word (or Search Term) that is used as part of a Query in a Keyword Search. Keyword Expansion: See Query Expansion. Keyword Search: A search in which all Documents that contain one or more specific Keywords are returned. Kleen: Kleen Prods. LLC v. Packaging Corp. of Am., Case No. 1:10-cv , various Pleadings and Tr. (N.D. Ill. 2012). A federal case in which plaintiffs sought to compel defendants to use Content-Based Advanced Analytics (CBAA) for their production, after defendants had already employed complex Boolean Searches to identify Responsive Documents. Defendants advanced Elusion scores of 5%, based on a Judgmental Sample of custodians, to defend the reasonableness of the Boolean Search. After two days of evidentiary hearings before (and many conferences with) Magistrate Judge Nan R. Nolan, plaintiffs withdrew their request for CBAA, without prejudice. Knowledge Engineering: The process of capturing the expertise of a Subject Matter Expert in a form (typically a Rule Base) that can be executed by a computer to emulate the human s judgment.

464 22 FEDERAL COURTS LAW REVIEW [Vol. 7 Label / Labeled / Labeling: See Code / Coded / Coding. Latent Semantic Analysis (LSA): See Latent Semantic Indexing. Latent Semantic Indexing (LSI): A Feature Engineering Algorithm that uses linear algebra to group together correlated Features. For example, Windows, Gates, Ballmer might be one group, while Windows, Gates, Doors might be another. Latent Semantic Indexing underlies many Concept Search tools. While Latent Semantic Indexing is used for Feature Engineering in some Technology-Assisted Review tools, it is not, per se, a Technology-Assisted Review method. Also referred to as Latent Semantic Analysis. Linear Review: A Document-by-Document Manual Review in which the Documents are examined in a prescribed order, typically chronological order. Logistic Regression: A state-of-the-art Supervised Learning Algorithm that estimates the Probability that a Document is Relevant, based on the Features it contains. Machine Learning: The use of a computer Algorithm to organize or Classify Documents by analyzing their Features. In the context of Technology-Assisted Review, Supervised Learning Algorithms (e.g., Support Vector Machines, Logistic Regression, Nearest Neighbor, and Bayesian Classifiers) are used to infer Relevance or Non-Relevance of Documents based on the Coding of Documents in a Training Set. In Electronic Discovery generally, Unsupervised Learning Algorithms are used for Clustering, Near-Duplicate Detection, and Concept Search. Manual Review: The practice of having human reviewers individually read and Code the Documents in a Collection for Responsiveness, particular issues, privilege, and/or confidentiality. Margin of Error: The maximum amount by which a Point Estimate might likely deviate from the true value, typically expressed as plus or minus a percentage, with a particular Confidence Level. For example, one might express a Statistical Estimate as 30% of the Documents in the Population are Relevant, plus or minus 3%, with 95% confidence. This means that the Point Estimate is 30%, the Margin of Error is 3%, the Confidence Interval is 27% to 33%, and the Confidence Level is 95%. Using Gaussian Estimation, the Margin of Error is one-half of the size of the Confidence Interval. It is important to note that when the Margin of Error is expressed as a percentage, it refers to a percentage of the Population, not to a

465 2013] Glossary of Technology-Assisted Review 23 percentage of the Point Estimate. In the current example, if there are one million Documents in the Document Population, the Statistical Estimate may be restated as 300,000 Documents in the Population are Relevant, plus or minus 30,000 Documents, with 95% confidence ; or, alternatively, between 270,000 and 330,000 Documents in the Population are Relevant, with 95% confidence. The Margin of Error is commonly misconstrued to be a percentage of the Point Estimate. However, it would be incorrect to interpret the Confidence Interval in this example to mean that 300,000 Documents in the Population are Relevant, plus or minus 9,000 Documents. The fact that a Margin of Error of plus or minus 3% has been achieved is not, by itself, evidence of a precise Statistical Estimate when the Prevalence of Relevant Documents is low. Miss / Missed: A Relevant Document that is not identified as Relevant by a search or review effort. Also referred to as a False Negative. Miss Rate: The fraction (or proportion) of truly Relevant Documents that are not identified as Relevant by a search or review effort. Miss Rate = 100% Recall. Also referred to as the False Negative Rate. Model: See Statistical Model. Mutual F 1 : See Jaccard Index. Naïve Bayes: A Supervised Learning Algorithm in which the relative frequency of words (or other Features) in Relevant and Non-Relevant Training Examples is used to estimate the likelihood that a new Document containing those words (or other Features) is Relevant. Naïve Bayes relies on the simplistic assumption that the words in a Document occur with independent Probabilities, with the consequence that it tends to yield extremely low or extremely high estimates. NDLON: National Day Laborer Organizing Network v. U.S. Immigration and Customs Enforcement Agency, Case No. 10-Civ-3488 (SAS), 2012 WL (S.D.N.Y. July 13, 2012), a Freedom of Information Act (FOIA) case in which District Judge Shira A. Scheindlin held that most custodians cannot be trusted to run effective searches because designing legally sufficient electronic searches in the discovery or FOIA contexts is not part of their daily responsibilities, and stated (in dicta) that beyond the use of keyword search, parties can (and frequently should) rely on latent semantic indexing, statistical probability models, and machine learning to find responsive documents. Through iterative learning, these methods (known as computer-assisted or predictive coding) allow humans to teach computers what documents are and are not responsive to a particular FOIA

466 24 FEDERAL COURTS LAW REVIEW [Vol. 7 or discovery request and they can significantly increase the effectiveness and efficiency of searches. Near-Duplicate Detection: An industry-specific term generally used to describe a method of grouping together nearly identical Documents. Near-Duplicate Detection is a variant of Clustering in which the similarity among Documents in the same group is very strong. It is typically used to reduce review costs, and to ensure consistent Coding. Also referred to as Near-Deduplication. Near-Deduplication: See Near-Duplicate Detection. Nearest Neighbor: A Supervised Learning Algorithm in which a new Document is Classified by finding the most similar Document in the Training Set, and assuming that the correct Coding for the new Document is the same as the most similar one in the Training Set. Negative Predictive Value (NPV): The fraction (Proportion) of Documents that are identified as Non-Relevant by a search or review effort, that are in fact Non-Relevant. The complement of Precision; that is, Negative Predictive Value is computed the same way as Precision when the definitions of Relevant and Non-Relevant are transposed. N-Gram: N consecutive words or characters treated as a Feature. In the phrase, To be or not to be, a word Bigram (i.e., 2-gram) would be to be ; a word Trigram (i.e., 3-gram) would be to be or ; a Quad-Gram (i.e., 4- gram) would be to be or not ; and so on. See also Shingling. Non-Relevant / Not Relevant: In Information Retrieval, a Document is considered Non-Relevant (or Not Relevant) if it does not meet the Information Need of the search or review effort. The synonym irrelevant is rarely used in Information Retrieval. Normal Distribution: The bell curve of classical statistics. The number of Relevant Documents in a Sample tends to obey a Normal (Gaussian) Distribution, provided the Sample size is large enough to capture a substantial number of Relevant and Non-Relevant Documents. In this situation, Gaussian Estimation is reasonably accurate. If the Sample size is insufficiently large to capture a substantial number of both Relevant and Non-Relevant Documents (as a rule of thumb, at least 12 of each), the Binomial Distribution better characterizes the number of Relevant Documents in the Sample, and Binomial Estimation is more appropriate. Also referred to as a Gaussian Distribution.

467 2013] Glossary of Technology-Assisted Review 25 Null Set: The set of Documents that are not returned by a search process, or that are identified as Not Relevant by a review process. Ontology: A representation of the relationships among words and their meanings that is richer than a Taxonomy. For example, an Ontology can represent the fact that a wheel is a part of a bicycle, that gold is yellow, and so on. Overlap: See Jaccard Index. Parallel Trial: An Experimental Design for comparing two search or review processes using the same Document Collection and Information Need, in which both processes are applied concurrently but independently, and then the results of the two efforts are compared. (Cf. Crossover Trial.) Pattern Matching: The science of designing computer Algorithms to recognize natural phenomena like parts of speech, faces, or spoken words. Point Estimate: The most likely value for a Population characteristic. When combined with a Margin of Error (or Confidence Interval) and a Confidence Level, it reflects a Statistical Estimate. Population: See Document Population. Positive Agreement: The Probability that, if one reviewer Codes a Document as Relevant, a second independent reviewer will also Code the Document as Relevant. Empirical studies show that Positive Agreement rates of 70% are typical, and Positive Agreement rates of 80% are rare. Positive Agreement should not be confused with Agreement (which is a less informative measure) or Overlap (which is a numerically smaller measure that conveys similar information). Under the assumption that the two reviewers are equally likely to err, Overlap is roughly equal to the square of Positive Agreement. That is, if Positive Agreement is 70%, Overlap is roughly 70% 70% = 49%. Positive Predictive Value (PPV): See Precision. Positive Predictive Value is a term used in Signal Detection Theory; Precision is the equivalent term in Information Retrieval. Precision: The fraction of Documents identified as Relevant by a search or review effort, that are in fact Relevant. Also referred to as Positive Predictive Value.

468 26 FEDERAL COURTS LAW REVIEW [Vol. 7 Precision-Recall Curve: The curve representing the tradeoff between Precision and Recall for a given search or review effort, depending on the chosen Cutoff value. See Recall-Precision Curve. Precision-Recall Tradeoff: The notion that most search strategies can be adjusted to increase Precision at the expense of Recall, or vice versa. At one extreme, 100% Recall could be achieved by a search that returned the entire Document Population, but Precision would be low (equal to Prevalence). At the other extreme, 100% Precision could be achieved by a search that returned a single Relevant Document, but Recall would be low (equal to 1/N, where N is the number of Relevant Documents in the Document Population). More generally, a broader search returning many Documents will have higher Recall and lower Precision, while a narrower search returning fewer Documents will have lower Recall and higher Precision. A Precision-Recall Curve illustrates the Precision-Recall Tradeoff for a particular search method. Predictive Coding: An industry-specific term generally used to describe a Technology-Assisted Review process involving the use of a Machine Learning Algorithm to distinguish Relevant from Non-Relevant Documents, based on Subject Matter Expert(s) Coding of a Training Set of Documents. See Supervised Learning and Active Learning. Prevalence: The fraction of Documents in a Population that are Relevant to an Information Need. Also referred to as Richness or Yield. Prioritization / Prioritized: See Relevance Ranking. Probabilistic Latent Semantic Analysis: A variant of Latent Semantic Analysis based on conditional Probability rather than on correlation. Probability: The fraction (proportion) of times that a particular outcome would occur, should the same action be repeated under the same conditions an infinite number of times. For example, if one were to flip a fair coin, the Probability of it landing heads is one-half, or 50%; as one repeats this action indefinitely, the fraction of times that the coin lands heads will become indistinguishable from 50%. If one were to flip two fair coins, the Probability of both landing heads is one-quarter, or 25%. Proportion: The fraction of a set of Documents having some particular property (typically Relevance). Proportionality: Pursuant to Federal Rules of Civil Procedure 26(b)(2)(B), 26(b)(2)(C), 26(g)(1)(B)(iii), and other federal and state procedural rules,

469 2013] Glossary of Technology-Assisted Review 27 the legal doctrine that Electronically Stored Information may be withheld from production if the cost and burden of producing it exceeds its potential value to the resolution of the matter. Proportionality has been interpreted in the case law to apply to preservation as well as production. Quad-Gram: An N-Gram where N = 4 (i.e., a 4-gram). Quality Assurance: A method to ensure, after the fact, that a search or review effort has achieved reasonable results. Quality Control: Ongoing methods to ensure, during a search or review effort, that reasonable results are being achieved. Query: A formal search command provided as input to a search tool. Query Expansion: The process of adding Search Terms to a Query to improve Recall, often at the expense of decreased Precision. Random Sample / Random Sampling: A subset of the Document Population selected by a method that is equally likely to select any Document from the Document Population for inclusion in the Sample; the Sample resulting from such action. Random Sampling is the basis of Statistical Estimation. RAND Study: A 2012 study (Nicholas M. Pace & Laura Zakaras, Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery, RAND Institute for Civil Justice (2012)), indicating that Document review accounts for 73% of Electronic Discovery costs, and concluding that [t]he exponential growth in digital information, which shows no signs of slowing, makes a computer-categorized review strategy, such as predictive coding, not only a cost-effective choice but perhaps the only reasonable way to handle many large-scale productions. Ranking: See Relevance Ranking. Recall: The fraction of Relevant Documents that are identified as Relevant by a search or review effort. Recall-Precision Curve: See Precision-Recall Curve. Recall-Precision Tradeoff: See Precision-Recall Tradeoff.

470 28 FEDERAL COURTS LAW REVIEW [Vol. 7 Receiver Operating Characteristic Curve (ROC): In Signal Detection Theory, a graph of the tradeoff between True Positive Rate and False Positive Rate, as the Cutoff is varied. Relevance Feedback: An Active Learning process in which the Documents with the highest likelihood of Relevance are coded by a human, and added to the Training Set. Relevance Ranking: A search method in which the results are ranked from the most likely to the least likely to be Relevant to an Information Need; the result of such ranking. Google Web Search is an example of Relevance Ranking. Relevance / Relevant: In Information Retrieval, a Document is considered Relevant if it meets the Information Need of the search or review effort. Responsiveness: A Document that is Relevant to an Information Need expressed by a particular request for production or subpoena in a civil, criminal, or regulatory matter. Richness: See Prevalence or Yield. Richmond Journal Study / Richmond Study: See JOLT Study. Roitblat, Kershaw, and Oot: Authors of the JASIST (a.k.a. EDI) Study. Rolling Collection / Rolling Ingestion: A process in which the Document Collection is periodically augmented as new, potentially Relevant Documents are identified and gathered. Whenever the Document Collection is augmented, the results of prior search or review efforts must be supplemented to account for the new Documents. Rolling Production: A process in which Responsive Documents are delivered incrementally to a requesting party to provide timely, partial satisfaction of a Document request. Rule: A formal statement of one or more criteria used to determine a particular outcome, e.g., whether to code a Document as Relevant or Non- Relevant. Rule Base: A set of Rules created by an expert to emulate the human decision-making process for the purposes of Classifying Documents in the context of Electronic Discovery.

471 2013] Glossary of Technology-Assisted Review 29 Sample / Sampling: A subset of the Document Population used to assess some characteristic of the Population; the act of generating such a subset of the Document Population. See Interval Sample, Judgmental Sample, Random Sample, Statistical Sample, or Systematic Sample. Sample Size: The number of Documents drawn at random that are used to calculate a Statistical Estimate. Search Term: See Keyword. Sedona / Sedona Conference: The Sedona Conference (https://thesedona conference.org) is a nonprofit, 501(c)(3) research and educational institute, founded in 1997 by Richard G. Braman, dedicated to the advanced study of law and policy in the areas of antitrust, complex litigation, and intellectual property rights. Sedona sponsors a preeminent think-tank in the area of Electronic Discovery known as Working Group 1 on Electronic Document Retention and Production. Sedona is well known for its thoughtful, balanced, and free publications, such as The Sedona Conference Glossary: E-Discovery & Digital Information Management (Third Edition, Sept. 2010), The Sedona Principles Addressing Electronic Document Production, Second Edition (June 2007), and The Sedona Conference Cooperation Proclamation (July 2008). Seed Set: The initial Training Set provided to the learning Algorithm in an Active Learning process. The Documents in the Seed Set may be selected based on Random Sampling or Judgmental Sampling. Some commentators use the term more restrictively to refer only to Documents chosen using Judgmental Sampling. Other commentators use the term generally to mean any Training Set, including the final Training Set in Iterative Training, or the only Training Set in non-iterative Training. Sensitivity: See True Positive Rate. Shingling: A Feature Engineering method in which the Features consist of all N-Grams in a text, for some number N. For example, the Trigram Shingling of the text To be or not to be consists of the features to be or ; be or not ; or not to ; and not to be. Note that the Features overlap one another in the text, suggesting the metaphor of roof shingles. Signal Detection Theory: Invented at the same time and in conjunction with radar, the science of distinguishing true observations from spurious ones. Signal Detection Theory is widely used in radio engineering and medical diagnostic testing. The terms True Positive, True Negative, False Positive, False Negative, Sensitivity, Specificity, Receiver Operating

472 30 FEDERAL COURTS LAW REVIEW [Vol. 7 Characteristic Curve, Area Under the ROC Curve, and Internal Response Curve, all arise from Signal Detection Theory. Significance / Significant: The confirmation, with a given Confidence Level, of a prior hypothesis, using a Statistical Estimate. The result is said to be Statistically Significant if all values within the Confidence Interval for the desired Confidence Level (typically 95%) are consistent with the hypothesis being true, and inconsistent with it being false. For example, if the hypothesis is that fewer than 300,000 Documents are Relevant, and a Statistical Estimate shows that 290,000 Documents are Relevant, plus or minus 5,000 Documents, we say that the result is Significant. On the other hand, if the Statistical Estimate shows that 290,000 Documents are Relevant, plus or minus 15,000 Documents, we say that the result is not Significant, because the Confidence Interval includes values (i.e., the values between 300,000 and 305,000) that contradict the hypothesis. Specificity: See True Negative Rate. Statistical Estimate: A quantitative estimate of a Population characteristic using Statistical Estimation. It is generally expressed as a Point Estimate accompanied by a Margin of Error and a Confidence Level, or as a Confidence Interval accompanied by a Confidence Level. Statistical Estimation: The act of estimating the Proportion of a Document Population that has a particular characteristic, based on the Proportion of a Random Sample that has the same characteristic. Methods of Statistical Estimation include Binomial Estimation and Gaussian Estimation. Statistically Significant / Statistical Significance: See Significance. Statistical Model: A mathematical abstraction of the Document Population that removes irrelevant characteristics while largely preserving those of interest for a particular purpose. For the purpose of computing Recall, a Statistical Model need only consider whether or not the Documents are Relevant, and whether or not the Documents are Coded Relevant, not any other characteristics of the Documents. Statistical Sample / Statistical Sampling: A method in which a Sample of the Document Population is drawn at random, so that statistical properties of the Sample may be extrapolated to the entire Document Population; the Sample resulting from such action. Stemming: In Keyword or Boolean Search, or Feature Engineering, the process of equating all forms of the same root word. For example, the

473 2013] Glossary of Technology-Assisted Review 31 words stem, stemming, stemmed, and stemmable would all be treated as equivalent, and would each yield the same result when used as Search Terms in a Query. In some search systems, stemming is implicit, and in others, it must be made explicit through particular Query syntax. Stop Word: A common word that is eliminated from Indexing. Eliminating Stop Words from Indexing dramatically reduces the size of the Index, while only marginally affecting the search process in most circumstances. Examples of Stop Words include a, the, of, but, and not. Because phrases and names such as To be or not to be, and The Who, contain exclusively Stop Words that would not be Indexed, they would not be identified (or identifiable) through a Keyword Search. Subject Matter Expert(s): One or more individuals (typically, but not necessarily, attorneys) who are familiar with the Information Need and can render an authoritative determination as to whether a Document is Relevant or not. Supervised Learning: A Machine Learning method in which the learning Algorithm infers how to distinguish between Relevant and Non-Relevant Documents using a Training Set. Supervised Learning can be a stand-alone process, or used repeatedly in an Active Learning process. Support Vector Machine: A state-of-the-art Supervised Learning Algorithm that separates Relevant from Non-Relevant Documents using geometric methods (i.e., geometry). Each Document is considered to be a point in [hyper]space, whose coordinates are determined from the Features contained in the Document. The Support Vector Machine finds a [hyper]plane that best separates Relevant from Non-Relevant Training Examples. Documents outside the Training Set (i.e., uncoded Documents from the Document Collection) are then Classified as Relevant or not, depending on which side of the [hyper]plane they fall on. Although a Support Vector Machine does not calculate a Probability of Relevance, one may infer that the Classification of Documents closer to the [hyper]plane is less certain than for those that are far from the [hyper]plane. Synthetic Document: An industry-specific term generally used to describe an artificial Document created by either the requesting party or the producing party, as part of a Technology-Assisted Review process, for use as a Training Example for a Machine Learning Algorithm. Synthetic Documents are contrived Documents in which one party imagines what the evidence might look like and relies on the Machine Learning Algorithm to find actual Documents that are similar to the artificial Document.

474 32 FEDERAL COURTS LAW REVIEW [Vol. 7 Systematic Sample / Systematic Sampling: A Sampling method in which every Nth Document (for some fixed number N) is selected, when the Documents are considered in some prescribed order; the Sample resulting from such action. A Systematic Sample is random (and hence a true Statistical Sample) only when the prescribed order is itself random. Sometimes referred to as an Interval Sample / Interval Sampling. TAR: See Technology-Assisted Review. Taxonomy: A hierarchical organizational scheme that arranges the meanings of words into classes and subclasses. For example, vehicles, aircraft, and ships are modes of transportation; cars, trucks, and bicycles are vehicles, and Fords and Chryslers are cars. Technology-Assisted Review (TAR): A process for Prioritizing or Coding a Collection of Documents using a computerized system that harnesses human judgments of one or more Subject Matter Expert(s) on a smaller set of Documents and then extrapolates those judgments to the remaining Document Collection. Some TAR methods use Machine Learning Algorithms to distinguish Relevant from Non-Relevant Documents, based on Training Examples Coded as Relevant or Non-Relevant by the Subject Matter Experts(s), while other TAR methods derive systematic Rules that emulate the expert(s) decision-making process. TAR processes generally incorporate Statistical Models and/or Sampling techniques to guide the process and to measure overall system effectiveness. Term Expansion: See Query Expansion. Term Frequency and Inverse Document Frequency (TF-IDF): An enhancement to the Bag of Words method in which each word has a weight based on Term Frequency the number of times the word appears in the Document and Inverse Document Frequency the reciprocal of the number of Documents in which the word occurs. Thesaurus Expansion: In Keyword or Boolean Search, replacing a single Search Term by a list of its synonyms, as listed in a thesaurus. Threshold: See Cutoff. Training Example: One Document from a Training Set. Training Set: A Sample of Documents coded by one or more Subject Matter Expert(s) as Relevant or Non-Relevant, from which a Machine

475 2013] Glossary of Technology-Assisted Review 33 Learning Algorithm then infers how to distinguish between Relevant and Non-Relevant Documents beyond those in the Training Set. TREC: The Text REtrieval Conference, sponsored by the National Institute of Standards and Technology (NIST), which has run since 1992, to support research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies. In particular, the TREC workshop series has the following goals: to encourage research in information retrieval based on large test Collections; to increase communication among industry, academia, and government by creating an open forum for the exchange of research ideas; to speed the transfer of technology from research labs into commercial products by demonstrating substantial improvements in retrieval methodologies on real-world problems; and to increase the availability of appropriate evaluation techniques for use by industry and academia, including development of new evaluation techniques more applicable to current systems. TREC Legal Track: From 2006 through 2011, TREC included a Legal Track, which sought to assess the ability of information retrieval techniques to meet the needs of the legal profession for tools and methods capable of helping with the retrieval of electronic business records, principally for use as evidence in civil litigation. Trigram: An N-Gram where N = 3 (i.e., a 3-gram). True Negative (TN): A Non-Relevant Document that is correctly identified as Non-Relevant by a search or review effort. True Negative Rate (TNR): The fraction (or Proportion) of Non-Relevant Documents that are correctly identified as Non-Relevant by a search or review effort. True Positive (TP): A Relevant Document that is correctly identified as Relevant by a search or review effort. True Positive Rate (TPR): The fraction (or Proportion) of Relevant Documents that are correctly identified as Relevant by a search or review effort. True Positive Rate is a term used in Signal Detection Theory; Recall is the equivalent term in Information Retrieval. Uncertainty Sampling: An Active Learning approach in which the Machine Learning Algorithm selects the Documents as to which it is least

476 34 FEDERAL COURTS LAW REVIEW [Vol. 7 certain about Relevance, for Coding by the Subject Matter Expert(s), and addition to the Training Set. Unsupervised Learning: A Machine Learning method in which the learning Algorithm infers categories of similar Documents without any training by a Subject Matter Expert. Examples of Unsupervised Learning methods include Clustering and Near-Duplicate Detection. Validation: The act of confirming that a process has achieved its intended purpose. Validation may involve Statistical or Judgmental Sampling. Vertical Deduplication: Deduplication within a custodian; identical copies of a Document held by different custodians are not Deduplicated. (Cf. Horizontal Deduplication.) Yield: See Prevalence or Richness.

477 Technology Assisted Review

Xact Data Discovery. Xact Data Discovery. Xact Data Discovery. Xact Data Discovery. ediscovery for DUMMIES LAWYERS. MDLA TTS August 23, 2013

Xact Data Discovery. Xact Data Discovery. Xact Data Discovery. Xact Data Discovery. ediscovery for DUMMIES LAWYERS. MDLA TTS August 23, 2013 MDLA TTS August 23, 2013 ediscovery for DUMMIES LAWYERS Kate Burke Mortensen, Esq. kburke@xactdatadiscovery.com Scott Polus, Director of Forensic Services spolus@xactdatadiscovery.com 1 Where Do I Start??

More information

FEDERAL PRACTICE. In some jurisdictions, understanding the December 1, 2006 Amendments to the Federal Rules of Civil Procedure is only the first step.

FEDERAL PRACTICE. In some jurisdictions, understanding the December 1, 2006 Amendments to the Federal Rules of Civil Procedure is only the first step. A BNA, INC. DIGITAL DISCOVERY & E-EVIDENCE! VOL. 7, NO. 11 232-235 REPORT NOVEMBER 1, 2007 Reproduced with permission from Digital Discovery & e-evidence, Vol. 7, No. 11, 11/01/2007, pp. 232-235. Copyright

More information

Discussion of Electronic Discovery at Rule 26(f) Conferences: A Guide for Practitioners

Discussion of Electronic Discovery at Rule 26(f) Conferences: A Guide for Practitioners Discussion of Electronic Discovery at Rule 26(f) Conferences: A Guide for Practitioners INTRODUCTION Virtually all modern discovery involves electronically stored information (ESI). The production and

More information

UNDERSTANDING E DISCOVERY A PRACTICAL GUIDE. 99 Park Avenue, 16 th Floor New York, New York 10016 www.devoredemarco.com

UNDERSTANDING E DISCOVERY A PRACTICAL GUIDE. 99 Park Avenue, 16 th Floor New York, New York 10016 www.devoredemarco.com UNDERSTANDING E DISCOVERY A PRACTICAL GUIDE 1 What is ESI? Information that exists in a medium that can only be read through the use of computers Examples E-mail Word Documents Databases Spreadsheets Multimedia

More information

GUIDELINES FOR USE OF THE MODEL AGREEMENT REGARDING DISCOVERY OF ELECTRONICALLY STORED INFORMATION

GUIDELINES FOR USE OF THE MODEL AGREEMENT REGARDING DISCOVERY OF ELECTRONICALLY STORED INFORMATION GUIDELINES FOR USE OF THE MODEL AGREEMENT REGARDING DISCOVERY OF ELECTRONICALLY STORED INFORMATION Experience increasingly demonstrates that discovery of electronically stored information ( ESI poses challenges

More information

Reduce Cost and Risk during Discovery E-DISCOVERY GLOSSARY

Reduce Cost and Risk during Discovery E-DISCOVERY GLOSSARY 2016 CLM Annual Conference April 6-8, 2016 Orlando, FL Reduce Cost and Risk during Discovery E-DISCOVERY GLOSSARY Understanding e-discovery definitions and concepts is critical to working with vendors,

More information

Electronic Discovery and the New Amendments to the Federal Rules of Civil Procedure: A Guide For In-House Counsel and Attorneys

Electronic Discovery and the New Amendments to the Federal Rules of Civil Procedure: A Guide For In-House Counsel and Attorneys Electronic Discovery and the New Amendments to the Federal Rules of Civil Procedure: A Guide For In-House Counsel and Attorneys By Ronald S. Allen, Esq. As technology has evolved, the federal courts have

More information

ediscovery Technology That Works for You

ediscovery Technology That Works for You ediscovery Technology That Works for You Peace of Mind for Serious ediscovery ediscovery demands options, and having one of the industry s most comprehensive portfolios of proprietary and best-of-breed

More information

E-Discovery Technology Considerations

E-Discovery Technology Considerations E-Discovery Technology Considerations Presented by: Dave Howard Oregon Department of Justice Deputy CIO Topics E-Discovery Process Overview Sources of Electronically Stored Information (ESI) Data Maps

More information

DISCOVERY OF ELECTRONICALLY-STORED INFORMATION IN STATE COURT: WHAT TO DO WHEN YOUR COURT S RULES DON T HELP

DISCOVERY OF ELECTRONICALLY-STORED INFORMATION IN STATE COURT: WHAT TO DO WHEN YOUR COURT S RULES DON T HELP DISCOVERY OF ELECTRONICALLY-STORED INFORMATION IN STATE COURT: WHAT TO DO WHEN YOUR COURT S RULES DON T HELP Presented by Frank H. Gassler, Esq. Written by Jeffrey M. James, Esq. Over the last few years,

More information

2004 E-Discovery Developments: Year in Review

2004 E-Discovery Developments: Year in Review 2004 E-Discovery Developments: Year in Review Sean Foley, Esq., Legal Consultant Michele C.S. Lange, Esq., Staff Attorney, Legal Technologies January 20, 2005 Presenters Sean Foley, Esq., Legal Consultant

More information

Electronic Discovery: Litigation Holds, Data Preservation and Production

Electronic Discovery: Litigation Holds, Data Preservation and Production Electronic Discovery: Litigation Holds, Data Preservation and Production April 27, 2010 Daniel Munsch, Assistant General Counsel John Lerchey, Coordinator for Incident Response 0 E-Discovery Rules Federal

More information

Cyber Tech & E-Commerce

Cyber Tech & E-Commerce MEALEY S LITIGATION REPORT Cyber Tech & E-Commerce The Duty To Preserve Data Stored Temporarily In Ram: Is The Sky Really Falling? by J. Alexander Lawrence Morrison & Foerster New York, New York A commentary

More information

In-House Solutions to the E-Discovery Conundrum

In-House Solutions to the E-Discovery Conundrum 125 In-House Solutions to the E-Discovery Conundrum Retta A. Miller Carl C. Butzer Jackson Walker L.L.P. April 21, 2007 www.pointmm.com I. OVERVIEW OF THE RULES GOVERNING ELECTRONICALLY- STORED INFORMATION

More information

Proactive Data Management for ediscovery

Proactive Data Management for ediscovery Proactive Data Management for ediscovery Simon Taylor Snr. Director Information Management CommVault Systems Inc. Why ediscovery sucks for IT The US Federal Rules of Civil Procedure Rule 34(a), (b) Definition

More information

General Items Of Thought

General Items Of Thought ESI PROTOCOLS & CASE LONG BUDGETS General Items Of Thought What s a GB =??? What Are Sources Of Stored Data? What s BYOD mean??? The Human Factor Is At Play! Litigation Hold Duty Arises When? Zubulake

More information

Presented By: Attorney/Legal Technologist Cosgrave Vergeer Kester LLP. Paine Hamblen LLP 805 SW Broadway, 8 th Floor

Presented By: Attorney/Legal Technologist Cosgrave Vergeer Kester LLP. Paine Hamblen LLP 805 SW Broadway, 8 th Floor Everything You Wanted to Know About ESI and E-Discovery but Were Afraid to Ask Jason M. Pistacchio Presented By: Gregory S. Johnson Attorney Attorney/Legal Technologist Cosgrave Vergeer Kester LLP Paine

More information

COURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Fall 2014

COURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Fall 2014 COURSE DESCRIPTION AND SYLLABUS LITIGATING IN THE DIGITAL AGE: ELECTRONIC CASE MANAGEMENT (994-001) Professors:Mark Austrian Christopher Racich Fall 2014 Introduction The ubiquitous use of computers, the

More information

Navigating Information Governance and ediscovery

Navigating Information Governance and ediscovery Navigating Information Governance and ediscovery Implementing Processes & Technology to Reduce Downstream ediscovery Cost and Risk Shannon Smith General Counsel, Globanet March 11 12, 2013 Agenda 1 Overview

More information

CORPORATE RECORD RETENTION IN AN ELECTRONIC AGE (Outline)

CORPORATE RECORD RETENTION IN AN ELECTRONIC AGE (Outline) CORPORATE RECORD RETENTION IN AN ELECTRONIC AGE (Outline) David J. Chavolla, Esq. and Gary L. Kemp, Esq. Casner & Edwards, LLP 303 Congress Street Boston, MA 02210 A. Document and Record Retention Preservation

More information

Electronic documents questionnaire

Electronic documents questionnaire Electronic documents questionnaire (Civil Procedure Rules Practice Direction 31B) WARNING: Unless the court makes some other order, the answers given in this document may only be used for the purposes

More information

ESI DEMYSTIFIED. Streamlining the E-Discovery Process Through Internal Processes and Controls. Melinda Burrows Bruce Cosgrove*

ESI DEMYSTIFIED. Streamlining the E-Discovery Process Through Internal Processes and Controls. Melinda Burrows Bruce Cosgrove* ESI DEMYSTIFIED Streamlining the E-Discovery Process Through Internal Processes and Controls Melinda Burrows Bruce Cosgrove* The widespread proliferation of electronically stored information (so-called

More information

Predictability in E-Discovery

Predictability in E-Discovery Predictability in E-Discovery Presented by: John G. Roman, Jr. National Manager, Practice Group Technology Services Nixon Peabody LLP Tom Barce Assistant Director of Practice Support Fulbright & Jaworski

More information

Case 2:14-cv-02159-KHV-JPO Document 12 Filed 07/10/14 Page 1 of 10 IN THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF KANSAS

Case 2:14-cv-02159-KHV-JPO Document 12 Filed 07/10/14 Page 1 of 10 IN THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF KANSAS Case 2:14-cv-02159-KHV-JPO Document 12 Filed 07/10/14 Page 1 of 10 IN THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF KANSAS KYLE ALEXANDER, and DYLAN SYMINGTON, on behalf of themselves and all those

More information

Amendments to Federal Rules of Civil Procedure. electronically stored information. 6 Differences from Paper Documents

Amendments to Federal Rules of Civil Procedure. electronically stored information. 6 Differences from Paper Documents Amendments to Federal Rules of Civil Procedure Electronic Discovery effective Dec. 1, 2006 Copyright David A. Devine GROH EGGERS, LLC Rules amended: 16, 26, 33, 34, 37 & 45 Sources of information: Rules

More information

REINHART. Labor & Employment E-News E-NEWSLETTER ATTORNEYS:

REINHART. Labor & Employment E-News E-NEWSLETTER ATTORNEYS: REINHART E-NEWSLETTER ATTORNEYS: ROBERT K. SHOLL, CHAIR JEFFREY P. CLARK JOHN H. ZAWADSKY LYNN M. STATHAS DAVID J. SISSON CHRISTOPHER P. BANASZAK ROBERT J. MUTEN DARYLL J. NEUSER SUSAN B. WOODS JENNIFER

More information

E-Discovery and Electronically Stored Information (ESI):

E-Discovery and Electronically Stored Information (ESI): E-Discovery and Electronically Stored Information (ESI): How Can It Help or Hinder a Case? Rosevelie Márquez Morales Harris Beach PLLC New York, NY Rosevelie Márquez Morales is a partner at Harris Beach

More information

Florida E-Discovery 2013

Florida E-Discovery 2013 Florida E-Discovery 2013 Christopher.Hopkins @Akerman.com Palm Beach Bar Association Employment Law Committee Florida E-Discovery 2013 Download This PPT: InternetLawCommentary.com Palm Beach Bar Association

More information

for Insurance Claims Professionals

for Insurance Claims Professionals A Practical Guide to Understanding ediscovery for Insurance Claims Professionals ediscovery Defined and its Relationship to an Insurance Claim Simply put, ediscovery (or Electronic Discovery) refers to

More information

A Brief Overview of ediscovery in California

A Brief Overview of ediscovery in California What is ediscovery? Electronic discovery ( ediscovery ) is discovery of electronic information in litigation. ediscovery in California is governed generally by the Civil Discovery Act. In 2009, the California

More information

Best Practices for Enforcing Legal Holds on E-Mail and Electronic Data through Proactive Archiving Sponsored by Symantec

Best Practices for Enforcing Legal Holds on E-Mail and Electronic Data through Proactive Archiving Sponsored by Symantec WHITE PAPER: Best Practices for legal holds Confidence in a connected world. Best Practices for Enforcing Legal Holds on E-Mail and Electronic Data through Proactive Archiving Sponsored by Symantec Jennifer

More information

Legal Arguments & Response Strategies for E-Discovery

Legal Arguments & Response Strategies for E-Discovery Legal Arguments & Response Strategies for E-Discovery The tools to craft strategic discovery requests & mitigate the risks and burdens of production. Discussion Outline Part I Strategies for Requesting

More information

Measures Regarding Litigation Holds and Preservation of Electronically Stored Information (ESI)

Measures Regarding Litigation Holds and Preservation of Electronically Stored Information (ESI) University of California, Merced Measures Regarding Litigation Holds and Preservation of Electronically Stored Information (ESI) Responsible Officials: Executive Vice Chancellor and Provost Vice Chancellor

More information

ediscovery 101 Myth Busting October 29, 2009 Olivia Gerroll ediscovery Solutions Group Director

ediscovery 101 Myth Busting October 29, 2009 Olivia Gerroll ediscovery Solutions Group Director ediscovery 101 Myth Busting October 29, 2009 Olivia Gerroll ediscovery Solutions Group Director Background Olivia Gerroll, ediscovery Solutions Group Director Over sixteen years of experience in litigation

More information

ZL UNIFIED ARCHIVE A Project Manager s Guide to E-Discovery. ZL TECHNOLOGIES White Paper

ZL UNIFIED ARCHIVE A Project Manager s Guide to E-Discovery. ZL TECHNOLOGIES White Paper ZL UNIFIED ARCHIVE A Project Manager s Guide to E-Discovery ZL TECHNOLOGIES White Paper PAGE 1 A project manager s guide to e-discovery In civil litigation, the parties in a dispute are required to provide

More information

Article originally appeared in the Fall 2011 issue of The Professional Engineer

Article originally appeared in the Fall 2011 issue of The Professional Engineer Article originally appeared in the Fall 2011 issue of The Professional Engineer Electronic Discovery in Litigation By Douglas P. Jeremiah, P.E., Esq. Your firm is involved in litigation and you get the

More information

E-Discovery Best Practices

E-Discovery Best Practices José Ramón González-Magaz jrgonzalez@steptoe.com E-Discovery Best Practices www.steptoe.com November 10, 2010 Importance of E-Discovery 92% of all data is ESI. Source: Berkeley Study. 97 billion e-mails

More information

BEST PRACTICES FOR PREPARING YOUR BUSINESS FOR E-DISCOVERY

BEST PRACTICES FOR PREPARING YOUR BUSINESS FOR E-DISCOVERY BEST PRACTICES FOR PREPARING YOUR BUSINESS FOR E-DISCOVERY I. Background The Federal Rules of Civil Procedure provide for document production in the discovery process. Until recently, all types of documents

More information

www.salixdata.com 513-381-2679

www.salixdata.com 513-381-2679 Electronic Discovery Presented by: Jonathan Adams www.salixdata.com 513-381-2679 Our Goal Explain E-Discovery in layman s terms Equip you to be able to add value to your organization SALIX is the region

More information

Best Practices Page 1

Best Practices Page 1 BEST PRACTICES FOR ELECTRONIC DISCOVERY IN CRIMINAL CASES Western District of Washington Adopted March 21, 2013 These best practices reflect recommendations adopted in February 2012 by the Department of

More information

IN THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF COLORADO

IN THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF COLORADO Regents of the University of Colorado, The v. Allergan, Inc. et al Doc. 69 Civil Action No. 1:14-cv-01562-MSK-NYW IN THE UNITED STATES DISTRICT COURT FOR THE DISTRICT OF COLORADO THE REGENTS OF THE UNIVERSITY

More information

LEGAL HOLD OBLIGATIONS FOR DISTRICT EMPLOYEES

LEGAL HOLD OBLIGATIONS FOR DISTRICT EMPLOYEES LEGAL HOLD OBLIGATIONS FOR DISTRICT EMPLOYEES INSERT YOUR NAME HERE Place logo or logotype here, Otherwise delete this text box. AGENDA.. Federal Rules of Civil Procedure What is a legal hold? What are

More information

The E-Discovery Process

The E-Discovery Process POOLING PROVISIONS The E-Discovery Process A publication of Nevada Public Agency Insurance Pool The e- discovery process the search of electronic records for use as legal evidence can cost thousands of

More information

How to Win the Battle Over Electronic Discovery in Employment Cases. By Philip L. Gordon, Esq.

How to Win the Battle Over Electronic Discovery in Employment Cases. By Philip L. Gordon, Esq. How to Win the Battle Over Electronic Discovery in Employment Cases By Philip L. Gordon, Esq. IMPORTANT NOTICE This publication is not a do-it-yourself guide to resolving employment disputes or handling

More information

A PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE

A PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE A PRIMER ON THE NEW ELECTRONIC DISCOVERY PROVISIONS IN THE ALABAMA RULES OF CIVIL PROCEDURE Effective February 1, 2010, the Alabama Rules of Civil Procedure were amended to provide for and accommodate

More information

E-DISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK

E-DISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK E-DISCOVERY: BURDENSOME, EXPENSIVE, AND FRAUGHT WITH RISK If your company is involved in civil litigation, the Federal Rules of Civil Procedure regarding preservation and production of electronic documents

More information

Digital Forensics, ediscovery and Electronic Evidence

Digital Forensics, ediscovery and Electronic Evidence Digital Forensics, ediscovery and Electronic Evidence By Digital Forensics What Is It? Forensics is the use of science and technology to investigate and establish facts in a court of law. Digital forensics

More information

B. Preservation is not limited to simply avoiding affirmative acts of destruction because day-to-day operations routinely alter or destroy evidence.

B. Preservation is not limited to simply avoiding affirmative acts of destruction because day-to-day operations routinely alter or destroy evidence. This is a sample approach to developing a sound document collection process, referenced at Section II(7)(vi) of the Guidelines on Best Practices for Litigating Cases Before the Court of Chancery. It should

More information

Rule 30(b)(6) Depositions in Electronic Discovery. Discovering What There Is to Discover

Rule 30(b)(6) Depositions in Electronic Discovery. Discovering What There Is to Discover : Discovering What There Is to Discover One of the challenges in electronic discovery is identifying the various sources of electronically stored information (ESI) that could potentially be relevant to

More information

California Electronic Discovery Rules. William W. Belt, Jr.

California Electronic Discovery Rules. William W. Belt, Jr. California Electronic Discovery Rules William W. Belt, Jr. July 16, 2009 Today s speaker and some notes... Bill Belt William.Belt@LeClairRyan.com Welcome. With the high number of attendees, please note

More information

E- Discovery in Criminal Law

E- Discovery in Criminal Law E- Discovery in Criminal Law ! An e-discovery Solution for the Criminal Context Criminal lawyers often lack formal procedures to guide them through preservation, collection and analysis of electronically

More information

ESI and Predictive Coding

ESI and Predictive Coding Beijing Boston Brussels Chicago Frankfurt Hong Kong ESI and Predictive Coding Houston London Los Angeles Moscow Munich New York Palo Alto Paris São Paulo Charles W. Schwartz Chris Wycliff December 13,

More information

ACADEMIC AFFAIRS COUNCIL ******************************************************************************

ACADEMIC AFFAIRS COUNCIL ****************************************************************************** ACADEMIC AFFAIRS COUNCIL AGENDA ITEM: 8.D DATE: March 15, 2007 ****************************************************************************** SUBJECT: Electronic Records Discovery Electronic records management

More information

DEFAULT STANDARD FOR DISCOVERY, INCLUDING DISCOVERY OF ELECTRONICALLY STORED INFORMATION {"ESI")

DEFAULT STANDARD FOR DISCOVERY, INCLUDING DISCOVERY OF ELECTRONICALLY STORED INFORMATION {ESI) DEFAULT STANDARD FOR DISCOVERY, INCLUDING DISCOVERY OF ELECTRONICALLY STORED INFORMATION {"ESI") 1. General Provisions a. Cooperation. Parties are expected to reach agreements cooperatively on how to conduct

More information

(Previously published in The Legal Intelligencer, November 8, 2011) New Cost Guidelines for E-Discovery by Peter Vaira

(Previously published in The Legal Intelligencer, November 8, 2011) New Cost Guidelines for E-Discovery by Peter Vaira (Previously published in The Legal Intelligencer, November 8, 2011) New Cost Guidelines for E-Discovery by Peter Vaira In a recent case in the Eastern District, Judge Legrome Davis upheld court costs of

More information

E-Discovery Guidance for Federal Government Professionals Summer 2014

E-Discovery Guidance for Federal Government Professionals Summer 2014 E-Discovery Guidance for Federal Government Professionals Summer 2014 Allison Stanton Director, E-Discovery, FOIA, & Records Civil Division, Department of Justice Adam Bain Senior Trial Counsel Civil Division,

More information

102 ediscovery Shakedown: Lowering your Risk. Kindred Healthcare

102 ediscovery Shakedown: Lowering your Risk. Kindred Healthcare 102 ediscovery Shakedown: Lowering your Risk Long-Term Care Session HCCA Compliance Institute April 27, 2009 Las Vegas, Nevada Presented by: Diane Kissel, Manager IS Risk & Compliance Kindred Healthcare,

More information

E-discovery: Federal Rules of Civil Procedure and their Implications for Public Sector Corrections Departments

E-discovery: Federal Rules of Civil Procedure and their Implications for Public Sector Corrections Departments E-discovery: Federal Rules of Civil Procedure and their Implications for Public Sector Corrections Departments Andres De Aguero, Senior Lead, Deloitte Consulting LLP David F. Axelrod, Director, Deloitte

More information

plantemoran.com What School Personnel Administrators Need to know

plantemoran.com What School Personnel Administrators Need to know plantemoran.com Data Security and Privacy What School Personnel Administrators Need to know Tomorrow s Headline Let s hope not District posts confidential data online (Tech News, May 18, 2007) In one of

More information

E-Discovery Toolkit for Educational Institutions

E-Discovery Toolkit for Educational Institutions E-Discovery Toolkit for Educational Institutions The e-discovery process the search of electronic records for use as legal evidence can cost educational institutions thousands of dollars and hours. The

More information

Litigation Support. Learn How to Talk the Talk. solutions. Document management

Litigation Support. Learn How to Talk the Talk. solutions. Document management Document management solutions Litigation Support glossary of Terms Learn How to Talk the Talk Covering litigation support from A to Z. Designed to help you come up to speed quickly on key terms and concepts,

More information

Five Steps to Ensure a Technically Accurate Document Production

Five Steps to Ensure a Technically Accurate Document Production Five Steps to Ensure a Technically Accurate Document Production by Elwood Clark Lawyers spend a lot of time focusing on the legal aspects of a document production, including properly defining the scope

More information

C. All responses should reflect an inquiry into actual employee practices, and not just the organization s policies.

C. All responses should reflect an inquiry into actual employee practices, and not just the organization s policies. Questionnaire on Electronically Stored Information (May 2014) Comment The Questionnaire is intended to be a comprehensive set of questions about a company s computer systems. The extent to which you should

More information

Electronic Discovery and Disclosure:

Electronic Discovery and Disclosure: Electronic Discovery and Disclosure: Managing & Producing Electronic Information Washington State Office of the Attorney General 1 Overview of Presentation EDD (Electronic Discovery and Disclosure) ESI

More information

Best Practices in Electronic Record Retention

Best Practices in Electronic Record Retention A. Principles For Document Management Policies Arthur Anderson, LLD v. U.S., 544 U.S. 696 (2005) ( Document retention policies, which are created in part to keep certain information from getting into the

More information

6/7/2010. Managing the E-Discovery Process: Guidance for Federal Agencies

6/7/2010. Managing the E-Discovery Process: Guidance for Federal Agencies Managing the E-Discovery Process: Guidance for Federal Agencies Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Message from the Associate Attorney General Message from Magistrate Judge Paul

More information

Electronic Discovery

Electronic Discovery Electronic Discovery L. Amy Blum, Esq. UCLA University of California, Los Angeles 1 Topics Not Covered Best practices for E-mail E use and retention in the ordinary course of business Records Disposition

More information

Elements of a Good Document Retention Policy. Discovery Services WHITE PAPER

Elements of a Good Document Retention Policy. Discovery Services WHITE PAPER Elements of a Good Document Retention Policy Discovery Services WHITE PAPER Document retention especially the retention of electronic data has become a hot topic in the legal industry. In the wake of several

More information

ARCHIVING FOR EXCHANGE 2013

ARCHIVING FOR EXCHANGE 2013 White Paper ARCHIVING FOR EXCHANGE 2013 A Comparison with EMC SourceOne Email Management Abstract Exchange 2013 is the latest release of Microsoft s flagship email application and as such promises to deliver

More information

A White Paper from AccessData Group. The Future of Mobile E-Discovery

A White Paper from AccessData Group. The Future of Mobile E-Discovery A White Paper from AccessData Group The Future of Mobile E-Discovery Contents 1. The changing landscape of e-discovery 2. New expectations in the courtroom 3. Mobile discovery within corporations 4. MPE+

More information

How to Manage Costs and Expectations for Successful E-Discovery: Best Practices

How to Manage Costs and Expectations for Successful E-Discovery: Best Practices How to Manage Costs and Expectations for Successful E-Discovery: Best Practices Mukesh Advani, Esq., Advisory Board Member, UBIC North America, Inc. UBIC North America, Inc. 3 Lagoon Dr., Ste. 180, Redwood

More information

OVERCOME REGULATORY DATA RETENTION CHALLENGES WITH COMPLIANCE ARCHIVING

OVERCOME REGULATORY DATA RETENTION CHALLENGES WITH COMPLIANCE ARCHIVING OVERCOME REGULATORY DATA RETENTION CHALLENGES WITH COMPLIANCE ARCHIVING EXECUTIVE SUMMARY A modern enterprise must retain information for years in order to comply with ever-changing government and industry

More information

A White Paper from AccessData Group. The Future of Mobile E-Discovery

A White Paper from AccessData Group. The Future of Mobile E-Discovery A White Paper from AccessData Group The Future of Mobile E-Discovery Contents 1. The changing landscape of e-discovery 2. New expectations in the courtroom 3. Mobile discovery within corporations 4. MPE+

More information

E-Discovery Basics For the RIM Professional. Learning Objectives 5/18/2015. What is Electronic Discovery?

E-Discovery Basics For the RIM Professional. Learning Objectives 5/18/2015. What is Electronic Discovery? E-Discovery Basics For the RIM Professional By: Andy Sokol, CEDS, CSDS Adding A New Service Offering For Your Legal & Corporate Clients Learning Objectives What is Electronic Discovery? How Does E-Discovery

More information

Strategies for Implementing an Effective and Defensible Legal Hold Workflow

Strategies for Implementing an Effective and Defensible Legal Hold Workflow Strategies for Implementing an Effective and Defensible Legal Hold Workflow Who should read this paper Corporate Counsel, IT/Legal Liaisons, and Messaging Administrators involved in the preservation of

More information

Metadata, Electronic File Management and File Destruction

Metadata, Electronic File Management and File Destruction Metadata, Electronic File Management and File Destruction By David Outerbridge, Torys LLP A. Metadata What is Metadata? Metadata is usually defined as data about data. It is a level of extra information

More information

[DESCRIPTION OF CLAIM, INCLUDING RELEVANT ACTORS, EVENTS, DATES, LOCATIONS, PRODUCTS, ETC.]

[DESCRIPTION OF CLAIM, INCLUDING RELEVANT ACTORS, EVENTS, DATES, LOCATIONS, PRODUCTS, ETC.] What follows isn t the perfect preservation letter for your case, so don t simply treat it as a form. Use it as a drafting aid that flags issues unique to EDD, but tailor your preservation demand to the

More information

Social Media. e-discovery and Litigation Issues in the Advanced Age of Technology. September 26-29, 2010 ACI-NA 19 th Annual Conference and Exhibition

Social Media. e-discovery and Litigation Issues in the Advanced Age of Technology. September 26-29, 2010 ACI-NA 19 th Annual Conference and Exhibition Social Media and Litigation Issues in the Advanced Age of Technology TOPIC When Worlds Collide Litigation 3 1 Social Media 2 EARLY ESI MANAGEMENT Kircoska v. United States (circa 1970) Request for Production

More information

E-DISCOVERY & PRESERVATION OF ELECTRONIC EVIDENCE. Ana Maria Martinez April 14, 2011

E-DISCOVERY & PRESERVATION OF ELECTRONIC EVIDENCE. Ana Maria Martinez April 14, 2011 E-DISCOVERY & PRESERVATION OF ELECTRONIC EVIDENCE Ana Maria Martinez April 14, 2011 This presentation does not present the views of the U.S. Department of Justice. This presentation is not legal advice.

More information

WHITE PAPER: CUSTOMIZE WHITE PAPER: BEST PRACTICES FOR ARCHIVING. Best Practices for Defining and Establishing Effective Archive Retention Policies

WHITE PAPER: CUSTOMIZE WHITE PAPER: BEST PRACTICES FOR ARCHIVING. Best Practices for Defining and Establishing Effective Archive Retention Policies WHITE PAPER: CUSTOMIZE WHITE PAPER: BEST PRACTICES FOR ARCHIVING Confidence in a connected world. Best Practices for Defining and Establishing Effective Archive Retention Policies Sponsored by Symantec

More information

CAPABILITY STATEMENT LEGAL TECHNOLOGIES AND COMPUTER FORENSICS. www.controlrisks.com DECEMBER 2013

CAPABILITY STATEMENT LEGAL TECHNOLOGIES AND COMPUTER FORENSICS. www.controlrisks.com DECEMBER 2013 LEGAL TECHNOLOGIES AND COMPUTER FORENSICS DECEMBER 2013 Copyright Control Risks. All rights reserved. This document cannot be reproduced without the express written permission of Control Risks. Any reproduction

More information

Symantec Enterprise Vault.cloud Overview

Symantec Enterprise Vault.cloud Overview Fact Sheet: Archiving and ediscovery Introduction The data explosion that has burdened corporations and governments across the globe for the past decade has become increasingly expensive and difficult

More information

THE IMPACT OF THE ELECTRONIC DISCOVERY RULES ON THE EEOC PROCESS

THE IMPACT OF THE ELECTRONIC DISCOVERY RULES ON THE EEOC PROCESS THE IMPACT OF THE ELECTRONIC DISCOVERY RULES ON THE EEOC PROCESS Cynthia L. Gibson, Esq. Katz, Teller, Brant & Hild 255 East Fifth Street Suite 2400 Cincinnati, OH 45202 (513) 977-3418 cgibson@katzteller.com

More information

How to Avoid The Biggest Electronic Evidence Mistakes. Ken Jones Senior Technology Architect Pileum Corporation

How to Avoid The Biggest Electronic Evidence Mistakes. Ken Jones Senior Technology Architect Pileum Corporation How to Avoid The Biggest Electronic Evidence Mistakes Ken Jones Senior Technology Architect Pileum Corporation Why is Proper Handling of Electronic Data Important? Most of the evidence in your case isn

More information

E-Discovery for Paralegals: Definition, Application and FRCP Changes. April 27, 2007 IPE Seminar

E-Discovery for Paralegals: Definition, Application and FRCP Changes. April 27, 2007 IPE Seminar E-Discovery for Paralegals: Definition, Application and FRCP Changes April 27, 2007 IPE Seminar Initial Disclosures ESI Electronically Stored Information FRCP 26(a)(1)(B) all ESI must be disclosed initially

More information

E-Discovery: The New Federal Rules of Civil Procedure A Practical Approach for Employers

E-Discovery: The New Federal Rules of Civil Procedure A Practical Approach for Employers MARCH 7, 2007 E-Discovery: The New Federal Rules of Civil Procedure A Practical Approach for Employers By Tara Daub and Christopher Gegwich News of the recent amendments to the Federal Rules of Civil Procedure

More information

HOT TIPS ABOUT ESI. Hot Tips From the Coolest Family Law Attorneys Friday, September 27, 2013

HOT TIPS ABOUT ESI. Hot Tips From the Coolest Family Law Attorneys Friday, September 27, 2013 HOT TIPS ABOUT ESI Hot Tips From the Coolest Family Law Attorneys Friday, September 27, 2013 Melissa Fuller Brown 145 King Street, Ste. 405 Charleston, SC 29401 843-722- 8900 melissa@melissa- brown.com

More information

Social Media & ediscovery: Untangling the Tweets for the Trials

Social Media & ediscovery: Untangling the Tweets for the Trials Social Media & ediscovery: Untangling the Tweets for the Trials Jack Halprin, Esq. Vice President, ediscovery & Compliance POWER PROTECT PROMOTE Agenda Social Media Sites: What, Who, Where & How Discoverability

More information

What Happens When Litigation Starts? How Do You Get People Not To Generate the Bad Documents?

What Happens When Litigation Starts? How Do You Get People Not To Generate the Bad Documents? Document Retention and Destruction in Oregon What Happens When Litigation Starts? How Do You Get People Not To Generate the Bad Documents? Timothy W. Snider (503) 294-9557 twsnider@stoel.com Stoel Rives

More information

Hong Kong High Court Procedure E-Discovery: Practice Direction Effective September 1, 2014

Hong Kong High Court Procedure E-Discovery: Practice Direction Effective September 1, 2014 CLIENT MEMORANDUM Hong Kong High Court Procedure E-Discovery: Practice Direction Effective September 1, 2014 August 28, 2014 Mandatory application of e-discovery Mandatory application of e-discovery to

More information

Director, Value Engineering

Director, Value Engineering Director, Value Engineering April 25 th, 2012 Copyright OpenText Corporation. All rights reserved. This publication represents proprietary, confidential information pertaining to OpenText product, software

More information

Understanding ediscovery and Electronically Stored Information (ESI)

Understanding ediscovery and Electronically Stored Information (ESI) Copyright The information transmitted in this document is intended only for the addressee and may contain confidential and/or privileged material. Any interception, review, retransmission, dissemination

More information

Electronic Discovery

Electronic Discovery Electronic Discovery e-discovery: A Primer Mauricio Perry, CRM, CEDS Mauricio Perry, CEDS, CRM 1 Disclaimer I am not a lawyer The ideas exposed here are not to be construed as legal advice but are educational

More information

Making Sense of E-Discovery: 10 Plain Steps for Producing ESI

Making Sense of E-Discovery: 10 Plain Steps for Producing ESI Making Sense of E-Discovery: 10 Plain Steps for Producing ESI The following article provides a practical guide to producing electronically stored information (ESI) that lawyers can apply immediately in

More information

E-Discovery: New to California 1

E-Discovery: New to California 1 E-Discovery: New to California 1 Patrick O Donnell and Martin Dean 2 Introduction The New Electronic Discovery Act The new Electronic Discovery Act, Assembly Bill 5 (Evans), has modernized California law

More information

Ethics in Technology and ediscovery Stuff You Know, But Aren t Thinking About

Ethics in Technology and ediscovery Stuff You Know, But Aren t Thinking About Ethics in Technology and ediscovery Stuff You Know, But Aren t Thinking About Kelly H Twigger, Esq. Oil and Gas Symposium Arkansas Law Review October 16-17, 2014 Overview In the last two decades, business

More information

CMA Shipping 2015. Ethics and E-Discovery in Shipping Disputes

CMA Shipping 2015. Ethics and E-Discovery in Shipping Disputes CMA Shipping 2015 Ethics and E-Discovery in Shipping Disputes March 25, 2015 Vincent J. Foley, Holland & Knight LLP (212) 513-3357 vincent.foley@hklaw.com CMA Shipping 2015 Ethics and E-Discovery for Shipping

More information