Using Visual Analytics in E-Discovery and E-Disclosure Cases

Transcription

1 Using Visual Analytics in E-Discovery and E-Disclosure Cases VASS 2012: Second UKVAC International Visual Analytics Summer School Middlesex University London, U.K. September 3, 2012 Jason R. Baron Director of Litigation Office of General Counsel US National Archives and Records Administration Washington, D.C.

2 Overview What is e-discovery and e-disclosure in the law? The As is Model of performing searches Case example: US v Phillip Morris Moving Beyond Boolean: the emergence of clustering algorithms and supervised learning alternatives to keyword searching The Promise of Visual Analytics Strategic challenges 2

3

4 Searching the Haystack. 4

5 to find relevant needles 5

6 ends up like searching in a maze 6

7 is still the 800 lb. gorilla of ediscovery (whether in the clouds or not)

8 Reality: The era of Big Data in Litigation has just begun. Lehman Brothers Investigation billion page universe (3 petabytes) -- Examiner narrowed collection by selecting key custodians, using dozens of Boolean searches -- Reviewed 5 million docs (40 million pages using 70 contract attorneys) Source: Report of Anton R. Valukas, Examiner, In re Lehman Brothers Holdings Inc., et al., Chapter 11 Case No (U.S. Bankruptcy Ct. S.D.N.Y. March 11, 2010), Vol. 7, Appx. 5, at

9 U.S. v. Philip Morris et al. Civil lawsuit brought by Clinton Administration against tobacco companies in 1999 Racketeering allegation that companies have conspired since 1953 to defraud the American public as to the true health effects of smoking 1,726 Requests to Produce from tobacco companies for tobacco-related records (including ) from 30 federal agencies 32 million Clinton-era records held by National Archives

10 Case Study: U.S. v. Philip Morris (con t) Employing a limited feedback loop Original set of 12 keywords searched unilaterally After informal negotiations, additional terms explored Sampling against database to find noisy terms generating too many false positives (Marlboro, PMI, TI, etc.) Report back and consensus on what additional terms would be in search protocol. 10

11 Query Terms Round 1 Round 2 tobacco cigarette smoking <tar> nicotine Smokeless Liggett PMI (Philip Morris Institute) MSA (Master Settlement Agreement) ETS Synar Amendment Philip Morris B&W R.J. Reynolds (Brown & Williamson) BAT Industries TI (Tobacco Institute) Liggett group Brown and Williamson (Environmental Tobacco Smoke)

12 False Positives Relevant Smoking Policy s White House Counsel VP Chief of Staff Ron Klain Office of the U.S. Trade Rep. OMB

13 Suppressing False Positives Marlboro vs. Upper Marlboro, Maryland Philip Morris Institute vs. Presidential Management Intern (PMI) program Master Settlement Agreement vs. Medical Savings Accounts and Metropolitan Standard Areas Tobacco Institute (TI) vs. do re me.

14 Example of Boolean search string from U.S. v. Philip Morris (((master settlement agreement OR msa) AND NOT (medical savings account OR metropolitan standard area)) OR s OR (ets AND NOT educational testing service) OR (liggett AND NOT sharon a. liggett) OR atco OR lorillard OR (pmi AND NOT presidential management intern) OR pm usa OR rjr OR (b&w AND NOT photo*) OR phillip morris OR batco OR ftc test method OR star scientific OR vector group OR joe camel OR (marlboro AND NOT upper marlboro)) AND NOT (tobacco* OR cigarette* OR smoking OR tar OR nicotine OR smokeless OR synar amendment OR philip morris OR r.j. reynolds OR ("brown and williamson") OR ("brown & williamson") OR bat industries OR liggett group) 14

15 Clinton White House search request Tobacco Policy 20 million s ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ National Archives hired 25 persons ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ 100,000 relevant docs, with 20,000 privileged for 6 months ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ ~~~~~~~~ 200,000 docs with hits

16 A Hypothetical 1 billion s, 25% with attachments Reviewed at 50 per hour Would take 100 people, 10 hrs per day, 7 days a week, 52 weeks a year. 54 YEARS TO COMPLETE At $100/hr, $ 2 billion in cost Even 1% (10 million docs) 28 weeks and $20 million in cost.. 16

17 Judge Grimm writing for the U.S. District Court for the District of Maryland [W]hile it is universally acknowledged that keyword searches are useful tools for search and retrieval of ESI, all keyword searches are not created equal; and there is a growing body of literature that highlights the risks associated with conducting an unreliable or inadequate keyword search or relying on such searches for privilege review. Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251 (D. Md. 2008); see id., text accompanying nn. 9 & 10 (citing to Sedona Search Commentary & TREC Legal Track research project) 17

18 Judge Facciola writing for the U.S. District Court for the District of Columbia Whether search terms or keywords will yield the information sought is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguistics. See George L. Paul & Jason R. Baron, Information Inflation: Can the Legal System Adapt?', 13 RICH. J.L. & TECH.. 10 (2007) * * * Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread. -- U.S. v. O'Keefe, 537 F.Supp.2d 14, 24 D.D.C. 2008). 18

19 Boolean Retrieval The Model documents represented by descriptors descriptors originally manually assigned concepts from controlled vocabulary modern implementations generally use words in text as descriptors information need represented by descriptors structured with Boolean operators modern implementations include more operators than just AND, OR, NOT a match occurs if and only if doc satisfies Boolean expression fuzzy match systems use descriptor weights, relax strict binary interpretation Pros and cons good: transparency---clear exactly why doc retrieved bad: little control over retrieved set size; no ranking; searchers must learn query language From Ellen Voorhees, Georgetown 2009

20 Vector Space Model The Model documents represented as vectors in N-dimensional space where N is number of terms in the document set term is usually a word (stem); but might be phrase or thesaurus class terms are weighted based on frequency and distribution of occurrences information need is natural language text mapped in same space matching is similarity between query and doc vectors example similarity: cosine of angle between vectors allows documents to be ranked by decreasing similarity Pros and Cons good: less brittle than pure Boolean bad: less transparency---depending on weights, a doc with few query terms can be ranked higher than a doc with many From Ellen Voorhees, Georgetown 2009

21 Vector Similarities Document-Document similarity docs are similar to the extent they contain the same terms doc pairs with maximal similarity detects duplicates document clustering cluster hypothesis: Closely associated documents tend to be relevant to the same requests. thus, do retrieval based on returning whole clusters since usually much more information in doc-doc comparison than doc-query Term-Term similarity terms are similar to the extent the occur in the same documents term clustering query expansion provide bottom-up description of document set D1 D2 D3 D4 D5 D6 T1 T2 T3 T From Ellen Voorhees, Georgetown 2009

22 Emerging New Strategies: Predictive Analytics Improved review and case assessment: cluster docs thru use of software with minimal human intervention at front end to code seeded data set Slide adapted from Gartner Conference June 23, 2010 Washington, D.C. 22

23 Judicial endorsement of predictive analytics in document review by Judge Peck in Da Silva Moore v. Publicis Groupe (SDNY Feb. 24, 2012) This opinion appears to be the first in which a Court has approved of the use of computer-assisted review.... What the Bar should take away from this Opinion is that computer-assisted review is an available tool and should be seriously considered for use in largedata-volume cases where it may save the producing party (or both parties) significant amounts of legal fees in document review. Counsel no longer have to worry about being the first or guinea pig for judicial acceptance of computer-assisted review... Computer-assisted review can now be considered judicially-approved for use in appropriate cases.

24 The da Silva Moore Protocol Supervised learning Random sampling Establishment of seed set Issue tags Iteration Random sampling of docs deemed irrelevant 24

25 The e-discovery world of tomorrow optimizing visual analytics.

26 Visual Analysis Examples (Presentation by Dr. Victoria Lemieux, Univ. British Columbia, at Society of American Archivist Annual Mtg. 2010, Washington, D.C.) With acknowledgments to Jeffrey Heer, Exploring Enron, Adam Perer, Contrasting Portraits, and Fernanda Viegas, Conversations, 26

27

28 28

29 Showing Data changes over time Source: Pacific Northwest National Laboratory.

30 Another River chart The colors represent word frequency. Susan Havre. Pacific Northwest National Laboratory.

31 Bar charts Jeffrey Heer: researcher at Berkeley. The Enron s. Documents can be viewed as a function of time, and then organized by color based on author, topic, keywords, etc.

32 Where keywords can t go: networks Source: Carsten Görg and John Stasko, Georgia Institute of Technology.

33 Social Graphs Have the Potential to Provide Greater Insight into Corporate Relationships: Who is Talking To Whom?

34 Important Individuals To Zoom in on For Evidentiary Purposes

35 Network maps cont d

36 Links of near infinite complexity

37 More links examples

38 Concentrating on message stream between specific individuals

39 Wikipedia and philosophy connections: Simon Raper, UK Statistician and Bayesian modeling/va hobbyist.

40

41

42 Treemap distribution in data repository: file sizes and distribution in directories Credit: Maria Esteva et al., Texas Adv. Computing Center

43 Where are electronic messages coming from inside a corporation?

44 Types of documents found in given repository

45 Hypothetical: noticing spikes in communications between 2 units in a corporation

46 Spikes in conversations about specific topics of evidentiary interest

47 Hypothetical: Conversations on Per Hour on Topic X xxxxxxxxxxxx

48 Challenges for VA in E-Discovery/ E-Disclosure Cases a. Ease of use b. Analytical reasoning techniques c. Data input d. Visual representation and intepretation e. Scalability f. Tech-savvy-ness of lawyers and judges (or lack thereof) g. Defensibility in the context of an adversary system

49 Question A research topic of interest: how can use of Visual Analytics change the kind of questions asked in e-discovery cases? Example: From Produce all documents on Topic X TO interrogatories asking: What was the first time that there was a spike in communications on topic X. How many messages were recorded and between whom? Who were the senior most individuals who knew on date A that topic X was being discussed? How did the knowledge of Topic X spread through the institution? 49

50 Strategic Challenges Convincing the legal community that advanced forms of automated searches including visual analytics are not just desirable but necessary in response to large volume of documents to search.

51 Challenges (cont.) Designing an overall review process which maximizes the potential to find responsive documents in a large data collection (no matter which search tool is used), and using sampling and other visual analytic techniques to test hypotheses early on.

52 Challenges (cont.) Having parties understand that the use of visual analytics does not guarantee all responsive documents will be identified in a large data collection.

53 Challenges (cont.) Being open to using new and evolving search and information retrieval methods and tools, including advances in visual analytics based on increasing powerful data mining and AI techniques.

54 Problem: Innovative Thinking 54

55 References Topic Classification and Visualization Articles V. Lemieux and J. Baron, Overcoming the Digital Tsunami in e-discovery: is Visual Analysis the Answer?, Canadian J. of Law & Tech. (Spring 2011) D. Hillard et al., Computer-Assisted Topic Classification for Mixed-Methods Social Science Research, J. of Info. Tech. 4:4 (2007), available at C. Gorg & J. Stasko, Jigsaw: Investigative Analysis on Text Document Collections through Visualization, available at Background Law Review Referencing Autocategorization & Advanced Search J. Baron, Law in the Age of Exabytes: Some Further Thoughts on Information Inflation and Current Issues in E-Discovery Search, 17 Richmond J. Law & Technology (2011), see Predictive Coding Report N. Pace & L. Zakaras, Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery, RAND Report (2012), available at 55

56 Jason R. Baron Director of Litigation Office of General Counsel National Archives and Records Administration 8601 Adelphi Road # 3110 College Park, MD (301) jason.baron@nara.gov 56