Archiving the Web and Beyond: A Look at Twi8er and Facebook (and some other things too)

Transcription

1 Archiving the Web and Beyond: A Look at Twi8er and Facebook (and some other things too) July 23, 2014 Benn Joseph Manuscript Librarian Northwestern University Library

2 Outline of discussion Digital preservanon management (in general). How (and why) we archive websites at Northwestern University (and how to make the case). Social media archiving and the story of the Twi8er archive at LC. Behind the scenes: how to get your data out. Technology and tools: how do we do it? Sustaining the collecnon: ensuring preservanon and understanding the costs.

3 h8p://xkcd.com/1360/ Digital preservanon management

4 Backups vs. Archival Storage Backups keep your computer working and files safe Archival Storage keeps the content accessible for future computers and users

5 An Archival storage would comprise? Objects = (files + metadata) Storage system may include a variety of formats Metadata: idennficanon and descripnon Makes object understandable Unique idennfier == traceable in Nme

6 Metadata uniquely idennfies digital objects Unique IdenNfiers can also store descripnve data EX: MFM121587

7 Arranging digital objects Digital photos tossed in a folder for future archiving, credit: Keri Myers (a volunteer archivist with NDIIPP).

8 Lots of Copies let us save what remains not by vaults and locks which fence them from the public eye and use in consigning them to the waste of Nme, but by such a mulnplicanon of copies, as shall place them beyond the reach of accident. - Thomas Jefferson, 1791

9 How many? 2-6 COPIES! At least 2 copies in 2 places Consider File size Personally IdenNfiable InformaNon (PII)

10 Offline Storage

11 Online Storage Cloud and collaboranve storage

12 Dangers! to digital objects Change and loss accidental and intennonal DPOE Baseline Modules: Identify, version 3.0

13 Dangers! to digital objects Obsolescence as technology evolves DPOE Baseline Modules: Identify, version 3.0

14 Dangers! to digital objects Inappropriate access e.g., confidennal data DPOE Baseline Modules: Identify, version 2.0, Nov 2011

15 Dangers! to digital objects Non- compliance standards and requirements DPOE Baseline Modules: Identify, version 2.0, Nov 2011

16 Dangers! to digital objects Disasters emergencies of all kinds DPOE Baseline Modules: Identify, version 2.0, Nov 2011

17 Everyday ProtecNon Know where your content is located Onsite and offsite; online and offline (administranve metadata) Know who can have access to it Know who accesses your secure informanon For staff, depositors, users (AuthenNcaNon) DPOE Baseline Modules: Identify, version 2.0, Nov 2011

18 Web Archiving and how we archive websites at NU Web Archiving Service Product of the California Digital Library SubscripNon- based service Offers support Mildly customizable

19 WASsup? A li8le background on our Web Archiving Service implementanon 377 GB of data used (1TB available). 231 crawled sites. Planning began in 2010, we began using in June UA is insntunonal admin, I am the only project admin and we really only have two projects (one that takes in 99% of what we crawl; the other is dark).

20 Program of African Studies Newsle8er: only being made available online (no longer printed)

21 Web Archiving Task Force (WATF)

22 QuesNons remained

23 New library website launched August 2010 Old (top) and new (right) sites

24 Legacy student organizanon websites

25 AdreNUline Circus

26

27 Feinberg School of Medicine

28 Setng up crawl in WAS panel

29

30

31

32

33 PRINCIPLE: It is fair use to create topically based collecnons of websites and other material from the Internet and to make them available for scholarly use. LIMITATIONS: Captured material should be represented as it was captured, with appropriate informanon on mode of harvesnng and date. To the extent reasonably possible, the legal proprietors of the sites in quesnon should be idennfied according to the prevailing convennons of a8ribunon. Libraries should provide copyright owners with a simple tool for registering objecnons to making items from such a collecnon available online, and respond to such objecnons promptly.

34 Social media Terms of Service Twi?er: API available for just about all of it. Does not expressly prohibit crawlers (although robots.txt does) Facebook: You will not collect users' content or informanon, or otherwise access Facebook, using automated means (such as harvesnng bots, robots, spiders, or scrapers) without our prior permission. (API available).

35 SOCIAL MEDIA archiving and the story of the Twi?er archive at the Library of Congress (with images of what social media may look like).

36 The Library of Congress Agreement with Twi?er The Library s agreement with Twi8er announced April 14, 2010 provided that: Twi8er would donate a collecnon consisnng of all public tweets from the Twi8er service from its incepnon to the date of the agreement, an archive of 21 billion tweets that occurred between 2006 and Any addinonal materials Twi8er provides to the Library would be governed by the terms of the agreement unless both parnes agree to different terms in advance of receiving such addinonal materials. The Library could make available any pornon of the collecnon six months axer it was originally posted on Twi8er to bona fide researchers. A researcher must sign a nonficanon prohibinng commercial use and redistribunon of the collecnon. The Library cannot provide a substannal pornon of the collecnon on its web site in a form that can be easily downloaded.

37 Transfer of Data to the Library of Congress In December, 2010, Twi8er named a Colorado- based company, Gnip, as the delivery agent for moving data to the Library. Shortly thereaxer, the Library and Gnip began to agree on specificanons and processes for the transfer of files current tweets - on an ongoing basis. In February 2011, transfer of current tweets was ininated and began with tweets from December On February 28, 2012, the Library received the archive through Gnip in three compressed files totaling 2.3 terabytes. When uncompressed the files total 20 terabytes. The files contained approximately 21 billion tweets, each with more than 50 accompanying metadata fields, such as place and descripnon. As of December 1, 2012, the Library has received more than 150 billion addinonal tweets and corresponding metadata, for a total including the archive of approximately 170 billion tweets totaling terabytes for two compressed copies.

38 Behind the scenes how to get your data out!

39 Perez, Sarah. This is What a Tweet Looks Like

40 Twi?er archive download Download twi?er

41 Twi?er archive download Download twi?er

42 Facebook archive download Download Facebook informaoon

43 Facebook archive download

44 Download Google data Download and migrate from Google products, blogs, instagram, etc.

45 Download Google data What s included?

46 Save Page As Save As using web browser

47 Technology and tools how do we do it?

48 Tools the Library of Congress is documennng ALL of Twi8er. I work with SF/F authors and encourage them to either do periodic screencaps, prinnng to PDF, or imaging of some sort of their websites and blogs using what capture tools I can find. I also encourage them to submit their URLs to The Internet Archive in the hopes of addinonal snapshots. Lynne Thomas, Northern Illinois University. Wayback machine works for many websites, but not for social media.

49 tools

50 Facebook robots.txt

51 Tools All- inclusive tools commonly used by academic insntunons: WAS and Archive- It. WAS works good for simple websites, but not social media. Archive- It might work (Umbra). Neither allow for much customizanon of crawl opnons. What else is there?

52 Tools Command- line interfaces (wget, curl, and others) are more customizable: (STEP1) mkdir /websitedl/ cd /websitedl/ (STEP2) wget - - limit- rate=200k - - no- clobber - - convert- links - - random- wait - r - p - E - e robots=off - U mozilla h8p://

53 Tools Facebook and Twi8er both have APIs (use curl) Facebook s Graph API named axer the idea of a social graph a representanon of the informanon on Facebook composed of: nodes (basically "things" such as a User, a Photo, a Page, a Comment) edges (the connecnons between those "things", such as a Page's Photos, or a Photo's Comments) fields (info about those "things", such as the birthday of a User, or the name of a Page).

54 Tools HTTrack (Windows allows for similar commands as wget) Sitecrawler (Mac) Social Feed Manager (developed at George Washington University, a django applicanon for managing mulnple feeds of social media data)

55 Experimental tools WARCreate, a Chrome plugin WAIL (Web Archiving IntegraNon Layer) Archive Facebook (Firefox plugin)

56 Tools: subscripoon services Using subscripnon services can be much easier but also much more expensive!

57 So there are many challenges. Internet Archive. Challenges of CollecNng and Preserving the Social Web h8p://blogs.loc.gov/digitalpreservanon/files/ 2014/01/NDSA_CWG_120413_Carpenter.pdf.

58 What methods will you use for collecnng your data? How oxen will you collect it? Where will you put it? How will you provide access to it?

59 Sustaining the collecoon Finding aids for web archives (use SAA presentaoon and UT- San Antonio examples)

60

61 Sustaining the collecoon Bagit specificaoon Using checksums CreaOng durable digital objects

62 Sustaining the collecoon How would you talk about the costs of a web archiving or social media archiving project?

63

64 Thank you! Benn Joseph b-

65 How does the federal government collect social media? Methods to capture social media records include: Using web crawling or other soxware to create local versions of sites; Using web capture tools to capture social media; Using pla~orm specific applicanon programming interfaces (APIs) to pull content; Using RSS Feeds, aggregators, or manual methods to capture content; and Using tools built into some social media pla~orms to export content.

66 LC NDSA also gives personal archiving guidance for websites, blogs, and social media. Iden%fy where you have Websites Decide which informaoon has long- term value Export the selected informaoon Organize the informaoon Make copies and manage them in different places