HathiTrust Research Center Architecture. Data subsystem

Transcription

1 HathiTrust Research Center Architecture Data subsystem

2 Web portal Desktop SEASR client CI logon (NCSA) Access control (e.g. Grouper) framework Non- consump?ve Data capsules Task deployment SEASR analy?cs service Meandre Orches- tra?on WSO2 registry - services, collec?ons, data capsule images Authorita?ve volume store (Cassandra) WSO2 Enterprise service bus Programma?c access (e.g., Bamboo) NCSA local resources University of Michigan Replicated volume stores Future Grid Penguin on Demand NCSA HPC resources Page/volume tree (file system) rsync HathiTrust corpus

3 Web portal Desktop SEASR client CI logon (NCSA) Access control (e.g. Grouper) framework Non- consump?ve Data capsules Task deployment SEASR analy?cs service Meandre Orches- tra?on WSO2 registry - services, collec?ons, data capsule images Authorita?ve volume store (Cassandra) WSO2 Enterprise service bus Programma?c access (e.g., Bamboo) NCSA local resources University of Michigan Replicated volume stores Future Grid Penguin on Demand NCSA HPC resources Page/volume tree (file system) rsync HathiTrust corpus

4 Web portal Desktop SEASR client CI logon (NCSA) Access control (e.g. Grouper) framework Non- consump?ve Data capsules Task deployment SEASR analy?cs service Meandre Orches- tra?on WSO2 registry - services, collec?ons, data capsule images Authorita?ve volume store (Cassandra) WSO2 Enterprise service bus Programma?c access (e.g., Bamboo) NCSA local resources University of Michigan Replicated volume stores Future Grid Penguin on Demand NCSA HPC resources Page/volume tree (file system) rsync HathiTrust corpus

5 Solr quick introduc?on Lucene is a high- performance, full- featured text search engine library Solr is a web service frontend to Lucene Index consists of documents and document consists of fields which are name/value pair

6 HTRC Solr Has both bibliographic informa?on and full- text OCR scan 29 fields volume ID,?tle, author, several reference IDs (ISBN, ISSN, callnumber, etc), and full text Basic search like term query, wildcard, fuzzy query, phrase query and range query: Example: OCR: war, search documents containing the word war in text Term Vector is enabled to get word frequency and offset for each word : Occurences posi?on and offset

7 Filtered Term Vector Default Term Vector is massive O(5MB) per volume Extremely slow response for mul?ple volumes We extended Solr to filter unwanted words to enhance response speed significantly. Reduced term vector to O(80KB) per volume.

8 Web portal Desktop SEASR client CI logon (NCSA) Access control (e.g. Grouper) framework Non- consump?ve Data capsules Task deployment SEASR analy?cs service Meandre Orches- tra?on WSO2 registry - services, collec?ons, data capsule images Authorita?ve volume store (Cassandra) WSO2 Enterprise service bus Programma?c access (e.g., Bamboo) NCSA local resources University of Michigan Replicated volume stores Future Grid Penguin on Demand NCSA HPC resources Page/volume tree (file system) rsync HathiTrust corpus

9 Ingest Procedure Use rsync to pull filesystem data from HT main collec?on. Too many small text files... Parse structural metadata (METS) ordering of page, page checksum (and verifica?on); some metadata stored to NoSQL. Analyze delta logs to push incremental changes to NoSQL store

10 Rsync root Split pairtree list Bib metdata Collec?on namespace 1 pairtree_root Collec?on namespace 2 pairtree_root HTRC Text Corpora Ingest Workflow pairtree pairtree HathiTrust (remote) HathiTrust Research Center (local) Rsync split pairtree list Parallel rsync of the rest using split tree list Push modified volume s from pairtree to nosql Update collec?ons list pairtree pairtree Delta logs Split pairtree list Bib metdata pairtree_root Collec?on namespace 1 pairtree_root Collec?on namespace 2 Cassandra nosql repository Rsync root

11 Web portal Desktop SEASR client CI logon (NCSA) Access control (e.g. Grouper) framework Non- consump?ve Data capsules Task deployment SEASR analy?cs service Meandre Orches- tra?on WSO2 registry - services, collec?ons, data capsule images Authorita?ve volume store (Cassandra) WSO2 Enterprise service bus Programma?c access (e.g., Bamboo) NCSA local resources University of Michigan Replicated volume stores Future Grid Penguin on Demand NCSA HPC resources Page/volume tree (file system) rsync HathiTrust corpus

12 NoSQL Repository U?lizing Cassandra as a storage space for our text collec?ons and related metadata Aggregates small texts Allows us to manage flexible schemas Key- value based column store Offers good scalability, redundancy, and performance

13 Cassandra Schema Key: (volume ID) metadata Inu /001 Inu /xxx Inu copyright public Page count 16 What s up doc? f Rabbits 7 aabbcc metadata Inu /001 Inu /xxx Inu copyright Page count In- copyright b!2b 6 7effdd A ques?on 10 deadbeef Each row represents a volume Row key is the volume ID Each row contains many columns First column contains metadata ahributes about the volume Each subsequent column family is a page, key is page ID Page- specific columns contain page s and metadata about the page

14 Cassandra Schema Key: (volume ID) metadata Inu /001 Inu /xxx Inu copyright public Page count 16 What s up doc? f Rabbits 7 aabbcc metadata Inu /001 Inu /xxx Inu copyright Page count In- copyright b!2b 6 7effdd A ques?on 10 deadbeef Pros Cons Works well for all access primi?ves Well organized metadata no repe??ons Volume level versioning could follow similar schema, but version number needs to be concatenated to volume ID for historical versions Subcolumn families cannot be indexed Extra metadata are picked up even when only page s are needed Must store historical versions of volumes as deltas; naïve transla?on of the above format to historical versioning would have high cost in space