Working with Text Collections

Transcription

1 Working with Text Collections MINITEX Library Information Network Bibliographic & Technical Services (BATS) Presented by: Sara Ring October, 2008 Outline Review & Exercises Configuring our Collections Creating our Project/s Demonstration & Exercises Importing and searching full text PDF Document Postcard Monograph Editing a compound object 2 1

2 System Architecture Acquisition stations or clients (1 50) Now includes JPEG2000 Web-based editor OCR extension Server Administration tools Management Security Statistics Register with WorldCat Custom Web interfaces Web browsers Access Display Administer 3 Customize interfaces Configuring a Collection What s a Collection? A group of objects (items) that Share the same metadata schema Live on the same CONTENTdm server How many Collections can I have? Up to 200 collections per server How many items can be in a collection? 16 million items per collection 4 2

3 Populating a collection Through the use of a Project What s a CONTENTdm Project? A workspace on your personal computer Into which you import up to 5000 items at a time Where items reside until you upload to the server A group of settings that are applied to the items E.g., image display resolution, file format, branding E.g., automatic metadata input How many Projects can I have at one time? Limited only by your disk space on the workstation 5 RELATIONSHIP of Collection to Projects A single Collection Many Projects Project 2 Project 1 Collection Project 3 3

4 What s a CONTENTdm object or item? Display any file format: Viewed with a Web browser natively or viewed via a plug-in Including: JPEG, JPEG2000, TIFF, PDF, WAV or MP3 audio, AVI or MPEG video, html, MrSID Simple items e.g., images, sound files We ll load a PDF as a simple item today 7 CONTENTdm Compound Objects CONTENTdm defined classes Documents We will load a Document Postcards We will load a handwritten postcard Monographs (Structured documents) We will load a book with chapters Picture Cube (six-sided id d views) We will not be working with this type of object We ll load a PDF as a compound object today as well 8 4

5 Dublin Core Metadata Element Set 9 Review: Basics of CONTENTdm Simple and Qualified Dublin Core element sets offered 100 fields per collection Only DC.Title required to create a record Dublin Core is basis for cross-collection searching Text is stored in a metadata field 128,000 characters per full text search field 200 collections/server i.e., i e 200 different metadata schema 10 5

6 Configuring Your Collection 11 Metadata Additions Administrative fields View and configure six administrative metadata fields Full resolution OCLC number Date created Date modified CONTENTdm number CONTENTdm file name Each field can be designated as searchable and mapped to Dublin Core Field names can be changed and exposed in the collection interface 12 6

7 Metadata Additions Administrative fields Default settings Hidden Not searchable No DC mapping Some configuration options do not apply Controlled vocabulary Large field Data type cannot be changed Content in fields is system generated Full resolution and OCLC number fields can be edited 13 Administrative fields 14 7

8 Administrative fields 15 Administrative fields 16 8

9 Metadata Additions Shared Controlled Vocabulary Share controlled vocabularies between fields Within a single collection or across multiple collections Any controlled vocabulary can be shared Changes to a shared vocabulary are accessible from all fields using it Administration similar to standard controlled vocabulary Add, delete, browse and verify New administrative functions for sharing View list of fields using shared controlled vocabulary Change sharing 17 Metadata Additions Shared Controlled Vocabulary Controlled lld vocabulary bl must be shared before bf it is accessible from other fields fild Name shared controlled vocabularies for identification Name cannot be changed after creation Stop sharing a controlled vocabulary at any time Local copy of vocabulary is made for that field Cannot delete a shared controlled vocabulary that is used by more than one field Shared vocabulary is only deleted when setting is changed to do not share in the last field using it 18 9

10 Administrative fields 19 Administrative fields 20 10

11 Administrative fields 21 Exercise 1 Configure Your Own Collections Change the name of your collection and enter a description Add 2 fields Tag Transcript Make fields searchable 22 11

12 Review: Populating Collections Acquisition Station Projects (PC client) Add from CONTENTdm Administration (Browser- based) Connexion digital import (WorldCat cataloging client function) 23 Review: Acquisition Station - PC client Project workspace Project settings Tools to manage Image settings Metadata settings 24 12

13 Template Creator Use Projects as customized build templates Can treat sets of objects destined for a specific Collection in a consistent fashion Examples If you re working with a batch of digital objects that all have the same Publisher To insert information from the digital file Set the Identifier field to read the file name upon import and automatically insert it in the field 25 Review: Add web based function Platform independent Simple item add function may be used for single import of: Images -.jpg,.jp2,.tif (if bandwidth allows) PDF - single and multi-page Audio Video Can import compound objects via the web based Add function, but it is clunky to do so 26 13

14 Review - Exercise 2 Define a Project Review Template Creator Adding Metadata 27 Working with Text 28 14

15 Providing searchable text 29 Remember: Metadata fields can be made searchable In addition, full-text, extracted from the digital object itself can be stored in a metadata field designated as Full text search data type, in any of three ways: 1. Extracted (by server) from PDFs (if embedded to begin with) 2. Imported as.txt transcript Typescripted from handwritten or OCR d in advance (external OCR engine) 3. Generated by OCR on-the-fly (integrated ABBYY FineReader ) Excerpted from Creating and managing text collections using CONTENTdm 30 15

16 First things First - Prepare the Collection For importing searchable text items, whether singly or in batch at minimum: 1. One empty, searchable field is configured as Full text search data type to hold text 2. Configure to treat PDFs as compound objects or simple items. 3. Collection is configured to provide Full Resolution file management. 4. Other fields are made searchable, hidden, moved, or added, as needed. 5. OPTIONAL: the Web templates are adjusted to suppress display of components of compound objects in search results. 31 Importing PDFs 32 16

17 PDF - Prepare the items The PDF has been created with searchable text embedded. Beware: Not all PDFs are created equal! Multiple Ways to Handle PDFs Treat it as a single item Treat it as a compound object See the Tutorial in your packet Using PDF File in CONTENTdm 34 17

18 IMPORT PDFs AS SINGLE OBJECTS 35 Import as a Single Item Make sure that the convert multiple-page PDF files to compound objects is turned OFF at the Collection Level (by default it is off) 36 18

19 PDF Imported as a single item End Result 37 Generic PDF thumbnails display until uploaded to the server 38 19

20 Thumbnail creation & text extraction 39 PDFs and Text Extractions Text is extracted from the PDF and inserted into the full text search field when the item is added to a collection Collection must have a full text search field Full text search field must be empty when item is added to collection PDF must have embedded text PDF cannot be encrypted or locked 40 20

21 New with version 4.3 Inline display PDF files display inline in both PC and Mac browsers Single item viewer Compound object viewer Page & text viewer PDF display in single item viewer is configurable Default display is Access this item link Change by editing the S_SHOW_ITEMVIEW_PDF variable in the global style file 41 Inline Display 42 21

22 PDF Search Term Highlighting Search term highlighting Search terms highlighted when view PDF Single item viewer Compound object viewer Page & text viewer Adobe Reader controls highlighting behavior Only supported in Adobe Reader All occurrences of words are highlighted 43 PDF Search term highlighting 44 22

23 IMPORT PDFs AS COMPOUND OBJECTS 45 Importing PDFs as Compound Objects Compound object conversion Multiple-page PDF files can automatically be converted to compound objects when added to a collection Enable functionality per collection Default setting is disabled Enable by editing collection configuration settings in CONTENTdm Administrationn When not enabled multiple-page PDF files will be processed as a single item (thumbnail generation, text extraction, displayed in item viewer) 46 23

24 Compound object conversion 47 PDFs as Compound Objects Compound object conversion When compound object conversion is enabled, CONTENTdm: Creates a compound object based on the page order of the PDF. Generates a page-level metadata record for each page. Extracts text from the PDF, converts it to UTF-8, and inserts it into the full text field of the associated page level record. Generates thumbnail images of each page of the PDF. The thumbnail image of the first page will also be used for the compound object. Retains the original PDF file for export and printing. Displays the PDF compound object in a compound object viewer with each page of the PDF accessible from the left navigation menu. Highlights search terms in the PDF. Provides an option to select a subset of the PDF to print or save

25 PDFs as Compound Objects (cont.) Compound object conversion Conversion must be enabled for the collection The PDF must have more than one page All processing occurs when the PDF is added to the collection The PDF cannot be encrypted or locked Text extraction requires an empty, full text search field PDF compound objects have special properties Pages are virtual,,generated upon request Pages cannot be deleted or exported separately Permissions apply to the entire compound object Structure of PDF compound object cannot be edited 49 Compound Object Conversion 50 25

26 Compound Object Conversion 51 Compound Object Conversion 52 26

27 Printing and downloading PDFs Complete print version Original PDF file retained for printing and saving Subset of print version Select a subset of pages from the PDF to view, save, or print Select all pages with search hits or pick individual pages or page ranges Do not have to wait for large download if only need a few pages Also available for non-pdf compound objects when they have been processed using the OCR extension 53 Printing and Downloading 54 27

28 PDFs as Compound Objects When PDFs are converted to compound objects: It reduces the size of file that is downloaded for viewing An entire PDF may be several MB but individual pages are much smaller View a page within large PDF without downloading the full document Increase speed of access to view Provides full text indexing by page not document No secondary search required to find specific content in PDF Print only the information you need Better end-user experience! 55 PDFs as Compound Objects Quick and efficient PDF pages of compound object do not count against total number of items on the server Ideal for born digital documents Theses, dissertations, government documents, e-publications, and more CAUTION: Not ideal for scanned images, maps, newspapers, etc. Slow download times No embedded text 56 28

29 Working with PDFs Large file download A PDF over 20 MB will not load inline in any of the viewers Single item viewer if entire PDF is over 20 MB Compound object viewer if single page of PDF is over 20 MB Page & text viewer if single page of PDF is over 20 MB Subset viewer is selected subset is over 20 MB Complete print version if entire PDF is over 20 MB Download prompt displays with option to save or open the file outside of browser File can download in background File can be opened while download is in process 57 Exercise 3 Import PDF as Compound Object Configure your Collection to convert multi-page PDFs to compound objects Import PDFs from the Exercises Folder Upload to the Server View the PDFs in your Collection and look at the searching and printing options 58 29

30 IMPORT A DOCUMENT 59 Exercise 4: Import A Document See Exercise 4 in your Packets 30

31 Tips on working with Documents All files in the root directory will be imported in ascending alphanumeric order bell1.jpg will sort before bell2.jpg 001_Introduction will sort before 002_Appendix If you use text files for transcript text make sure the image file name corresponds to the correct text file name 001_page1.jpg corresponds to 001_page1.txt Make sure your text files and image files are in separate directories Tips on working with Documents (cont.) Enable one of your Collection Fields as data type Full text search The template creator is best used for format type, file names, and other administrative metadata in page-level metadata 31

32 POSTCARDS Postcard Collections 32

33 Postcard Collections - Customizations Cuban Postcard Collection 33

34 Exercise 5 Import A Postcard Browse to the Postcard Folder within your Exercise Folder Exercise 5 End Result 34

35 Tips on Working with Postcards Label the front and back image file names with number prefixes so they sort correctly E.g. 01_front and 02_back Enable one of your Collection Fields as data type Full text search Can use a tab delimited file to provide object metadata and page level metadata OR enter in metadata after import. MONOGRAPHS 35

36 Working with Monographs CONTENTdm supports up to nine levels of hierarchy within a monograph If using directories to define document structure, all image files must be in a single directory. If you have text files associated with your monograph, they must be stored in a separate directory Working with Monographs Files shown below the subdirectories will be the first to appear in the imported monograph 36

37 Working with Monographs In this case there are no files within each subdirectory (no sections within each chapter for example) Exercise 6 Import A Monograph Browse to the Monograph Folder within your Exercises Folder on the Desktop Follow the steps in the Import Compound Objects Wizard for working with a monograph. 37

38 Working with Monographs Click on Preview You can do any last minute editing here before upload. Working with Monographs - Final Results 38

39 Tips on working with Monographs Label the image file names with number prefixes so they sort correctly E.g. 001_Page1 and 002_Page2 Enable one of your Collection Fields as data type Full text search You can also choose to import metadata and define the monograph structure using a tab-delimited text file Getting Help with Text User Support Center Downloading the appropriate Acquisition Station JPEG2000 Installing, activating the OCR extension Tutorials to study Help files related to text works Write contentdmsupport@oclc.org 78 39

40 EDITING A COMPOUND OBJECT Editing Compound Objects See Exercise 7 in your packets For more on Adding and Editing compound object via web administration, see: (note: need a user support center log on and password to view 40

41 RESOURCES AND SUPPORT 81 Resources 82 41

42 Support Local system administrator is contact MINITEX, WiLS, or your local RSP (user support) Where technical support needed, the designated contact person calls/writes DiMeMa/OCLC directly (877) Toll free 83 CONTENTdm Users Share Join the local Upper Midwest CONTENTdm listserv: Join the national listserv: 84 42

43 Questions? Bibliographic & Technical Services MINITEX Library Information Network (800) (612)