CoLang 2014 Data Management and Archiving Course. Session 2. Nick Thieberger University of Melbourne

Transcription

1 CoLang 2014 Data Management and Archiving Course Session 2 Nick Thieberger University of Melbourne

2 Quiz In a morning recording session you recorded two speakers, each telling a story, then recorded your questions to both of them together to describe what they told you, now in English. This was done in 3 audio files, 2 video files (of which the audio is also recorded) and 3 photographs. In addition you have handwritten notes. Later a speaker (who you are paying for her assistance) will transcribe the media. How would you prepare this dataset? What files would there be? What naming conventions would you use? How would you name the transcripts?

3 Briefly Why Toolbox, or Flex?

4 Toolbox IGT Id number - Media reference Text line Parsed Morphemic line Gloss line Free translation line

5 Parse function

6 Lookup function

7 Lexical database functions

8

9 Tracking text processing in Toolbox

10 Tracking text processing in Toolbox

11 How to get from the field to analysis to the archive? Using good tools, for example: Transcriber, Elan Toolbox Fieldworks Why are these good tools?

12 Transcription with timealignment Necessary step in building a corpus from which to make generalizations. No extra cost to use time-alignment Index of media Many possible outputs Tools create simple text files that encode the relationship of the text to the timecodes in the media

13 Other transcription tools

14 Other annotation tools Anvil ATLAS CLAN CSLU Toolkit The MATE Workbench Multitool SignStream SmartKom SyncWRITER TalkBank Transana Voicewalker List of tools at this wiki:

15 What is so good about these programs: Toolbox, Flex, Elan etc.? They produce good textual outputs Simple (but structured) text can be easily converted and archived

16 Working form The form in which information is stored as it is created and edited. Archival form The form in which information is stored for access long into the future. Presentation form The form in which information is presented to the public. Simons (2004)

17 Working form The form in which information is stored as it is created and edited. Can include notes, not all of which may be useful later. E.g., files being processed (in Elan, Toolbox or Flex) and ancillary files (.typ,.prj, etc) that are only necessary while working to create the annotated data.

18 Archival form The form in which information is stored for access long into the future. Highest resolution form of the data

19 Presentation form The form in which information is presented to the public. Derived from working or archival form. May be compressed and arranged to make it easier to deliver and interpret.

20 Reuse Recording >Transcript Preg nafkal skot namer nig Emlakul. NT A

21 Reuse Recording >Transcript > Interlinear \id 061:005 \aud NT A \tx Preg nafkal skot namẽr \mr preg nafkal skot namẽr \mg make war nig Emlakul. nig Emlakul with people of p.name \fg To fight the people of Malakula.

22 Book production based on IGT

23 Playable media, produced in Elan Text, interlinearised in Fieldworks

24 Longevity of character encoding Problems of legacy fonts fonts as variable representation of one underlying form e.g., ASCII character N could be represented by ŋ if IPATimes was selected as the font

25 Need for documents to be legible! Characters must be portable so that the documents are legible International standard for character encoding - unicode

26 Unicode issues - Choose characters carefully! cf. Mia Kalish s article in LD&C (2) - mixing different code sets can lead to sorting issues. à - option a à - a plus combining diacritic 0300 à - U-00E0

27 Using Unicode Setting up keyboards for more extensive unicode entry Windows XP: Tavultesoft Keyman: SIL IPA keyboard for Keyman: Microsoft Keyboard Layout Creator: globaldev/tools/msklc.mspx MacOS X: Ukelele: Linux: Keyboard Mapping for Linux; Handy tools for inserting characters:

28 Using Unicode Entering characters Windows XP: Windows XP character map - To Find Character Map: Programs > Accessories >System Tools > Character Map Use Alt-+-XXXX (e.g. Alt+00E9 [hex code]). Some applications (e.g. MS Word) support typing the hex code followed by Alt-x. Mac OS X: Mac - character palette Set up the Unicode Hex Input Keyboard (International Preferences). Use Option-XXXX (e.g. Option-00E9).

29 Questions / discussion?

30 Archiving

31 Archives, old places with old stuff

32 According to Jacobs & Humphrey (2004), Data archiving is a process, not an end state where data is simply turned over to a repository at the conclusion of a study. Rather, data archiving should begin early in a project and incorporate a schedule for depositing products over the course of a project s life cycle and the creation and preservation of ac- curate metadata, ensuring the usability of the research data itself. Such practices would incorporate archiving as part of the research method. Jacobs, James A., & Charles Humphrey. (2004). Preserving Research Data. Communications of the ACM 47(9):

33 Archiving If we make records of languages we need some way of making sure they last

34 Reusable and interoperable research data DVDs of field recordings archived directly from the northern Philippines

35 Why archive? Our responsibility to ensure longterm access and availability of the data we record For the speakers and their descendants Centrality of data in language documentation

36 Endangered data Digital word processing is our most advanced writing technology to date, but it is also the most ephemeral. Hardware and software technologies are changing so rapidly that a typical storage medium or file format is obsolete within 5 to 10 years. Digital records of endangered languages are in danger of dying out before the languages themselves

37 What should we do? 1. Put the materials into an enduring file format. 2. Provide sufficient metadata to make the files discoverable. 3. Deposit the materials with an archive that will make a practice of migrating them to new storage media as needed.

38 Archiving formats Text - XML, pdf, txt Media Audio pcm/wav/bwf Images TIF Video JPEG2000, MXF How do we get our data into these formats?