Crowdsourcing the transcription of archival data

Transcription

1 INSTITUTE FOR MATHEMATICAL BEHAVIORAL SCIENCES UC IRVINE Crowdsourcing the transcription of archival data Kimberly A. Jameson 1, Sean Tauber 1, Prutha S. Deshpande 2, Stephanie M. Chang 3, and Sergio Gago 3 1 Institute for Mathematical Behavioral Sciences, 2 Cognitive Sciences, and 3 Calit2 University of California, Irvine

2 UCI ColCat Project Collaborators: Prutha Deshpande Sean Tauber Stephanie Chang Sergio Gago Nathan Benjamin Yang Jiao Brian Huynh Han Ke Ram Bhakta Zhimin Xiang Ian Harris Funding and Support for the archive project: Calit2 at UCI. University of California Pacific Rim Research Program, (K.A. Jameson, PI). National Science Foundation (#SMA , K.A. Jameson, PI). UCI s UROP Program Awards. IRB Approvals HS# and

3 UCI ColCat Project Collaborators: Prutha S. Deshpande CogSci Sean Tauber IMBS Sergio Gago Calit2 Stephanie M. Chang Calit2

4 Talk Overview Background on an important problem in Cognitive Science. The domain under consideration: Color categorization. Creating a new database using internet-based procedures. Features of the internet-based research problem and solution approaches that may generalize elsewhere. Modeling the problem and developing appropriate analyses. Preliminary results from empirical tests. Summary.

5 Research on how concepts are represented across linguistic groups Individual concept formation and the sharing and transmission of concepts within and across groups. E.g., Kinship terminology

6 Concept formation across language groups E.g., Kinship terminology:

7 Concept formation across language groups E.g., Kinship terminology:

8 In what ways are representations of concepts similar across individuals and language groups? and What are the various ways concepts vary across individuals and language groups?

9 How do the world s languages map the color appearances we all see in our environments?

10 Basic Color Terms (1969) Brent Berlin Paul Kay Basic Color Terms being described as the smallest set of simple words with which the speaker can name any color.

11 Courtesy of Lindsey & Brown (2006). PNAS, 102.

12 Image Credit: Lindsey & Brown (2006). PNAS, 102.

13 Basic Color Terms (1969) (1) Found all languages tested had systems including 11 or fewer basic color words (e.g., English): red, yellow, green, blue, orange, purple, pink, brown, grey, black and white. (Terms such as crimson, blonde and royal blue are not considered to be basic.) (2) Provided a sequence by which languages adopted subsets of the 11 basic color categories.

14 Color concept universals like this were made popular by Berlin & Kay, and by several other investigators, still, there are instances where different societies have evolved different conventions for color naming... IMBS workshop UC Irvine 12/04/2015

15 Image Credit: Lindsey & Brown (2006). PNAS, 102.

16 Berinmo (5 words) Image Courtesy Credit: of Lindsey Kay & Regier & Brown (2007). (2006). Cognition, PNAS,

17 Different numbers of Color Terms: n=3 T. Regier et al, PNAS 104, 2007

18 Different numbers of Color Terms: n=3 n=4 T. Regier et al, PNAS 104, 2007

19 Different numbers of Color Terms: n=3 n=4 n=5 T. Regier et al, PNAS 104, 2007

20 Different numbers of Color Terms: n=3 n=4 n=5 n=6 T. Regier et al, PNAS 104, 2007

21 The World Color Survey 110 languages; 25 speakers. Data collection ended in Digitalizing hand coded data took more than 23 years. A very valuable site of unembellished ascii data files:

22 World Color Survey Data Uses a Generic Format

23 The existing World Color Survey (WCS) database (2009) Beginning ~2003 the WCS database was made publicly available. Has been very widely cited in the last few years.

24 E.g., Focus selection task: Shown the chart, pinpoint the best example of each root they volunteered while naming.

25 Datafile Example foci.txt : Color chip selected as category best-exemplar (WCS datafiles do not include headers) Language Number Speaker Number Focus Number Term Abbrv. Coordinates of focus selection

26 Focus selections in two languages: English Korean Deshpande, P.S. (under review). Investigating Color Categorization Behaviors in Korean- English Bilinguals. UCI Undergraduate Research Journal (submitted June, 2015).

27 See Poster: An Affordance Based Approach to Large Data-Set Navigation. The WCS data is awesome, but Nathan a platform with a GUI for empirically investigating and analyzing such data would be even better, and a site with rigorous on-board research tools would also be a big plus. We were given a chance to do this Jameson, K. A., Benjamin, N. A., Chang, S.M., Deshpande, P. S., Gago, S., Harris, I. G., Jiao, Y., and Tauber, S. (2015). Mesoamerican Color Survey Digital Archive. In Encyclopedia of Color Science and Technology, (Ronnier Luo, Ed.). Springer: Berlin / Heidelberg. ISBN: (Online). DOI /

28 The Robert E. MacLaury Archive ~23,000 pages of raw color categorization data that includes: 116 dialects from indigenous Mesoamerican societies (261 surveys), and ~130 additional surveys from a variety of languages (across Africa, Asia, the Americas and Europe).

29 R. E. MacLaury s Dissertation: Color in MesoAmerica, Vol. I: A Theory of Composite Categorization. (1986) Book: Color and Cognition in Mesoamerica: Constructing Categories as Vantages. (1997)

30 The mesoamerican portion of the REM archive: 33 within Mexico City 37 within Oaxaca 30 within Guatemala Jameson et al. (2015). ECST.

31 Chinantec language diversity in the MCS

32 Chinantec language diversity in the MCS Developing Vigorous Endangered Jameson et al. (2015). ECST.

33 Features of our transcription problem that may be general: The data has a constrained structure and format. (unlike typical historical records transcription tasks) It s a perceptual identification/reproduction problem: e.g., identify handwritten characters/symbols in a standardized template or form and reproduce them via keyboard input. transcription of large blocks of data can be broken into small tasks and transcribed by OCR or crowdsourcing methods. See Poster: Optical Character Recognition of Handwritten Tabular Data. Yang

34 Focus selection task: Shown the chart, pinpoint the best example of each root they volunteered while naming.

35 Focus selection task: Shown the chart, pinpoint the best example of each root they volunteered while naming. Problem: Convert THIS into a data addressable file

36 Problem: Convert THIS into a data addressable file American English Data

37 DATA... continues up to

38 Challenges of our transcription job: Concepts. How they apply everywhere There s a classic example color. There s an existing database. There s a chance to do better. Crowdsourcing can help greatly Why OCR doesn't work. Handwriting that is not prose. The reason is its a perceptual problem. Crowdsourcing lets us break the problem into pieces and solve it piecewise.

39 Features of our problem and approach that may apply elsewhere: The perceptual nature of our tasks differ from general information surveys or opinion-poll data e.g., response bias is likely to be itembased rather than the usual informant-based form, perhaps allowing more than one possible decision strategy. In large-scale efforts there s a need to automate quantification and evaluation of the goodness of the transcribed product. Minimize response bias by partitioning larger tasks into smaller, distributed, tasks that are answered by several subjects and reassembled into a whole lends itself to crowdsourced approaches. By definition, while crowdsourcing makes Big Data possible, an intelligent model of data aggregation (like CCT) may permit trading off smarter for bigger, giving a more economical approach to accurately deriving robust results using internet-based crowdsourcing methods. National Science Foundation (#SMA , K.A. Jameson, PI).

40 Features of our problem and approach that may apply elsewhere: The perceptual nature of our tasks differ from general information surveys or opinion-poll data e.g., response bias is likely to be itembased rather than the usual informant-based form, perhaps allowing more than one possible decision strategy. In large-scale efforts there s a need to automate quantification and evaluation of the goodness of the transcribed product. Minimize response bias by partitioning larger tasks into smaller, distributed, tasks that are answered by several subjects and reassembled into a whole lends itself to crowdsourced approaches. By definition, while crowdsourcing makes Big Data possible, an intelligent model of data aggregation (like CCT) may permit trading off smarter for bigger, giving a more economical approach to accurately deriving robust results using internet-based crowdsourcing methods. National Science Foundation (#SMA , K.A. Jameson, PI).

41 Features of our problem and approach that may apply elsewhere: The perceptual nature of our tasks differ from general information surveys or opinion-poll data e.g., response bias is likely to be itembased rather than the usual informant-based form, perhaps allowing more than one possible decision strategy. In large-scale efforts there s a need to automate quantification and evaluation of the goodness of the transcribed product. Minimize response bias by partitioning larger tasks into smaller, distributed, tasks that are answered by several subjects and reassembled into a whole lends itself to crowdsourced approaches. By definition, while crowdsourcing makes Big Data possible, an intelligent model of data aggregation (like CCT) may permit trading off smarter for bigger, giving a more economical approach to accurately deriving robust results using internet-based crowdsourcing methods. National Science Foundation (#SMA , K.A. Jameson, PI).

42 Features of our problem and approach that may apply elsewhere: The perceptual nature of our tasks differ from general information surveys or opinion-poll data e.g., response bias is likely to be itembased rather than the usual informant-based form, perhaps allowing more than one possible decision strategy. In large-scale efforts there s a need to automate quantification and evaluation of the goodness of the transcribed product. Minimize response bias by partitioning larger tasks into smaller, distributed, tasks that are answered by several subjects and reassembled into a whole lends itself to crowdsourced approaches. While crowdsourcing makes Big Data possible, an intelligent model of data aggregation (like CCT ) may permit trading off smarter data for bigger data, giving a more economical approach to accurately deriving robust results using internet-based crowdsourcing methods. National Science Foundation (#SMA , K.A. Jameson, PI).

43 Batchelder and Romney (1988) Test theory without an answerkey. Psychometrika. Cultural consensus analyses of a cognitive-perceptual task For tasks evaluating new characters designed to extend the 26 letters of the English alphabet, consensus analyses objectively identified expert typeface designers with higher competence compared to college undergraduates. Jameson & Romney (1990). Consensus on Semiotic Models of Alphabetic Systems. J. of Quant. Anthro.

44 * Automating archive transcription: Task and Judgments Design 1: OCR verification (pattern recognition) - 2-AFC yes/no Design 2: OCR verification (training data) - free response Design 3: Crowdsource verification - 2-AFC match/no-match Design 4: Naming ranges 1 - free response + confidence Design 5: Naming ranges 2 - N-AFC + confidence Design 6: Focus transcription 1 - free response + confidence Design 7: Focus transcription 2 - free response free response = a recaptcha task. Poster title: Designing Crowdsourcing Methods for the Transcription of Handwritten Documents. Stephanie

45 E.g., internet-based transcription task:

46 Cultural Consensus Theory (CCT) to aggregate the data Automate piece-wise crowdsourced transcription designs for analysis with CCT to derive the correct transcription. Enrich the model underlying Dichtomous Bayesian form of CCT (Oravecz, et al. 2014) to handle N-alternative forcedchoice data formats. As a result, employ smarter analyses of smaller samples, using CCT s formal process model, that produce solutions as robust as those from large amounts of averaged data. Deshpande, Tauber., Chang, Gago & Jameson. (in preparation). Digitizing a large corpus of handwritten documents using crowdsourcing and cultural consensus theory. See Poster: A Cultural Consensus Theory Analysis of Crowdsourced Transcription Data. Prutha

47 Results:

48 Results: Task 4 n=30

49 Results: Task 4 n=30 hi, hl

50 Results: Task 4 n=30

51 Inferring the true transcription Mode? (Bayesian) Cultural Consensus Theory (CCT) (Oravecz, Vandekerckhove & Batchelder, 2014) (Batchelder & Romney, 1988)

52 Cultural Consensus Theory (CCT) Test theory without an answer key (Batchelder & Romney, 1988) Allows us to infer: shared latent cultural knowledge (true transcription) individual ability item difficulty response bias

53 Cultural Consensus Theory (CCT) Usually applied to dichotomous (true/false) data. Other formats have been explored with Bayesian framework but not multiple choice / free response (to our knowledge). Not typically applied to perceptual identification (although, see Jameson 1990)

54 Dichotomous CCT Multiple Choice CCT

55 Dichotomous CCT Multiple Choice CCT Observed Data

56 Dichotomous CCT Multiple Choice CCT Observed Data

57 Dichotomous CCT Multiple Choice CCT Observed Data Latent Parameters

58 Dichotomous CCT Multiple Choice CCT Observed Data Latent Parameters

59 Dichotomous CCT Multiple Choice CCT Observed Data Latent Parameters

60 Dichotomous CCT Multiple Choice CCT Observed Data Latent Parameters

61 Dichotomous CCT Multiple Choice CCT Observed Data Latent Parameters (subject-wise bias)

62 Examples of perceptually confusable stimuli

63 Response bias: Individuals or items? subject-wise bias item-wise bias

64 Response bias: Individuals or items? subject-wise bias item-wise bias

65 CCT Answer Key: Task 4

66 CCT Answer Key: Task 4

67 CCT Answer Key: Task 4

68 CCT Answer Key: Task 4

69 CCT Answer Key: Task 4

70 subject-wise posteriors Answer 4 (Z4) Answer 16 (Z16) Answer 125 (Z125) Subject 0 bias (g0)

71 subject-wise posteriors Answer 4 (Z4) Answer 16 (Z16) Answer 125 (Z125) Subject 0 bias (g0)

72 item-wise posteriors Answer 4 (Z4) Item 4 bias (g4) Answer 16 (Z16) Item 16 bias (g16) Answer 125 (Z125) Item 125 bias (g125)

73 task 4 subject-wise model predictions

74 task 4 subject-wise model predictions item-wise model predictions

75 task 7 subject-wise model predictions

76 task 7 subject-wise model predictions item-wise model predictions

77 Can we use fewer informants? CCT was designed to work on small (6-10) sized subject samples typical of anthropological studies. Would the patterns of results reported for Task 4 be possible with a sample smaller than 30 participants? Method Answer Key Estimate %-correct Mean Competence Mean Item Difficulty Trial 1-8 participants 100% Trial 2-8 participants 100% Trial 3-8 participants 100% Trial 4-8 participants 100% Trial 5-8 participants 100% Participants 100% Preliminary trends suggests 8 participants may be as informative as 30.

78 Discussion points Two (or more) response-strategy subcultures? Confidence data can help CCT results Quantitative model evaluation Item + individual bias component? Automation and integration with other server-side processes (Python module vs. R, Matlab)

79 Results Summary: These preliminary results suggest two novel approaches, piece-wise crowdsourcing and CCT data handling, can be used to accurately transcribe a large corpus of ethnographic data. By using internet-based methods, it appears we can a avoid 20+ year manual transcription job and derive an accurate and unbiased database of great value to investigations of concept formation across language groups. The economical way in which we modeled this perceptuallybased transcription problem seems likely to generalize to other internet-based tasks that require extraction and evaluation of targets embedded in distracting information, and our novel use of CCT analyses seem promising for intelligently aggregating smaller subsets of crowdsourced responses to address large data handling problems.

80 Thanks for Listening!! Funding and Support for the archive project: Calit2 at UCI. University of California Pacific Rim Research Program, (K.A. Jameson, PI). National Science Foundation (#SMA , K.A. Jameson, PI). UCI s UROP Program Awards. IRB Approvals HS# and