Cultural Trends and language change

Transcription

1 Cultural Trends and language change Gosse Bouma Information Science University of Groningen NHL 2015/03 Gosse Bouma 1/25

2 Popularity of Wolf in English books Gosse Bouma 2/25

3 Google Books Ngrams Digital Library Google Books is a project where books are scanned and turned into text using OCR (Optical Character Recognition) and made searchable with Google search. currently approx. 20 M books, mostly English, mostly since 1800 Google Books Ngrams: Valuable resource for cultural and linguistic studies Gosse Bouma 3/25

4 Google Books Google Books & ngrams viewer The Google Labs N-gram Viewer is the first tool of its kind, capable of precisely and rapidly quantifying cultural trends based on massive quantities of data. It is a gateway to culturomics! The browser is designed to enable you to examine the frequency of words (banana) or phrases ( United States of America ) in books over time. You ll be searching through over 5.2 million books: 4% of all books ever published! A-users-guide-to-culturomics Gosse Bouma 4/25

5 Popularity of various *isms Gosse Bouma 5/25

6 Google Books Ngrams Viewer Jean-Baptiste Michel et al., Quantitative Analysis of Culture Using Millions of Digitized Books. Science books.html Gosse Bouma 6/25

7 Google Books Ngrams Viewer Examples freedom, liberty vampire, werewolf, zombie computer, phone, radio, gun radio, television, internet best, beft Gosse Bouma 7/25

8 Google Books 2.0 The USA is/are Do people think of the United States as a singular or plural entity? And is this constant over time? Gosse Bouma 8/25

9 Google Books 2.0 The USA is/are Do people think of the United States as a singular or plural entity? And is this constant over time? We can answer this question by computing how often the United states is followed by examples of frequent plural verbs and divide the result by the overall frequency of the United States, We can do the same for the United states followed by a frequent singular verb. Gosse Bouma 8/25

10 Google Books 2.0 The USA is/are Do people think of the United States as a singular or plural entity? And is this constant over time? We can answer this question by computing how often the United states is followed by examples of frequent plural verbs and divide the result by the overall frequency of the United States, We can do the same for the United states followed by a frequent singular verb. The result is shown here: Notice the use of arithmetic to sum and divide frequencies. Gosse Bouma 8/25

11 Determiners and Country Names "The Ukraine" is incorrect both grammatically and politically, says Oksana Kyzyma of the Embassy of Ukraine in London. "Ukraine is both the conventional short and long name of the country," she says. "This name is stated in the Ukrainian Declaration of Independence and Constitution." Gosse Bouma 9/25

12 Google Books 2.0 Spelling,phrasal expressions recognise, recognize (try) gone missing (try) graduated college vs graduated from college Search using part of speech experience_noun vs. experience_verb (try) to always _VERB_, never, quickly, boldly Arithmetic (The United States have + The United States are)/the United States (The United States has + The United States is)/the United States try Gosse Bouma 10/25

13 Literary Applications First Person Fiction Ted Underwood suggests there is a sharp drop in the percentage of first person narratives around 1800 Can we investigate this using corpus linguistics? we-dont-already-know-the-broad-outlines-of-literary-history Gosse Bouma 11/25

14 Literary Applications Given novels that are clearly written with a 1st or 3rd person narrator Which words do occur significantly more often in 1st or 3rd person novels? Gosse Bouma 12/25

15 Literary Applications Given a large collection of fiction books: Does the ratio between 1st and 3rd person pronouns change over time? Gosse Bouma 13/25

16 Using Syntax Dependency Relations What are frequent direct objects of drink? drink => *_NOUN What things are magnificent? *_NOUN_ => magnificent Gosse Bouma 14/25

17 Google Ngrams Google BOOKS ngram viewer uses books Google NGRAMS is Web data Gosse Bouma 15/25

18 Dutch Twitter Corpus Since 2011 the Information Science Department of the University of Groningen has been collecting Dutch language tweets. The goal is to collect a representative sample of all tweets posted in Dutch. We estimate that our method captures approximately 40-60% of the relevant tweets. Gosse Bouma 16/25

19 Dutch Twitter Corpus Since 2011 the Information Science Department of the University of Groningen has been collecting Dutch language tweets. The goal is to collect a representative sample of all tweets posted in Dutch. We estimate that our method captures approximately 40-60% of the relevant tweets. RieksOsinga #CTAboutaleb op Mooie en inspirerende woorden. AHPOIESZ Vandaag ons winnend concept gepresenteerd aan College van is er een probleem met de mail? Krijg namelijk een 500 interland server Error. Gelukkig! ICT geeft ook aan dat er geen storing, dat scheelt;-) Fijne zondag nog! NHL_Hogeschool Klaar voor collegereeks Met twee bedrijven Gosse Bouma 16/25

20 Spelling Variation How often is eens written as is? Dit kan uiteraard wel is voorkomen in de statistiek. Source Dutch Twitter Ngram counts, Query wel [is,eens] %en Gosse Bouma 17/25

21 Spelling Variation How often is eens written as is? Dit kan uiteraard wel is voorkomen in de statistiek. Source Dutch Twitter Ngram counts, Query wel [is,eens] %en trigram count perc wel eens voorkomen wel is voorkomen Gosse Bouma 17/25

22 Spelling Variation How often is eens written as is? Dit kan uiteraard wel is voorkomen in de statistiek. Source Dutch Twitter Ngram counts, Query wel [is,eens] %en trigram count perc wel eens voorkomen wel is voorkomen wel eens gebeuren 2, wel is gebeuren Gosse Bouma 17/25

23 Spelling Variation How often is eens written as is? Dit kan uiteraard wel is voorkomen in de statistiek. Source Dutch Twitter Ngram counts, Query wel [is,eens] %en trigram count perc wel eens voorkomen wel is voorkomen wel eens gebeuren 2, wel is gebeuren wel eens zien 34, wel is zien 10, Gosse Bouma 17/25

24 Spelling Variation zeςma Zeς ma is a discourse marker of Arabic etymology that is used in North Africa as well as in the French and Dutch varieties spoken by the North African diaspora in Europe...On the internet, users either omit ς or use another character, e.g. the digit 3. (Bouwmans, 2003) Source Dutch Twitter Ngram counts, Query ze%ma Gosse Bouma 18/25

25 Spelling Variation zeςma Zeς ma is a discourse marker of Arabic etymology that is used in North Africa as well as in the French and Dutch varieties spoken by the North African diaspora in Europe...On the internet, users either omit ς or use another character, e.g. the digit 3. (Bouwmans, 2003) Source Dutch Twitter Ngram counts, Query ze%ma hits word hits word 76,218 zehma 2,845 Zegma 45,561 ze3ma 2,149 ze3hma 29,944 zegma 1,707 zemma 15,058 Zehma 1,568 ZEHMA 8,448 Ze3ma 1,553 zema (and at least 20 other spelling variants) Gosse Bouma 18/25

26 Language Change: Popularity of der (her/there) ik ga der geld geven voor der verjaardag Mag je der vaseline opdoen? Dutch Twitter Ngram counts 2014 Gosse Bouma 19/25

27 Ngram statistics Why use ngram counts? Given enough data, ngram frequencies are often sufficient to study variation and trends Dutch Twitter Corpus # (Million) tweets 2,500 tokens 28,000 unigrams bigrams trigrams grams grams Gosse Bouma 20/25

28 Comparable Tools and Resources Twitter viewers (Univ Groningen, twiqs.nl) : links to actual tweets, trends, metadata slow for large periods and/or frequent ngrams Google Web 1T 5-Gram Database for European languages: ngram counts for 133 billion words of Dutch webtext Regex search, collocations: Corpus Frequency counts Keuleers et al (2010), word frequencies based on Dutch subtitles... Rovereto Twitter n-gram corpus with demographic metadata Herdagdelen (2013) : a Twitter-based dataset using n-grams, thereby overcoming the limitations on the redistribution of raw tweets n-gram counts for 75 million English tweets With gender-of-author and time-of-posting Gosse Bouma 21/25

29 Twitter Ngrams Web Interface Raw ngram counts ( ) Limited regex support, export results as csv, collocations, associations Run your own experiment : Download ngrams data Evert (2010), Google Web 1T 5-Grams Made Easy (but not for the computer) Gosse Bouma 22/25

30 Twitter Ngrams Web Interface - Trends Relative frequencies per month For ngrams occurring at least once in each month Using sqllite + Google Tables Gosse Bouma 23/25

31 Twitter vs Google Web Ngrams een meisje/liedje/... die/dat noun Twitter %die web %die ratio meisje liedje kind type bedrijf boek nummer geld filmpje ding Twitter: , Web Ngrams: 2008 Gosse Bouma 24/25

32 Enjoy! Gosse Bouma 25/25