How To Create A Global Data Science System For People Of Every Language

Transcription

1 Bringing Data Science to the Speakers of Every Language!!!!!!!! Robert Munro, PhD! CEO, Idibon!

2 CEO, Idibon! Global text analytics in 50+ languages! About me: technology and global development! Working with leaders in industry & social good! Industry: CTO / CIO! Energy infrastructure in Liberia and Sierra Leone! Global epidemic tracking! Crowdsourcing and natural language processing for disaster response! Other! Ph.D. in NLP from Stanford! Bicycled 20+ countries!

3 Recommendations for language processing for social good! Look beyond English! Inherent benefit understanding and support speakers of every language!! Employ people in those languages! Crowdsourced workers speak 100s of languages, and want to use them!! Embrace the variation! You can t rely on consistent spellings, but you can learn to model the diversity!!

4 How many languages are in the connected world?!.'/%0#-%102-%30#*+"45% &%'(%)#*+,#+"-% !"#$%

5 How many languages are in the connected world?!.'/%0#-%102-%30#*+"45% &%'(%)#*+,#+"-% !"#$%

6 How many languages are in the connected world?! &%'(%)#*+,#+"-% !"#$%

7 How many languages are in the connected world?! 6,7*+%#%80'*"%2*%10"% 0#*4-%'(%"9"$:'*"%'*% 10"%8)#*"1%2-%10"%"#-:% 8#$1% 5000 ;*4"$-1#*42*+% "9"$:'*"%2-%+'2*+% 1'%<"%='$"% 3'=8)23#1"4% 2000 &%'(%)#*+,#+"-% !"#$%

8 Every human communication this year! Source: Ethnologue, Nationalencyklopedin

9 7% of our communications are digital, most is still direct spoken language!

10 If every online picture is worth a thousand words, it would double social media.! Every picture!

11 Every 3 months, the world's text messages exceed the word count of every book.! Every book. Ever.! Source: Google Books

12 Print communication is smaller than anything shown.!!

13 Print communication is smaller than anything shown.! Ditto any one social network.!

14 The Twitter firehose is about the size of the dot above the i in English.!! Beyond the processing capacity of most organizations.!! Might not be a representative sample of all human activity for your area of interest.!

15 There are more than 6,000 other languages.! Only the top 1% are shown.!

16 No language from the Americas made the cut.! Quechua!

17 spam would be larger than every block except spoken Mandarin ( ).! Source: Mashable

18 Short messages (SMS and IM) make up 2% of the world s communications.! The largest and most linguistically diverse form of written communication that has ever existed.!! # PhDs focused on processing large volumes of short messages in low resource languages?!!!!!!!!1!

19 If the Facebook like is a one-word language it is in the top 5% of languages by word count.!

20 Your browser probably won't show Sundanese script! (!!!!!!!)!

21 Sundanese speakers outnumber the populations New York, London, Tokyo and Moscow.! Combined.!

22 You misread Sundanese as "Sudanese" which is a variety of Arabic! We have a blind spot for knowing about the existence of languages.!

23 This is the breakdown of languages that most of our data is moving towards!

24 B#*,#$:%CDE%DFCF% % January 12, 2010! G'-1%)'3#)%-"$923"-%(#2)"4E% <,1%='-1%3"))H1'/"$-% $"=#2*"4%(,*3A'*#)I%% %

25 Messages start streaming in!

26 Messages start streaming in!

27 Mission 4636! Fanm gen tranche pou fè yon pitit nan Delmas 31 ( , ) Fanm gen tranche pou fè yon pitit nan Delmas 31 Undergoing children delivery Delmas , Emergency!"##$%"& '($)#*$'"+,& -$'"%.(/0"+&1& %".*.-$'"+ Fanm gen tranche pou fè yon pitit nan Delmas 31 Undergoing children delivery Delmas , Emergency 2.-$'/.)&/#& ("3/)"+&1& $-'/.)$4*"&/'"5#& $("&/+")'/3/"+

28 Global collaboration! 2,000 volunteers, transferred to paid workers in Haiti!

29 J'821#)%K#3$"HL'",$% 8',%)2%$"-"9/#% =',*%=#)#4%"%)#8% % NK#3$"HL'",$%.'-821#)%/0230% )'3#1"4%2*%102-% 92))#+"%'(%M@#8%2-% $"#4:%1'%$"3"29"% 10'-"%/0'%#$"% 2*O,$"4I%P0"$"('$"E% /"%#$"%#-@2*+% 10'-"%/0'%#$"%-23@% 1'%$"8'$1%1'%10#1% 0'-821#)IQ% %

30 J'821#)%K#3$"HL'",$% 8',%)2%$"-"9/#% =',*%=#)#4%"%)#8% % NK#3$"HL'",$%.'-821#)%/0230% )'3#1"4%2*%102-% 92))#+"%'(%M@#8%2-% $"#4:%1'%$"3"29"% 10'-"%/0'%#$"% 2*O,$"4I%P0"$"('$"E% /"%#$"%#-@2*+% 10'-"%/0'%#$"%-23@% 1'%$"8'$1%1'%10#1% 0'-821#)IQ% %

31 J'821#)%K#3$"HL'",$% 8',%)2%$"-"9/#% =',*%=#)#4%"%)#8% % NK#3$"HL'",$%.'-821#)%/0230% )'3#1"4%2*%102-% 92))#+"%'(%M@#8%2-% $"#4:%1'%$"3"29"% 10'-"%/0'%#$"% 2*O,$"4I%P0"$"('$"E% /"%#$"%#-@2*+% 10'-"%/0'%#$"%-23@% 1'%$"8'$1%1'%10#1% 0'-821#)IQ% %

32 Local knowledge! Workers collaborating to find locations: Dalila: I need Thomassin Apo please Apo: Kenscoff Route: Lat: , Long: Apo: This Area after Petion-Ville and Pelerin 5 is not on Google Map. We have no streets name Feedback from responders: "just got emergency SMS, child delivery, USCG are acting, and, the GPS coordinates of the location we got from someone of your team were 100% accurate!" Dalila Apo Haiti responders The ability for someone to make a real-time difference at any other place in the world: Apo: I know this place like my pocket Dalila: thank God u was here ( , ) here = anywhere

33 How do we automate processing the world s data?!

34 English! Generations of standardization in spelling and simple morphology! Whole words suitable as features for NLP systems! Most other languages! Relatively complex morphology! Less (observed) standardized spellings! More dialectal variation!

35 Haitian Krèyol! No standard (wide-spread) spellings! More or less French spellings! More or less phonetic spellings! Frequent words (esp pronouns) are shortened and compounded! Regional slang / abbreviations!

36 Haitian Krèyol! =R-2E%="-2E%% =R32E%="$32% % % % L%#%8%H%.%#%S%1%2%"%*% % T # 8 # : 2 - : " * %

37 The extent of the subword variation! >30 spellings of odwala ( patient ) in Chichewa! >50% variants of odwala occur only once in the data used here:! Affixes and incorporation! kwaodwala -> kwa + odwala! ndiodwala -> ndi odwala (official ngodwala not present)! Phonological/Orthographic! odwara -> odwala! ndiwodwala -> ndi (w) odwala!

38 Chichewa! The word odwala ( patient ) in 600 text-messages in Chichewa and the English translations!

39 Modeling the variation gives accurate results!!"#$$%&'!% $%!(*%1%,-$3/*%!(/;&/$2"#0#!245 A/#!/H/0+%??#&#0%(#;!/ 211;1?/*#()/1%*/ $2??%:2?!"#$%&'!% $%!(*%+% $%!6(*%+% &'!()*'+' G5/."2!(#&3/D12"#0(;1?!"# 6&'!///$%!6(*%+%,7./!22"/$2"#0#!285 9%(2:;13/</7=2>'2?(/&;1/%#"8!"# 6&'!///$%!6(*%+%,7./!22"/$2"#0#!285 9%(2:;13/</7=2>'2?(/&;1/%#"8 A/#!/!" 0+%??#&#0%(#;!/ 211;1/D;?(6D1;02??#!:I/ #$%&'()* *#()/?0%+2I

40 Comparison with English! >33,$#3:U%G23$'H(% 6"$3"*1%'(%1$#2*2*+%4#1#%

41 Taking it to the world!

42 The benefits of understanding everyone!.,=#*%42-"#-"-%"$#423#1"4%2*%10"%)#-1%vw%:"#$-u%% -=#))8'Y% X*3$"#-"%2*%#2$%1$#9")%2*%10"%)#-1%VW%:"#$-U%

43 Reports of strange new illnesses pre-date official records!.x_% 4"3#4"-% [`W%=2))2'*% 2*("31"4]%.CZW%[a2$4%\),]%% [bwf^%(#1#)]%.czc%[k/2*"%\),]% ='*10-% [CF^%'(%/'$)4% 2*("31"4]%

44 but the reports are in 1000s of languages! cf^%'(%"3')'+23#)%429"$-21:% cf^%'(%)2*+,2-a3%429"$-21:% -!0"""" 1"""" # 23, ("""" -""""."""" / + +&""" '("" )"""" *"""" ' +!"""" # $"""" % &"""" # 1 H5N1

45 Crowdsourcing, big data, and expert analysts! G'-1%2*('$=#A'*%2-%2*%8)#2*%)#*+,#+"U%% % -!0"""" 1"""" # 23, ("""" -""""."""" / + +&""" '("" )"""" *"""" ' +!"""" # $"""" % &"""" # 1 H5N1

46 i"8'$1-% =2))2'*-%8"$%4#:U% =#*:%)#*+,#+"-E% =,30%*'2-"% Digital Disease Discovery! a2+%d#1#% =#302*"%)"#$*2*+U% "Y1$#3A'*E%e)1"$2*+% f%8$2'$2ag#a'*% L$'/4-',$32*+% 10',-#*4-%'(% *#A9"H)#*+,#+"% >*#):-1-%% -"9"$#)% 4'=#2*% "Y8"$1-% h)'<#)% ='*21'$2*+% K#("$%/'$)4%

47 The impact of scalable monitoring! \',*4%02-1'$23#)%-2+*#)-%10#1%8$"H4#1"4%!""#$%&'$(& % NXl=%B#3?,2%B"$#-%/210%1'4#:l-%3')4%#*4%m,%$"8'$1%III% #3$'--%10"%=24H>1)#*A3%-1#1"-E%#%)2n)"%<21%'(%#*%2*3$"#-"Q% January 4, 2008 CNN Weather! %%

48 The impact of scalable monitoring! M$+#*2g#A'*I% % N/"%/"$"%#<)"%1'%8,))%2*%=,30%$230"$%4#1#%($'=%#%)#$+"$% *,=<"$%'(%-',$3"-E%-'%/"%@*"/%*'1%O,-1%0'/%=#*:%8"'8)"% /"$"%2*("31"4E%<,1%/0#1%@2*4%'(%1$#*-8'$1%10":%1''@%/0"*% 10":%/"*1%($'=%10"2$%92))#+"%1'%10"%0'-821#)%2*%10"%*"#$"-1% =#2*%1'/*IQ%."/%*0&1(+*"2&34&!%+%*5$&6--%7/$8&"+&9:;#&<505&5+,&!$"/5$&<%=%$">7%+0?& & 6#%&@&#%+,%*&@&=;$$5#%&@&A%+0&0"&B"->;05$I% P02-%2-%8"$-'*#)):%24"*A(:2*+%f%3',)4%)"#4%1'%8"$-"3,A'*I% M8"*%4#1#%/',)4%0#9"%#%*"+#A9"%2=8#31I%

49 The impact of scalable monitoring! oldli%.'/%4'%/"%='a9#1"%2*('$=#a'*%8$'3"--2*+5% % % % % % % % % G#$+2*-%#$"%-=#))U%'*):%('$H8$'e1%<2+%4#1#%#*4% 3$'/4-',$32*+%3#*%0#9"%#%-,-1#2*"4%2=8#31%

50 Idibon s current work! Hurricane Sandy! Idibon s CTO ran FEMA s Aerial Damage Assessments.! We have >1,000,000 manual tags on communications.! MIT Humanitarian Response Lab! Identifying reports about supplyline interruptions.! Research data from a combination of crowdsourcing and natural language processing!!

51 Recommendations for language processing for social good! Look beyond English! Inherent benefit understanding and support speakers of every language!! Employ people in those languages! Crowdsourced workers speak 100s of languages, and want to use them!! Embrace the variation! You can t rely on consistent spellings, but you can learn to model the diversity!!

52 Thank you!!!!!!!!! Robert Munro, PhD! CEO, Idibon!