# INFO 2950 Intro to Data Science. Lecture 17: Power Laws and Big Data

Save this PDF as:

Size: px
Start display at page:

Download "INFO 2950 Intro to Data Science. Lecture 17: Power Laws and Big Data"

## Transcription

1 INFO 2950 Intro to Data Science Lecture 17: Power Laws and Big Data Paul Ginsparg Cornell University, Ithaca, NY 29 Oct /25

2 Power Laws in log-log space y = cx k (k=1/2,1,2) log 10 y = k log 10 x +log 10 c sqrt(x) x x**2 100 sqrt(x) x x** /25

3 Zipf s law Now we have characterized the growth of the vocabulary in collections. We also want to know how many frequent vs. infrequent terms we should expect in a collection. In natural language, there are a few very frequent terms and very many very rare terms. Zipf s law (linguist/philologist George Zipf, 1935): The i th most frequent term has frequency proportional to 1/i. cf i 1 i cf i is collection frequency: the number of occurrences of the term t i in the collection. 3/25

4 s law Zipf s law: the frequency of any word is inversely proportional to its rank in the frequency table. Thus the most frequent word will occur approximately twice as often as the second most frequent word, which occurs twice as often as the fourth most frequent word, etc. Brown Corpus: the : 7% of all word occurrences (69,971 of >1M). of : 3.5% of words (36,411) and : 2.9% (28,852) Only 135 vocabulary items account for half the Brown Corpus. The Brown University Standard Corpus of Present-Day American English is a carefully compiled selection of current American English, totaling about a million words drawn from a wide variety of sources... for many years among the most-cited resources in the field. 4/25

5 Zipf s law Zipf s law: The i th most frequent term has frequency proportional to 1/i. cf i 1 i cf is collection frequency: the number of occurrences of the term in the collection. So if the most frequent term (the) occurs cf 1 times, then the second most frequent term (of) has half as many occurrences cf 2 = 1 2 cf and the third most frequent term (and) has a third as many occurrences cf 3 = 1 3 cf 1 etc. Equivalent: cf i = ci k and logcf i = logc+klogi (for k = 1) Example of a power law 5/25

6 Power Laws in log-log space y = cx k (k=1/2,1,2) log 10 y = k log 10 x +log 10 c /sqrt(x) 100/x 100/x** /sqrt(x) 100/x 100/x** /25

7 Zipf s law for Reuters log10 cf log10 rank Fit far from perfect, but nonetheless key insight: Few frequent terms, many rare terms. 7/25

8 more from s law A plot of word frequency in Wikipedia (27 Nov 2006). The plot is in log-log coordinates. x is rank of a word in the frequency table; y is the total number of the words occurrences. Most popular words are the, of and and, as expected. Zipf s law corresponds to the upper linear portion of the curve, roughly following the green (1/x) line. 8/25

9 Power laws more generally E.g., consider power law distributions of the form cr k, describing the number of book sales versus sales-rank r of a book, or the number of Wikipedia edits made by the r th most frequent contributor to Wikipedia. Amazon book sales: cr k, k.87 number of Wikipedia edits: cr k, k 1.7 (More on power laws and the long tail here: Networks, Crowds, and Markets: Reasoning About a Highly Connected World by David Easley and Jon Kleinberg Chpt 18: 9/25

10 Wikipedia edits/month Amazon sales/week / r^{1.7} / r^{.87} Normalization given by the roughly 1 sale/week for the 200,000th ranked Amazon title: 40916r.87 and by the 10 edits/month for the 1000th ranked Wikipedia editor: r User Book rank r 1e+07 1e+06 Wikipedia edits/month Amazon sales/week / r^{1.7} / r^{.87} Long tail: about a quarter of Amazon book sales estimated to come from the long tail, i.e., those outside the top 100,000 bestselling titles e+06 User Book rank r 10/25

11 Another Wikipedia count (15 May 2010) All articles in the English version of Wikipedia, 21GB in XML format (five hours to parse entire file, extract data from markup language, filter numbers, special characters, extract statistics): Total tokens (words, no numbers): T = 1,570,455,731 Unique tokens (words, no numbers): M = 5,800,280 11/25

12 Word frequency distribution follows Zipf s law 12/25

13 rank 1 50 (86M-3M), stop words (the, of, and, in, to, a, is,...) rank 51 3K (2.4M-56K), frequent words (university, January, tea, sharp,...) rank 3K 200K (56K-118), words from large comprehensive dictionaries (officiates, polytonality, neologism,...) above rank 50K mostly Long Tail words rank 200K 5.8M (117-1), terms from obscure niches, misspelled words, transliterated words from other languages, new words and non-words (euprosthenops, eurotrochilus, lokottaravada,...) 13/25

14 Some selected words and associated counts Google Twitter 894 domain domainer 22 Wikipedia Wiki Obama Oprah 3885 Moniker 4974 GoDaddy /25

15 Project Gutenberg (per billion) Over 36,000 items (Jun 2011), average of > 50 new e-books / week the of and to in I that was his he it with is for as had you not be her ,000 th 15/25

16 Bowtie structure of the web A.Broder,R.Kumar,F.Maghoul,P.Raghavan,S.Rajagopalan,S. Stata, A. Tomkins, and J. Wiener. Graph structure in the web. Computer Networks, 33: , Strongly connected component (SCC) in the center Lots of pages that get linked to, but don t link (OUT) Lots of pages that link to other pages, but don t get linked to (IN) Tendrils, tubes, islands # of in-links (in-degree) averages 8 15, not randomly distributed (Poissonian), instead a power law: # pages with in-degree i is 1/i α, α /25

17 Poisson Distribution Bernoulli process with N trials, each probability p of success: ( ) N p(m) = p m (1 p) N m. m Probability p(m) of m successes, in limit N very large and p small, parametrized by just µ = Np (µ = mean number of successes). N! For N m, we have (N m)! = N(N 1) (N m+1) Nm, so ( N) m N! m!, and m!(n m)! Nm p(m) 1 m! Nm( µ ) m ( 1 µ ) N m µ m ( lim 1 µ ) N = e µ µm N N m! N N m! (ignore (1 µ/n) m since by assumption N µm). N dependence drops out for N, with average µ fixed (p 0). The form p(m) = e µµm m! is known as a Poisson distribution (properly normalized: m=0 p(m) = e µ µ m m=0 m! = e µ e µ = 1). 17/25

18 Compare to power law p(m) 1/m /25 Poisson Distribution for µ = 10 p(m) = e 1010m m!

19 Power Law p(m) 1/m 2.1 and Poisson p(m) = e 1010m m! /25

20 Power Law p(m) 1/m 2.1 and Poisson p(m) = e 1010m 1 m! 0.1 (log log scale) e-05 1e-06 1e-07 1e-08 1e /25

21 Power law distributions Slide credit: Dragomir Radev 21/25

22 Examples Moby Dick scientific papers AOL users visiting sites 97 bestsellers AT&T customers on 1 day California /25

23 Moon Solar flares wars ( ) richest individuals 2003 US family names 1990 US cities /25

24 Power law in networks! For many interesting graphs, the distribution over node degree follows a power law film actors telephone call graph networks sexual contacts WWW internet peer-to-peer metabolic network protein interactions exponent! (in/out degree) / / Slide credit: Dragomir Radev 24/25

25 Next Time: More Statistical Methods Peter Norvig, How to Write a Spelling Corrector (See video: The Unreasonable Effectiveness of Data, given 23 Sep 2010.) Additional related references: A. Halevy, P. Norvig, F. Pereira, The Unreasonable Effectiveness of Data, Intelligent Systems Mar/Apr 2009 (copy at resources/unrealdata.pdf) P. Norvig, Natural Language Corpus Data 25/25

### Power-laws Scale free networks. Based on slides by Lada Adamic (UMichigan)

Power-laws Scale free networks Based on slides by Lada Adamic (UMichigan) Outline n Power law distributions n Fitting n what kinds of processes generate power laws? n Barabasi-Albert model for scale-free

### Information Retrieval

Information Retrieval ETH Zürich, Fall 2012 Thomas Hofmann LECTURE 4 INDEX COMPRESSION 10.10.2012 Information Retrieval, ETHZ 2012 1 Overview 1. Dictionary Compression 2. Zipf s Law 3. Posting List Compression

### 6.207/14.15: Networks Lecture 6: Growing Random Networks and Power Laws

6.207/14.15: Networks Lecture 6: Growing Random Networks and Power Laws Daron Acemoglu and Asu Ozdaglar MIT September 28, 2009 1 Outline Growing random networks Power-law degree distributions: Rich-Get-Richer

### Web Search Engines: Solutions

Web Search Engines: Solutions Problem 1: A. How can the owner of a web site design a spider trap? Answer: He can set up his web server so that, whenever a client requests a URL in a particular directory

### W. Heath Rushing Adsurgo LLC. Harness the Power of Text Analytics: Unstructured Data Analysis for Healthcare. Session H-1 JTCC: October 23, 2015

W. Heath Rushing Adsurgo LLC Harness the Power of Text Analytics: Unstructured Data Analysis for Healthcare Session H-1 JTCC: October 23, 2015 Outline Demonstration: Recent article on cnn.com Introduction

### Introduction to Bayesian Classification (A Practical Discussion) Todd Holloway Lecture for B551 Nov. 27, 2007

Introduction to Bayesian Classification (A Practical Discussion) Todd Holloway Lecture for B551 Nov. 27, 2007 Naïve Bayes Components ML vs. MAP Benefits Feature Preparation Filtering Decay Extended Examples

### Chapter 18. Power Laws and Rich-Get-Richer Phenomena. 18.1 Popularity as a Network Phenomenon

From the book Networks, Crowds, and Markets: Reasoning about a Highly Connected World. By David Easley and Jon Kleinberg. Cambridge University Press, 2010. Complete preprint on-line at http://www.cs.cornell.edu/home/kleinber/networks-book/

### A discussion of Statistical Mechanics of Complex Networks P. Part I

A discussion of Statistical Mechanics of Complex Networks Part I Review of Modern Physics, Vol. 74, 2002 Small Word Networks Clustering Coefficient Scale-Free Networks Erdös-Rényi model cover only parts

### Graph models for the Web and the Internet. Elias Koutsoupias University of Athens and UCLA. Crete, July 2003

Graph models for the Web and the Internet Elias Koutsoupias University of Athens and UCLA Crete, July 2003 Outline of the lecture Small world phenomenon The shape of the Web graph Searching and navigation

### Lecture 1b: Text, terms, and bags of words

Lecture 1b: Text, terms, and bags of words William Webber (william@williamwebber.com) COMP90042, 2014, Semester 1, Lecture 1b Corpus, document, term Body of text referred to as corpus Corpus regarded as

### Search Engines. Stephen Shaw 18th of February, 2014. Netsoc

Search Engines Stephen Shaw Netsoc 18th of February, 2014 Me M.Sc. Artificial Intelligence, University of Edinburgh Would recommend B.A. (Mod.) Computer Science, Linguistics, French,

### C Fall 2007 Electronic Commerce Tuesday-Thursday

C20.0038 Fall 2007 Electronic Commerce Tuesday-Thursday Professor Anindya Ghose aghose@stern.nyu.edu, (212) 998 0807, http://pages.stern.nyu.edu/~aghose/ Course dates: September 2007 through December 2007

### Creating Synthetic Temporal Document Collections for Web Archive Benchmarking

Creating Synthetic Temporal Document Collections for Web Archive Benchmarking Kjetil Nørvåg and Albert Overskeid Nybø Norwegian University of Science and Technology 7491 Trondheim, Norway Abstract. In

### The Shape of the Network. The Shape of the Internet. Why study topology? Internet topologies. Early work. More on topologies..

The Shape of the Internet Slides assembled by Jeff Chase Duke University (thanks to and ) The Shape of the Network Characterizing shape : AS-level topology: who connects to whom Router-level topology:

### 1 o Semestre 2007/2008

Departamento de Engenharia Informática Instituto Superior Técnico 1 o Semestre 2007/2008 Outline 1 2 3 4 5 Outline 1 2 3 4 5 Exploiting Text How is text exploited? Two main directions Extraction Extraction

### Random graphs and complex networks

Random graphs and complex networks Remco van der Hofstad Honours Class, spring 2008 Complex networks Figure 2 Ye a s t p ro te in in te ra c tio n n e tw o rk. A m a p o f p ro tein p ro tein in tera c

### CMU SCS Large Graph Mining Patterns, Tools and Cascade analysis

Large Graph Mining Patterns, Tools and Cascade analysis Christos Faloutsos CMU Roadmap Introduction Motivation Why big data Why (big) graphs? Patterns in graphs Tools: fraud detection on e-bay Conclusions

### Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations

Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations Jurij Leskovec, CMU Jon Kleinberg, Cornell Christos Faloutsos, CMU 1 Introduction What can we do with graphs? What patterns

Server Load Prediction Suthee Chaidaroon (unsuthee@stanford.edu) Joon Yeong Kim (kim64@stanford.edu) Jonghan Seo (jonghan@stanford.edu) Abstract Estimating server load average is one of the methods that

### Extracting Information from Social Networks

Extracting Information from Social Networks Aggregating site information to get trends 1 Not limited to social networks Examples Google search logs: flu outbreaks We Feel Fine Bullying 2 Bullying Xu, Jun,

### Examining the Six Degrees in Film: George Clooney as the New Kevin Bacon

GerardoPelayoGarcía,16gpg1,gpg1@williams.edu GaryChen,18gc5,gc5@williams.edu MichaelZuo,18mz4,mz4@williams.edu ExaminingtheSixDegreesinFilm: GeorgeClooneyastheNewKevinBacon Introduction SixDegreesofKevinBacon

### Graph Theory and Complex Networks: An Introduction. Chapter 08: Computer networks

Graph Theory and Complex Networks: An Introduction Maarten van Steen VU Amsterdam, Dept. Computer Science Room R4.20, steen@cs.vu.nl Chapter 08: Computer networks Version: March 3, 2011 2 / 53 Contents

### End of Life Content Report November 2014. Produced By The NHS Choices Reporting Team CH.NHSChoices-Reporting@nhs.net

End of Life Content Report November 2014 Produced By The NHS Choices Reporting Team CH.NHSChoices-Reporting@nhs.net End of Life Dashboard Page 1 Overall Choices Site Visits Tag cloud showing end of life

### Information Retrieval Statistics of Text

Information Retrieval Statistics of Text James Allan University of Massachusetts, Amherst (NTU ST770-A) Fall 2002 All slides copyright Bruce Croft and/or James Allan Outline Zipf distribution Vocabulary

### General Network Analysis: Graph-theoretic. COMP572 Fall 2009

General Network Analysis: Graph-theoretic Techniques COMP572 Fall 2009 Networks (aka Graphs) A network is a set of vertices, or nodes, and edges that connect pairs of vertices Example: a network with 5

### Web Usage in Client-Server Design

Web Search Web Usage in Client-Server Design A client (e.g., a browser) communicates with a server via http Hypertext transfer protocol: a lightweight and simple protocol asynchronously carrying a variety

### CSE373: Data Structures and Algorithms Lecture 3: Math Review; Algorithm Analysis. Linda Shapiro Winter 2015

CSE373: Data Structures and Algorithms Lecture 3: Math Review; Algorithm Analysis Linda Shapiro Today Registration should be done. Homework 1 due 11:59 pm next Wednesday, January 14 Review math essential

### IC05 Introduction on Networks &Visualization Nov. 2009.

IC05 Introduction on Networks &Visualization Nov. 2009 Overview 1. Networks Introduction Networks across disciplines Properties Models 2. Visualization InfoVis Data exploration

### So today we shall continue our discussion on the search engines and web crawlers. (Refer Slide Time: 01:02)

Internet Technology Prof. Indranil Sengupta Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture No #39 Search Engines and Web Crawler :: Part 2 So today we

### MATHEMATICS FOR ENGINEERS STATISTICS TUTORIAL 4 PROBABILITY DISTRIBUTIONS

MATHEMATICS FOR ENGINEERS STATISTICS TUTORIAL 4 PROBABILITY DISTRIBUTIONS CONTENTS Sample Space Accumulative Probability Probability Distributions Binomial Distribution Normal Distribution Poisson Distribution

### Network Analysis Basics and applications to online data

Network Analysis Basics and applications to online data Katherine Ognyanova University of Southern California Prepared for the Annenberg Program for Online Communities, 2010. Relational data Node (actor,

### Extension of Zipf s Law to Words and Phrases

Extension of Zipf s Law to Words and Phrases Abstract Le Quan Ha, E. I. Sicilia-Garcia, Ji Ming, F. J. Smith School Computer Science Queen s University of Belfast Belfast BT7 NN, Northern Ireland q.le@qub.ac.uk

### Scale-Free Extreme Programming

Scale-Free Extreme Programming Kent Beck, Three Rivers Institute A question that has vexed Extreme Programming from its inception is, How does it scale? Sometimes the scaling question is asked out of legitimate

### AP Physics 1 and 2 Lab Investigations

AP Physics 1 and 2 Lab Investigations Student Guide to Data Analysis New York, NY. College Board, Advanced Placement, Advanced Placement Program, AP, AP Central, and the acorn logo are registered trademarks

### Efficient Identification of Starters and Followers in Social Media

Efficient Identification of Starters and Followers in Social Media Michael Mathioudakis Department of Computer Science University of Toronto mathiou@cs.toronto.edu Nick Koudas Department of Computer Science

### 9 Retention. From Code to Product gidgreen.com/course

9 Retention From Code to Product gidgreen.com/course Lecture 9 Positioning Home pages Other web pages App store pages Ongoing retention From Code to Product Lecture 9 Retention Slide 2 gidgreen.com/course

### Protein Prospector and Ways of Calculating Expectation Values

Protein Prospector and Ways of Calculating Expectation Values 1/16 Aenoch J. Lynn; Robert J. Chalkley; Peter R. Baker; Mark R. Segal; and Alma L. Burlingame University of California, San Francisco, San

### WHERE DOES THE 10% CONDITION COME FROM?

1 WHERE DOES THE 10% CONDITION COME FROM? The text has mentioned The 10% Condition (at least) twice so far: p. 407 Bernoulli trials must be independent. If that assumption is violated, it is still okay

### Predictive Modeling Techniques in Insurance

Predictive Modeling Techniques in Insurance Tuesday May 5, 2015 JF. Breton Application Engineer 2014 The MathWorks, Inc. 1 Opening Presenter: JF. Breton: 13 years of experience in predictive analytics

### AT&T Global Network Client for Windows Product Support Matrix January 29, 2015

AT&T Global Network Client for Windows Product Support Matrix January 29, 2015 Product Support Matrix Following is the Product Support Matrix for the AT&T Global Network Client. See the AT&T Global Network

### Today s Class. Kinds of Networks. What s a Network? Normal Distribution. Properties of Networks. What are networks Properties of networks

Today s Class NBA 600 Networks and the Internet Class 3, Mon 10/22 Prof. Dan Huttenlocher What are networks Properties of networks Attributes of individuals in a network Very different from those of individuals

### CS345a: Data Mining Jure Leskovec and Anand Rajaraman Stanford University

CS345a: Data Mining Jure Leskovec and Anand Rajaraman Stanford University Friday 5:30 at Gates B12 5:30 7:30pm You will learn and get hands on experience on: Login to Amazon EC2 and request a cluster Run

Introduction to Hadoop 1 What is Hadoop? the big data revolution extracting value from data cloud computing 2 Understanding MapReduce the word count problem more examples MCS 572 Lecture 24 Introduction

### Search Engine Architecture I

Search Engine Architecture I Software Architecture The high level structure of a software system Software components The interfaces provided by those components The relationships between those components

### MATH4427 Notebook 2 Spring 2016. 2 MATH4427 Notebook 2 3. 2.1 Definitions and Examples... 3. 2.2 Performance Measures for Estimators...

MATH4427 Notebook 2 Spring 2016 prepared by Professor Jenny Baglivo c Copyright 2009-2016 by Jenny A. Baglivo. All Rights Reserved. Contents 2 MATH4427 Notebook 2 3 2.1 Definitions and Examples...................................

### Network Working Group

Network Working Group Request for Comments: 2413 Category: Informational S. Weibel OCLC Online Computer Library Center, Inc. J. Kunze University of California, San Francisco C. Lagoze Cornell University

### 1 Maximum likelihood estimation

COS 424: Interacting with Data Lecturer: David Blei Lecture #4 Scribes: Wei Ho, Michael Ye February 14, 2008 1 Maximum likelihood estimation 1.1 MLE of a Bernoulli random variable (coin flips) Given N

### Analyzing Big Data with AWS

Analyzing Big Data with AWS Peter Sirota, General Manager, Amazon Elastic MapReduce @petersirota What is Big Data? Computer generated data Application server logs (web sites, games) Sensor data (weather,

2. DATA AND EXERCISES (Geos2911 students please read page 8) 2.1 Data set The data set available to you is an Excel spreadsheet file called cyclones.xls. The file consists of 3 sheets. Only the third is

### A Corpus Linguistics-based Approach for Estimating Arabic Online Content

A Corpus Linguistics-based Approach for Estimating Arabic Online Content Anas Tawileh Systematics Consulting anas@systematics.ca Mansour Al Ghamedi King Abdulaziz City for Science and Technology mghamdi@kacst.edu.sa

### Appendix C: Graphs. Vern Lindberg

Vern Lindberg 1 Making Graphs A picture is worth a thousand words. Graphical presentation of data is a vital tool in the sciences and engineering. Good graphs convey a great deal of information and can

### The Internet Is Like A Jellyfish

The Internet Is Like A Jellyfish Michalis Faloutsos UC Riverside Joint work with: Leslie Tauro, Georgos Siganos (UCR) Chris Palmer(CMU) Big Picture: Modeling the Internet Topology Traffic Protocols Routing,

### BINOMIAL DISTRIBUTION

MODULE IV BINOMIAL DISTRIBUTION A random variable X is said to follow binomial distribution with parameters n & p if P ( X ) = nc x p x q n x where x = 0, 1,2,3..n, p is the probability of success & q

### INTRO TO BIG DATA. Djoerd Hiemstra. http://www.cs.utwente.nl/~hiemstra. Big Data in Clinical Medicinel, 30 June 2014

INTRO TO BIG DATA Big Data in Clinical Medicinel, 30 June 2014 Djoerd Hiemstra http://www.cs.utwente.nl/~hiemstra WHY BIG DATA? 2 Source: http://en.wikipedia.org/wiki/mount_everest 3 19 May 2012: 234 people

### Mobile Completes the Web

Mobile Completes the Web Presented by Daniel Appelquist Senior Technology Strategist, Vodafone Group Fundamentos Web, Contents 1. Customer Insight 2. Real Numbers 3. Market Trends and Disruptive Innovation

### Business Challenges and Research Directions of Management Analytics in the Big Data Era

Business Challenges and Research Directions of Management Analytics in the Big Data Era Abstract Big data analytics have been embraced as a disruptive technology that will reshape business intelligence,

### Economic Statistics (ECON2006), Statistics and Research Design in Psychology (PSYC2010), Survey Design and Analysis (SOCI2007)

COURSE DESCRIPTION Title Code Level Semester Credits 3 Prerequisites Post requisites Introduction to Statistics ECON1005 (EC160) I I None Economic Statistics (ECON2006), Statistics and Research Design

### The Big Data Paradigm Shift. Insight Through Automation

The Big Data Paradigm Shift Insight Through Automation Agenda The Problem Emcien s Solution: Algorithms solve data related business problems How Does the Technology Work? Case Studies 2013 Emcien, Inc.

### 77 Top SEO Ranking Factors

77 Top SEO Ranking Factors If you ve downloaded this resource, it suggests that you re likely looking to improve your website s search engine rankings and get more new customers for your business. Keep

### On the Value of a Social Network

On the Value of a Social Network Sandeep Chalasani Abstract Different laws have been proposed for the value of a social network. According to Metcalfe s law, the value of a network is proportional to n

### 2 Binomial, Poisson, Normal Distribution

2 Binomial, Poisson, Normal Distribution Binomial Distribution ): We are interested in the number of times an event A occurs in n independent trials. In each trial the event A has the same probability

### Chi-Square Test. J. Savoy Université de Neuchâtel

Chi-Square Test J. Savoy Université de Neuchâtel C. D. Manning & H. Schütze : Foundations of statistical natural language processing. The MIT Press. Cambridge (MA) 1 Discriminating Features How can we

### CPSC 340: Machine Learning and Data Mining. Mark Schmidt University of British Columbia Fall 2015

CPSC 340: Machine Learning and Data Mining Mark Schmidt University of British Columbia Fall 2015 Outline 1) Intro to Machine Learning and Data Mining: Big data phenomenon and types of data. Definitions

### Non-Parametric Tests (I)

Lecture 5: Non-Parametric Tests (I) KimHuat LIM lim@stats.ox.ac.uk http://www.stats.ox.ac.uk/~lim/teaching.html Slide 1 5.1 Outline (i) Overview of Distribution-Free Tests (ii) Median Test for Two Independent

### Visualising Space Time Dynamics: Plots and Clocks, Graphs and Maps

The International Symposium on Spatial Temporal Data Mining & Geo Computation at UCL, July 19 23, 2011 Visualising Space Time Dynamics: Plots and Clocks, Graphs and Maps Michael Batty Martin Austwick Ollie

### Facebook Ads: Local Advertisers. A Guide for. Marketing Research and Intelligence Series. From the Search Engine People. Search Engine People

Facebook Ads: A Guide for Local Advertisers From the Marketing Research and Intelligence Series Ajax, Ontario Canada L1Z 1E2 By: Helen M. Overland Date: October 27 th, 2009 press@searchenginepeople.com

### Graph Mining Techniques for Social Media Analysis

Graph Mining Techniques for Social Media Analysis Mary McGlohon Christos Faloutsos 1 1-1 What is graph mining? Extracting useful knowledge (patterns, outliers, etc.) from structured data that can be represented

### Tutorial, IEEE SERVICE 2014 Anchorage, Alaska

Tutorial, IEEE SERVICE 2014 Anchorage, Alaska Big Data Science: Fundamental, Techniques, and Challenges (Data Mining on Big Data) 2014. 6. 27. By Neil Y. Yen Presented by Incheon Paik University of Aizu

### HOSPIRA (HSP US) HISTORICAL COMMON STOCK PRICE INFORMATION

30-Apr-2004 28.35 29.00 28.20 28.46 28.55 03-May-2004 28.50 28.70 26.80 27.04 27.21 04-May-2004 26.90 26.99 26.00 26.00 26.38 05-May-2004 26.05 26.69 26.00 26.35 26.34 06-May-2004 26.31 26.35 26.05 26.26

### Language Entropy: A Metric for Characterization of Author Programming Language Distribution

Language Entropy: A Metric for Characterization of Author Programming Language Distribution Daniel P. Delorey Google, Inc. delorey@google.com Jonathan L. Krein, Alexander C. MacLean SEQuOIA Lab, Brigham

### First Foray into Text Analysis with R

Chapter 2 First Foray into Text Analysis with R Abstract In this chapter readers learn how to load, tokenize, and search a text. Several methods for exploring word frequencies and lexical makeup are introduced.

### VERIFYING HEAPS LAW USING GOOGLE BOOKS NGRAM DATA

VERIFYING HEAPS LAW USING GOOGLE BOOKS NGRAM DATA Vladimir V. Bochkarev, Eduard Yu.Lerner, Anna V. Shevlyakova Kazan Federal University, Kremlevskaya str.18, Kazan 420018, Russia E-mail: vladimir.bochkarev@kpfu.ru

### CS 345 Data Mining Lecture 1. Introduction to Web Mining

CS 345 Data Mining Lecture 1 Introduction to Web Mining What is Web Mining? Discovering useful information from the World-Wide Web and its usage patterns Applications Web search e.g., Google, Yahoo, Vertical

### 99.37, 99.38, 99.38, 99.39, 99.39, 99.39, 99.39, 99.40, 99.41, 99.42 cm

Error Analysis and the Gaussian Distribution In experimental science theory lives or dies based on the results of experimental evidence and thus the analysis of this evidence is a critical part of the

### Doing Multidisciplinary Research in Data Science

Doing Multidisciplinary Research in Data Science Assoc.Prof. Abzetdin ADAMOV CeDAWI - Center for Data Analytics and Web Insights Qafqaz University aadamov@qu.edu.az http://ce.qu.edu.az/~aadamov 16 May

### SOCIAL ENGAGEMENT BENCHMARK REPORT THE SALESFORCE MARKETING CLOUD. Metrics from 3+ Million Twitter* Messages Sent Through Our Platform

THE SALESFORCE MARKETING CLOUD SOCIAL ENGAGEMENT BENCHMARK REPORT Metrics from 3+ Million Twitter* Messages Sent Through Our Platform *All trademarks, service marks, and trade names are the property of

### Data Mining in Web Search Engine Optimization and User Assisted Rank Results

Data Mining in Web Search Engine Optimization and User Assisted Rank Results Minky Jindal Institute of Technology and Management Gurgaon 122017, Haryana, India Nisha kharb Institute of Technology and Management

### Six Degrees of Separation in Online Society

Six Degrees of Separation in Online Society Lei Zhang * Tsinghua-Southampton Joint Lab on Web Science Graduate School in Shenzhen, Tsinghua University Shenzhen, Guangdong Province, P.R.China zhanglei@sz.tsinghua.edu.cn

### Overlay and P2P Networks. On power law networks. Prof. Sasu Tarkoma

Overlay and P2P Networks On power law networks Prof. Sasu Tarkoma 28.1.2013 Contents Zipf s law and power law distributions Robustness of networks Scale free and small worlds Search in small worlds Freenet

### Measurement and Metrics Fundamentals. SE 350 Software Process & Product Quality

Measurement and Metrics Fundamentals Lecture Objectives Provide some basic concepts of metrics Quality attribute metrics and measurements Reliability, validity, error Correlation and causation Discuss

### Geographical Classification of Documents Using Evidence from Wikipedia

Geographical Classification of Documents Using Evidence from Wikipedia Rafael Odon de Alencar (odon.rafael@gmail.com) Clodoveu Augusto Davis Jr. (clodoveu@dcc.ufmg.br) Marcos André Gonçalves (mgoncalv@dcc.ufmg.br)

### Twitter Analytics: Architecture, Tools and Analysis

Twitter Analytics: Architecture, Tools and Analysis Rohan D.W Perera CERDEC Ft Monmouth, NJ 07703-5113 S. Anand, K. P. Subbalakshmi and R. Chandramouli Department of ECE, Stevens Institute Of Technology

### Introduction to Networks and Business Intelligence

Introduction to Networks and Business Intelligence Prof. Dr. Daning Hu Department of Informatics University of Zurich Sep 17th, 2015 Outline Network Science A Random History Network Analysis Network Topological

### Statistical Distributions in Astronomy

Data Mining In Modern Astronomy Sky Surveys: Statistical Distributions in Astronomy Ching-Wa Yip cwyip@pha.jhu.edu; Bloomberg 518 From Data to Information We don t just want data. We want information from

### Approximating PageRank from In-Degree

Approximating PageRank from In-Degree Santo Fortunato 1,2,Marián Boguñá 3, Alessandro Flammini 1, and Filippo Menczer 1 1 School of Informatics, Indiana University Bloomington, IN 47406, USA 2 Complex

### Applying Machine Learning to Stock Market Trading Bryce Taylor

Applying Machine Learning to Stock Market Trading Bryce Taylor Abstract: In an effort to emulate human investors who read publicly available materials in order to make decisions about their investments,

### Predicting the Stock Market with News Articles

Predicting the Stock Market with News Articles Kari Lee and Ryan Timmons CS224N Final Project Introduction Stock market prediction is an area of extreme importance to an entire industry. Stock price is

### Chapter 3 RANDOM VARIATE GENERATION

Chapter 3 RANDOM VARIATE GENERATION In order to do a Monte Carlo simulation either by hand or by computer, techniques must be developed for generating values of random variables having known distributions.

### Visualization and Modeling of Structural Features of a Large Organizational Email Network

Visualization and Modeling of Structural Features of a Large Organizational Email Network Benjamin H. Sims Statistical Sciences (CCS-6) Email: bsims@lanl.gov Nikolai Sinitsyn Physics of Condensed Matter

### Basic indexing pipeline

Information Retrieval Document Parsing Basic indexing pipeline Documents to be indexed. Friends, Romans, countrymen. Tokenizer Token stream. Friends Romans Countrymen Linguistic modules Modified tokens.

### Applying Big Data approaches to Competitive Intelligence challenges

Applying Big Data approaches to Competitive Intelligence challenges THOMSON REUTERS IP & SCIENCE PHARMA CI EUROPE CONFERENCE & EXHIBITION TIM MILLER 19 FEBRUARY 2014 BIG DATA, NOT JUST ABOUT VOLUMES Patient

### Search and Information Retrieval

Search and Information Retrieval Search on the Web 1 is a daily activity for many people throughout the world Search and communication are most popular uses of the computer Applications involving search

Logistics Big Data Algorithmic Introduction Prof. Yuval Shavitt Contact: shavitt@eng.tau.ac.il Final grade: 4 6 home assignments (will try to include programing assignments as well): 2% Exam 8% Big Data

### Statistics I for QBIC. Contents and Objectives. Chapters 1 7. Revised: August 2013

Statistics I for QBIC Text Book: Biostatistics, 10 th edition, by Daniel & Cross Contents and Objectives Chapters 1 7 Revised: August 2013 Chapter 1: Nature of Statistics (sections 1.1-1.6) Objectives

### Big Data Scoring. April 2014

Big Data Scoring April 2014 There was 5 exabytes of information created between the dawn of civilization through 2003 that much information is now created every 2 days - Eric Schmidt, Google CEO 2 Why

### Principles of Hypothesis Testing for Public Health

Principles of Hypothesis Testing for Public Health Laura Lee Johnson, Ph.D. Statistician National Center for Complementary and Alternative Medicine johnslau@mail.nih.gov Fall 2011 Answers to Questions

### Probit Analysis By: Kim Vincent

Probit Analysis By: Kim Vincent Quick Overview Probit analysis is a type of regression used to analyze binomial response variables. It transforms the sigmoid dose-response curve to a straight line that

### Evaluating the impact of research online with Google Analytics

Public Engagement with Research Online Evaluating the impact of research online with Google Analytics The web provides extensive opportunities for raising awareness and discussion of research findings