# INFO 2950 Intro to Data Science. Lecture 17: Power Laws and Big Data

Size: px
Start display at page:

Download "INFO 2950 Intro to Data Science. Lecture 17: Power Laws and Big Data"

## Transcription

1 INFO 2950 Intro to Data Science Lecture 17: Power Laws and Big Data Paul Ginsparg Cornell University, Ithaca, NY 29 Oct /25

2 Power Laws in log-log space y = cx k (k=1/2,1,2) log 10 y = k log 10 x +log 10 c sqrt(x) x x**2 100 sqrt(x) x x** /25

3 Zipf s law Now we have characterized the growth of the vocabulary in collections. We also want to know how many frequent vs. infrequent terms we should expect in a collection. In natural language, there are a few very frequent terms and very many very rare terms. Zipf s law (linguist/philologist George Zipf, 1935): The i th most frequent term has frequency proportional to 1/i. cf i 1 i cf i is collection frequency: the number of occurrences of the term t i in the collection. 3/25

4 s law Zipf s law: the frequency of any word is inversely proportional to its rank in the frequency table. Thus the most frequent word will occur approximately twice as often as the second most frequent word, which occurs twice as often as the fourth most frequent word, etc. Brown Corpus: the : 7% of all word occurrences (69,971 of >1M). of : 3.5% of words (36,411) and : 2.9% (28,852) Only 135 vocabulary items account for half the Brown Corpus. The Brown University Standard Corpus of Present-Day American English is a carefully compiled selection of current American English, totaling about a million words drawn from a wide variety of sources... for many years among the most-cited resources in the field. 4/25

5 Zipf s law Zipf s law: The i th most frequent term has frequency proportional to 1/i. cf i 1 i cf is collection frequency: the number of occurrences of the term in the collection. So if the most frequent term (the) occurs cf 1 times, then the second most frequent term (of) has half as many occurrences cf 2 = 1 2 cf and the third most frequent term (and) has a third as many occurrences cf 3 = 1 3 cf 1 etc. Equivalent: cf i = ci k and logcf i = logc+klogi (for k = 1) Example of a power law 5/25

6 Power Laws in log-log space y = cx k (k=1/2,1,2) log 10 y = k log 10 x +log 10 c /sqrt(x) 100/x 100/x** /sqrt(x) 100/x 100/x** /25

7 Zipf s law for Reuters log10 cf log10 rank Fit far from perfect, but nonetheless key insight: Few frequent terms, many rare terms. 7/25

8 more from s law A plot of word frequency in Wikipedia (27 Nov 2006). The plot is in log-log coordinates. x is rank of a word in the frequency table; y is the total number of the words occurrences. Most popular words are the, of and and, as expected. Zipf s law corresponds to the upper linear portion of the curve, roughly following the green (1/x) line. 8/25

9 Power laws more generally E.g., consider power law distributions of the form cr k, describing the number of book sales versus sales-rank r of a book, or the number of Wikipedia edits made by the r th most frequent contributor to Wikipedia. Amazon book sales: cr k, k.87 number of Wikipedia edits: cr k, k 1.7 (More on power laws and the long tail here: Networks, Crowds, and Markets: Reasoning About a Highly Connected World by David Easley and Jon Kleinberg Chpt 18: 9/25

10 Wikipedia edits/month Amazon sales/week / r^{1.7} / r^{.87} Normalization given by the roughly 1 sale/week for the 200,000th ranked Amazon title: 40916r.87 and by the 10 edits/month for the 1000th ranked Wikipedia editor: r User Book rank r 1e+07 1e+06 Wikipedia edits/month Amazon sales/week / r^{1.7} / r^{.87} Long tail: about a quarter of Amazon book sales estimated to come from the long tail, i.e., those outside the top 100,000 bestselling titles e+06 User Book rank r 10/25

11 Another Wikipedia count (15 May 2010) All articles in the English version of Wikipedia, 21GB in XML format (five hours to parse entire file, extract data from markup language, filter numbers, special characters, extract statistics): Total tokens (words, no numbers): T = 1,570,455,731 Unique tokens (words, no numbers): M = 5,800,280 11/25

12 Word frequency distribution follows Zipf s law 12/25

13 rank 1 50 (86M-3M), stop words (the, of, and, in, to, a, is,...) rank 51 3K (2.4M-56K), frequent words (university, January, tea, sharp,...) rank 3K 200K (56K-118), words from large comprehensive dictionaries (officiates, polytonality, neologism,...) above rank 50K mostly Long Tail words rank 200K 5.8M (117-1), terms from obscure niches, misspelled words, transliterated words from other languages, new words and non-words (euprosthenops, eurotrochilus, lokottaravada,...) 13/25

14 Some selected words and associated counts Google Twitter 894 domain domainer 22 Wikipedia Wiki Obama Oprah 3885 Moniker 4974 GoDaddy /25

15 Project Gutenberg (per billion) Over 36,000 items (Jun 2011), average of > 50 new e-books / week the of and to in I that was his he it with is for as had you not be her ,000 th 15/25

16 Bowtie structure of the web A.Broder,R.Kumar,F.Maghoul,P.Raghavan,S.Rajagopalan,S. Stata, A. Tomkins, and J. Wiener. Graph structure in the web. Computer Networks, 33: , Strongly connected component (SCC) in the center Lots of pages that get linked to, but don t link (OUT) Lots of pages that link to other pages, but don t get linked to (IN) Tendrils, tubes, islands # of in-links (in-degree) averages 8 15, not randomly distributed (Poissonian), instead a power law: # pages with in-degree i is 1/i α, α /25

17 Poisson Distribution Bernoulli process with N trials, each probability p of success: ( ) N p(m) = p m (1 p) N m. m Probability p(m) of m successes, in limit N very large and p small, parametrized by just µ = Np (µ = mean number of successes). N! For N m, we have (N m)! = N(N 1) (N m+1) Nm, so ( N) m N! m!, and m!(n m)! Nm p(m) 1 m! Nm( µ ) m ( 1 µ ) N m µ m ( lim 1 µ ) N = e µ µm N N m! N N m! (ignore (1 µ/n) m since by assumption N µm). N dependence drops out for N, with average µ fixed (p 0). The form p(m) = e µµm m! is known as a Poisson distribution (properly normalized: m=0 p(m) = e µ µ m m=0 m! = e µ e µ = 1). 17/25

18 Compare to power law p(m) 1/m /25 Poisson Distribution for µ = 10 p(m) = e 1010m m!

19 Power Law p(m) 1/m 2.1 and Poisson p(m) = e 1010m m! /25

20 Power Law p(m) 1/m 2.1 and Poisson p(m) = e 1010m 1 m! 0.1 (log log scale) e-05 1e-06 1e-07 1e-08 1e /25

21 Power law distributions Slide credit: Dragomir Radev 21/25

22 Examples Moby Dick scientific papers AOL users visiting sites 97 bestsellers AT&T customers on 1 day California /25

23 Moon Solar flares wars ( ) richest individuals 2003 US family names 1990 US cities /25

24 Power law in networks! For many interesting graphs, the distribution over node degree follows a power law film actors telephone call graph networks sexual contacts WWW internet peer-to-peer metabolic network protein interactions exponent! (in/out degree) / / Slide credit: Dragomir Radev 24/25

25 Next Time: More Statistical Methods Peter Norvig, How to Write a Spelling Corrector (See video: The Unreasonable Effectiveness of Data, given 23 Sep 2010.) Additional related references: A. Halevy, P. Norvig, F. Pereira, The Unreasonable Effectiveness of Data, Intelligent Systems Mar/Apr 2009 (copy at resources/unrealdata.pdf) P. Norvig, Natural Language Corpus Data 25/25

### Web Search Engines: Solutions

Web Search Engines: Solutions Problem 1: A. How can the owner of a web site design a spider trap? Answer: He can set up his web server so that, whenever a client requests a URL in a particular directory

### W. Heath Rushing Adsurgo LLC. Harness the Power of Text Analytics: Unstructured Data Analysis for Healthcare. Session H-1 JTCC: October 23, 2015

W. Heath Rushing Adsurgo LLC Harness the Power of Text Analytics: Unstructured Data Analysis for Healthcare Session H-1 JTCC: October 23, 2015 Outline Demonstration: Recent article on cnn.com Introduction

### Introduction to Bayesian Classification (A Practical Discussion) Todd Holloway Lecture for B551 Nov. 27, 2007

Introduction to Bayesian Classification (A Practical Discussion) Todd Holloway Lecture for B551 Nov. 27, 2007 Naïve Bayes Components ML vs. MAP Benefits Feature Preparation Filtering Decay Extended Examples

### Chapter 18. Power Laws and Rich-Get-Richer Phenomena. 18.1 Popularity as a Network Phenomenon

From the book Networks, Crowds, and Markets: Reasoning about a Highly Connected World. By David Easley and Jon Kleinberg. Cambridge University Press, 2010. Complete preprint on-line at http://www.cs.cornell.edu/home/kleinber/networks-book/

### A discussion of Statistical Mechanics of Complex Networks P. Part I

A discussion of Statistical Mechanics of Complex Networks Part I Review of Modern Physics, Vol. 74, 2002 Small Word Networks Clustering Coefficient Scale-Free Networks Erdös-Rényi model cover only parts

### Graph models for the Web and the Internet. Elias Koutsoupias University of Athens and UCLA. Crete, July 2003

Graph models for the Web and the Internet Elias Koutsoupias University of Athens and UCLA Crete, July 2003 Outline of the lecture Small world phenomenon The shape of the Web graph Searching and navigation

### Search Engines. Stephen Shaw 18th of February, 2014. Netsoc

Search Engines Stephen Shaw Netsoc 18th of February, 2014 Me M.Sc. Artificial Intelligence, University of Edinburgh Would recommend B.A. (Mod.) Computer Science, Linguistics, French,

### Creating Synthetic Temporal Document Collections for Web Archive Benchmarking

Creating Synthetic Temporal Document Collections for Web Archive Benchmarking Kjetil Nørvåg and Albert Overskeid Nybø Norwegian University of Science and Technology 7491 Trondheim, Norway Abstract. In

### C Fall 2007 Electronic Commerce Tuesday-Thursday

C20.0038 Fall 2007 Electronic Commerce Tuesday-Thursday Professor Anindya Ghose aghose@stern.nyu.edu, (212) 998 0807, http://pages.stern.nyu.edu/~aghose/ Course dates: September 2007 through December 2007

### The Shape of the Network. The Shape of the Internet. Why study topology? Internet topologies. Early work. More on topologies..

The Shape of the Internet Slides assembled by Jeff Chase Duke University (thanks to and ) The Shape of the Network Characterizing shape : AS-level topology: who connects to whom Router-level topology:

### 1 o Semestre 2007/2008

Departamento de Engenharia Informática Instituto Superior Técnico 1 o Semestre 2007/2008 Outline 1 2 3 4 5 Outline 1 2 3 4 5 Exploiting Text How is text exploited? Two main directions Extraction Extraction

### General Network Analysis: Graph-theoretic. COMP572 Fall 2009

General Network Analysis: Graph-theoretic Techniques COMP572 Fall 2009 Networks (aka Graphs) A network is a set of vertices, or nodes, and edges that connect pairs of vertices Example: a network with 5

### Random graphs and complex networks

Random graphs and complex networks Remco van der Hofstad Honours Class, spring 2008 Complex networks Figure 2 Ye a s t p ro te in in te ra c tio n n e tw o rk. A m a p o f p ro tein p ro tein in tera c

### CMU SCS Large Graph Mining Patterns, Tools and Cascade analysis

Large Graph Mining Patterns, Tools and Cascade analysis Christos Faloutsos CMU Roadmap Introduction Motivation Why big data Why (big) graphs? Patterns in graphs Tools: fraud detection on e-bay Conclusions

### Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations

Graphs over Time Densification Laws, Shrinking Diameters and Possible Explanations Jurij Leskovec, CMU Jon Kleinberg, Cornell Christos Faloutsos, CMU 1 Introduction What can we do with graphs? What patterns

### Extracting Information from Social Networks

Extracting Information from Social Networks Aggregating site information to get trends 1 Not limited to social networks Examples Google search logs: flu outbreaks We Feel Fine Bullying 2 Bullying Xu, Jun,

### Examining the Six Degrees in Film: George Clooney as the New Kevin Bacon

GerardoPelayoGarcía,16gpg1,gpg1@williams.edu GaryChen,18gc5,gc5@williams.edu MichaelZuo,18mz4,mz4@williams.edu ExaminingtheSixDegreesinFilm: GeorgeClooneyastheNewKevinBacon Introduction SixDegreesofKevinBacon

Server Load Prediction Suthee Chaidaroon (unsuthee@stanford.edu) Joon Yeong Kim (kim64@stanford.edu) Jonghan Seo (jonghan@stanford.edu) Abstract Estimating server load average is one of the methods that

### Network Analysis Basics and applications to online data

Network Analysis Basics and applications to online data Katherine Ognyanova University of Southern California Prepared for the Annenberg Program for Online Communities, 2010. Relational data Node (actor,

### End of Life Content Report November 2014. Produced By The NHS Choices Reporting Team CH.NHSChoices-Reporting@nhs.net

End of Life Content Report November 2014 Produced By The NHS Choices Reporting Team CH.NHSChoices-Reporting@nhs.net End of Life Dashboard Page 1 Overall Choices Site Visits Tag cloud showing end of life

### Information Retrieval Statistics of Text

Information Retrieval Statistics of Text James Allan University of Massachusetts, Amherst (NTU ST770-A) Fall 2002 All slides copyright Bruce Croft and/or James Allan Outline Zipf distribution Vocabulary

### CSE373: Data Structures and Algorithms Lecture 3: Math Review; Algorithm Analysis. Linda Shapiro Winter 2015

CSE373: Data Structures and Algorithms Lecture 3: Math Review; Algorithm Analysis Linda Shapiro Today Registration should be done. Homework 1 due 11:59 pm next Wednesday, January 14 Review math essential

### So today we shall continue our discussion on the search engines and web crawlers. (Refer Slide Time: 01:02)

Internet Technology Prof. Indranil Sengupta Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture No #39 Search Engines and Web Crawler :: Part 2 So today we

### IC05 Introduction on Networks &Visualization Nov. 2009.

IC05 Introduction on Networks &Visualization Nov. 2009 Overview 1. Networks Introduction Networks across disciplines Properties Models 2. Visualization InfoVis Data exploration

### Graph Theory and Complex Networks: An Introduction. Chapter 08: Computer networks

Graph Theory and Complex Networks: An Introduction Maarten van Steen VU Amsterdam, Dept. Computer Science Room R4.20, steen@cs.vu.nl Chapter 08: Computer networks Version: March 3, 2011 2 / 53 Contents

### AT&T Global Network Client for Windows Product Support Matrix January 29, 2015

AT&T Global Network Client for Windows Product Support Matrix January 29, 2015 Product Support Matrix Following is the Product Support Matrix for the AT&T Global Network Client. See the AT&T Global Network

### Scale-Free Extreme Programming

Scale-Free Extreme Programming Kent Beck, Three Rivers Institute A question that has vexed Extreme Programming from its inception is, How does it scale? Sometimes the scaling question is asked out of legitimate

### Predictive Modeling Techniques in Insurance

Predictive Modeling Techniques in Insurance Tuesday May 5, 2015 JF. Breton Application Engineer 2014 The MathWorks, Inc. 1 Opening Presenter: JF. Breton: 13 years of experience in predictive analytics

### AP Physics 1 and 2 Lab Investigations

AP Physics 1 and 2 Lab Investigations Student Guide to Data Analysis New York, NY. College Board, Advanced Placement, Advanced Placement Program, AP, AP Central, and the acorn logo are registered trademarks

### 9 Retention. From Code to Product gidgreen.com/course

9 Retention From Code to Product gidgreen.com/course Lecture 9 Positioning Home pages Other web pages App store pages Ongoing retention From Code to Product Lecture 9 Retention Slide 2 gidgreen.com/course

### WHERE DOES THE 10% CONDITION COME FROM?

1 WHERE DOES THE 10% CONDITION COME FROM? The text has mentioned The 10% Condition (at least) twice so far: p. 407 Bernoulli trials must be independent. If that assumption is violated, it is still okay

Introduction to Hadoop 1 What is Hadoop? the big data revolution extracting value from data cloud computing 2 Understanding MapReduce the word count problem more examples MCS 572 Lecture 24 Introduction

### Protein Prospector and Ways of Calculating Expectation Values

Protein Prospector and Ways of Calculating Expectation Values 1/16 Aenoch J. Lynn; Robert J. Chalkley; Peter R. Baker; Mark R. Segal; and Alma L. Burlingame University of California, San Francisco, San

### MATH4427 Notebook 2 Spring 2016. 2 MATH4427 Notebook 2 3. 2.1 Definitions and Examples... 3. 2.2 Performance Measures for Estimators...

MATH4427 Notebook 2 Spring 2016 prepared by Professor Jenny Baglivo c Copyright 2009-2016 by Jenny A. Baglivo. All Rights Reserved. Contents 2 MATH4427 Notebook 2 3 2.1 Definitions and Examples...................................

### Analyzing Big Data with AWS

Analyzing Big Data with AWS Peter Sirota, General Manager, Amazon Elastic MapReduce @petersirota What is Big Data? Computer generated data Application server logs (web sites, games) Sensor data (weather,

### 1 Maximum likelihood estimation

COS 424: Interacting with Data Lecturer: David Blei Lecture #4 Scribes: Wei Ho, Michael Ye February 14, 2008 1 Maximum likelihood estimation 1.1 MLE of a Bernoulli random variable (coin flips) Given N

### Network Working Group

Network Working Group Request for Comments: 2413 Category: Informational S. Weibel OCLC Online Computer Library Center, Inc. J. Kunze University of California, San Francisco C. Lagoze Cornell University

### A Corpus Linguistics-based Approach for Estimating Arabic Online Content

A Corpus Linguistics-based Approach for Estimating Arabic Online Content Anas Tawileh Systematics Consulting anas@systematics.ca Mansour Al Ghamedi King Abdulaziz City for Science and Technology mghamdi@kacst.edu.sa

### Efficient Identification of Starters and Followers in Social Media

Efficient Identification of Starters and Followers in Social Media Michael Mathioudakis Department of Computer Science University of Toronto mathiou@cs.toronto.edu Nick Koudas Department of Computer Science

### Business Challenges and Research Directions of Management Analytics in the Big Data Era

Business Challenges and Research Directions of Management Analytics in the Big Data Era Abstract Big data analytics have been embraced as a disruptive technology that will reshape business intelligence,

### The Internet Is Like A Jellyfish

The Internet Is Like A Jellyfish Michalis Faloutsos UC Riverside Joint work with: Leslie Tauro, Georgos Siganos (UCR) Chris Palmer(CMU) Big Picture: Modeling the Internet Topology Traffic Protocols Routing,

### INTRO TO BIG DATA. Djoerd Hiemstra. http://www.cs.utwente.nl/~hiemstra. Big Data in Clinical Medicinel, 30 June 2014

INTRO TO BIG DATA Big Data in Clinical Medicinel, 30 June 2014 Djoerd Hiemstra http://www.cs.utwente.nl/~hiemstra WHY BIG DATA? 2 Source: http://en.wikipedia.org/wiki/mount_everest 3 19 May 2012: 234 people

2. DATA AND EXERCISES (Geos2911 students please read page 8) 2.1 Data set The data set available to you is an Excel spreadsheet file called cyclones.xls. The file consists of 3 sheets. Only the third is

### Mobile Completes the Web

Mobile Completes the Web Presented by Daniel Appelquist Senior Technology Strategist, Vodafone Group Fundamentos Web, Contents 1. Customer Insight 2. Real Numbers 3. Market Trends and Disruptive Innovation

### Economic Statistics (ECON2006), Statistics and Research Design in Psychology (PSYC2010), Survey Design and Analysis (SOCI2007)

COURSE DESCRIPTION Title Code Level Semester Credits 3 Prerequisites Post requisites Introduction to Statistics ECON1005 (EC160) I I None Economic Statistics (ECON2006), Statistics and Research Design

### Today s Class. Kinds of Networks. What s a Network? Normal Distribution. Properties of Networks. What are networks Properties of networks

Today s Class NBA 600 Networks and the Internet Class 3, Mon 10/22 Prof. Dan Huttenlocher What are networks Properties of networks Attributes of individuals in a network Very different from those of individuals

### 77 Top SEO Ranking Factors

77 Top SEO Ranking Factors If you ve downloaded this resource, it suggests that you re likely looking to improve your website s search engine rankings and get more new customers for your business. Keep

### The Big Data Paradigm Shift. Insight Through Automation

The Big Data Paradigm Shift Insight Through Automation Agenda The Problem Emcien s Solution: Algorithms solve data related business problems How Does the Technology Work? Case Studies 2013 Emcien, Inc.

### CPSC 340: Machine Learning and Data Mining. Mark Schmidt University of British Columbia Fall 2015

CPSC 340: Machine Learning and Data Mining Mark Schmidt University of British Columbia Fall 2015 Outline 1) Intro to Machine Learning and Data Mining: Big data phenomenon and types of data. Definitions

### Facebook Ads: Local Advertisers. A Guide for. Marketing Research and Intelligence Series. From the Search Engine People. Search Engine People

Facebook Ads: A Guide for Local Advertisers From the Marketing Research and Intelligence Series Ajax, Ontario Canada L1Z 1E2 By: Helen M. Overland Date: October 27 th, 2009 press@searchenginepeople.com

### The Impact of YouTube Recommendation System on Video Views

The Impact of YouTube Recommendation System on Video Views Renjie Zhou, Samamon Khemmarat, Lixin Gao College of Computer Science and Technology Department of Electrical and Computer Engineering Harbin

### Visualising Space Time Dynamics: Plots and Clocks, Graphs and Maps

The International Symposium on Spatial Temporal Data Mining & Geo Computation at UCL, July 19 23, 2011 Visualising Space Time Dynamics: Plots and Clocks, Graphs and Maps Michael Batty Martin Austwick Ollie

### Graph Mining Techniques for Social Media Analysis

Graph Mining Techniques for Social Media Analysis Mary McGlohon Christos Faloutsos 1 1-1 What is graph mining? Extracting useful knowledge (patterns, outliers, etc.) from structured data that can be represented

### Tutorial, IEEE SERVICE 2014 Anchorage, Alaska

Tutorial, IEEE SERVICE 2014 Anchorage, Alaska Big Data Science: Fundamental, Techniques, and Challenges (Data Mining on Big Data) 2014. 6. 27. By Neil Y. Yen Presented by Incheon Paik University of Aizu

### Non-Parametric Tests (I)

Lecture 5: Non-Parametric Tests (I) KimHuat LIM lim@stats.ox.ac.uk http://www.stats.ox.ac.uk/~lim/teaching.html Slide 1 5.1 Outline (i) Overview of Distribution-Free Tests (ii) Median Test for Two Independent

### First Foray into Text Analysis with R

Chapter 2 First Foray into Text Analysis with R Abstract In this chapter readers learn how to load, tokenize, and search a text. Several methods for exploring word frequencies and lexical makeup are introduced.

### 2 Binomial, Poisson, Normal Distribution

2 Binomial, Poisson, Normal Distribution Binomial Distribution ): We are interested in the number of times an event A occurs in n independent trials. In each trial the event A has the same probability

### Six Degrees of Separation in Online Society

Six Degrees of Separation in Online Society Lei Zhang * Tsinghua-Southampton Joint Lab on Web Science Graduate School in Shenzhen, Tsinghua University Shenzhen, Guangdong Province, P.R.China zhanglei@sz.tsinghua.edu.cn

### Measurement and Metrics Fundamentals. SE 350 Software Process & Product Quality

Measurement and Metrics Fundamentals Lecture Objectives Provide some basic concepts of metrics Quality attribute metrics and measurements Reliability, validity, error Correlation and causation Discuss

### Data Mining in Web Search Engine Optimization and User Assisted Rank Results

Data Mining in Web Search Engine Optimization and User Assisted Rank Results Minky Jindal Institute of Technology and Management Gurgaon 122017, Haryana, India Nisha kharb Institute of Technology and Management

### Twitter Analytics: Architecture, Tools and Analysis

Twitter Analytics: Architecture, Tools and Analysis Rohan D.W Perera CERDEC Ft Monmouth, NJ 07703-5113 S. Anand, K. P. Subbalakshmi and R. Chandramouli Department of ECE, Stevens Institute Of Technology

### Predicting the Stock Market with News Articles

Predicting the Stock Market with News Articles Kari Lee and Ryan Timmons CS224N Final Project Introduction Stock market prediction is an area of extreme importance to an entire industry. Stock price is

### Approximating PageRank from In-Degree

Approximating PageRank from In-Degree Santo Fortunato 1,2,Marián Boguñá 3, Alessandro Flammini 1, and Filippo Menczer 1 1 School of Informatics, Indiana University Bloomington, IN 47406, USA 2 Complex

### Statistical Distributions in Astronomy

Data Mining In Modern Astronomy Sky Surveys: Statistical Distributions in Astronomy Ching-Wa Yip cwyip@pha.jhu.edu; Bloomberg 518 From Data to Information We don t just want data. We want information from

### Doing Multidisciplinary Research in Data Science

Doing Multidisciplinary Research in Data Science Assoc.Prof. Abzetdin ADAMOV CeDAWI - Center for Data Analytics and Web Insights Qafqaz University aadamov@qu.edu.az http://ce.qu.edu.az/~aadamov 16 May

### Applying Machine Learning to Stock Market Trading Bryce Taylor

Applying Machine Learning to Stock Market Trading Bryce Taylor Abstract: In an effort to emulate human investors who read publicly available materials in order to make decisions about their investments,

### CS 345 Data Mining Lecture 1. Introduction to Web Mining

CS 345 Data Mining Lecture 1 Introduction to Web Mining What is Web Mining? Discovering useful information from the World-Wide Web and its usage patterns Applications Web search e.g., Google, Yahoo, Vertical

### 99.37, 99.38, 99.38, 99.39, 99.39, 99.39, 99.39, 99.40, 99.41, 99.42 cm

Error Analysis and the Gaussian Distribution In experimental science theory lives or dies based on the results of experimental evidence and thus the analysis of this evidence is a critical part of the

### Introduction to Networks and Business Intelligence

Introduction to Networks and Business Intelligence Prof. Dr. Daning Hu Department of Informatics University of Zurich Sep 17th, 2015 Outline Network Science A Random History Network Analysis Network Topological

### SOCIAL ENGAGEMENT BENCHMARK REPORT THE SALESFORCE MARKETING CLOUD. Metrics from 3+ Million Twitter* Messages Sent Through Our Platform

THE SALESFORCE MARKETING CLOUD SOCIAL ENGAGEMENT BENCHMARK REPORT Metrics from 3+ Million Twitter* Messages Sent Through Our Platform *All trademarks, service marks, and trade names are the property of

Logistics Big Data Algorithmic Introduction Prof. Yuval Shavitt Contact: shavitt@eng.tau.ac.il Final grade: 4 6 home assignments (will try to include programing assignments as well): 2% Exam 8% Big Data

### Search and Information Retrieval

Search and Information Retrieval Search on the Web 1 is a daily activity for many people throughout the world Search and communication are most popular uses of the computer Applications involving search

### Basic indexing pipeline

Information Retrieval Document Parsing Basic indexing pipeline Documents to be indexed. Friends, Romans, countrymen. Tokenizer Token stream. Friends Romans Countrymen Linguistic modules Modified tokens.

### Applying Big Data approaches to Competitive Intelligence challenges

Applying Big Data approaches to Competitive Intelligence challenges THOMSON REUTERS IP & SCIENCE PHARMA CI EUROPE CONFERENCE & EXHIBITION TIM MILLER 19 FEBRUARY 2014 BIG DATA, NOT JUST ABOUT VOLUMES Patient

### Big Data Scoring. April 2014

Big Data Scoring April 2014 There was 5 exabytes of information created between the dawn of civilization through 2003 that much information is now created every 2 days - Eric Schmidt, Google CEO 2 Why

### Chapter 3 RANDOM VARIATE GENERATION

Chapter 3 RANDOM VARIATE GENERATION In order to do a Monte Carlo simulation either by hand or by computer, techniques must be developed for generating values of random variables having known distributions.

### Professor Yashar Ganjali Department of Computer Science University of Toronto. yganjali@cs.toronto.edu http://www.cs.toronto.

Professor Yashar Ganjali Department of Computer Science University of Toronto yganjali@cs.toronto.edu http://www.cs.toronto.edu/~yganjali Announcements Final project proposal Guidelines have been posted

### Gutenberg 3.2 Ebook-Piracy Report

Gutenberg 3.2 Ebook-Piracy Report What does piracy cost the publishers? Ersatzraten (replacement rates) Manuel Bonik Dr. Andreas Schaale Pic. from ref. [1] Berlin, September 2012 Prologue The report Gutenberg

### Probit Analysis By: Kim Vincent

Probit Analysis By: Kim Vincent Quick Overview Probit analysis is a type of regression used to analyze binomial response variables. It transforms the sigmoid dose-response curve to a straight line that

### Capturing Meaningful Competitive Intelligence from the Social Media Movement

Capturing Meaningful Competitive Intelligence from the Social Media Movement Social media has evolved from a creative marketing medium and networking resource to a goldmine for robust competitive intelligence

### Chapter 29 Scale-Free Network Topologies with Clustering Similar to Online Social Networks

Chapter 29 Scale-Free Network Topologies with Clustering Similar to Online Social Networks Imre Varga Abstract In this paper I propose a novel method to model real online social networks where the growing

### Evaluating the impact of research online with Google Analytics

Public Engagement with Research Online Evaluating the impact of research online with Google Analytics The web provides extensive opportunities for raising awareness and discussion of research findings

### Introduction to Predictive Analytics. Dr. Ronen Meiri ronen@dmway.com

Introduction to Predictive Analytics Dr. Ronen Meiri Outline From big data to predictive analytics Predictive Analytics vs. BI Intelligent platforms What can we do with it. The modeling process. Example

### media kit 2014 PUBLISH / DEVELOP Global Mobile Ad Network

media kit 2014 PUBLISH / DEVELOP Global Mobile Ad Network WHY MOBILE PUBLISHING Proliferation of smartphone devices and tablets is shifting the way that customers use Internet, making advertising a key

### Big Data Analytics Building Blocks. Simple Data Storage (SQLite)

http://poloclub.gatech.edu/cse6242 CSE6242 / CX4242: Data & Visual Analytics Big Data Analytics Building Blocks. Simple Data Storage (SQLite) Duen Horng (Polo) Chau Georgia Tech Partly based on materials

### Social Media Statistics. What is going on out there

Social Media Statistics What is going on out there What is Social Media? It s a great big machine allowing people to talk to each other, about everything and nothing, all the time, where ever the are or..

### Statistics I for QBIC. Contents and Objectives. Chapters 1 7. Revised: August 2013

Statistics I for QBIC Text Book: Biostatistics, 10 th edition, by Daniel & Cross Contents and Objectives Chapters 1 7 Revised: August 2013 Chapter 1: Nature of Statistics (sections 1.1-1.6) Objectives

### BeeSocial. Create A Buzz About Your Business. Social Media Marketing. Bee Social Marketing is part of Genacom, Inc. www.genacom.

BeeSocial M A R K E T I N G Create A Buzz About Your Business Social Media Marketing Bee Social Marketing is part of Genacom, Inc. www.genacom.com What is Social Media Marketing? Social Media Marketing

### BINOMIAL DISTRIBUTION

MODULE IV BINOMIAL DISTRIBUTION A random variable X is said to follow binomial distribution with parameters n & p if P ( X ) = nc x p x q n x where x = 0, 1,2,3..n, p is the probability of success & q

### BIG DATA IN SCIENCE & EDUCATION

BIG DATA IN SCIENCE & EDUCATION SURFsara Data & Computing Infrastructure Event, 12 March 2014 Djoerd Hiemstra http://www.cs.utwente.nl/~hiemstra WHY BIG DATA? 2 Source: Jimmy Lin & http://en.wikipedia.org/wiki/mount_everest

### Using R for Social Media Analytics

Using R for Social Media Analytics Presentation to Tools for Teaching and Learning of Social Media Analytics Blue Sky workshop, 2015 International Communication Association conference (San Juan, Puerto

### Concentration of Trading in S&P 500 Stocks. Recent studies and news reports have noted an increase in the average correlation

Concentration of Trading in S&P 500 Stocks Recent studies and news reports have noted an increase in the average correlation between stock returns in the U.S. markets over the last decade, especially for

### MINFS544: Business Network Data Analytics and Applications

MINFS544: Business Network Data Analytics and Applications March 30 th, 2015 Daning Hu, Ph.D., Department of Informatics University of Zurich F Schweitzer et al. Science 2009 Stop Contagious Failures in

### GENERAL OVERVIEW OF VARIOUS SSO SYSTEMS: ACTIVE DIRECTORY, GOOGLE & FACEBOOK

Antti Pyykkö, Mikko Malinen, Oskari Miettinen GENERAL OVERVIEW OF VARIOUS SSO SYSTEMS: ACTIVE DIRECTORY, GOOGLE & FACEBOOK TJTSE54 Assignment 29.4.2008 Jyväskylä University Department of Computer Science

### Should HR Care About Big Data? E D COHEN, L E A R NING I NDUSTRY CONSULTA NT

Should HR Care About Big Data? E D COHEN, L E A R NING I NDUSTRY CONSULTA NT Tweeting? Please mention: #BigData (800) 263-6317 or (805) 690-5753 For More Info / To Register / To Access Archive: Search

### Big Data Analytics. Prof. Dr. Lars Schmidt-Thieme

Big Data Analytics Prof. Dr. Lars Schmidt-Thieme Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany 33. Sitzung des Arbeitskreises Informationstechnologie,

### Analysis of Internet Topologies: A Historical View

Analysis of Internet Topologies: A Historical View Mohamadreza Najiminaini, Laxmi Subedi, and Ljiljana Trajković Communication Networks Laboratory http://www.ensc.sfu.ca/cnl Simon Fraser University Vancouver,

### Bret Phillips. Web Consultant. bretwp@webdevils.biz. @bretwp. http://webdevils.biz http://bretphillips.com. Web Devils offers the following services:

Bret Phillips Web Consultant bretwp@webdevils.biz @bretwp http://webdevils.biz http://bretphillips.com Web Devils offers the following services: Web Development Social Media Management Search Engine Optimization