DATA ANALYSIS I. Data: Probabilistic View

Transcription

1 DATA ANALYSIS I Data: Probabilistic View

2 Sources Leskovec, J., Rajaraman, A., Ullman, J. D. (2014). Mining of massive datasets. Cambridge University Press. [5-7] Zaki, M. J., Meira Jr, W. (2014). Data Mining and Analysis: Fundamental Concepts and Algorithms. Cambridge University Press. [13-25]

3 Assumption: The Data is Random Suppose you have a certain amount of data, and you look for events of a certain type within that data. You can expect events of this type to occur, even if the data is completely random, and the number of occurrences of these events will grow as the size of the data grows. This essentially means that an algorithm or method you think is useful for finding the events returns more false positives.

4 Example Suppose there are believed to be some evil-doers out there, and we want to detect them. We have reason to believe that periodically, evil-doers gather at a hotel to plot their evil. There are one billion people who might be evil-doers. Everyone goes to a hotel one day in 100. A hotel holds 100 people. There are 100,000 hotels. We shall examine hotel records for 1000 days. There is 250,000 pairs of people who look like evil-doers, even though they are not.

5 Bonferroni s Principle Calculate the expected number of occurrences of the events you are looking for, on the assumption that data is random. If this number is significantly larger than the number of real instances you hope to find, then you must expect almost anything you find to be bogus, i.e., a statistical artifact rather than evidence of what you are looking for. Bonferroni s principle says that we may only detect the expected events by looking for events that are so rare that they are unlikely to occur in random data.

6 Random Variable The probabilistic view of the data assumes that each numeric attribute X is a random variable, defined as a function that assigns a real number to each outcome of an experiment. Formally, X is a function X: O R, where O, the domain of X, is the set of all possible outcomes of the experiment, and R, the range of X, is the set of real numbers. If the outcomes are numeric, and represent the observed values of the random variable, then X: O O is simply the identity function: X(v) = v for all v O.

7 Discrete x Continuous Discrete random variable: A variable can take on only a finite or countably infinite number of values in its range. Continuous random variable: A variable can take on any value in its range.

8 Iris Dataset: Sepal Length

9 Random Variable: Sepal Length All n = 150 values of this attribute lie in the range [4.3,7.9], with centimeters as the unit of measurement. Let us assume that these constitute the set of all possible outcomes O. We can consider the attribute X1 to be a continuous random variable, given as the identity function X1(v) = v.

10 Probability Mass Function Probability mass function: If X is discrete, the probability mass function of X is defined as The function f gives the probability P(X = x) that the random variable X has the exact value x.

11 Bernoulli distribution Short and long sepal lengths? We can define a discrete random variable A as follows: A(v) = 0 if v < 7 1 if v 7 Probabilities: f (1) = P(A = 1) = 13 / 150 = = p f (0) = P(A = 0) = 137 / 150 = = 1 p

12 Binomial Distribution Bernoulli trial is a random experiment with exactly two possible outcomes, "success" and "failure", in which the probability of success is the same every time the experiment is conducted. Let us consider another discrete random variable B, denoting the number of Irises with long sepal length in m independent Bernoulli trials with probability of success p.

13 Example The probability of observing exactly k = 2 Irises with long sepal length in m = 10 trials is given as

14 Probability Density Function Probability density function: If X is continuous, the probability density function, which specifies the probability that the variable X takes on values in any interval [a,b] R is defined as Probability mass is spread so thinly over the range of values that it can be measured only over intervals [a,b] R.

15 Probability Density Function The density function f must satisfy the basic laws of probability:

16 Sepal Length: Distribution?

17 Normal Distribution A random variable X has a normal distribution, with the parameters mean μ and variance σ 2, if the probability density function of X is given as follows:

18 Sepal Length [μ = 5.84, σ 2 = 0.681]

19 Cumulative Distribution Function Cumulative distribution function: For any random variable X, whether discrete or continuous, we can define the cumulative distribution function F: R [0,1], which gives the probability of observing a value at most some given value x.

20 Discrete x Continuous When X is discrete When X is continuous

21 Cumulative Distribution Function [Normal Distribution]

22 Bivariate Random Variables Instead of considering each attribute as a random variable, we can also perform pair-wise analysis by considering a pair of attributes, X 1 and X 2, as a bivariate random variable. Joint probability mass function (discrete random variables)

23 Bivariate Distributions Consider the sepal length and sepal width attributes in the Iris dataset. Let A denote the random variable corresponding to long sepal length (at least 7 cm) and the random variable B corresponding to long sepal width (at least 3.5 cm). Let X = (A B) be a discrete bivariate random variable.

24 Probability Mass Function X1 (long sepal length), X2 (long sepal width)

25 Density Function Joint probability density function (continuous random variables)

26 Bivariate Normal Density

27 Multivariate Random Variables A d-dimensional multivariate random variable (vector random variable) X = (X1,X2,...,Xd ) T is defined as a function that assigns a vector of real numbers to each outcome in the sample space. Joint probability mass function (discrete random variables) Joint probability density function (continuous random variables)