# The Analysis of Data. Volume 1. Probability. Guy Lebanon

Save this PDF as:

Size: px
Start display at page:

Download "The Analysis of Data. Volume 1. Probability. Guy Lebanon"

## Transcription

1 The Analysis of Data Volume 1 Probability Guy Lebanon First Edition 2012

2 Probability. The Analysis of Data, Volume 1. First Edition, First Printing, Copyright 2013 by Guy Lebanon. All rights reserved. This book or any portion thereof may not be reproduced or used in any manner whatsoever without the express written permission of the author.

3 In memory of Alex Lebanon

4

5 Contents Contents 5 Preface 9 Mathematical Notations 14 1 Basic Definitions Sample Space and Events The Probability Function The Classical Probability Model on Finite Spaces The Classical Model on Continuous Spaces Conditional Probability and Independence Basic Combinatorics for Probability Probability and Measure Theory* Notes Exercises Random Variables Basic Definitions Functions of a Random Variable Discrete g(x) Continuous g(x) Expectation and Variance Moments and the Moment Generating Function Random Variables and Measurable Functions* Notes Exercises Important Random Variables The Bernoulli Trial Random Variable The Binomial Random Variable The Geometric Random Variable The Hypergeometric Random Variable The Negative Binomial Random Variable

6 6 CONTENTS 3.6 The Poisson Random Variable The Uniform Random Variable The Exponential Random Variable The Gaussian Random Variable The Gamma and χ 2 Distributions The t Random Variable The Beta Random Variable Mixture Random Variable The Empirical Random Variable The Smoothed Empirical Random Variable Notes Exercises Random Vectors Basic Definitions Joint Pmf, Pdf, and Cdf Functions Marginal Random Vectors Functions of a Random Vector Conditional Probabilities and Random Vectors Moments Conditional Expectations Moment Generating Function Random Vectors and Independent σ-algebras* Notes Exercises Important Random Vectors The Multinomial Random Vector The Multivariate Normal Random Vector The Dirichlet Random Vector Mixture Random Vectors The Exponential Family Random Vector Notes Exercises Random Processes Basic Definitions Random Processes and Marginal Distributions Moments One Dimensional Random Walk Random Processes and Measure Theory* The Borel-Cantelli Lemmas and the Zero-One Law* Notes Exercises

7 CONTENTS 7 7 Important Random Processes Markov Chains Basic Definitions Examples Transience, Persistence, and Irreducibility Persistence and Transience of the Random Walk Process Periodicity in Markov Chains The Stationary Distribution Poisson Processes Postulates and Di erential Equation Relationship to Poisson Distribution Relationship to Exponential Distribution Relationship to Binomial Distribution Relationship to Uniform Distribution Gaussian Processes The Wiener Process Notes Exercises Limit Theorems Modes of Stochastic Convergence Relationships Between Modes of Convergences Dominated Convergence Theorem for Vectors* Sche e s Theorem The Portmanteau Theorem The Law of Large Numbers The Characteristic Function* Levy s Continuity Theorem The Central Limit Theorem Continuous Mapping Theorem Slustky s Theorems Notes Exercises A Set Theory 223 A.1 Basic Definitions A.2 Functions A.3 Cardinality A.4 Limits of Sets A.5 Notes A.6 Exercises

8 8 CONTENTS B Metric Spaces 237 B.1 Basic Definitions B.2 Limits B.3 Continuity B.4 The Euclidean Space B.5 Growth of Functions B.6 Notes B.7 Exercises C Linear Algebra 257 C.1 Basic Definitions C.2 Rank C.3 Eigenvalues, Determinant, and Trace C.4 Positive Semi-Definite Matrices C.5 Singular Value Decomposition C.6 Notes C.7 Exercises D Di erentiation 279 D.1 Univariate Di erentiation D.2 Taylor Expansion and Power Series D.3 Notes D.4 Exercises E Measure Theory* 289 E.1 σ-algebras* E.2 The Measure Function* E.3 Caratheodory s Extension Theorem* E.3.1 Dynkin s Theorem* E.3.2 Outer Measure* E.3.3 The Extension Theorem* E.4 Independent σ-algebras* E.5 Important Measure Functions* E.5.1 Discrete Measure Functions* E.5.2 The Lebesgue Measure* E.5.3 The Lebesgue-Stieltjes Measure* E.6 Measurability of Functions* E.7 Notes F Integration 307 F.1 Riemann Integral F.2 Integration and Di erentiation F.3 The Lebesgue Integral* F.3.1 Relation between the Riemann and the Lebesgue Integrals* 325 F.3.2 Transformed Measures* F.4 Product Measures*

9 CONTENTS 9 F.5 Integration over Product Spaces* F.5.1 The Lebesgue Measure over R d * F.6 Multivariate Di erentiation and Integration F.7 Notes Bibliography 337 Index 341

10 10 CONTENTS

11 Preface The Analysis of Data Project The Analysis of Data (TAOD) project provides educational material in the area of data analysis. The project features comprehensive coverage of all relevant disciplines including probability, statistics, computing, and machine learning. The content is almost self-contained and includes mathematical prerequisites and basic computing concepts. The R programming language is used to demonstrate the contents. Full code is available, facilitating reproducibility of experiments and letting readers experiment with variations of the code. The presentation is mathematically rigorous, and includes derivations and proofs in most cases. HTML versions are freely available on the website Hardcopies are available at a ordable prices. Volume 1: Probability This volume focuses on probability theory. There are many excellent textbooks on probability, and yet this book di ers from others in several ways. Probability theory is a wide field. This book focuses on the parts of probability that are most relevant for statistics and machine learning. The book contains almost all of the mathematical prerequisites, including set theory, metric spaces, linear algebra, di erentiation, integration, and measure theory. Almost all results in the book appear with a proof. 11

12 12 CONTENTS Probability textbooks are typically either elementary or advanced. This book strikes a balance by attempting to avoid measure theory where possible, but resorting to measure theory and other advanced material in a few places where they are essential. The book uses R to illustrate concepts. Full code is available in the book, facilitating reproducibility of experiments and letting readers experiment with variations of the code. I am not aware of a single textbook that covers the material from probability theory that is necessary and sufficient for an in-depth understanding of statistics and machine learning. This book represents my best e ort in that direction. Since this book is part of a series of books on data analysis, it does not include any statistics or machine learning. Such content is postponed to future volumes. Website A companion website ( contains an HTML version of this book, errata, and additional multimedia material. The website will also link to additional TAOD volumes as they become available. Mathematical Appendices A large part of the book contains six appendices on mathematical prerequisites. Probability requires knowledge of many branches of mathematics, including calculus, linear algebra, set theory, metric spaces, measure, and Lebesgue integration. Instead of referring the reader to a large collection of math textbooks we include here all of the necessary prerequisites. References are provided in the notes sections at the end of each chapter for additional resources. Dependencies The diagram below indicates the dependencies between the di erent chapters of the book. Appendix chapters are shaded and dependencies between appendix chapters and regular chapters are marked by dashed arrows. It is not essential to strictly adhere to this dependency as many chapters require only a brief familiarity with some issues in the chapters that they depend on.

13 CONTENTS 13 A B C 1 D E 2 F Starred sections correspond to material that requires measure theory, or that can be better appreciated with knowledge of measure theory. R Code The book contains many fragments of R code, aimed to illustrate probability theory and its applications. The code is included so that the reader can reproduce the results as well as modify the code and run variations of it. In order to appreciate the code and modify it, the reader will need a basic understanding of the R programming language and R graphics. A good introduction to R is available from the CRAN website at Alternatively, two chapters from volume 2 of the analysis of data series (R Programming and R Graphics) are freely available online at The book uses a variety of R graphics packages, including base, lattice, and ggplot2. The most frequently used package is ggplot2, which is described in detail in [49] (see also and the R Graphics chapter available at To ensure that the code fragments run locally as they appear in the book, install and bring into scope the required packages using the install.packages and library R functions. For example, to install and load the ggplot2 package type the following commands in the R prompt.

14 14 CONTENTS # this is a comment install.packages("ggplot2") # installs the package ggplot2 library("ggplot2") # brings ggplot2 into scope The code fragments throughout the book are annotated with output displayed by the R interpreter. This output is displayed following two hash symbols which are interpreted as comments by R (see below). a = pi print(a) # note the ## symbols preceding the output below ## [1] print(a + 1) ## [1] This format makes it easy to copy a code fragment from an HTML page and paste it directly into R (the output following the hash symbols will be interpreted as comments and not produce a syntax error). Acknowledgements The following people made technical suggestions that helped improve the contents: Krishnakumar Balasubramanian, Rohit Banga, Joshua Dillon, Sanjeet Hajarnis, Oded Green, Yi Mao, Seungyeon Kim, Joonseok Lee, Fuxin Li, Nishant Mehta, Yaron Rachlin, Parikshit Ram, Kaushik Rangadurai, Neil Slagle, Mingxuan Sun, Brian Steber, Gena Tang, Long Tran. In addition, many useful comments were received through a discussion board during my fall 2011 class computational data analysis at Georgia Tech. These comments were mostly anonymized, but some commentators who identified themselves appear above. Katharina Probst, Neil Slagle and Laura Usselman edited portions of this book, and made many useful suggestions. The book features a combination of text, equations, graphs, and R code, made possible by the knitr package. I thank Yihui Xie for implementing knitr, and for his help through the knitr Google discussion group. Katharina Probst helped with web development and design.

15 Mathematical Notations Logic 8 for all 9 there exists ) implies, if and only if def = defined as For example, 8a >0, 1/a > 0 reads for every a>0wehave1/a > 0 and 8a >0, 9b >0, a/b =1 ) b/a = 1 reads for all a>0 there exists b>0 such that, whenever a/b = 1, we also have b/a = 1. Sets and Functions (Chapter A) sample space! an element of the sample space I A indicator function I A (x) =1ifx 2 A and 0 otherwise δ ij Kronercker s delta: δ ij = 1 if i = j and 0 otherwise N natural numbers {1, 2, 3,...} Z integers {..., 2, 1, 0, 1, 2,...} Q rational numbers A B Cartesian product of two sets A k repeated Cartesian product (k times) of a set: A A A c complement of a set: \ A 2 A power set of the set A A number of elements in a finite set f 1 (A) pre-image of the function f: {x : f(x) 2 A} f g f(x) =g(x) for all x x (n) sequence of elements in a set A n % A convergence of a sequence of increasing sets A n & A convergence of a sequence of decreasing sets Combinatorics (Section 1.6) n! factorial function n(n 1) 2 1 (n) r n!/(n r)! n n!/(r!(n r)!) r 15

16 16 CONTENTS Metric spaces (Chapter B) R real numbers R d set of d-dimensional vectors of real numbers B a (x) open ball of radius a centered at x x vector in R d the i-component of the vector x sequence of vectors in R d the i-component of the vector x (j) hx, yi inner product P i x iy i kxk Euclidean norm p Pi x2 i d(x, y) Euclidean distance pp i (x i y i ) 2 x (n)! x convergence of a sequence f n! f pointwise convergence of a sequence of functions f n % f pointwise convergence of an increasing sequence of functions f n & f pointwise convergence of a decreasing sequence of functions O(f) big O growth notation o(f) little o growth notation x i x (n) x (j) i Note in particular that we denote vectors in bold face, for example x, and the scalar components of such vectors using subscripts (non-bold face) x = (x 1,...,x d ). We refer to sequence of vectors using superscripts, for example x (n) and their scalar components as x (n) =(x (n) 1,...,x(n) d ). Probability (Chapters 1, 2, 4, 8) P probability function P E probability conditioned on the event E F X cumulative distribution function (cdf) corresponding to the RV X f X probability density function (pdf) corresponding to the RV X p X probability mass function (pmf) corresponding to the RV X E expectation Var variance Cov covariance std standard deviation distributed according to iid independent identically distributed (iid) sampling P k simplex of probability functions over = {1,...,k} i.o. infinitely often as! convergence with probability 1 p! convergence in probability convergence in distribution

17 CONTENTS 17 A > A n A n ij A 1 I tr A det A row A col A dim A rank A diag(v) A B Matrices (Chapter C) matrix transpose the matrix A raised to the n-power the i, j element of the matrix A n inverse of matrix A identity matrix trace of the matrix A determinant of the matrix A row space of the matrix A column space of the matrix A dimension of a linear space rank of the matrix A diagonal matrix whose diagonal is given by the vector v Kronecker product of two matrices A, B All vectors are assumed to be column vectors unless stated explicitly otherwise. For example, if x is an arbitrary vector and A a matrix the expression x > Ax represents a scalar. df /dx d 2 f/dx 2 i rf 2 j r 2 f Di erentiation (Chapter D) derivative second order derivative partial derivative gradient vector of partial derivatives of f : R k! R Jacobian matrix of partial derivatives of f : R k! R m second order partial derivative Hessian matrix of second order partial derivatives of f : R k! R We consider the gradient vector rf as a row vector. This ensures that the notations rf and rf are consistent. Measure and Integration (Chapters E, F) σ(c) σ algebra generated by the set of sets C µ R measure R dµ Lebesgue integral with respect to measure µ R d P Lebesgue integral with respect to probability measure P dfx Lebesgue integral with respect to probability measure R corresponding to cdf F X dx Lebesgue integral with respect to Lebesgue measure or Riemann integral B(R d ) Borel σ-algebra over metric space R d F 1 F 1 product σ-algebra µ 1 µ 2 product measure a.e. almost everywhere (except on a set of measure zero)

18 18 CONTENTS

### Master s Theory Exam Spring 2006

Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem

### MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix.

MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix. Inverse matrix Definition. Let A be an n n matrix. The inverse of A is an n n matrix, denoted

### Mathematical Background

Appendix A Mathematical Background A.1 Joint, Marginal and Conditional Probability Let the n (discrete or continuous) random variables y 1,..., y n have a joint joint probability probability p(y 1,...,

### Alabama Department of Postsecondary Education

Date Adopted 1998 Dates reviewed 2007, 2011, 2013 Dates revised 2004, 2008, 2011, 2013, 2015 Alabama Department of Postsecondary Education Representing Alabama s Public Two-Year College System Jefferson

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### PCHS ALGEBRA PLACEMENT TEST

MATHEMATICS Students must pass all math courses with a C or better to advance to the next math level. Only classes passed with a C or better will count towards meeting college entrance requirements. If

### 1 Sets and Set Notation.

LINEAR ALGEBRA MATH 27.6 SPRING 23 (COHEN) LECTURE NOTES Sets and Set Notation. Definition (Naive Definition of a Set). A set is any collection of objects, called the elements of that set. We will most

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### 13 MATH FACTS 101. 2 a = 1. 7. The elements of a vector have a graphical interpretation, which is particularly easy to see in two or three dimensions.

3 MATH FACTS 0 3 MATH FACTS 3. Vectors 3.. Definition We use the overhead arrow to denote a column vector, i.e., a linear segment with a direction. For example, in three-space, we write a vector in terms

### Mathematics (MAT) MAT 061 Basic Euclidean Geometry 3 Hours. MAT 051 Pre-Algebra 4 Hours

MAT 051 Pre-Algebra Mathematics (MAT) MAT 051 is designed as a review of the basic operations of arithmetic and an introduction to algebra. The student must earn a grade of C or in order to enroll in MAT

### MATH 240 Fall, Chapter 1: Linear Equations and Matrices

MATH 240 Fall, 2007 Chapter Summaries for Kolman / Hill, Elementary Linear Algebra, 9th Ed. written by Prof. J. Beachy Sections 1.1 1.5, 2.1 2.3, 4.2 4.9, 3.1 3.5, 5.3 5.5, 6.1 6.3, 6.5, 7.1 7.3 DEFINITIONS

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components

Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### Notes for STA 437/1005 Methods for Multivariate Data

Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.

### 1 Introduction to Matrices

1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

### Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of

### LINEAR ALGEBRA W W L CHEN

LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008 This chapter is available free to all individuals, on understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,

### F Matrix Calculus F 1

F Matrix Calculus F 1 Appendix F: MATRIX CALCULUS TABLE OF CONTENTS Page F1 Introduction F 3 F2 The Derivatives of Vector Functions F 3 F21 Derivative of Vector with Respect to Vector F 3 F22 Derivative

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### L10: Probability, statistics, and estimation theory

L10: Probability, statistics, and estimation theory Review of probability theory Bayes theorem Statistics and the Normal distribution Least Squares Error estimation Maximum Likelihood estimation Bayesian

### Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

### Machine Learning Math Essentials

Machine Learning Math Essentials Jeff Howbert Introduction to Machine Learning Winter 2012 1 Areas of math essential to machine learning Machine learning is part of both statistics and computer science

### The Exponential Family

The Exponential Family David M. Blei Columbia University November 3, 2015 Definition A probability density in the exponential family has this form where p.x j / D h.x/ expf > t.x/ a./g; (1) is the natural

### SF2940: Probability theory Lecture 8: Multivariate Normal Distribution

SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2015 Timo Koski Matematisk statistik 24.09.2015 1 / 1 Learning outcomes Random vectors, mean vector, covariance matrix,

### Random Vectors and the Variance Covariance Matrix

Random Vectors and the Variance Covariance Matrix Definition 1. A random vector X is a vector (X 1, X 2,..., X p ) of jointly distributed random variables. As is customary in linear algebra, we will write

### Inner product. Definition of inner product

Math 20F Linear Algebra Lecture 25 1 Inner product Review: Definition of inner product. Slide 1 Norm and distance. Orthogonal vectors. Orthogonal complement. Orthogonal basis. Definition of inner product

### MATHEMATICS (CLASSES XI XII)

MATHEMATICS (CLASSES XI XII) General Guidelines (i) All concepts/identities must be illustrated by situational examples. (ii) The language of word problems must be clear, simple and unambiguous. (iii)

### Math 5311 Gateaux differentials and Frechet derivatives

Math 5311 Gateaux differentials and Frechet derivatives Kevin Long January 26, 2009 1 Differentiation in vector spaces Thus far, we ve developed the theory of minimization without reference to derivatives.

### MATHEMATICAL METHODS OF STATISTICS

MATHEMATICAL METHODS OF STATISTICS By HARALD CRAMER TROFESSOK IN THE UNIVERSITY OF STOCKHOLM Princeton PRINCETON UNIVERSITY PRESS 1946 TABLE OF CONTENTS. First Part. MATHEMATICAL INTRODUCTION. CHAPTERS

### Chapter 2: Systems of Linear Equations and Matrices:

At the end of the lesson, you should be able to: Chapter 2: Systems of Linear Equations and Matrices: 2.1: Solutions of Linear Systems by the Echelon Method Define linear systems, unique solution, inconsistent,

### For a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )

Probability Review 15.075 Cynthia Rudin A probability space, defined by Kolmogorov (1903-1987) consists of: A set of outcomes S, e.g., for the roll of a die, S = {1, 2, 3, 4, 5, 6}, 1 1 2 1 6 for the roll

### CHAPTER 5. Product Measures

54 CHAPTER 5 Product Measures Given two measure spaces, we may construct a natural measure on their Cartesian product; the prototype is the construction of Lebesgue measure on R 2 as the product of Lebesgue

### Cofactor Expansion: Cramer s Rule

Cofactor Expansion: Cramer s Rule MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 2015 Introduction Today we will focus on developing: an efficient method for calculating

### Determinants. Dr. Doreen De Leon Math 152, Fall 2015

Determinants Dr. Doreen De Leon Math 52, Fall 205 Determinant of a Matrix Elementary Matrices We will first discuss matrices that can be used to produce an elementary row operation on a given matrix A.

### MATHEMATICS (MATH) 3. Provides experiences that enable graduates to find employment in sciencerelated

194 / Department of Natural Sciences and Mathematics MATHEMATICS (MATH) The Mathematics Program: 1. Provides challenging experiences in Mathematics, Physics, and Physical Science, which prepare graduates

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### UNIT 2 MATRICES - I 2.0 INTRODUCTION. Structure

UNIT 2 MATRICES - I Matrices - I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress

### CITY UNIVERSITY LONDON. BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION

No: CITY UNIVERSITY LONDON BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION ENGINEERING MATHEMATICS 2 (resit) EX2005 Date: August

### Introduction to Matrix Algebra

Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary

### The Delta Method and Applications

Chapter 5 The Delta Method and Applications 5.1 Linear approximations of functions In the simplest form of the central limit theorem, Theorem 4.18, we consider a sequence X 1, X,... of independent and

### A Tutorial on Probability Theory

Paola Sebastiani Department of Mathematics and Statistics University of Massachusetts at Amherst Corresponding Author: Paola Sebastiani. Department of Mathematics and Statistics, University of Massachusetts,

### State of Stress at Point

State of Stress at Point Einstein Notation The basic idea of Einstein notation is that a covector and a vector can form a scalar: This is typically written as an explicit sum: According to this convention,

### Mathematics of Cryptography

CHAPTER 2 Mathematics of Cryptography Part I: Modular Arithmetic, Congruence, and Matrices Objectives This chapter is intended to prepare the reader for the next few chapters in cryptography. The chapter

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct

### Undergraduate Notes in Mathematics. Arkansas Tech University Department of Mathematics

Undergraduate Notes in Mathematics Arkansas Tech University Department of Mathematics An Introductory Single Variable Real Analysis: A Learning Approach through Problem Solving Marcel B. Finan c All Rights

### Vector Spaces; the Space R n

Vector Spaces; the Space R n Vector Spaces A vector space (over the real numbers) is a set V of mathematical entities, called vectors, U, V, W, etc, in which an addition operation + is defined and in which

### 4. MATRICES Matrices

4. MATRICES 170 4. Matrices 4.1. Definitions. Definition 4.1.1. A matrix is a rectangular array of numbers. A matrix with m rows and n columns is said to have dimension m n and may be represented as follows:

### Lecture Notes 1. Brief Review of Basic Probability

Probability Review Lecture Notes Brief Review of Basic Probability I assume you know basic probability. Chapters -3 are a review. I will assume you have read and understood Chapters -3. Here is a very

### Probability and Statistics

CHAPTER 2: RANDOM VARIABLES AND ASSOCIATED FUNCTIONS 2b - 0 Probability and Statistics Kristel Van Steen, PhD 2 Montefiore Institute - Systems and Modeling GIGA - Bioinformatics ULg kristel.vansteen@ulg.ac.be

### Introduction to General and Generalized Linear Models

Introduction to General and Generalized Linear Models General Linear Models - part I Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK-2800 Kgs. Lyngby

### 1 Introduction. 2 Matrices: Definition. Matrix Algebra. Hervé Abdi Lynne J. Williams

In Neil Salkind (Ed.), Encyclopedia of Research Design. Thousand Oaks, CA: Sage. 00 Matrix Algebra Hervé Abdi Lynne J. Williams Introduction Sylvester developed the modern concept of matrices in the 9th

### 4. Joint Distributions

Virtual Laboratories > 2. Distributions > 1 2 3 4 5 6 7 8 4. Joint Distributions Basic Theory As usual, we start with a random experiment with probability measure P on an underlying sample space. Suppose

### BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

### NOV - 30211/II. 1. Let f(z) = sin z, z C. Then f(z) : 3. Let the sequence {a n } be given. (A) is bounded in the complex plane

Mathematical Sciences Paper II Time Allowed : 75 Minutes] [Maximum Marks : 100 Note : This Paper contains Fifty (50) multiple choice questions. Each question carries Two () marks. Attempt All questions.

### ECE302 Spring 2006 HW7 Solutions March 11, 2006 1

ECE32 Spring 26 HW7 Solutions March, 26 Solutions to HW7 Note: Most of these solutions were generated by R. D. Yates and D. J. Goodman, the authors of our textbook. I have added comments in italics where

### UNIT I: RANDOM VARIABLES PART- A -TWO MARKS

UNIT I: RANDOM VARIABLES PART- A -TWO MARKS 1. Given the probability density function of a continuous random variable X as follows f(x) = 6x (1-x) 0

### 2.1: Determinants by Cofactor Expansion. Math 214 Chapter 2 Notes and Homework. Evaluate a Determinant by Expanding by Cofactors

2.1: Determinants by Cofactor Expansion Math 214 Chapter 2 Notes and Homework Determinants The minor M ij of the entry a ij is the determinant of the submatrix obtained from deleting the i th row and the

### (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular.

Theorem.7.: (Properties of Triangular Matrices) (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular. (b) The product

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### INDISTINGUISHABILITY OF ABSOLUTELY CONTINUOUS AND SINGULAR DISTRIBUTIONS

INDISTINGUISHABILITY OF ABSOLUTELY CONTINUOUS AND SINGULAR DISTRIBUTIONS STEVEN P. LALLEY AND ANDREW NOBEL Abstract. It is shown that there are no consistent decision rules for the hypothesis testing problem

### Introduction to Probability

Introduction to Probability EE 179, Lecture 15, Handout #24 Probability theory gives a mathematical characterization for experiments with random outcomes. coin toss life of lightbulb binary data sequence

### MA106 Linear Algebra lecture notes

MA106 Linear Algebra lecture notes Lecturers: Martin Bright and Daan Krammer Warwick, January 2011 Contents 1 Number systems and fields 3 1.1 Axioms for number systems......................... 3 2 Vector

### Exercises with solutions (1)

Exercises with solutions (). Investigate the relationship between independence and correlation. (a) Two random variables X and Y are said to be correlated if and only if their covariance C XY is not equal

### Introduction to Matrix Algebra I

Appendix A Introduction to Matrix Algebra I Today we will begin the course with a discussion of matrix algebra. Why are we studying this? We will use matrix algebra to derive the linear regression model

### The Dirichlet-Multinomial and Dirichlet-Categorical models for Bayesian inference

The Dirichlet-Multinomial and Dirichlet-Categorical models for Bayesian inference Stephen Tu tu.stephenl@gmail.com 1 Introduction This document collects in one place various results for both the Dirichlet-multinomial

### NOTES on LINEAR ALGEBRA 1

School of Economics, Management and Statistics University of Bologna Academic Year 205/6 NOTES on LINEAR ALGEBRA for the students of Stats and Maths This is a modified version of the notes by Prof Laura

### Probabilities and Random Variables

Probabilities and Random Variables This is an elementary overview of the basic concepts of probability theory. 1 The Probability Space The purpose of probability theory is to model random experiments so

### Pattern Analysis. Logistic Regression. 12. Mai 2009. Joachim Hornegger. Chair of Pattern Recognition Erlangen University

Pattern Analysis Logistic Regression 12. Mai 2009 Joachim Hornegger Chair of Pattern Recognition Erlangen University Pattern Analysis 2 / 43 1 Logistic Regression Posteriors and the Logistic Function Decision

### Mathematics Course 111: Algebra I Part IV: Vector Spaces

Mathematics Course 111: Algebra I Part IV: Vector Spaces D. R. Wilkins Academic Year 1996-7 9 Vector Spaces A vector space over some field K is an algebraic structure consisting of a set V on which are

### Inner Product Spaces

Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and

### 3. INNER PRODUCT SPACES

. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.

### SEMESTER PLANS FOR MATH COURSES, FOR MAJORS OUTSIDE MATH.

SEMESTER PLANS FOR MATH COURSES, FOR MAJORS OUTSIDE MATH. CONTENTS: AP calculus credit and Math Placement levels. List of semester math courses. Student pathways through the semester math courses Transition

### Chapter 3: DISCRETE RANDOM VARIABLES AND PROBABILITY DISTRIBUTIONS

Chapter 3: DISCRETE RANDOM VARIABLES AND PROBABILITY DISTRIBUTIONS Part 4: Geometric Distribution Negative Binomial Distribution Hypergeometric Distribution Sections 3-7, 3-8 The remaining discrete random

### P (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )

Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =

### Problem Set 4: Covariance functions and Gaussian random fields

Problem Set : Covariance functions and Gaussian random fields GEOS 7: Inverse Problems and Parameter Estimation, Carl Tape Assigned: February 9, 15 Due: February 1, 15 Last compiled: February 5, 15 Overview

### MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo. 3 MT426 Notebook 3 3. 3.1 Definitions... 3. 3.2 Joint Discrete Distributions...

MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo c Copyright 2004-2012 by Jenny A. Baglivo. All Rights Reserved. Contents 3 MT426 Notebook 3 3 3.1 Definitions............................................

### Linear Algebra Review. Vectors

Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete

### Stochastic Processes and Advanced Mathematical Finance. Laws of Large Numbers

Steven R. Dunbar Department of Mathematics 203 Avery Hall University of Nebraska-Lincoln Lincoln, NE 68588-0130 http://www.math.unl.edu Voice: 402-472-3731 Fax: 402-472-8466 Stochastic Processes and Advanced

### M2S1 Lecture Notes. G. A. Young http://www2.imperial.ac.uk/ ayoung

M2S1 Lecture Notes G. A. Young http://www2.imperial.ac.uk/ ayoung September 2011 ii Contents 1 DEFINITIONS, TERMINOLOGY, NOTATION 1 1.1 EVENTS AND THE SAMPLE SPACE......................... 1 1.1.1 OPERATIONS

### Mathematics Notes for Class 12 chapter 3. Matrices

1 P a g e Mathematics Notes for Class 12 chapter 3. Matrices A matrix is a rectangular arrangement of numbers (real or complex) which may be represented as matrix is enclosed by [ ] or ( ) or Compact form

### Logistic Regression (1/24/13)

STA63/CBB540: Statistical methods in computational biology Logistic Regression (/24/3) Lecturer: Barbara Engelhardt Scribe: Dinesh Manandhar Introduction Logistic regression is model for regression used

### SF2940: Probability theory Lecture 8: Multivariate Normal Distribution

SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2014 Timo Koski () Mathematisk statistik 24.09.2014 1 / 75 Learning outcomes Random vectors, mean vector, covariance

### Chapter 1 - Matrices & Determinants

Chapter 1 - Matrices & Determinants Arthur Cayley (August 16, 1821 - January 26, 1895) was a British Mathematician and Founder of the Modern British School of Pure Mathematics. As a child, Cayley enjoyed

### Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers

Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications

### STAT 830 Convergence in Distribution

STAT 830 Convergence in Distribution Richard Lockhart Simon Fraser University STAT 830 Fall 2011 Richard Lockhart (Simon Fraser University) STAT 830 Convergence in Distribution STAT 830 Fall 2011 1 / 31

### QUALITY ENGINEERING PROGRAM

QUALITY ENGINEERING PROGRAM Production engineering deals with the practical engineering problems that occur in manufacturing planning, manufacturing processes and in the integration of the facilities and

### Presentation 3: Eigenvalues and Eigenvectors of a Matrix

Colleen Kirksey, Beth Van Schoyck, Dennis Bowers MATH 280: Problem Solving November 18, 2011 Presentation 3: Eigenvalues and Eigenvectors of a Matrix Order of Presentation: 1. Definitions of Eigenvalues

### Institute of Actuaries of India Subject CT3 Probability and Mathematical Statistics

Institute of Actuaries of India Subject CT3 Probability and Mathematical Statistics For 2015 Examinations Aim The aim of the Probability and Mathematical Statistics subject is to provide a grounding in

### Torgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances

Torgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances It is possible to construct a matrix X of Cartesian coordinates of points in Euclidean space when we know the Euclidean

### Helpsheet. Giblin Eunson Library MATRIX ALGEBRA. library.unimelb.edu.au/libraries/bee. Use this sheet to help you:

Helpsheet Giblin Eunson Library ATRIX ALGEBRA Use this sheet to help you: Understand the basic concepts and definitions of matrix algebra Express a set of linear equations in matrix notation Evaluate determinants

### SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation

SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas Email: M.Vidyasagar@utdallas.edu September 19, 2015 Outline

### A SURVEY ON CONTINUOUS ELLIPTICAL VECTOR DISTRIBUTIONS

A SURVEY ON CONTINUOUS ELLIPTICAL VECTOR DISTRIBUTIONS Eusebio GÓMEZ, Miguel A. GÓMEZ-VILLEGAS and J. Miguel MARÍN Abstract In this paper it is taken up a revision and characterization of the class of

### Appendix 3 IB Diploma Programme Course Outlines

Appendix 3 IB Diploma Programme Course Outlines The following points should be addressed when preparing course outlines for each IB Diploma Programme subject to be taught. Please be sure to use IBO nomenclature

### The Characteristic Polynomial

Physics 116A Winter 2011 The Characteristic Polynomial 1 Coefficients of the characteristic polynomial Consider the eigenvalue problem for an n n matrix A, A v = λ v, v 0 (1) The solution to this problem

### x1 x 2 x 3 y 1 y 2 y 3 x 1 y 2 x 2 y 1 0.

Cross product 1 Chapter 7 Cross product We are getting ready to study integration in several variables. Until now we have been doing only differential calculus. One outcome of this study will be our ability