The Analysis of Data. Volume 1. Probability. Guy Lebanon


 Millicent Byrd
 2 years ago
 Views:
Transcription
1 The Analysis of Data Volume 1 Probability Guy Lebanon First Edition 2012
2 Probability. The Analysis of Data, Volume 1. First Edition, First Printing, Copyright 2013 by Guy Lebanon. All rights reserved. This book or any portion thereof may not be reproduced or used in any manner whatsoever without the express written permission of the author.
3 In memory of Alex Lebanon
4
5 Contents Contents 5 Preface 9 Mathematical Notations 14 1 Basic Definitions Sample Space and Events The Probability Function The Classical Probability Model on Finite Spaces The Classical Model on Continuous Spaces Conditional Probability and Independence Basic Combinatorics for Probability Probability and Measure Theory* Notes Exercises Random Variables Basic Definitions Functions of a Random Variable Discrete g(x) Continuous g(x) Expectation and Variance Moments and the Moment Generating Function Random Variables and Measurable Functions* Notes Exercises Important Random Variables The Bernoulli Trial Random Variable The Binomial Random Variable The Geometric Random Variable The Hypergeometric Random Variable The Negative Binomial Random Variable
6 6 CONTENTS 3.6 The Poisson Random Variable The Uniform Random Variable The Exponential Random Variable The Gaussian Random Variable The Gamma and χ 2 Distributions The t Random Variable The Beta Random Variable Mixture Random Variable The Empirical Random Variable The Smoothed Empirical Random Variable Notes Exercises Random Vectors Basic Definitions Joint Pmf, Pdf, and Cdf Functions Marginal Random Vectors Functions of a Random Vector Conditional Probabilities and Random Vectors Moments Conditional Expectations Moment Generating Function Random Vectors and Independent σalgebras* Notes Exercises Important Random Vectors The Multinomial Random Vector The Multivariate Normal Random Vector The Dirichlet Random Vector Mixture Random Vectors The Exponential Family Random Vector Notes Exercises Random Processes Basic Definitions Random Processes and Marginal Distributions Moments One Dimensional Random Walk Random Processes and Measure Theory* The BorelCantelli Lemmas and the ZeroOne Law* Notes Exercises
7 CONTENTS 7 7 Important Random Processes Markov Chains Basic Definitions Examples Transience, Persistence, and Irreducibility Persistence and Transience of the Random Walk Process Periodicity in Markov Chains The Stationary Distribution Poisson Processes Postulates and Di erential Equation Relationship to Poisson Distribution Relationship to Exponential Distribution Relationship to Binomial Distribution Relationship to Uniform Distribution Gaussian Processes The Wiener Process Notes Exercises Limit Theorems Modes of Stochastic Convergence Relationships Between Modes of Convergences Dominated Convergence Theorem for Vectors* Sche e s Theorem The Portmanteau Theorem The Law of Large Numbers The Characteristic Function* Levy s Continuity Theorem The Central Limit Theorem Continuous Mapping Theorem Slustky s Theorems Notes Exercises A Set Theory 223 A.1 Basic Definitions A.2 Functions A.3 Cardinality A.4 Limits of Sets A.5 Notes A.6 Exercises
8 8 CONTENTS B Metric Spaces 237 B.1 Basic Definitions B.2 Limits B.3 Continuity B.4 The Euclidean Space B.5 Growth of Functions B.6 Notes B.7 Exercises C Linear Algebra 257 C.1 Basic Definitions C.2 Rank C.3 Eigenvalues, Determinant, and Trace C.4 Positive SemiDefinite Matrices C.5 Singular Value Decomposition C.6 Notes C.7 Exercises D Di erentiation 279 D.1 Univariate Di erentiation D.2 Taylor Expansion and Power Series D.3 Notes D.4 Exercises E Measure Theory* 289 E.1 σalgebras* E.2 The Measure Function* E.3 Caratheodory s Extension Theorem* E.3.1 Dynkin s Theorem* E.3.2 Outer Measure* E.3.3 The Extension Theorem* E.4 Independent σalgebras* E.5 Important Measure Functions* E.5.1 Discrete Measure Functions* E.5.2 The Lebesgue Measure* E.5.3 The LebesgueStieltjes Measure* E.6 Measurability of Functions* E.7 Notes F Integration 307 F.1 Riemann Integral F.2 Integration and Di erentiation F.3 The Lebesgue Integral* F.3.1 Relation between the Riemann and the Lebesgue Integrals* 325 F.3.2 Transformed Measures* F.4 Product Measures*
9 CONTENTS 9 F.5 Integration over Product Spaces* F.5.1 The Lebesgue Measure over R d * F.6 Multivariate Di erentiation and Integration F.7 Notes Bibliography 337 Index 341
10 10 CONTENTS
11 Preface The Analysis of Data Project The Analysis of Data (TAOD) project provides educational material in the area of data analysis. The project features comprehensive coverage of all relevant disciplines including probability, statistics, computing, and machine learning. The content is almost selfcontained and includes mathematical prerequisites and basic computing concepts. The R programming language is used to demonstrate the contents. Full code is available, facilitating reproducibility of experiments and letting readers experiment with variations of the code. The presentation is mathematically rigorous, and includes derivations and proofs in most cases. HTML versions are freely available on the website Hardcopies are available at a ordable prices. Volume 1: Probability This volume focuses on probability theory. There are many excellent textbooks on probability, and yet this book di ers from others in several ways. Probability theory is a wide field. This book focuses on the parts of probability that are most relevant for statistics and machine learning. The book contains almost all of the mathematical prerequisites, including set theory, metric spaces, linear algebra, di erentiation, integration, and measure theory. Almost all results in the book appear with a proof. 11
12 12 CONTENTS Probability textbooks are typically either elementary or advanced. This book strikes a balance by attempting to avoid measure theory where possible, but resorting to measure theory and other advanced material in a few places where they are essential. The book uses R to illustrate concepts. Full code is available in the book, facilitating reproducibility of experiments and letting readers experiment with variations of the code. I am not aware of a single textbook that covers the material from probability theory that is necessary and sufficient for an indepth understanding of statistics and machine learning. This book represents my best e ort in that direction. Since this book is part of a series of books on data analysis, it does not include any statistics or machine learning. Such content is postponed to future volumes. Website A companion website ( contains an HTML version of this book, errata, and additional multimedia material. The website will also link to additional TAOD volumes as they become available. Mathematical Appendices A large part of the book contains six appendices on mathematical prerequisites. Probability requires knowledge of many branches of mathematics, including calculus, linear algebra, set theory, metric spaces, measure, and Lebesgue integration. Instead of referring the reader to a large collection of math textbooks we include here all of the necessary prerequisites. References are provided in the notes sections at the end of each chapter for additional resources. Dependencies The diagram below indicates the dependencies between the di erent chapters of the book. Appendix chapters are shaded and dependencies between appendix chapters and regular chapters are marked by dashed arrows. It is not essential to strictly adhere to this dependency as many chapters require only a brief familiarity with some issues in the chapters that they depend on.
13 CONTENTS 13 A B C 1 D E 2 F Starred sections correspond to material that requires measure theory, or that can be better appreciated with knowledge of measure theory. R Code The book contains many fragments of R code, aimed to illustrate probability theory and its applications. The code is included so that the reader can reproduce the results as well as modify the code and run variations of it. In order to appreciate the code and modify it, the reader will need a basic understanding of the R programming language and R graphics. A good introduction to R is available from the CRAN website at Alternatively, two chapters from volume 2 of the analysis of data series (R Programming and R Graphics) are freely available online at The book uses a variety of R graphics packages, including base, lattice, and ggplot2. The most frequently used package is ggplot2, which is described in detail in [49] (see also and the R Graphics chapter available at To ensure that the code fragments run locally as they appear in the book, install and bring into scope the required packages using the install.packages and library R functions. For example, to install and load the ggplot2 package type the following commands in the R prompt.
14 14 CONTENTS # this is a comment install.packages("ggplot2") # installs the package ggplot2 library("ggplot2") # brings ggplot2 into scope The code fragments throughout the book are annotated with output displayed by the R interpreter. This output is displayed following two hash symbols which are interpreted as comments by R (see below). a = pi print(a) # note the ## symbols preceding the output below ## [1] print(a + 1) ## [1] This format makes it easy to copy a code fragment from an HTML page and paste it directly into R (the output following the hash symbols will be interpreted as comments and not produce a syntax error). Acknowledgements The following people made technical suggestions that helped improve the contents: Krishnakumar Balasubramanian, Rohit Banga, Joshua Dillon, Sanjeet Hajarnis, Oded Green, Yi Mao, Seungyeon Kim, Joonseok Lee, Fuxin Li, Nishant Mehta, Yaron Rachlin, Parikshit Ram, Kaushik Rangadurai, Neil Slagle, Mingxuan Sun, Brian Steber, Gena Tang, Long Tran. In addition, many useful comments were received through a discussion board during my fall 2011 class computational data analysis at Georgia Tech. These comments were mostly anonymized, but some commentators who identified themselves appear above. Katharina Probst, Neil Slagle and Laura Usselman edited portions of this book, and made many useful suggestions. The book features a combination of text, equations, graphs, and R code, made possible by the knitr package. I thank Yihui Xie for implementing knitr, and for his help through the knitr Google discussion group. Katharina Probst helped with web development and design.
15 Mathematical Notations Logic 8 for all 9 there exists ) implies, if and only if def = defined as For example, 8a >0, 1/a > 0 reads for every a>0wehave1/a > 0 and 8a >0, 9b >0, a/b =1 ) b/a = 1 reads for all a>0 there exists b>0 such that, whenever a/b = 1, we also have b/a = 1. Sets and Functions (Chapter A) sample space! an element of the sample space I A indicator function I A (x) =1ifx 2 A and 0 otherwise δ ij Kronercker s delta: δ ij = 1 if i = j and 0 otherwise N natural numbers {1, 2, 3,...} Z integers {..., 2, 1, 0, 1, 2,...} Q rational numbers A B Cartesian product of two sets A k repeated Cartesian product (k times) of a set: A A A c complement of a set: \ A 2 A power set of the set A A number of elements in a finite set f 1 (A) preimage of the function f: {x : f(x) 2 A} f g f(x) =g(x) for all x x (n) sequence of elements in a set A n % A convergence of a sequence of increasing sets A n & A convergence of a sequence of decreasing sets Combinatorics (Section 1.6) n! factorial function n(n 1) 2 1 (n) r n!/(n r)! n n!/(r!(n r)!) r 15
16 16 CONTENTS Metric spaces (Chapter B) R real numbers R d set of ddimensional vectors of real numbers B a (x) open ball of radius a centered at x x vector in R d the icomponent of the vector x sequence of vectors in R d the icomponent of the vector x (j) hx, yi inner product P i x iy i kxk Euclidean norm p Pi x2 i d(x, y) Euclidean distance pp i (x i y i ) 2 x (n)! x convergence of a sequence f n! f pointwise convergence of a sequence of functions f n % f pointwise convergence of an increasing sequence of functions f n & f pointwise convergence of a decreasing sequence of functions O(f) big O growth notation o(f) little o growth notation x i x (n) x (j) i Note in particular that we denote vectors in bold face, for example x, and the scalar components of such vectors using subscripts (nonbold face) x = (x 1,...,x d ). We refer to sequence of vectors using superscripts, for example x (n) and their scalar components as x (n) =(x (n) 1,...,x(n) d ). Probability (Chapters 1, 2, 4, 8) P probability function P E probability conditioned on the event E F X cumulative distribution function (cdf) corresponding to the RV X f X probability density function (pdf) corresponding to the RV X p X probability mass function (pmf) corresponding to the RV X E expectation Var variance Cov covariance std standard deviation distributed according to iid independent identically distributed (iid) sampling P k simplex of probability functions over = {1,...,k} i.o. infinitely often as! convergence with probability 1 p! convergence in probability convergence in distribution
17 CONTENTS 17 A > A n A n ij A 1 I tr A det A row A col A dim A rank A diag(v) A B Matrices (Chapter C) matrix transpose the matrix A raised to the npower the i, j element of the matrix A n inverse of matrix A identity matrix trace of the matrix A determinant of the matrix A row space of the matrix A column space of the matrix A dimension of a linear space rank of the matrix A diagonal matrix whose diagonal is given by the vector v Kronecker product of two matrices A, B All vectors are assumed to be column vectors unless stated explicitly otherwise. For example, if x is an arbitrary vector and A a matrix the expression x > Ax represents a scalar. df /dx d 2 f/dx 2 i rf 2 j r 2 f Di erentiation (Chapter D) derivative second order derivative partial derivative gradient vector of partial derivatives of f : R k! R Jacobian matrix of partial derivatives of f : R k! R m second order partial derivative Hessian matrix of second order partial derivatives of f : R k! R We consider the gradient vector rf as a row vector. This ensures that the notations rf and rf are consistent. Measure and Integration (Chapters E, F) σ(c) σ algebra generated by the set of sets C µ R measure R dµ Lebesgue integral with respect to measure µ R d P Lebesgue integral with respect to probability measure P dfx Lebesgue integral with respect to probability measure R corresponding to cdf F X dx Lebesgue integral with respect to Lebesgue measure or Riemann integral B(R d ) Borel σalgebra over metric space R d F 1 F 1 product σalgebra µ 1 µ 2 product measure a.e. almost everywhere (except on a set of measure zero)
18 18 CONTENTS
Master s Theory Exam Spring 2006
Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem
More informationMATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix.
MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix. Inverse matrix Definition. Let A be an n n matrix. The inverse of A is an n n matrix, denoted
More informationMathematical Background
Appendix A Mathematical Background A.1 Joint, Marginal and Conditional Probability Let the n (discrete or continuous) random variables y 1,..., y n have a joint joint probability probability p(y 1,...,
More informationAlabama Department of Postsecondary Education
Date Adopted 1998 Dates reviewed 2007, 2011, 2013 Dates revised 2004, 2008, 2011, 2013, 2015 Alabama Department of Postsecondary Education Representing Alabama s Public TwoYear College System Jefferson
More informationMATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +
More information15.062 Data Mining: Algorithms and Applications Matrix Math Review
.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop
More informationPCHS ALGEBRA PLACEMENT TEST
MATHEMATICS Students must pass all math courses with a C or better to advance to the next math level. Only classes passed with a C or better will count towards meeting college entrance requirements. If
More information1 Sets and Set Notation.
LINEAR ALGEBRA MATH 27.6 SPRING 23 (COHEN) LECTURE NOTES Sets and Set Notation. Definition (Naive Definition of a Set). A set is any collection of objects, called the elements of that set. We will most
More informationSimilarity and Diagonalization. Similar Matrices
MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that
More information13 MATH FACTS 101. 2 a = 1. 7. The elements of a vector have a graphical interpretation, which is particularly easy to see in two or three dimensions.
3 MATH FACTS 0 3 MATH FACTS 3. Vectors 3.. Definition We use the overhead arrow to denote a column vector, i.e., a linear segment with a direction. For example, in threespace, we write a vector in terms
More informationMathematics (MAT) MAT 061 Basic Euclidean Geometry 3 Hours. MAT 051 PreAlgebra 4 Hours
MAT 051 PreAlgebra Mathematics (MAT) MAT 051 is designed as a review of the basic operations of arithmetic and an introduction to algebra. The student must earn a grade of C or in order to enroll in MAT
More informationMATH 240 Fall, Chapter 1: Linear Equations and Matrices
MATH 240 Fall, 2007 Chapter Summaries for Kolman / Hill, Elementary Linear Algebra, 9th Ed. written by Prof. J. Beachy Sections 1.1 1.5, 2.1 2.3, 4.2 4.9, 3.1 3.5, 5.3 5.5, 6.1 6.3, 6.5, 7.1 7.3 DEFINITIONS
More informationMATRIX ALGEBRA AND SYSTEMS OF EQUATIONS
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a
More informationEigenvalues, Eigenvectors, Matrix Factoring, and Principal Components
Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they
More informationStatistical Machine Learning
Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes
More informationNotes for STA 437/1005 Methods for Multivariate Data
Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.
More information1 Introduction to Matrices
1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns
More informationLinear Algebra Notes for Marsden and Tromba Vector Calculus
Linear Algebra Notes for Marsden and Tromba Vector Calculus ndimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of
More informationLINEAR ALGEBRA W W L CHEN
LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008 This chapter is available free to all individuals, on understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,
More informationF Matrix Calculus F 1
F Matrix Calculus F 1 Appendix F: MATRIX CALCULUS TABLE OF CONTENTS Page F1 Introduction F 3 F2 The Derivatives of Vector Functions F 3 F21 Derivative of Vector with Respect to Vector F 3 F22 Derivative
More information2.3 Convex Constrained Optimization Problems
42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions
More informationL10: Probability, statistics, and estimation theory
L10: Probability, statistics, and estimation theory Review of probability theory Bayes theorem Statistics and the Normal distribution Least Squares Error estimation Maximum Likelihood estimation Bayesian
More informationContinued Fractions and the Euclidean Algorithm
Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction
More informationMachine Learning Math Essentials
Machine Learning Math Essentials Jeff Howbert Introduction to Machine Learning Winter 2012 1 Areas of math essential to machine learning Machine learning is part of both statistics and computer science
More informationThe Exponential Family
The Exponential Family David M. Blei Columbia University November 3, 2015 Definition A probability density in the exponential family has this form where p.x j / D h.x/ expf > t.x/ a./g; (1) is the natural
More informationSF2940: Probability theory Lecture 8: Multivariate Normal Distribution
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2015 Timo Koski Matematisk statistik 24.09.2015 1 / 1 Learning outcomes Random vectors, mean vector, covariance matrix,
More informationRandom Vectors and the Variance Covariance Matrix
Random Vectors and the Variance Covariance Matrix Definition 1. A random vector X is a vector (X 1, X 2,..., X p ) of jointly distributed random variables. As is customary in linear algebra, we will write
More informationInner product. Definition of inner product
Math 20F Linear Algebra Lecture 25 1 Inner product Review: Definition of inner product. Slide 1 Norm and distance. Orthogonal vectors. Orthogonal complement. Orthogonal basis. Definition of inner product
More informationMATHEMATICS (CLASSES XI XII)
MATHEMATICS (CLASSES XI XII) General Guidelines (i) All concepts/identities must be illustrated by situational examples. (ii) The language of word problems must be clear, simple and unambiguous. (iii)
More informationMath 5311 Gateaux differentials and Frechet derivatives
Math 5311 Gateaux differentials and Frechet derivatives Kevin Long January 26, 2009 1 Differentiation in vector spaces Thus far, we ve developed the theory of minimization without reference to derivatives.
More informationMATHEMATICAL METHODS OF STATISTICS
MATHEMATICAL METHODS OF STATISTICS By HARALD CRAMER TROFESSOK IN THE UNIVERSITY OF STOCKHOLM Princeton PRINCETON UNIVERSITY PRESS 1946 TABLE OF CONTENTS. First Part. MATHEMATICAL INTRODUCTION. CHAPTERS
More informationChapter 2: Systems of Linear Equations and Matrices:
At the end of the lesson, you should be able to: Chapter 2: Systems of Linear Equations and Matrices: 2.1: Solutions of Linear Systems by the Echelon Method Define linear systems, unique solution, inconsistent,
More informationFor a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )
Probability Review 15.075 Cynthia Rudin A probability space, defined by Kolmogorov (19031987) consists of: A set of outcomes S, e.g., for the roll of a die, S = {1, 2, 3, 4, 5, 6}, 1 1 2 1 6 for the roll
More informationCHAPTER 5. Product Measures
54 CHAPTER 5 Product Measures Given two measure spaces, we may construct a natural measure on their Cartesian product; the prototype is the construction of Lebesgue measure on R 2 as the product of Lebesgue
More informationCofactor Expansion: Cramer s Rule
Cofactor Expansion: Cramer s Rule MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 2015 Introduction Today we will focus on developing: an efficient method for calculating
More informationDeterminants. Dr. Doreen De Leon Math 152, Fall 2015
Determinants Dr. Doreen De Leon Math 52, Fall 205 Determinant of a Matrix Elementary Matrices We will first discuss matrices that can be used to produce an elementary row operation on a given matrix A.
More informationMATHEMATICS (MATH) 3. Provides experiences that enable graduates to find employment in sciencerelated
194 / Department of Natural Sciences and Mathematics MATHEMATICS (MATH) The Mathematics Program: 1. Provides challenging experiences in Mathematics, Physics, and Physical Science, which prepare graduates
More informationDiagonal, Symmetric and Triangular Matrices
Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by
More informationUNIT 2 MATRICES  I 2.0 INTRODUCTION. Structure
UNIT 2 MATRICES  I Matrices  I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress
More informationCITY UNIVERSITY LONDON. BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION
No: CITY UNIVERSITY LONDON BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION ENGINEERING MATHEMATICS 2 (resit) EX2005 Date: August
More informationIntroduction to Matrix Algebra
Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra  1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary
More informationThe Delta Method and Applications
Chapter 5 The Delta Method and Applications 5.1 Linear approximations of functions In the simplest form of the central limit theorem, Theorem 4.18, we consider a sequence X 1, X,... of independent and
More informationA Tutorial on Probability Theory
Paola Sebastiani Department of Mathematics and Statistics University of Massachusetts at Amherst Corresponding Author: Paola Sebastiani. Department of Mathematics and Statistics, University of Massachusetts,
More informationState of Stress at Point
State of Stress at Point Einstein Notation The basic idea of Einstein notation is that a covector and a vector can form a scalar: This is typically written as an explicit sum: According to this convention,
More informationMathematics of Cryptography
CHAPTER 2 Mathematics of Cryptography Part I: Modular Arithmetic, Congruence, and Matrices Objectives This chapter is intended to prepare the reader for the next few chapters in cryptography. The chapter
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 6 Three Approaches to Classification Construct
More informationUndergraduate Notes in Mathematics. Arkansas Tech University Department of Mathematics
Undergraduate Notes in Mathematics Arkansas Tech University Department of Mathematics An Introductory Single Variable Real Analysis: A Learning Approach through Problem Solving Marcel B. Finan c All Rights
More informationVector Spaces; the Space R n
Vector Spaces; the Space R n Vector Spaces A vector space (over the real numbers) is a set V of mathematical entities, called vectors, U, V, W, etc, in which an addition operation + is defined and in which
More information4. MATRICES Matrices
4. MATRICES 170 4. Matrices 4.1. Definitions. Definition 4.1.1. A matrix is a rectangular array of numbers. A matrix with m rows and n columns is said to have dimension m n and may be represented as follows:
More informationLecture Notes 1. Brief Review of Basic Probability
Probability Review Lecture Notes Brief Review of Basic Probability I assume you know basic probability. Chapters 3 are a review. I will assume you have read and understood Chapters 3. Here is a very
More informationProbability and Statistics
CHAPTER 2: RANDOM VARIABLES AND ASSOCIATED FUNCTIONS 2b  0 Probability and Statistics Kristel Van Steen, PhD 2 Montefiore Institute  Systems and Modeling GIGA  Bioinformatics ULg kristel.vansteen@ulg.ac.be
More informationIntroduction to General and Generalized Linear Models
Introduction to General and Generalized Linear Models General Linear Models  part I Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK2800 Kgs. Lyngby
More information1 Introduction. 2 Matrices: Definition. Matrix Algebra. Hervé Abdi Lynne J. Williams
In Neil Salkind (Ed.), Encyclopedia of Research Design. Thousand Oaks, CA: Sage. 00 Matrix Algebra Hervé Abdi Lynne J. Williams Introduction Sylvester developed the modern concept of matrices in the 9th
More information4. Joint Distributions
Virtual Laboratories > 2. Distributions > 1 2 3 4 5 6 7 8 4. Joint Distributions Basic Theory As usual, we start with a random experiment with probability measure P on an underlying sample space. Suppose
More informationBANACH AND HILBERT SPACE REVIEW
BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but
More informationNOV  30211/II. 1. Let f(z) = sin z, z C. Then f(z) : 3. Let the sequence {a n } be given. (A) is bounded in the complex plane
Mathematical Sciences Paper II Time Allowed : 75 Minutes] [Maximum Marks : 100 Note : This Paper contains Fifty (50) multiple choice questions. Each question carries Two () marks. Attempt All questions.
More informationECE302 Spring 2006 HW7 Solutions March 11, 2006 1
ECE32 Spring 26 HW7 Solutions March, 26 Solutions to HW7 Note: Most of these solutions were generated by R. D. Yates and D. J. Goodman, the authors of our textbook. I have added comments in italics where
More informationUNIT I: RANDOM VARIABLES PART A TWO MARKS
UNIT I: RANDOM VARIABLES PART A TWO MARKS 1. Given the probability density function of a continuous random variable X as follows f(x) = 6x (1x) 0
More information2.1: Determinants by Cofactor Expansion. Math 214 Chapter 2 Notes and Homework. Evaluate a Determinant by Expanding by Cofactors
2.1: Determinants by Cofactor Expansion Math 214 Chapter 2 Notes and Homework Determinants The minor M ij of the entry a ij is the determinant of the submatrix obtained from deleting the i th row and the
More information(a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular.
Theorem.7.: (Properties of Triangular Matrices) (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular. (b) The product
More informationOverview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model
Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written
More informationINDISTINGUISHABILITY OF ABSOLUTELY CONTINUOUS AND SINGULAR DISTRIBUTIONS
INDISTINGUISHABILITY OF ABSOLUTELY CONTINUOUS AND SINGULAR DISTRIBUTIONS STEVEN P. LALLEY AND ANDREW NOBEL Abstract. It is shown that there are no consistent decision rules for the hypothesis testing problem
More informationIntroduction to Probability
Introduction to Probability EE 179, Lecture 15, Handout #24 Probability theory gives a mathematical characterization for experiments with random outcomes. coin toss life of lightbulb binary data sequence
More informationMA106 Linear Algebra lecture notes
MA106 Linear Algebra lecture notes Lecturers: Martin Bright and Daan Krammer Warwick, January 2011 Contents 1 Number systems and fields 3 1.1 Axioms for number systems......................... 3 2 Vector
More informationExercises with solutions (1)
Exercises with solutions (). Investigate the relationship between independence and correlation. (a) Two random variables X and Y are said to be correlated if and only if their covariance C XY is not equal
More informationIntroduction to Matrix Algebra I
Appendix A Introduction to Matrix Algebra I Today we will begin the course with a discussion of matrix algebra. Why are we studying this? We will use matrix algebra to derive the linear regression model
More informationThe DirichletMultinomial and DirichletCategorical models for Bayesian inference
The DirichletMultinomial and DirichletCategorical models for Bayesian inference Stephen Tu tu.stephenl@gmail.com 1 Introduction This document collects in one place various results for both the Dirichletmultinomial
More informationNOTES on LINEAR ALGEBRA 1
School of Economics, Management and Statistics University of Bologna Academic Year 205/6 NOTES on LINEAR ALGEBRA for the students of Stats and Maths This is a modified version of the notes by Prof Laura
More informationProbabilities and Random Variables
Probabilities and Random Variables This is an elementary overview of the basic concepts of probability theory. 1 The Probability Space The purpose of probability theory is to model random experiments so
More informationPattern Analysis. Logistic Regression. 12. Mai 2009. Joachim Hornegger. Chair of Pattern Recognition Erlangen University
Pattern Analysis Logistic Regression 12. Mai 2009 Joachim Hornegger Chair of Pattern Recognition Erlangen University Pattern Analysis 2 / 43 1 Logistic Regression Posteriors and the Logistic Function Decision
More informationMathematics Course 111: Algebra I Part IV: Vector Spaces
Mathematics Course 111: Algebra I Part IV: Vector Spaces D. R. Wilkins Academic Year 19967 9 Vector Spaces A vector space over some field K is an algebraic structure consisting of a set V on which are
More informationInner Product Spaces
Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and
More information3. INNER PRODUCT SPACES
. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.
More informationSEMESTER PLANS FOR MATH COURSES, FOR MAJORS OUTSIDE MATH.
SEMESTER PLANS FOR MATH COURSES, FOR MAJORS OUTSIDE MATH. CONTENTS: AP calculus credit and Math Placement levels. List of semester math courses. Student pathways through the semester math courses Transition
More informationChapter 3: DISCRETE RANDOM VARIABLES AND PROBABILITY DISTRIBUTIONS
Chapter 3: DISCRETE RANDOM VARIABLES AND PROBABILITY DISTRIBUTIONS Part 4: Geometric Distribution Negative Binomial Distribution Hypergeometric Distribution Sections 37, 38 The remaining discrete random
More informationP (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )
Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =
More informationProblem Set 4: Covariance functions and Gaussian random fields
Problem Set : Covariance functions and Gaussian random fields GEOS 7: Inverse Problems and Parameter Estimation, Carl Tape Assigned: February 9, 15 Due: February 1, 15 Last compiled: February 5, 15 Overview
More informationMT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo. 3 MT426 Notebook 3 3. 3.1 Definitions... 3. 3.2 Joint Discrete Distributions...
MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo c Copyright 20042012 by Jenny A. Baglivo. All Rights Reserved. Contents 3 MT426 Notebook 3 3 3.1 Definitions............................................
More informationLinear Algebra Review. Vectors
Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete
More informationStochastic Processes and Advanced Mathematical Finance. Laws of Large Numbers
Steven R. Dunbar Department of Mathematics 203 Avery Hall University of NebraskaLincoln Lincoln, NE 685880130 http://www.math.unl.edu Voice: 4024723731 Fax: 4024728466 Stochastic Processes and Advanced
More informationM2S1 Lecture Notes. G. A. Young http://www2.imperial.ac.uk/ ayoung
M2S1 Lecture Notes G. A. Young http://www2.imperial.ac.uk/ ayoung September 2011 ii Contents 1 DEFINITIONS, TERMINOLOGY, NOTATION 1 1.1 EVENTS AND THE SAMPLE SPACE......................... 1 1.1.1 OPERATIONS
More informationMathematics Notes for Class 12 chapter 3. Matrices
1 P a g e Mathematics Notes for Class 12 chapter 3. Matrices A matrix is a rectangular arrangement of numbers (real or complex) which may be represented as matrix is enclosed by [ ] or ( ) or Compact form
More informationLogistic Regression (1/24/13)
STA63/CBB540: Statistical methods in computational biology Logistic Regression (/24/3) Lecturer: Barbara Engelhardt Scribe: Dinesh Manandhar Introduction Logistic regression is model for regression used
More informationSF2940: Probability theory Lecture 8: Multivariate Normal Distribution
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2014 Timo Koski () Mathematisk statistik 24.09.2014 1 / 75 Learning outcomes Random vectors, mean vector, covariance
More informationChapter 1  Matrices & Determinants
Chapter 1  Matrices & Determinants Arthur Cayley (August 16, 1821  January 26, 1895) was a British Mathematician and Founder of the Modern British School of Pure Mathematics. As a child, Cayley enjoyed
More informationVariance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers
Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications
More informationSTAT 830 Convergence in Distribution
STAT 830 Convergence in Distribution Richard Lockhart Simon Fraser University STAT 830 Fall 2011 Richard Lockhart (Simon Fraser University) STAT 830 Convergence in Distribution STAT 830 Fall 2011 1 / 31
More informationQUALITY ENGINEERING PROGRAM
QUALITY ENGINEERING PROGRAM Production engineering deals with the practical engineering problems that occur in manufacturing planning, manufacturing processes and in the integration of the facilities and
More informationPresentation 3: Eigenvalues and Eigenvectors of a Matrix
Colleen Kirksey, Beth Van Schoyck, Dennis Bowers MATH 280: Problem Solving November 18, 2011 Presentation 3: Eigenvalues and Eigenvectors of a Matrix Order of Presentation: 1. Definitions of Eigenvalues
More informationInstitute of Actuaries of India Subject CT3 Probability and Mathematical Statistics
Institute of Actuaries of India Subject CT3 Probability and Mathematical Statistics For 2015 Examinations Aim The aim of the Probability and Mathematical Statistics subject is to provide a grounding in
More informationTorgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances
Torgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances It is possible to construct a matrix X of Cartesian coordinates of points in Euclidean space when we know the Euclidean
More informationHelpsheet. Giblin Eunson Library MATRIX ALGEBRA. library.unimelb.edu.au/libraries/bee. Use this sheet to help you:
Helpsheet Giblin Eunson Library ATRIX ALGEBRA Use this sheet to help you: Understand the basic concepts and definitions of matrix algebra Express a set of linear equations in matrix notation Evaluate determinants
More informationSYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation
SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas Email: M.Vidyasagar@utdallas.edu September 19, 2015 Outline
More informationA SURVEY ON CONTINUOUS ELLIPTICAL VECTOR DISTRIBUTIONS
A SURVEY ON CONTINUOUS ELLIPTICAL VECTOR DISTRIBUTIONS Eusebio GÓMEZ, Miguel A. GÓMEZVILLEGAS and J. Miguel MARÍN Abstract In this paper it is taken up a revision and characterization of the class of
More informationAppendix 3 IB Diploma Programme Course Outlines
Appendix 3 IB Diploma Programme Course Outlines The following points should be addressed when preparing course outlines for each IB Diploma Programme subject to be taught. Please be sure to use IBO nomenclature
More informationThe Characteristic Polynomial
Physics 116A Winter 2011 The Characteristic Polynomial 1 Coefficients of the characteristic polynomial Consider the eigenvalue problem for an n n matrix A, A v = λ v, v 0 (1) The solution to this problem
More informationx1 x 2 x 3 y 1 y 2 y 3 x 1 y 2 x 2 y 1 0.
Cross product 1 Chapter 7 Cross product We are getting ready to study integration in several variables. Until now we have been doing only differential calculus. One outcome of this study will be our ability
More informationMatrix Inverse and Determinants
DM554 Linear and Integer Programming Lecture 5 and Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1 2 3 4 and Cramer s rule 2 Outline 1 2 3 4 and
More informationSTAT/MTHE 353: Probability II. STAT/MTHE 353: Multiple Random Variables. Review. Administrative details. Instructor: TamasLinder
STAT/MTHE 353: Probability II STAT/MTHE 353: Multiple Random Variables Administrative details Instructor: TamasLinder Email: linder@mast.queensu.ca T. Linder ueen s University Winter 2012 O ce: Je ery
More information