Probability for Statistics and Machine Learning: Fundamentals and Advanced Topics. Anirban DasGupta

Transcription

1 Probability for Statistics and Machine Learning: Fundamentals and Advanced Topics Anirban DasGupta

2 Contents 1 Review of Univariate Probability ExperimentsandSampleSpaces Conditional Probability and Independence IntegerValuedandDiscreteRandomVariables CDF and Independence Expectation and Moments Inequalities Generating and Moment Generating Functions Applications of Generating Functions to a Pattern Problem Standard Discrete Distributions Poisson Approximation to Binomial ContinuousRandomVariables Functions of a Continuous Random Variable Expectation and Moments Moments and the Tail of a CDF Moment Generating Function and Fundamental Inequalities InversionofanMGFandPost sformula Some Special Continuous Distributions Normal Distribution and Confidence Interval for a Mean Stein s Lemma Chernoff s Variance Inequality Various Characterizations of Normal Distributions Normal Approximations and Central Limit Theorem Binomial Confidence Interval Error of the CLT Normal Approximation to Poisson and Gamma Confidence Intervals Convergence of Densities and Edgeworth Expansions Exercises References Multivariate Discrete Distributions Bivariate Joint Distributions and Expectations of Functions Conditional Distributions and Conditional Expectations Examples on Conditional Distributions and Expectations Using Conditioning to Evaluate Mean and Variance Covariance and Correlation Multivariate Case Joint MGF Multinomial Distribution The Poissonization Technique I

3 2.7 Exercises Multidimensional Densities Joint Density Function and Its Role Expectation of Functions Bivariate Normal Conditional Densities and Expectations Examples on Conditional Densities and Expectations Posterior Densities, Likelihood Functions, and Bayes Estimates Bivariate Normal Conditional Distributions Useful Formulas and Characterizations for Bivariate Normal Computing Bivariate Normal Probabilities Conditional Expectation Given a Set and Borel s Paradox Exercises References Advanced Distribution Theory Convolutions and Examples Products and Quotients and the t and F Distribution Transformations Applications of Jacobian Formula Polar Coordinates in Two Dimensions n-dimensional Polar and Helmert s Transformation Efficient Spherical Calculations with Polar Coordinates Independence of Mean and Variance in Normal Case The t Confidence Interval The Dirichlet Distribution Picking a Point from the Surface of a Sphere Poincaré slemma Ten Important High Dimensional Formulas for Easy Reference Exercises References Multivariate Normal and Related Distributions Definition and Some Basic Properties Conditional Distributions ExchangeableNormalVariables Sampling Distributions Useful in Statistics Wishart Expectation Identities * Hotelling s T 2 and Distribution of Quadratic Forms Distribution of Correlation Coefficient Noncentral Distributions Some Important Inequalities for Easy Reference Exercises II

4 5.8 References Finite Sample Theory of Order Statistics and Extremes Basic Distribution Theory More Advanced Distribution Theory Quantile Transformation and Existence of Moments Spacings Exponential Spacings and Réyni s Representation Uniform Spacings Conditional Distributions and Markov Property Some Applications Records The Empirical CDF Distribution of the Multinomial Maximum Exercises References Essential Asymptotics and Applications Some Basic Notation and Convergence Concepts LawsofLargeNumbers Convergence Preservation Convergence in Distribution Preservation of Convergence and Statistical Applications Slutsky s Theorem Delta Theorem Variance Stabilizing Transformations Convergence of Moments Uniform Integrability The Moment Problem and Convergence in Distribution Approximation of Moments Convergence of Densities and Scheffé stheorem Exercises References Characteristic Functions and Applications Characteristic Functions of Standard Distributions Inversion and Uniqueness Taylor Expansions, Differentiability, and Moments ContinuityTheorems ProofoftheCLTandtheWLLN Producing Characteristic Functions Error of the Central Limit Theorem Lindeberg-Feller Theorem for General Independent Case Infinite Divisibility and Stable Laws III

5 8.10 Some Useful Inequalities Exercises References Asymptotics of Extremes and Order Statistics Central Order Statistics Single Order Statistic Two Statistical Applications Several Order Statistics Extremes Easily Applicable Limit Theorems The Convergence of Types Theorem Fisher-Tippett Family and Putting it Together Exercises References Markov Chains and Applications Notation and Basic Definitions Examples and Various Applications as a Model Chapman-Kolmogorov Equation Communicating Classes Gambler s Ruin First Passage, Recurrence and Transience Long Run Evolution and Stationary Distributions Exercises References Random Walks Random Walk on the Cubic Lattice Some Distribution Theory Recurrence and Transience Pólya s Formula for the Return Probability First Passage Time and Arc Sine Law The Local Time Practically Useful Generalizations Wald s Identity FateofaRandomWalk Chung-Fuchs Theorem Six Important Inequalities Exercises References IV

6 12 Brownian Motion and Gaussian Processes Preview of Connections to the Random Walk Basic Definitions Condition for a Gaussian Process to be Markov Explicit Construction of Brownian Motion Basic Distributional Properties Reflection Principle and Extremes Path Properties and Behavior Near Zero and Infinity FractalNatureofLevelSets The Dirichlet Problem and Boundary Crossing Probabilities Recurrence and Transience The Local Time of Brownian Motion Invariance Principle and Statistical Applications Strong Invariance Principle and the KMT Theorem Brownian Motion with Drift and Ornstein-Uhlenbeck Process Negative Drift and Density of Maximum Transition Density and the Heat Equation TheOrnstein-UhlenbeckProcess Exercises References Poisson Processes and Applications Notation Defining a Homogeneous Poisson Process Important Properties and Uses as a Statistical Model Linear Poisson Process and Brownian Motion: A Connection Higher Dimensional Poisson Point Processes The Mapping Theorem One Dimensional Nonhomogeneous Processes Campbell s Theorem and Shot Noise Poisson process and Stable Laws Exercises References Discrete Time Martingales and Concentration Inequalities Illustrative Examples and Applications in Statistics Stopping Times and Optional Stopping Stopping Times Optional Stopping Sufficient Conditions for Optional Stopping Theorem Applications of Optional Stopping Martingale and Concentration Inequalities Maximal Inequality V

7 Inequalities of Burkholder, Davis, and Gundy Inequalites of Hoeffding and Azuma Inequalities of McDiarmid and Devroye The Upcrossing Inequality Convergence of Martingales The Basic Convergence Theorem Convergence in L 1 and L Reverse Martingales Martingale Central Limit Theorem Exercises References Probability Metrics Standard Probability Metrics Useful in Statistics Basic Properties of the Metrics Metric Inequalities Differential Metrics for Parametric Families Fisher Information and Differential Metrics Rao s Geodesic Distances on Distributions Exercises References Empirical Processes and VC Theory Basic Notation and Definitions Classic Asymptotic Properties of the Empirical Process Invariance Principle and Statistical Applications WeightedEmpiricalProcess The Quantile Process Strong Approximations of the Empirical Process Vapnik-Chervonenkis Theory Basic Theory Concrete Examples CLTs for Empirical Measures and Applications Notation and Formulation Entropy Bounds and Specific CLTs Concrete Examples Maximal Inequalities and Symmetrization Connection to the Poisson Process Exercises References VI

8 17 Large Deviations Large Deviations for Sample Means The Cramér-Chernoff Theorem in R Properties of the Rate Function Cramér s Theorem for General Sets The Gärtner-Ellis Theorem The t-statistic Lipschitz Functions and Talagrand s Inequality Large Deviations in Continuous Time ContinuityofaGaussianProcess Metric Entropy of T and Tail of the Supremum Exercises References The Exponential Family and Statistical Applications One Parameter Exponential Family Definition and First Examples The Canonical Form and Basic Properties Convexity Properties Moments and Moment Generating Function Closure Properties Multiparameter Exponential Family Sufficiency and Completeness Neyman-Fisher Factorization and Basu s Theorem Applications of Basu s Theorem to Probability Curved Exponential Family Exercises References Simulation and Markov Chain Monte Carlo The Ordinary Monte Carlo Basic Theory and Examples Monte Carlo P -values Rao-Blackwellization Textbook Simulation Techniques Quantile Transformation and Accept-Reject Importance Sampling and its Asymptotic Properties Optimal Importance Sampling Distribution Algorithms for Simulating from Common Distributions Markov Chain Monte Carlo Reversible Markov Chains Metropolis Algorithms The Gibbs Sampler VII

9 19.5 Convergence of MCMC and Bounds on Errors Spectral Bounds Dobrushin s Inequality and Diaconis-Fill-Stroock Bound Drift and Minorization Methods MCMC on General Spaces General Theory and Metropolis Schemes Convergence Convergence of the Gibbs Sampler Practical Convergence Diagnostics Exercises References Useful Tools for Statistics and Machine Learning The Bootstrap Consistency of the Bootstrap Further Examples Higher Order Accuracy of the Bootstrap Bootstrap for Dependent Data The EM Algorithm The Algorithm and Examples Monotone Ascent and Convergence of EM Modifications of EM Kernels and Classification Smoothing by Kernels Some Common Kernels in Use Kernels for Statistical Classification Reproducing Kernel Hilbert Spaces Mercer s Theorem and Feature Maps Support Vector Machines Exercises References VIII

10 Suggested Courses with Different Themes Duration Theme Chapters 15 weeks Beginning Graduate 2-7, 9 15 weeks Advanced Graduate 7, 8, 10, 11, 12, 13, weeks Special topics for Statistics students 9, 10, 15, 16, 17, 18, weeks Special topics for Computer science students 4, 11, 14, 16, 17, 18, 19 8 weeks Summer course for Statistics students 11, 12, 14, 20 8 weeks Summer course for Computer science students 14, 16, 18, 20 8 weeks Summer course on Modelling and Simulation 4, 10, 13, 19 9