# Introduction. Independent Component Analysis

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Independent Component Analysis 1 Introduction Independent component analysis (ICA) is a method for finding underlying factors or components from multivariate (multidimensional) statistical data. What distinguishes ICA from other methods is that it looks for components that are both statistically independent, and nongaussian. Here we briefly introduce the basic concepts, applications, and estimation principles of ICA. 1.1 LINEAR REPRESENTATION OF MULTIVARIATE DATA The general statistical setting A long-standing problem in statistics and related areas is how to find a suitable representation of multivariate data. Representation here means that we somehow transform the data so that its essential structure is made more visible or accessible. In neural computation, this fundamental problem belongs to the area of unsupervised learning, since the representation must be learned from the data itself without any external input from a supervising teacher. A good representation is also a central goal of many techniques in data mining and exploratory data analysis. In signal processing, the same problem can be found in feature extraction, and also in the source separation problem that will be considered below. Let us assume that the data consists of a number of variables that we have observed together. Let us denote the number of variables by and the number of observations by. We can then denote the data by where the indices take the values and. The dimensions and can be very large. Copyright c 2001 John Wiley & Sons 1 All rights reserved

2 Independent Component Analysis A very general formulation of the problem can be stated as follows: What could be a function from an -dimensional space to an -dimensional space such that the transformed variables give information on the data that is otherwise hidden in the large data set. That is, the transformed variables should be the underlying factors or components that describe the essential structure of the data. It is hoped that these components correspond to some physical causes that were involved in the process that generated the data in the first place. In most cases, we consider linear functions only, because then the interpretation of the representation is simpler, and so is its computation. Thus, every component, say, is expressed as a linear combination of the observed variables: where the for (1.1) are some coefficients that define the representation. The problem can then be rephrased as the problem of determining the coefficients. Using linear algebra, we can express the linear transformation in Eq. (1.1) as a matrix multiplication. Collecting the coefficients in a matrix, the equation becomes.. (1.2) A basic statistical approach consists of considering the as a set of realizations of random variables. Thus each set is a sample of one random variable; let us denote the random variable by. In this framework, we could determine the matrix by the statistical properties of the transformed components. In the following sections, we discuss some statistical properties that could be used; one of them will lead to independent component analysis Dimension reduction methods One statistical principle for choosing the matrix is to limit the number of components to be quite small, maybe only 1 or 2, and to determine so that the contain as much information on the data as possible. This leads to a family of techniques called principal component analysis or factor analysis. In a classic paper, Spearman [409] considered data that consisted of school performance rankings given to schoolchildren in different branches of study, complemented by some laboratory measurements. Spearman then determined by finding a single linear combination such that it explained the maximum amount of the variation in the results. He claimed to find a general factor of intelligence, thus founding factor analysis, and at the same time starting a long controversy in psychology. Copyright c 2001 John Wiley & Sons 2 All rights reserved

3 Independent Component Analysis Fig. 1.1 The density function of the Laplacian distribution, which is a typical supergaussian distribution. For comparison, the gaussian density is given by a dashed line. The Laplacian density has a higher peak at zero, and heavier tails. Both densities are normalized to unit variance and have zero mean Independence as a guiding principle Another principle that has been used for determining is independence: the components should be statistically independent. This means that the value of any one of the components gives no information on the values of the other components. In fact, in factor analysis it is often claimed that the factors are independent, but this is only partly true, because factor analysis assumes that the data has a gaussian distribution. If the data is gaussian, it is simple to find components that are independent, because for gaussian data, uncorrelated components are always independent. In reality, however, the data often does not follow a gaussian distribution, and the situation is not as simple as those methods assume. For example, many real-world data sets have supergaussian distributions. This means that the random variables take relatively more often values that are very close to zero or very large. In other words, the probability density of the data is peaked at zero and has heavy tails (large values far from zero), when compared to a gaussian density of the same variance. An example of such a probability density is shown in Fig This is the starting point of ICA. We want to find statistically independent components, in the general case where the data is nongaussian. 1.2 BLIND SOURCE SEPARATION Let us now look at the same problem of finding a good representation, from a different viewpoint. This is a problem in signal processing that also shows the historical background for ICA. Copyright c 2001 John Wiley & Sons 3 All rights reserved

4 Independent Component Analysis Observing mixtures of unknown signals Consider a situation where there are a number of signals emitted by some physical objects or sources. These physical sources could be, for example, different brain areas emitting electric signals; people speaking in the same room, thus emitting speech signals; or mobile phones emitting their radio waves. Assume further that there are several sensors or receivers. These sensors are in different positions, so that each records a mixture of the original source signals with slightly different weights. For the sake of simplicity of exposition, let us say there are three underlying source signals, and also three observed signals. Denote by and the observed signals, which are the amplitudes of the recorded signals at time point, and by and the original signals. The are then weighted sums of the, where the coefficients depend on the distances between the sources and the sensors: (1.3) The are constant coefficients that give the mixing weights. They are assumed unknown, since we cannot know the values of without knowing all the properties of the physical mixing system, which can be extremely difficult in general. The source signals are unknown as well, since the very problem is that we cannot record them directly. As an illustration, consider the waveforms in Fig These are three linear mixtures of some original source signals. They look as if they were completely noise, but actually, there are some quite structured underlying source signals hidden in these observed signals. What we would like to do is to find the original signals from the mixtures and. This is the blind source separation (BSS) problem. Blind means that we know very little if anything about the original sources. We can safely assume that the mixing coefficients are different enough to make the matrix that they form invertible. Thus there exists a matrix with coefficients, such that we can separate the as (1.4) Such a matrix could be found as the inverse of the matrix that consists of the mixing coefficients in Eq. 1.3, if we knew those coefficients. Now we see that in fact this problem is mathematically similar to the one where we wanted to find a good representation for the random data in, as in (1.2). Indeed, we could consider each signal as a sample of a random variable, so that the value of the random variable is given by the amplitudes of that signal at the time points recorded. Copyright c 2001 John Wiley & Sons 4 All rights reserved

5 Independent Component Analysis Fig. 1.2 signals. The observed signals that are assumed to be mixtures of some underlying source Source separation based on independence The question now is: How can we estimate the coefficients in (1.4)? We want to obtain a general method that works in many different circumstances, and in fact provides one answer to the very general problem that we started with: finding a good representation of multivariate data. Therefore, we use very general statistical properties. All we observe is the signals and, and we want to find a matrix so that the representation is given by the original source signals, and. A surprisingly simple solution to the problem can be found by considering just the statistical independence of the signals. In fact, if the signals are not gaussian, it is enough to determine the coefficients, so that the signals! " # \$% & ' \$% & (1.5)! \$% \$%! \$% \$% are statistically independent. If the signals, and are independent, then they are equal to the original signals #, and. (They could be multiplied by some scalar constants, though, but this has little significance.) Using just this information on the statistical independence, we can in fact estimate the coefficient matrix for the signals in Fig What we obtain are the source signals in Fig (These signals were estimated by the FastICA algorithm that we shall meet in several chapters of this book.) We see that from a data set that seemed to be just noise, we were able to estimate the original source signals, using an algorithm that used the information on the independence only. These estimated signals are indeed equal to those that were used in creating the mixtures in Fig. 1.2 Copyright ( c 2001 John Wiley & Sons 5 All rights reserved

6 Independent Component Analysis Fig. 1.3 The estimates of the original source signals, estimated using only the observed mixture signals in Fig The original signals were found very accurately. (the original signals are not shown, but they really are virtually identical to what the algorithm found). Thus, in the source separation problem, the original signals were the independent components of the data set. 1.3 INDEPENDENT COMPONENT ANALYSIS Definition We have now seen that the problem of blind source separation boils down to finding a linear representation in which the components are statistically independent. In practical situations, we cannot in general find a representation where the components are really independent, but we can at least find components that are as independent as possible. This leads us to the following definition of ICA, which will be considered in more detail in Chapter 7. Given a set of observations of random variables, where is the time or sample index, assume that they are generated as a linear mixture of independent components:!!..! (1.6) where is some unknown matrix. Independent component analysis now consists of estimating both the matrix and the!#", when we only observe the ". Note Copyright \$ c 2001 John Wiley & Sons 6 All rights reserved

7 Independent Component Analysis that we assumed here that the number of independent components is equal to the number of observed variables; this is a simplifying assumption that is not completely necessary. Alternatively, we could define ICA as follows: find a linear transformation given by a matrix as in (1.2), so that the random variables are as independent as possible. This formulation is not really very different from the previous one, since after estimating, its inverse gives. It can be shown (see Section 7.5) that the problem is well-defined, that is, the model in (1.6) can be estimated if and only if the components are nongaussian. This is a fundamental requirement that also explains the main difference between ICA and factor analysis, in which the nongaussianity of the data is not taken into account. In fact, ICA could be considered as nongaussian factor analysis, since in factor analysis, we are also modeling the data as linear mixtures of some underlying factors Applications Due to its generality the ICA model has applications in many different areas, some of which are treated in Part IV. Some examples are: In brain imaging, we often have different sources in the brain emit signals that are mixed up in the sensors outside of the head, just like in the basic blind source separation model (Chapter 22). In econometrics, we often have parallel time series, and ICA could decompose them into independent components that would give an insight to the structure of the data set (Section 24.1). A somewhat different application is in image feature extraction, where we want to find features that are as independent as possible (Chapter 21) How to find the independent components It may be very surprising that the independent components can be estimated from linear mixtures with no more assumptions than their independence. Now we will try to explain briefly why and how this is possible; of course, this is the main subject of the book (especially of Part II). Uncorrelatedness is not enough The first thing to note is that independence is a much stronger property than uncorrelatedness. Considering the blind source separation problem, we could actually find many different uncorrelated representations of the signals that would not be independent and would not separate the sources. Uncorrelatedness in itself is not enough to separate the components. This is also the reason why principal component analysis (PCA) or factor analysis cannot separate the signals: they give components that are uncorrelated, but little more. Copyright c 2001 John Wiley & Sons 7 All rights reserved

8 Independent Component Analysis Fig. 1.4 A sample of independent components and with uniform distributions. Horizontal axis: ; vertical axis:. Fig. 1.5 Uncorrelated mixtures and. Horizontal axis: ; vertical axis:. Let us illustrate this with a simple example using two independent components with uniform distributions, that is, the components can have any values inside a certain interval with equal probability. Data from two such components are plotted in Fig The data is uniformly distributed inside a square due to the independence of the components. Now, Fig. 1.5 shows two uncorrelated mixtures of those independent components. Although the mixtures are uncorrelated, one sees clearly that the distributions are not the same. The independent components are still mixed, using an orthogonal mixing matrix, which corresponds to a rotation of the plane. One can also see that in Fig. 1.5 the components are not independent: if the component on the horizontal axis has a value that is near the corner of the square that is in the extreme right, this clearly restricts the possible values that the components on the vertical axis can have. In fact, by using the well-known decorrelation methods, we can transform any linear mixture of the independent components into uncorrelated components, in which case the mixing is orthogonal (this will be proven in Section 7.4.2). Thus, the trick in ICA is to estimate the orthogonal transformation that is left after decorrelation. This is something that classic methods cannot estimate because they are based on essentially the same covariance information as decorrelation. Figure 1.5 also gives a hint as to why ICA is possible. By locating the edges of the square, we could compute the rotation that gives the original components. In the following, we consider a couple more sophisticated methods for estimating ICA. Nonlinear decorrelation is the basic ICA method One way of stating how independence is stronger than uncorrelatedness is to say that independence implies nonlinear uncorrelatedness: If and are independent, then any nonlinear transformations and are uncorrelated (in the sense that their covariance is Copyright c 2001 John Wiley & Sons 8 All rights reserved

9 Independent Component Analysis zero). In contrast, for two random variables that are merely uncorrelated, such nonlinear transformations do not have zero covariance in general. Thus, we could attempt to perform ICA by a stronger form of decorrelation, by finding a representation where the are uncorrelated even after some nonlinear transformations. This gives a simple principle of estimating the matrix : ICA estimation principle 1: Nonlinear decorrelation. Find the matrix for any so that, the components and are uncorrelated, and the transformed components and are uncorrelated, where and are some suitable nonlinear functions. This is a valid approach to estimating ICA: If the nonlinearities are properly chosen, the method does find the independent components. In fact, computing nonlinear correlations between the two mixtures in Fig. 1.5, one would immediately see that the mixtures are not independent. Although this principle is very intuitive, it leaves open an important question: How should the nonlinearities and be chosen? Answers to this question can be found be using principles from estimation theory and information theory. Estimation theory provides the most classic method of estimating any statistical model: the maximum likelihood method (Chapter 9). Information theory provides exact measures of independence, such as mutual information (Chapter 10). Using either one of these theories, we can determine the nonlinear functions and in a satisfactory way. Independent components are the maximally nongaussian components Another very intuitive and important principle of ICA estimation is maximum nongaussianity (Chapter 8). The idea is that according to the central limit theorem, sums of nongaussian random variables are closer to gaussian that the original ones. Therefore, if we take a linear combination of the observed mixture variables (which, because of the linear mixing model, is a linear combination of the independent components as well), this will be maximally nongaussian if it equals one of the independent components. This is because if it were a real mixture of two or more components, it would be closer to a gaussian distribution, due to the central limit theorem. Thus, the principle can be stated as follows ICA estimation principle 2: Maximum nongaussianity. Find the local maxima of nongaussianity of a linear combination under the constraint that the variance of is constant. Each local maximum gives one independent component. To measure nongaussianity in practice, we could use, for example, the kurtosis. Kurtosis is a higher-order cumulant, which are some kind of generalizations of variance using higher-order polynomials. Cumulants have interesting algebraic and statistical properties which is why they have an important part in the theory of ICA. For example, comparing the nongaussianities of the components given by the axes in Figs. 1.4 and 1.5, we see that in Fig. 1.5 they are smaller, and thus Fig. 1.5 cannot give the independent components (see Chapter 8). An interesting point is that this principle of maximum nongaussianity shows the very close connection between ICA and an independently developed technique Copyright c 2001 John Wiley & Sons 9 All rights reserved

10 Independent Component Analysis called projection pursuit. In projection pursuit, we are actually looking for maximally nongaussian linear combinations, which are used for visualization and other purposes. Thus, the independent components can be interpreted as projection pursuit directions. When ICA is used to extract features, this principle of maximum nongaussianity also shows an important connection to sparse coding that has been used in neuroscientific theories of feature extraction (Chapter 21). The idea in sparse coding is to represent data with components so that only a small number of them are active at the same time. It turns out that this is equivalent, in some situations, to finding components that are maximally nongaussian. The projection pursuit and sparse coding connections are related to a deep result that says that ICA gives a linear representation that is as structured as possible. This statement can be given a rigorous meaning by information-theoretic concepts (Chapter 10), and shows that the independent components are in many ways easier to process than the original random variables. In particular, independent components are easier to code (compress) than the original variables. ICA estimation needs more than covariances There are many other methods for estimating the ICA model as well. Many of them will be treated in this book. What they all have in common is that they consider some statistics that are not contained in the covariance matrix (the matrix that contains the covariances between all pairs of the ). Using the covariance matrix, we can decorrelate the components in the ordinary linear sense, but not any stronger. Thus, all the ICA methods use some form of higher-order statistics, which specifically means information not contained in the covariance matrix. Earlier, we encountered two kinds of higher-order information: the nonlinear correlations and kurtosis. Many other kinds can be used as well. Numerical methods are important In addition to the estimation principle, one has to find an algorithm for implementing the computations needed. Because the estimation principles use nonquadratic functions, the computations needed usually cannot be expressed using simple linear algebra, and therefore they can be quite demanding. Numerical algorithms are thus an integral part of ICA estimation methods. The numerical methods are typically based on optimization of some objective functions. The basic optimization method is the gradient method. Of particular interest is a fixed-point algorithm called FastICA that has been tailored to exploit the particular structure of the ICA problem. For example, we could use both of these methods to find the maxima of the nongaussianity as measured by the absolute value of kurtosis. Copyright c 2001 John Wiley & Sons 10 All rights reserved

11 Independent Component Analysis 1.4 HISTORY OF ICA The technique of ICA, although not yet the name, was introduced in the early 1980s by J. Hérault, C. Jutten, and B. Ans [178, 179, 16]. As recently reviewed by Jutten [227], the problem first came up in 1982 in a neurophysiological setting. In a simplified model of motion coding in muscle contraction, the outputs and were two types of sensory signals measuring muscle contraction, and and were the angular position and velocity of a moving joint. Then it is not unreasonable to assume that the ICA model holds between these signals. The nervous system must be somehow able to infer the position and velocity signals from the measured responses. One possibility for this is to learn the inverse model using the nonlinear decorrelation principle in a simple neural network. Hérault and Jutten proposed a specific feedback circuit to solve the problem. This approach is covered in Chapter 12. All through the 1980s, ICA was mostly known among French researchers, with limited influence internationally. The few ICA presentations in international neural network conferences in the mid-1980s were largely buried under the deluge of interest in back-propagation, Hopfield networks, and Kohonen s Self-Organizing Map (SOM), which were actively propagated in those times. Another related field was higher-order spectral analysis, on which the first international workshop was organized in In this workshop, early papers on ICA by J.-F. Cardoso [60] and P. Comon [88] were given. Cardoso used algebraic methods, especially higher-order cumulant tensors, which eventually led to the JADE algorithm [72]. The use of fourth-order cumulants has been earlier proposed by J.-L. Lacoume [254]. In signal processing literature, classic early papers by the French group are [228, 93, 408, 89]. A good source with historical accounts and a more complete list of references is [227]. In signal processing, there had been earlier approaches in the related problem of blind signal deconvolution [114, 398]. In particular, the results used in multichannel blind deconvolution are very similar to ICA techniques. The work of the scientists in the 1980 s was extended by, among others, A. Cichocki and R. Unbehauen, who were the first to propose one of the presently most popular ICA algorithms [82, 85, 84]. Some other papers on ICA and signal separation from early 1990s are [57, 314]. The nonlinear PCA approach was introduced by the present authors in [332, 232]. However, until the mid-1990s, ICA remained a rather small and narrow research effort. Several algorithms were proposed that worked, usually in somewhat restricted problems, but it was not until later that the rigorous connections of these to statistical optimization criteria were exposed. ICA attained wider attention and growing interest after A.J. Bell and T.J. Sejnowski published their approach based on the infomax principle [35, 36] in the mid-90 s. This algorithm was further refined by S.-I. Amari and his co-workers using the natural gradient [12], and its fundamental connections to maximum likelihood estimation, as well as to the Cichocki-Unbehauen algorithm, were established. A couple of years later, the present authors presented the fixed-point or FastICA algorithm, [210, 192, Copyright c 2001 John Wiley & Sons 11 All rights reserved

12 Independent Component Analysis 197], which has contributed to the application of ICA to large-scale problems due to its computational efficiency. Since the mid-1990s, there has been a growing wave of papers, workshops, and special sessions devoted to ICA. The first international workshop on ICA was held in Aussois, France, in January 1999, and the second workshop followed in June 2000 in Helsinki, Finland. Both gathered more than 100 researchers working on ICA and blind signal separation, and contributed to the transformation of ICA to an established and mature field of research. Copyright c 2001 John Wiley & Sons 12 All rights reserved

### Nonlinear Blind Source Separation and Independent Component Analysis

Nonlinear Blind Source Separation and Independent Component Analysis Prof. Juha Karhunen Helsinki University of Technology Neural Networks Research Centre Espoo, Finland Helsinki University of Technology,

### Component Ordering in Independent Component Analysis Based on Data Power

Component Ordering in Independent Component Analysis Based on Data Power Anne Hendrikse Raymond Veldhuis University of Twente University of Twente Fac. EEMCS, Signals and Systems Group Fac. EEMCS, Signals

### An Introduction to Independent Component Analysis: InfoMax and FastICA algorithms

Tutorials in Quantitative Methods for Psychology 2010, Vol. 6(1), p. 31-38. An Introduction to Independent Component Analysis: InfoMax and FastICA algorithms Dominic Langlois, Sylvain Chartier, and Dominique

### Independent Component Analysis: Algorithms and Applications

Independent Component Analysis: Algorithms and Applications Aapo Hyvärinen and Erkki Oja Neural Networks Research Centre Helsinki University of Technology P.O. Box 5400, FIN-02015 HUT, Finland Neural Networks,

### Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches

Modelling, Extraction and Description of Intrinsic Cues of High Resolution Satellite Images: Independent Component Analysis based approaches PhD Thesis by Payam Birjandi Director: Prof. Mihai Datcu Problematic

### Independent component ordering in ICA time series analysis

Neurocomputing 41 (2001) 145}152 Independent component ordering in ICA time series analysis Yiu-ming Cheung*, Lei Xu Department of Computer Science and Engineering, The Chinese University of Hong Kong,

### Introduction to Principal Components and FactorAnalysis

Introduction to Principal Components and FactorAnalysis Multivariate Analysis often starts out with data involving a substantial number of correlated variables. Principal Component Analysis (PCA) is a

### The Classical Linear Regression Model

The Classical Linear Regression Model 1 September 2004 A. A brief review of some basic concepts associated with vector random variables Let y denote an n x l vector of random variables, i.e., y = (y 1,

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### EM Clustering Approach for Multi-Dimensional Analysis of Big Data Set

EM Clustering Approach for Multi-Dimensional Analysis of Big Data Set Amhmed A. Bhih School of Electrical and Electronic Engineering Princy Johnson School of Electrical and Electronic Engineering Martin

### Lecture 9: Continuous

CSC2515 Fall 2007 Introduction to Machine Learning Lecture 9: Continuous Latent Variable Models 1 Example: continuous underlying variables What are the intrinsic latent dimensions in these two datasets?

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Dimensionality Reduction: Principal Components Analysis

Dimensionality Reduction: Principal Components Analysis In data mining one often encounters situations where there are a large number of variables in the database. In such situations it is very likely

### CS 2750 Machine Learning. Lecture 1. Machine Learning. http://www.cs.pitt.edu/~milos/courses/cs2750/ CS 2750 Machine Learning.

Lecture Machine Learning Milos Hauskrecht milos@cs.pitt.edu 539 Sennott Square, x5 http://www.cs.pitt.edu/~milos/courses/cs75/ Administration Instructor: Milos Hauskrecht milos@cs.pitt.edu 539 Sennott

### Probability and Statistics

CHAPTER 2: RANDOM VARIABLES AND ASSOCIATED FUNCTIONS 2b - 0 Probability and Statistics Kristel Van Steen, PhD 2 Montefiore Institute - Systems and Modeling GIGA - Bioinformatics ULg kristel.vansteen@ulg.ac.be

### Chapter 15 Introduction to Linear Programming

Chapter 15 Introduction to Linear Programming An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Brief History of Linear Programming The goal of linear programming is to determine the values of

### Review Jeopardy. Blue vs. Orange. Review Jeopardy

Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round \$200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?

### Feature Extraction and Selection. More Info == Better Performance? Curse of Dimensionality. Feature Space. High-Dimensional Spaces

More Info == Better Performance? Feature Extraction and Selection APR Course, Delft, The Netherlands Marco Loog Feature Space Curse of Dimensionality A p-dimensional space, in which each dimension is a

### Summary of Probability

Summary of Probability Mathematical Physics I Rules of Probability The probability of an event is called P(A), which is a positive number less than or equal to 1. The total probability for all possible

### ELEC-E8104 Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems

Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems Minimum Mean Square Error (MMSE) MMSE estimation of Gaussian random vectors Linear MMSE estimator for arbitrarily distributed

### CORRELATED TO THE SOUTH CAROLINA COLLEGE AND CAREER-READY FOUNDATIONS IN ALGEBRA

We Can Early Learning Curriculum PreK Grades 8 12 INSIDE ALGEBRA, GRADES 8 12 CORRELATED TO THE SOUTH CAROLINA COLLEGE AND CAREER-READY FOUNDATIONS IN ALGEBRA April 2016 www.voyagersopris.com Mathematical

### Bayesian probability theory

Bayesian probability theory Bruno A. Olshausen arch 1, 2004 Abstract Bayesian probability theory provides a mathematical framework for peforming inference, or reasoning, using probability. The foundations

### Solving Simultaneous Equations and Matrices

Solving Simultaneous Equations and Matrices The following represents a systematic investigation for the steps used to solve two simultaneous linear equations in two unknowns. The motivation for considering

### SPECIAL PERTURBATIONS UNCORRELATED TRACK PROCESSING

AAS 07-228 SPECIAL PERTURBATIONS UNCORRELATED TRACK PROCESSING INTRODUCTION James G. Miller * Two historical uncorrelated track (UCT) processing approaches have been employed using general perturbations

### Principal Component Analysis Application to images

Principal Component Analysis Application to images Václav Hlaváč Czech Technical University in Prague Faculty of Electrical Engineering, Department of Cybernetics Center for Machine Perception http://cmp.felk.cvut.cz/

### The General Linear Model: Theory

Gregory Carey, 1998 General Linear Model - 1 The General Linear Model: Theory 1.0 Introduction In the discussion of multiple regression, we used the following equation to express the linear model for a

### Chapter R - Basic Algebra Operations (69 topics, due on 05/01/12)

Course Name: College Algebra 001 Course Code: R3RK6-CTKHJ ALEKS Course: College Algebra with Trigonometry Instructor: Prof. Bozyk Course Dates: Begin: 01/17/2012 End: 05/04/2012 Course Content: 288 topics

### Nonlinear Iterative Partial Least Squares Method

Numerical Methods for Determining Principal Component Analysis Abstract Factors Béchu, S., Richard-Plouet, M., Fernandez, V., Walton, J., and Fairley, N. (2016) Developments in numerical treatments for

### BLIND SOURCE SEPARATION OF SPEECH AND BACKGROUND MUSIC FOR IMPROVED SPEECH RECOGNITION

BLIND SOURCE SEPARATION OF SPEECH AND BACKGROUND MUSIC FOR IMPROVED SPEECH RECOGNITION P. Vanroose Katholieke Universiteit Leuven, div. ESAT/PSI Kasteelpark Arenberg 10, B 3001 Heverlee, Belgium Peter.Vanroose@esat.kuleuven.ac.be

### SIGNAL PROCESSING & SIMULATION NEWSLETTER

1 of 10 1/25/2008 3:38 AM SIGNAL PROCESSING & SIMULATION NEWSLETTER Note: This is not a particularly interesting topic for anyone other than those who ar e involved in simulation. So if you have difficulty

### The Method of Least Squares

Hervé Abdi 1 1 Introduction The least square methods (LSM) is probably the most popular technique in statistics. This is due to several factors. First, most common estimators can be casted within this

### 6.5. Log-linear Graphs. Introduction. Prerequisites. Learning Outcomes. Learning Style

Log-linear Graphs 6.5 Introduction In this block we employ our knowledge of logarithms to simplify plotting the relation between one variable and another. In particular we consider those situations in

### A Introduction to Matrix Algebra and Principal Components Analysis

A Introduction to Matrix Algebra and Principal Components Analysis Multivariate Methods in Education ERSH 8350 Lecture #2 August 24, 2011 ERSH 8350: Lecture 2 Today s Class An introduction to matrix algebra

### 10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES

55 CHAPTER NUMERICAL METHODS. POWER METHOD FOR APPROXIMATING EIGENVALUES In Chapter 7 we saw that the eigenvalues of an n n matrix A are obtained by solving its characteristic equation n c n n c n n...

### Course Name: Course Code: ALEKS Course: Instructor: Course Dates: Course Content: Textbook: Dates Objective Prerequisite Topics

Course Name: MATH 1204 Fall 2015 Course Code: N/A ALEKS Course: College Algebra Instructor: Master Templates Course Dates: Begin: 08/22/2015 End: 12/19/2015 Course Content: 271 Topics (261 goal + 10 prerequisite)

### Multivariate Analysis (Slides 13)

Multivariate Analysis (Slides 13) The final topic we consider is Factor Analysis. A Factor Analysis is a mathematical approach for attempting to explain the correlation between a large set of variables

### Speech Synthesis by Artificial Neural Networks (AI / Speech processing / Signal processing)

Speech Synthesis by Artificial Neural Networks (AI / Speech processing / Signal processing) Christos P. Yiakoumettis Department of Informatics University of Sussex, UK (Email: c.yiakoumettis@sussex.ac.uk)

### T-test & factor analysis

Parametric tests T-test & factor analysis Better than non parametric tests Stringent assumptions More strings attached Assumes population distribution of sample is normal Major problem Alternatives Continue

### Unknown n sensors x(t)

Appeared in journal: Neural Network World Vol.6, No.4, 1996, pp.515{523. Published by: IDG Co., Prague, Czech Republic. LOCAL ADAPTIVE LEARNING ALGORITHMS FOR BLIND SEPARATION OF NATURAL IMAGES Andrzej

### Infomax Algorithm for Filtering Airwaves in the Field of Seabed Logging

Research Journal of Applied Sciences, Engineering and Technology 7(14): 2914-2920, 2014 ISSN: 2040-7459; e-issn: 2040-7467 Maxwell Scientific Organization, 2014 Submitted: May 16, 2013 Accepted: June 08,

### DATA ANALYTICS USING R

DATA ANALYTICS USING R Duration: 90 Hours Intended audience and scope: The course is targeted at fresh engineers, practicing engineers and scientists who are interested in learning and understanding data

### , then the form of the model is given by: which comprises a deterministic component involving the three regression coefficients (

Multiple regression Introduction Multiple regression is a logical extension of the principles of simple linear regression to situations in which there are several predictor variables. For instance if we

### Regression Using Support Vector Machines: Basic Foundations

Regression Using Support Vector Machines: Basic Foundations Technical Report December 2004 Aly Farag and Refaat M Mohamed Computer Vision and Image Processing Laboratory Electrical and Computer Engineering

### Lecture 21: The Inverse of a Matrix

Lecture 21: The Inverse of a Matrix Winfried Just, Ohio University October 16, 2015 Review: Our chemical reaction system Recall our chemical reaction system A + 2B 2C A + B D A + 2C 2D B + D 2C If we write

### Section 1.1. Introduction to R n

The Calculus of Functions of Several Variables Section. Introduction to R n Calculus is the study of functional relationships and how related quantities change with each other. In your first exposure to

### NEW YORK STATE TEACHER CERTIFICATION EXAMINATIONS

NEW YORK STATE TEACHER CERTIFICATION EXAMINATIONS TEST DESIGN AND FRAMEWORK September 2014 Authorized for Distribution by the New York State Education Department This test design and framework document

### 6.2.8 Neural networks for data mining

6.2.8 Neural networks for data mining Walter Kosters 1 In many application areas neural networks are known to be valuable tools. This also holds for data mining. In this chapter we discuss the use of neural

### Generalized Neutral Portfolio

Generalized Neutral Portfolio Sandeep Bhutani bhutanister@gmail.com December 10, 2009 Abstract An asset allocation framework decomposes the universe of asset returns into factors either by Fundamental

### Applications to Data Smoothing and Image Processing I

Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is

### Time Domain and Frequency Domain Techniques For Multi Shaker Time Waveform Replication

Time Domain and Frequency Domain Techniques For Multi Shaker Time Waveform Replication Thomas Reilly Data Physics Corporation 1741 Technology Drive, Suite 260 San Jose, CA 95110 (408) 216-8440 This paper

### ANALYSIS AND APPLICATIONS OF LAPLACE /FOURIER TRANSFORMATIONS IN ELECTRIC CIRCUIT

www.arpapress.com/volumes/vol12issue2/ijrras_12_2_22.pdf ANALYSIS AND APPLICATIONS OF LAPLACE /FOURIER TRANSFORMATIONS IN ELECTRIC CIRCUIT M. C. Anumaka Department of Electrical Electronics Engineering,

### Natural Image Statistics

Aapo Hyvärinen Jarmo Hurri Patrik O. Hoyer Natural Image Statistics A probabilistic approach to early computational vision February 27, 2009 Springer Contents overview 1 Introduction...................................................

### Overview of Factor Analysis

Overview of Factor Analysis Jamie DeCoster Department of Psychology University of Alabama 348 Gordon Palmer Hall Box 870348 Tuscaloosa, AL 35487-0348 Phone: (205) 348-4431 Fax: (205) 348-8648 August 1,

### Linear Equations and Inverse Matrices

Chapter 4 Linear Equations and Inverse Matrices 4. Two Pictures of Linear Equations The central problem of linear algebra is to solve a system of equations. Those equations are linear, which means that

### MUSICAL INSTRUMENT FAMILY CLASSIFICATION

MUSICAL INSTRUMENT FAMILY CLASSIFICATION Ricardo A. Garcia Media Lab, Massachusetts Institute of Technology 0 Ames Street Room E5-40, Cambridge, MA 039 USA PH: 67-53-0 FAX: 67-58-664 e-mail: rago @ media.

### Higher Education Math Placement

Higher Education Math Placement Placement Assessment Problem Types 1. Whole Numbers, Fractions, and Decimals 1.1 Operations with Whole Numbers Addition with carry Subtraction with borrowing Multiplication

### Module 5: Multiple Regression Analysis

Using Statistical Data Using to Make Statistical Decisions: Data Multiple to Make Regression Decisions Analysis Page 1 Module 5: Multiple Regression Analysis Tom Ilvento, University of Delaware, College

### FEGYVERNEKI SÁNDOR, PROBABILITY THEORY AND MATHEmATICAL

FEGYVERNEKI SÁNDOR, PROBABILITY THEORY AND MATHEmATICAL STATIsTICs 4 IV. RANDOm VECTORs 1. JOINTLY DIsTRIBUTED RANDOm VARIABLEs If are two rom variables defined on the same sample space we define the joint

### 1 Short Introduction to Time Series

ECONOMICS 7344, Spring 202 Bent E. Sørensen January 24, 202 Short Introduction to Time Series A time series is a collection of stochastic variables x,.., x t,.., x T indexed by an integer value t. The

### CS 2750 Machine Learning. Lecture 1. Machine Learning. CS 2750 Machine Learning.

Lecture 1 Machine Learning Milos Hauskrecht milos@cs.pitt.edu 539 Sennott Square, x-5 http://www.cs.pitt.edu/~milos/courses/cs75/ Administration Instructor: Milos Hauskrecht milos@cs.pitt.edu 539 Sennott

### 4.1 Learning algorithms for neural networks

4 Perceptron Learning 4.1 Learning algorithms for neural networks In the two preceding chapters we discussed two closely related models, McCulloch Pitts units and perceptrons, but the question of how to

### L10: Probability, statistics, and estimation theory

L10: Probability, statistics, and estimation theory Review of probability theory Bayes theorem Statistics and the Normal distribution Least Squares Error estimation Maximum Likelihood estimation Bayesian

### Independent Component Analysis and Its Applications Tzyy-Ping Jung

Independent Component Analysis and Its Applications Tzyy-Ping Jung Swartz Center for Computational Neuroscience Institute for Neural Computation University of California, San Diego & Department of Computer

### C A R I B B E A N E X A M I N A T I O N S C O U N C I L REPORT ON CANDIDATES WORK IN THE SECONDARY EDUCATION CERTIFICATE EXAMINATION MAY/JUNE 2011

C A R I B B E A N E X A M I N A T I O N S C O U N C I L REPORT ON CANDIDATES WORK IN THE SECONDARY EDUCATION CERTIFICATE EXAMINATION MAY/JUNE 2011 MATHEMATICS GENERAL PROFICIENCY EXAMINATION Copyright

### Face Recognition using Principle Component Analysis

Face Recognition using Principle Component Analysis Kyungnam Kim Department of Computer Science University of Maryland, College Park MD 20742, USA Summary This is the summary of the basic idea about PCA

### PSS 27.2 The Electric Field of a Continuous Distribution of Charge

Chapter 27 Solutions PSS 27.2 The Electric Field of a Continuous Distribution of Charge Description: Knight Problem-Solving Strategy 27.2 The Electric Field of a Continuous Distribution of Charge is illustrated.

### The basic unit in matrix algebra is a matrix, generally expressed as: a 11 a 12. a 13 A = a 21 a 22 a 23

(copyright by Scott M Lynch, February 2003) Brief Matrix Algebra Review (Soc 504) Matrix algebra is a form of mathematics that allows compact notation for, and mathematical manipulation of, high-dimensional

### 1111: Linear Algebra I

1111: Linear Algebra I Dr. Vladimir Dotsenko (Vlad) Lecture 3 Dr. Vladimir Dotsenko (Vlad) 1111: Linear Algebra I Lecture 3 1 / 12 Vector product and volumes Theorem. For three 3D vectors u, v, and w,

### How to report the percentage of explained common variance in exploratory factor analysis

UNIVERSITAT ROVIRA I VIRGILI How to report the percentage of explained common variance in exploratory factor analysis Tarragona 2013 Please reference this document as: Lorenzo-Seva, U. (2013). How to report

### 11/20/2014. Correlational research is used to describe the relationship between two or more naturally occurring variables.

Correlational research is used to describe the relationship between two or more naturally occurring variables. Is age related to political conservativism? Are highly extraverted people less afraid of rejection

### Simple Regression Theory II 2010 Samuel L. Baker

SIMPLE REGRESSION THEORY II 1 Simple Regression Theory II 2010 Samuel L. Baker Assessing how good the regression equation is likely to be Assignment 1A gets into drawing inferences about how close the

Factor Analysis Advanced Financial Accounting II Åbo Akademi School of Business Factor analysis A statistical method used to describe variability among observed variables in terms of fewer unobserved variables

### Problem Set 4: Covariance functions and Gaussian random fields

Problem Set : Covariance functions and Gaussian random fields GEOS 7: Inverse Problems and Parameter Estimation, Carl Tape Assigned: February 9, 15 Due: February 1, 15 Last compiled: February 5, 15 Overview

### Advanced 3G and 4G Wireless Communication Prof. Aditya K. Jagannatham Department of Electrical Engineering Indian Institute of Technology, Kanpur

Advanced 3G and 4G Wireless Communication Prof. Aditya K. Jagannatham Department of Electrical Engineering Indian Institute of Technology, Kanpur Lecture - 3 Rayleigh Fading and BER of Wired Communication

### SCHOOL DISTRICT OF THE CHATHAMS CURRICULUM PROFILE

CONTENT AREA(S): Mathematics COURSE/GRADE LEVEL(S): Honors Algebra 2 (10/11) I. Course Overview In Honors Algebra 2, the concept of mathematical function is developed and refined through the study of real

### Random Projection-based Multiplicative Data Perturbation for Privacy Preserving Distributed Data Mining

Random Projection-based Multiplicative Data Perturbation for Privacy Preserving Distributed Data Mining Kun Liu Hillol Kargupta and Jessica Ryan Abstract This paper explores the possibility of using multiplicative

### LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12,

LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, 2000 45 4 Iterative methods 4.1 What a two year old child can do Suppose we want to find a number x such that cos x = x (in radians). This is a nonlinear

### Solutions to Exam in Speech Signal Processing EN2300

Solutions to Exam in Speech Signal Processing EN23 Date: Thursday, Dec 2, 8: 3: Place: Allowed: Grades: Language: Solutions: Q34, Q36 Beta Math Handbook (or corresponding), calculator with empty memory.

### Numerical Methods Lecture 2 Simultaneous Equations

Numerical Methods Lecture 2 Simultaneous Equations Topics: matrix operations solving systems of equations Matrix operations: Mathcad is designed to be a tool for quick and easy manipulation of matrix forms

### Understanding and Applying Kalman Filtering

Understanding and Applying Kalman Filtering Lindsay Kleeman Department of Electrical and Computer Systems Engineering Monash University, Clayton 1 Introduction Objectives: 1. Provide a basic understanding

### Common Core Unit Summary Grades 6 to 8

Common Core Unit Summary Grades 6 to 8 Grade 8: Unit 1: Congruence and Similarity- 8G1-8G5 rotations reflections and translations,( RRT=congruence) understand congruence of 2 d figures after RRT Dilations

### Introduction to Matrix Algebra

Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary

### 3.4. Solving Simultaneous Linear Equations. Introduction. Prerequisites. Learning Outcomes

Solving Simultaneous Linear Equations 3.4 Introduction Equations often arise in which there is more than one unknown quantity. When this is the case there will usually be more than one equation involved.

### Introduction to machine learning and pattern recognition Lecture 1 Coryn Bailer-Jones

Introduction to machine learning and pattern recognition Lecture 1 Coryn Bailer-Jones http://www.mpia.de/homes/calj/mlpr_mpia2008.html 1 1 What is machine learning? Data description and interpretation

### Rachel J. Goldberg, Guideline Research/Atlanta, Inc., Duluth, GA

PROC FACTOR: How to Interpret the Output of a Real-World Example Rachel J. Goldberg, Guideline Research/Atlanta, Inc., Duluth, GA ABSTRACT THE METHOD This paper summarizes a real-world example of a factor

### Ordered Pairs. Graphing Lines and Linear Inequalities, Solving System of Linear Equations. Cartesian Coordinates System.

Ordered Pairs Graphing Lines and Linear Inequalities, Solving System of Linear Equations Peter Lo All equations in two variables, such as y = mx + c, is satisfied only if we find a value of x and a value

### Linear Programming. Solving LP Models Using MS Excel, 18

SUPPLEMENT TO CHAPTER SIX Linear Programming SUPPLEMENT OUTLINE Introduction, 2 Linear Programming Models, 2 Model Formulation, 4 Graphical Linear Programming, 5 Outline of Graphical Procedure, 5 Plotting

### Lecture 8: Signal Detection and Noise Assumption

ECE 83 Fall Statistical Signal Processing instructor: R. Nowak, scribe: Feng Ju Lecture 8: Signal Detection and Noise Assumption Signal Detection : X = W H : X = S + W where W N(, σ I n n and S = [s, s,...,

### Prentice Hall Mathematics: Algebra 1 2007 Correlated to: Michigan Merit Curriculum for Algebra 1

STRAND 1: QUANTITATIVE LITERACY AND LOGIC STANDARD L1: REASONING ABOUT NUMBERS, SYSTEMS, AND QUANTITATIVE SITUATIONS Based on their knowledge of the properties of arithmetic, students understand and reason

### A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution

A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution PSYC 943 (930): Fundamentals of Multivariate Modeling Lecture 4: September

### Methodology for Emulating Self Organizing Maps for Visualization of Large Datasets

Methodology for Emulating Self Organizing Maps for Visualization of Large Datasets Macario O. Cordel II and Arnulfo P. Azcarraga College of Computer Studies *Corresponding Author: macario.cordel@dlsu.edu.ph

### ALGEBRA I A PLUS COURSE OUTLINE

ALGEBRA I A PLUS COURSE OUTLINE OVERVIEW: 1. Operations with Real Numbers 2. Equation Solving 3. Word Problems 4. Inequalities 5. Graphs of Functions 6. Linear Functions 7. Scatterplots and Lines of Best

### Quick Reference Guide to Linear Algebra in Quantum Mechanics

Quick Reference Guide to Linear Algebra in Quantum Mechanics Scott N. Walck September 2, 2014 Contents 1 Complex Numbers 2 1.1 Introduction............................ 2 1.2 Real Numbers...........................

### 3.4. Solving simultaneous linear equations. Introduction. Prerequisites. Learning Outcomes

Solving simultaneous linear equations 3.4 Introduction Equations often arise in which there is more than one unknown quantity. When this is the case there will usually be more than one equation involved.

### Pythagorean Theorem. Overview. Grade 8 Mathematics, Quarter 3, Unit 3.1. Number of instructional days: 15 (1 day = minutes) Essential questions

Grade 8 Mathematics, Quarter 3, Unit 3.1 Pythagorean Theorem Overview Number of instructional days: 15 (1 day = 45 60 minutes) Content to be learned Prove the Pythagorean Theorem. Given three side lengths,

### Factor analysis. Angela Montanari

Factor analysis Angela Montanari 1 Introduction Factor analysis is a statistical model that allows to explain the correlations between a large number of observed correlated variables through a small number

### Machine Learning and Data Mining. Regression Problem. (adapted from) Prof. Alexander Ihler

Machine Learning and Data Mining Regression Problem (adapted from) Prof. Alexander Ihler Overview Regression Problem Definition and define parameters ϴ. Prediction using ϴ as parameters Measure the error

### Stochastic Doppler shift and encountered wave period distributions in Gaussian waves

Ocean Engineering 26 (1999) 507 518 Stochastic Doppler shift and encountered wave period distributions in Gaussian waves G. Lindgren a,*, I. Rychlik a, M. Prevosto b a Department of Mathematical Statistics,