# Online k-means Clustering of Nonstationary Data

Save this PDF as:

Size: px
Start display at page:

Download "Online k-means Clustering of Nonstationary Data"

## Transcription

1 Prediction Proect Report Online -Means Clustering of Nonstationary Data Angie King May 17, 2012

2 1 Introduction and Motivation Machine learning algorithms often assume we have all our data before we try to learn from it; however, we often need to learn from data as we gather it over time. In my proect I will focus on the case where we want to learn by clustering. The setting where we have all the data ahead of time is called batch clustering ; here we will be interested in online clustering. Specifically, we will investigate algorithms for online clustering when the data is non-stationary. Consider a motivating example of a t-shirt retailer that receives online data about their sales. The retailer sells women s, men s, girls and boys t-shirts. Fashion across these groups is different; the colors, styles, and prices that are popular in women s t-shirts are quite different from those that are popular in boys t-shirts. Therefore the retailer would lie to find the average set of characteristics for each demographic in order to maret appropriately to each. However, as fashion changes over time, so do the characteristics of each demographic. At every point in time, the retailer would lie to be able to segment its maret data into correctly identified clusters and find the appropriate current cluster average. This is a problem of online clustering of non-stationary data. More formally, consider d- dimensional data points which arrive online and need to be clustered. While the cluster should remain intact, its center essentially shifts in d-space over time. In this paper I will argue why the standard -means cost obective is unsuitable in the case of non-stationary data, and will propose an alternative cost obective. I will discuss a standard algorithm for online clustering, and its shortcomings when data is non-stationary. I will introduce a simple variant of this algorithm which taes into account nonstationarity, and will compare the performance of these algorithms with respect to the optimal clustering for a simulated data set. I will then discuss performance guarantees, and provide a practical, rather than a theoretical, way of measuring an online clustering algorithm s performance. 2 Bacground and Literature Review Before we delve into online clustering of time-varying data, we will build a baseline for this problem by providing bacground and reviewing relevant literature. As in the t-shirt example, the entire data set to be processed may not be available when we need to begin learning; indeed, the data in the t-shirt example is not even finite. There are several modes in which the data may be available and we now define precisely what those are. In batch mode, a finite dataset is available from the beginning. This is the most commonly analyzed setting. Streaming mode refers to a setting where there is a finite amount of data to be processed but it arrives one data point at a time, and the entire dataset cannot be stored in memory. The online setting, which we will be concerned with here, departs from the previous two settings in that the there is an infinite amount of data to be analyzed, and 1

3 it arrives one data point at a time. Since there is an infinite amount of data, it is impossible to store all the data in memory. When clustering data in the batch setting, several natural obectives present themselves. The -center obective is to minimize the maximum distance from any point to its cluster s center. The -means obective is to minimize the mean squared distance from all points to their respective cluster centers. The -median obective is to minimize the distance from all points to their respective cluster centers. Depending on the data being analyzed, different obectives are appropriate in different scenarios. In our case we will focus on the -means obective. Even in the batch setting, finding the optimal -means clustering is an NP-hard problem [1]. Therefore heuristics are often used. The most common heuristic is often simply called the -means algorithm, however we will refer to it here as Lloyd s algorithm [7] to avoid confusion between the algorithm and the -clustering obective. In the batch setting, an algorithm s performance can be compared directly to the optimal clustering as measured with respect to the -means obective. Lloyd s algorithm, which is the most commonly used heuristic, can perform arbitrarily badly with respect to the cost of the optimal clustering [8]. However, there exist other heuristics for the -means obective which have a performance guarantee; for example, -means++ is a heuristic with an O(log- approximation for the cost of the optimal clustering [2]. Another algorithm, developed by Kanungo et al. in [6], gives a constant factor 50-approximation to the cost of the optimal clustering. However, in the online setting, it is much less clear how to analyze an algorithm s performance. In [5], Dasgupta proposed two possible methods for evaluating the performance of an online clustering scheme. The first is an approximation algorithm style bound: if at time t an algorithm sees a new data point x t and then outputs a set of centers Z t then an approximation algorithm would find some α 1 such that Cost(Z t α OPT, where OPT is the cost for the best centers for x 1,..., x t. Another type of performance metric is regret, which is a common framewor in online learning. Under a regret framewor, an algorithm would announce a set of centers Z t at time t, then recieve a new data point x t and incur a loss for that point equal to the squared distance from x t to its closest center in Z t. Then the cumulative regret up to time t is given by min xτ z 2 2 z Z τ τ t and we can compare this to the regret that would be achieved under the best centers for x 1,.., x t. Dasgupta acnowledges that it is an open problem to develop a good online algorithm for -means clustering [5] under either of these performance metrics. Indeed, although several online algorithms exist, almost nothing theoretical is nown about performance. 2

4 There is a standard online variant of Lloyd s algorithm which we will describe in detail in Section 4; lie Lloyd s algorithm in the batch case it can perform arbitrarily badly under either of the performance framewors outlined above. The only existing wor that proves performance guarantees for online -means clustering is by Choromansa and Monteleoni [4]. They consider a traditional online learning framewor where there are experts that the algorithm can learn from. In their setting, the experts are batch algorithms with nown performance guarantees. To get an approximation guarantee in the online sense, they use an analog to Dasgupta s concept of regret defined above [5] and prove the approximation bounds by comparing the online performance to the batch performance, which has a nown performance guarantee with respect to the cost of the optimal clustering. Although [4] is the only paper which provides performance bounds for online clustering, there is a growing literature in this area which focuses on heuristics for particular applications. In [3], rather than trying to minimize the -means obective, Barbah and Fyfe consider how to overcome the sensitivity of Lloyd s algorithm to initial conditions when Lloyd s algorithm is extended to an online setting. As motivated in Section 1, here we will be concerned with developing a heuristic for online clustering which performs well on time-varying data. The traditional -means obective is inadequate in the non-stationary setting, and it is not obvious what it should be replaced by. In this proect, we will propose a performance obective for the analog of -means clustering in the the non-stationary setting and provide ustification for the departure from the traditional -means performance obective. We then introduce a heuristic algorithm, and in lieu of proving a performance guarantee (which it should be clear by now is an open, hard problem in the online setting, let alone when the data is non-stationary we will test empirical performance of the algorithm on a simulated non-stationary data set. By using a simulated data set we have ground truth and can therefore calculate the optimal clairvoyant clustering of the data with respect to our performance measure. However, the findings would apply to any real data set which has similar properties of clusterability and nonstationarity; we discuss this in more detail in Section 4 and Section 5. 3 Cost obective in the non-stationary setting The traditional batch -means cost function is given by 2 Cost(C 1,..., C, z 1,..., z = x i z 2 {i:xi C} where C 1,..., C are the clusters and z 1,..., z are the cluster centers [8]. 3

5 In the online setting, we must consider the cost at any point in time. At time t, we have centers z1, t..., z t and clusters Ct 1,..., C t. Then Cost t (C t 1,..., C t, z t 1,..., z t = x t 2 i z 2 {i:xi C t } I claim that this obective is inappropriate in the non-stationary setting; this obective implicitly assumes that we want to find the best online clustering for all of the points seen so far, and at time t all points x 1,..., x t contribute equally to the cost of the algorithm. However, in the non-stationary setting, old data points should not be worth as much as new data points; the data may have shifted significantly over time. In order to tae this into account, I propose the following cost function for the non-stationary setting: Cost t (C1, t..., C, t z1, t..., z t = δ t i xi z 2 2 {i:xi C t } where δ (0, 1. Under this cost function, we want to find clusters which minimize the weighted squared distance from each point to its cluster center, where the weight is exponentially decreasing in the age of the point. t 4 Algorithms Since we are in an online setting with a theoretically infinite stream of data, it is impossible to store all of the data points seen so far. Therefore we assume that from one time period to the next, we can only eep O( pieces of information where is the number of cluster centers. In Online Lloyd s algorithm, at time step t, each point x 1,..., x t contributes equally to determine the updated centers. Here is the psuedocode for Online Lloyd s Algorithm: initialize the cluster centers z 1,..., z in any way create counters n 1,..., n and initialize them to zero loop get new data point x determine the closest center z i to x update the number of points in that cluster: n i n i update the cluster center: z i zi + (x n i z i end loop 1 I propose an alternative to updating the centers by the update rule z i zi + (x n i z i. Instead, I suggest using a discounted updating rule z i z i + α(x z i for α (0, 1. 4

6 This exponential smoothing heuristic should wor well when the data is naturally clusterable and the cluster centers are moving over time. However, as with Lloyd s algorithm, it may perform arbitrarily badly and is particularly sensitive to the initial cluster centers. Also, it may not be a good choice of algorithm if you do not now the number of clusters a priori. 5 Empirical Tests In order to run empirical tests, I simulated a data set. I wanted the data set to have the properties of having natural clusters and for these clusters to drift over time. Since I needed a data set with ground truth, I modified the well nown data set, iris, in order to have the time series property. Therefore the modified iris data had four dimensions, with 3 natural clusters whose centers drift over time. I tested Online Lloyd s algorithm on this dataset, as well as my exponentially smoothed version of Online Lloyd s. Additionally, I calculated the optimal clairvoyant cost. A plot of costs appears below for α = 0.5: 5

7 As expected, Online Lloyd s performs poorly as time goes on, but my exponential smoothing version of Online Lloyd s forgets the initial points and performs close to the optimal clairvoyant solution. 6 Practical Performance Guarantee It is a hard, open problem to mae performance guarantees for online clustering, let alone when the data is non-stationary. However, if we can eep trac of the cost that an algorithm is incurring over time, we can give a practical performance guarantee by comparing costs of different algorithms against each other. I claim that it is possible to eep trac of the cost without holding all points ever seen in memory. Specifically, I give the following result: Result 1. It is possible to calculate the cost in time period t + 1 recursively with only O( pieces of information from the previous time period. Proof. We will need to eep trac of the following pieces of information: Cost t = (Cost t 1, Cost t 2,...C t = the cost at time t for each center z t = (z1, t z2, t...z t, the cluster centers at time t sum-delta t = (sum-delta 1 t,..., sum-delta t i t where sum-delta t = { i:x C δ is a scalar i t } eeping trac of the sum of the discount factors of every point that has ever entered cluster up to time period t. At every time period we will multiply this number by δ. sum-delta-x t = (sum-delta-x 1 t,..., sum-delta-x t where sum-delta-x t = i {i:x C t δt x i } i is a vector eeping trac of the sum of every point that has ever entered cluster up to time period t, weighted by its discount factor. Let d be the dimension of the data points. Note that Cost t = i {i:x i Ct δt } x i z t+1 2 d 2 2 = {i:x C t } δt i =1 (x t i z i for every center. Suppose that at time t + 1 a new data point x t+1 arrives to cluster. First we update: sum-delta t+1 δsum-delta t sum-delta t+1 δsum-delta t sum-delta-x t+1 δsum-delta-x t sum-delta-x t+1 δsum-delta-x t+1 + x t+1 The new center z t+1 can be chosen in any way. Then the cost in time period t + 1 is given by Cost t+1 + δ = Costt. We can calculate Cost t+1 as follows. 6

8 Cost t+1 = δ t+1 i x i z t = t { i:x i C +1 } d δ t+1 i (x { i:x t+1 i C =1 } 2 i z t+1 = d δ t+1 i (x z t + z t z t+1 2 { i:x i C t+1 } =1 d i = δ t+1 i (xi z t 2 + 2(x z t (z t t ( z t i z +1 + z {i:x i C t+1 =1 } = δ d δ d t i (xi z + (x t+1 z t 2 t 2 {i:xi C t =1 } =1 t d d δ t+1 i +1 2(xi z (z z + δ (z z t t t+1 t i t t+1 2 {i:x i C t } =1 {i:xi C t =1 } ( d (z d 2 x t t+1 2 t+1 zt + sum-delta t+1 z =1 =1 = δcost t + ( d + 2 ( (z t t+1 2 t z sum-delta-x t+1 z sum-delta t+1 =1 Therefore, counting a point as a single piece of information, we can calculate the cost recursively from time t to time t + 1 by storing = O( pieces of information from one period to the next. Therefore, even without nowing the optimal cost, it is possible to run several online clustering algorithms in parallel, calculate the cost each one is incurring, and choose the best one for a specific dataset. 7 Conclusion In this paper we have explored online clustering, specifically for non-stationary data. Online clustering is a hard problem and performance guarantees are rare. When non-stationarity is 7

9 introduced, there is not even a standard agreed-upon framewor for measuring performance. Here, I have defined a cost function for the non-stationary setting, and demonstrated a simple exponential smoothing algorithm which performs well in practice. There is lots of room to invent algorithms for online clustering of non-stationary data, but little theoretical machinery to give performance guarantees. In lieu of this, I have shown that at the very least, we can compare algorithms against each other by eeping trac of the cost with only O( pieces of information. This gives a practical performance guarantee. This paper scratches the surface of online clustering for non-stationary data. In general, this corner of clustering research contains many open questions. 8

10 References [1] Daniel Aloise and Amit Deshpande, Pierre Hansen, and Preyas Popat. Np-hardness of euclidean sum-of- squares clustering. Machine Learning, 75: , May [2] D. Arthur and S Vassilvitsii. -means++: the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages , [3] W. Barbah and C. Fyfe. Online Clustering Algorithms. International Journal of Neural Systems (IJNS, 18(3:1 10, [4] Anna Choromansa and Claire Monteleoni. Online Clustering with Experts. In Journal of Machine Learning Research (JMLR Worshop and Conference Proceedings, ICML 2011 Worshop on Online Trading of Exploration and Exploitation 2, [5] Sanoy Dasgupta. Clustering in an Online/Streaming Setting. Technical report, Lecture notes from the course CSE 291: Topics in Unsupervised Learning, UCSD, Retrieved at cseweb.ucsd.edu/~dasgupta/291/lec6.pdf. [6] T. Kanungo and D. Mount, N. Netanyahu, C. Piato, R. Silverman and A. Wu. A Local Search Approximation Algorithm for -Means Clustering. Computational Geometry: Theory and Applications, [7] S. P. Lloyd. Least square quantization in pcm. Bell Telephone Laboratories Paper, [8] Cynthia Rudin and Seyda Ertein. Clustering. Technical report, MIT Course Notes, [9] Shi Zhong. Efficient Online Spherical K-means Clustering. In Journal of Machine Learning Research (JMLR Worshop and Conference Proceedings, IEEE Int. Joint Conf. Neural Networs (IJCNN 2005, pages , Montreal, August

11 MIT OpenCourseWare Prediction: Machine Learning and Statistics Spring 2012 For information about citing these materials or our Terms of Use, visit:

### Lecture 4 Online and streaming algorithms for clustering

CSE 291: Geometric algorithms Spring 2013 Lecture 4 Online and streaming algorithms for clustering 4.1 On-line k-clustering To the extent that clustering takes place in the brain, it happens in an on-line

### Lecture 6 Online and streaming algorithms for clustering

CSE 291: Unsupervised learning Spring 2008 Lecture 6 Online and streaming algorithms for clustering 6.1 On-line k-clustering To the extent that clustering takes place in the brain, it happens in an on-line

### Clustering. Danilo Croce Web Mining & Retrieval a.a. 2015/201 16/03/2016

Clustering Danilo Croce Web Mining & Retrieval a.a. 2015/201 16/03/2016 1 Supervised learning vs. unsupervised learning Supervised learning: discover patterns in the data that relate data attributes with

### Web Hosting Service Level Agreements

Chapter 5 Web Hosting Service Level Agreements Alan King (Mentor) 1, Mehmet Begen, Monica Cojocaru 3, Ellen Fowler, Yashar Ganjali 4, Judy Lai 5, Taejin Lee 6, Carmeliza Navasca 7, Daniel Ryan Report prepared

### K-Means Clustering Tutorial

K-Means Clustering Tutorial By Kardi Teknomo,PhD Preferable reference for this tutorial is Teknomo, Kardi. K-Means Clustering Tutorials. http:\\people.revoledu.com\kardi\ tutorial\kmean\ Last Update: July

### Ckmeans.1d.dp: Optimal k-means Clustering in One Dimension by Dynamic Programming

CONTRIBUTED RESEARCH ARTICLES 29 C.d.dp: Optimal k-means Clustering in One Dimension by Dynamic Programming by Haizhou Wang and Mingzhou Song Abstract The heuristic k-means algorithm, widely used for cluster

### IMPROVING PERFORMANCE OF RANDOMIZED SIGNATURE SORT USING HASHING AND BITWISE OPERATORS

Volume 2, No. 3, March 2011 Journal of Global Research in Computer Science RESEARCH PAPER Available Online at www.jgrcs.info IMPROVING PERFORMANCE OF RANDOMIZED SIGNATURE SORT USING HASHING AND BITWISE

### Lecture 20: Clustering

Lecture 20: Clustering Wrap-up of neural nets (from last lecture Introduction to unsupervised learning K-means clustering COMP-424, Lecture 20 - April 3, 2013 1 Unsupervised learning In supervised learning,

### Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization. Learning Goals. GENOME 560, Spring 2012

Why Taking This Course? Course Introduction, Descriptive Statistics and Data Visualization GENOME 560, Spring 2012 Data are interesting because they help us understand the world Genomics: Massive Amounts

### B490 Mining the Big Data. 2 Clustering

B490 Mining the Big Data 2 Clustering Qin Zhang 1-1 Motivations Group together similar documents/webpages/images/people/proteins/products One of the most important problems in machine learning, pattern

### 6.080 / 6.089 Great Ideas in Theoretical Computer Science Spring 2008

MIT OpenCourseWare http://ocw.mit.edu 6.080 / 6.089 Great Ideas in Theoretical Computer Science Spring 2008 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.

### CPSC 340: Machine Learning and Data Mining. K-Means Clustering Fall 2015

CPSC 340: Machine Learning and Data Mining K-Means Clustering Fall 2015 Admin Assignment 1 solutions posted after class. Tutorials for Assignment 2 on Monday. Random Forests Random forests are one of the

### Algorithmic Mechanism Design for Load Balancing in Distributed Systems

In Proc. of the 4th IEEE International Conference on Cluster Computing (CLUSTER 2002), September 24 26, 2002, Chicago, Illinois, USA, IEEE Computer Society Press, pp. 445 450. Algorithmic Mechanism Design

### Coding Information and Problems of Storage in Dynamical Systems

FACTA UNIVERSITATIS (NIŠ) SER.: ELEC. ENERG. vol. 7 December 004 355-363 Coding Information and Problems of Storage in Dynamical Systems Ilhem Djellit and Rezi Chemlal Abstract: A method of storage of

### Chapter 15 Introduction to Linear Programming

Chapter 15 Introduction to Linear Programming An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Brief History of Linear Programming The goal of linear programming is to determine the values of

### Offline sorting buffers on Line

Offline sorting buffers on Line Rohit Khandekar 1 and Vinayaka Pandit 2 1 University of Waterloo, ON, Canada. email: rkhandekar@gmail.com 2 IBM India Research Lab, New Delhi. email: pvinayak@in.ibm.com

### Simple and efficient online algorithms for real world applications

Simple and efficient online algorithms for real world applications Università degli Studi di Milano Milano, Italy Talk @ Centro de Visión por Computador Something about me PhD in Robotics at LIRA-Lab,

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### A Piggybacking Design Framework for Read-and Download-efficient Distributed Storage Codes

A Piggybacing Design Framewor for Read-and Download-efficient Distributed Storage Codes K V Rashmi, Nihar B Shah, Kannan Ramchandran, Fellow, IEEE Department of Electrical Engineering and Computer Sciences

### Bag of Pursuits and Neural Gas for Improved Sparse Coding

Bag of Pursuits and Neural Gas for Improved Sparse Coding Kai Labusch, Erhardt Barth, and Thomas Martinetz University of Lübec Institute for Neuro- and Bioinformatics Ratzeburger Allee 6 23562 Lübec, Germany

### SEARCH ENGINE WITH PARALLEL PROCESSING AND INCREMENTAL K-MEANS FOR FAST SEARCH AND RETRIEVAL

SEARCH ENGINE WITH PARALLEL PROCESSING AND INCREMENTAL K-MEANS FOR FAST SEARCH AND RETRIEVAL Krishna Kiran Kattamuri 1 and Rupa Chiramdasu 2 Department of Computer Science Engineering, VVIT, Guntur, India

### Module1. x 1000. y 800.

Module1 1 Welcome to the first module of the course. It is indeed an exciting event to share with you the subject that has lot to offer both from theoretical side and practical aspects. To begin with,

### Social Media Mining. Data Mining Essentials

Introduction Data production rate has been increased dramatically (Big Data) and we are able store much more data than before E.g., purchase data, social media data, mobile phone data Businesses and customers

### Online Convex Optimization

E0 370 Statistical Learning heory Lecture 19 Oct 22, 2013 Online Convex Optimization Lecturer: Shivani Agarwal Scribe: Aadirupa 1 Introduction In this lecture we shall look at a fairly general setting

### Induction Problems. Tom Davis November 7, 2005

Induction Problems Tom Davis tomrdavis@earthlin.net http://www.geometer.org/mathcircles November 7, 2005 All of the following problems should be proved by mathematical induction. The problems are not necessarily

### Introduction to Machine Learning Using Python. Vikram Kamath

Introduction to Machine Learning Using Python Vikram Kamath Contents: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Introduction/Definition Where and Why ML is used Types of Learning Supervised Learning Linear Regression

### Clustering UE 141 Spring 2013

Clustering UE 141 Spring 013 Jing Gao SUNY Buffalo 1 Definition of Clustering Finding groups of obects such that the obects in a group will be similar (or related) to one another and different from (or

### The Classes P and NP

The Classes P and NP We now shift gears slightly and restrict our attention to the examination of two families of problems which are very important to computer scientists. These families constitute the

### K-Cover of Binary sequence

K-Cover of Binary sequence Prof Amit Kumar Prof Smruti Sarangi Sahil Aggarwal Swapnil Jain Given a binary sequence, represent all the 1 s in it using at most k- covers, minimizing the total length of all

### 6.080 / 6.089 Great Ideas in Theoretical Computer Science Spring 2008

MIT OpenCourseWare http://ocw.mit.edu 6.080 / 6.089 Great Ideas in Theoretical Computer Science Spring 2008 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.

### A Multi-Layered Immune Inspired Approach to Data Mining

A Multi-Layered Immune Inspired Approach to Data Mining Thomas Knight and Jon Timmis Computing Laboratory University of Kent at Canterbury Canterbury, Kent, CT2 7NF, United Kingdom e-mail: {tpk1, jt6}@ukc.ac.uk

### CSE373: Data Structures and Algorithms Lecture 3: Math Review; Algorithm Analysis. Linda Shapiro Winter 2015

CSE373: Data Structures and Algorithms Lecture 3: Math Review; Algorithm Analysis Linda Shapiro Today Registration should be done. Homework 1 due 11:59 pm next Wednesday, January 14 Review math essential

### Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay

Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding

### Unsupervised Learning: Clustering with DBSCAN Mat Kallada

Unsupervised Learning: Clustering with DBSCAN Mat Kallada STAT 2450 - Introduction to Data Mining Supervised Data Mining: Predicting a column called the label The domain of data mining focused on prediction:

### Data Clustering for Forecasting

Data Clustering for Forecasting James B. Orlin MIT Sloan School and OR Center Mahesh Kumar MIT OR Center Nitin Patel Visiting Professor Jonathan Woo ProfitLogic Inc. 1 Overview of Talk Overview of Clustering

### MapReduce and Distributed Data Analysis. Sergei Vassilvitskii Google Research

MapReduce and Distributed Data Analysis Google Research 1 Dealing With Massive Data 2 2 Dealing With Massive Data Polynomial Memory Sublinear RAM Sketches External Memory Property Testing 3 3 Dealing With

### Azure Machine Learning, SQL Data Mining and R

Azure Machine Learning, SQL Data Mining and R Day-by-day Agenda Prerequisites No formal prerequisites. Basic knowledge of SQL Server Data Tools, Excel and any analytical experience helps. Best of all:

### LabVIEW Day 3: Arrays and Clusters

LabVIEW Day 3: Arrays and Clusters Vern Lindberg By now you should be getting used to LabVIEW. You should know how to Create a Constant, Control, or Indicator. I will assume you know how to create a new

### Degeneracy on K-means clustering

on K-means clustering Abdulrahman Alguwaizani Assistant Professor Math Department College of Science King Faisal University Kingdom of Saudi Arabia Office # 0096635899717 Mobile # 00966505920703 E-mail:

### Chapter 4: Non-Parametric Classification

Chapter 4: Non-Parametric Classification Introduction Density Estimation Parzen Windows Kn-Nearest Neighbor Density Estimation K-Nearest Neighbor (KNN) Decision Rule Gaussian Mixture Model A weighted combination

### Information Retrieval and Web Search Engines

Information Retrieval and Web Search Engines Lecture 7: Document Clustering December 10 th, 2013 Wolf-Tilo Balke and Kinda El Maarry Institut für Informationssysteme Technische Universität Braunschweig

### Big Data & Scripting Part II Streaming Algorithms

Big Data & Scripting Part II Streaming Algorithms 1, 2, a note on sampling and filtering sampling: (randomly) choose a representative subset filtering: given some criterion (e.g. membership in a set),

### K-Means Clustering. Clustering and Classification Lecture 8

K-Means Clustering Clustering and Lecture 8 Today s Class K-means clustering: What it is How it works What it assumes Pitfalls of the method (locally optimal results) 2 From Last Time If you recall the

### 6.080 / 6.089 Great Ideas in Theoretical Computer Science Spring 2008

MIT OpenCourseWare http://ocw.mit.edu 6.080 / 6.089 Great Ideas in Theoretical Computer Science Spring 008 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.

### Self Organizing Maps: Fundamentals

Self Organizing Maps: Fundamentals Introduction to Neural Networks : Lecture 16 John A. Bullinaria, 2004 1. What is a Self Organizing Map? 2. Topographic Maps 3. Setting up a Self Organizing Map 4. Kohonen

### Principles of demand management Airline yield management Determining the booking limits. » A simple problem» Stochastic gradients for general problems

Demand Management Principles of demand management Airline yield management Determining the booking limits» A simple problem» Stochastic gradients for general problems Principles of demand management Issues:»

### ARTIFICIAL INTELLIGENCE (CSCU9YE) LECTURE 6: MACHINE LEARNING 2: UNSUPERVISED LEARNING (CLUSTERING)

ARTIFICIAL INTELLIGENCE (CSCU9YE) LECTURE 6: MACHINE LEARNING 2: UNSUPERVISED LEARNING (CLUSTERING) Gabriela Ochoa http://www.cs.stir.ac.uk/~goc/ OUTLINE Preliminaries Classification and Clustering Applications

### Applications to Data Smoothing and Image Processing I

Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is

### AI: A Modern Approach, Chpts. 3-4 Russell and Norvig

AI: A Modern Approach, Chpts. 3-4 Russell and Norvig Sequential Decision Making in Robotics CS 599 Geoffrey Hollinger and Gaurav Sukhatme (Some slide content from Stuart Russell and HweeTou Ng) Spring,

### Load Balancing and Switch Scheduling

EE384Y Project Final Report Load Balancing and Switch Scheduling Xiangheng Liu Department of Electrical Engineering Stanford University, Stanford CA 94305 Email: liuxh@systems.stanford.edu Abstract Load

### Using the Triangle Inequality to Accelerate

Using the Triangle Inequality to Accelerate -Means Charles Elkan Department of Computer Science and Engineering University of California, San Diego La Jolla, California 92093-0114 ELKAN@CS.UCSD.EDU Abstract

### Elementary Number Theory and Methods of Proof. CSE 215, Foundations of Computer Science Stony Brook University http://www.cs.stonybrook.

Elementary Number Theory and Methods of Proof CSE 215, Foundations of Computer Science Stony Brook University http://www.cs.stonybrook.edu/~cse215 1 Number theory Properties: 2 Properties of integers (whole

### Theory of Computation Prof. Kamala Krithivasan Department of Computer Science and Engineering Indian Institute of Technology, Madras

Theory of Computation Prof. Kamala Krithivasan Department of Computer Science and Engineering Indian Institute of Technology, Madras Lecture No. # 31 Recursive Sets, Recursively Innumerable Sets, Encoding

### EFFICIENT K-MEANS CLUSTERING ALGORITHM USING RANKING METHOD IN DATA MINING

EFFICIENT K-MEANS CLUSTERING ALGORITHM USING RANKING METHOD IN DATA MINING Navjot Kaur, Jaspreet Kaur Sahiwal, Navneet Kaur Lovely Professional University Phagwara- Punjab Abstract Clustering is an essential

### Lecture 2: The Complexity of Some Problems

IAS/PCMI Summer Session 2000 Clay Mathematics Undergraduate Program Basic Course on Computational Complexity Lecture 2: The Complexity of Some Problems David Mix Barrington and Alexis Maciel July 18, 2000

### Automatic Labeling of Lane Markings for Autonomous Vehicles

Automatic Labeling of Lane Markings for Autonomous Vehicles Jeffrey Kiske Stanford University 450 Serra Mall, Stanford, CA 94305 jkiske@stanford.edu 1. Introduction As autonomous vehicles become more popular,

### Flat Clustering K-Means Algorithm

Flat Clustering K-Means Algorithm 1. Purpose. Clustering algorithms group a set of documents into subsets or clusters. The cluster algorithms goal is to create clusters that are coherent internally, but

### TD(0) Leads to Better Policies than Approximate Value Iteration

TD(0) Leads to Better Policies than Approximate Value Iteration Benjamin Van Roy Management Science and Engineering and Electrical Engineering Stanford University Stanford, CA 94305 bvr@stanford.edu Abstract

### Methodology for Emulating Self Organizing Maps for Visualization of Large Datasets

Methodology for Emulating Self Organizing Maps for Visualization of Large Datasets Macario O. Cordel II and Arnulfo P. Azcarraga College of Computer Studies *Corresponding Author: macario.cordel@dlsu.edu.ph

### A Survey of Kernel Clustering Methods

A Survey of Kernel Clustering Methods Maurizio Filippone, Francesco Camastra, Francesco Masulli and Stefano Rovetta Presented by: Kedar Grama Outline Unsupervised Learning and Clustering Types of clustering

### Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

### Degeneracy in Linear Programming

Degeneracy in Linear Programming I heard that today s tutorial is all about Ellen DeGeneres Sorry, Stan. But the topic is just as interesting. It s about degeneracy in Linear Programming. Degeneracy? Students

### Lecture 1 Newton s method.

Lecture 1 Newton s method. 1 Square roots 2 Newton s method. The guts of the method. A vector version. Implementation. The existence theorem. Basins of attraction. The Babylonian algorithm for finding

### Multiobjective Cloud Capacity Planning for Time- Varying Customer Demand

Multiobjective Cloud Capacity Planning for Time- Varying Customer Demand Brian Bouterse Department of Computer Science North Carolina State University Raleigh, NC, USA bmbouter@ncsu.edu Harry Perros Department

### Data Mining Practical Machine Learning Tools and Techniques

Ensemble learning Data Mining Practical Machine Learning Tools and Techniques Slides for Chapter 8 of Data Mining by I. H. Witten, E. Frank and M. A. Hall Combining multiple models Bagging The basic idea

### Map/Reduce Affinity Propagation Clustering Algorithm

Map/Reduce Affinity Propagation Clustering Algorithm Wei-Chih Hung, Chun-Yen Chu, and Yi-Leh Wu Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology,

### Chapter 11. 11.1 Load Balancing. Approximation Algorithms. Load Balancing. Load Balancing on 2 Machines. Load Balancing: Greedy Scheduling

Approximation Algorithms Chapter Approximation Algorithms Q. Suppose I need to solve an NP-hard problem. What should I do? A. Theory says you're unlikely to find a poly-time algorithm. Must sacrifice one

### Machine Learning using MapReduce

Machine Learning using MapReduce What is Machine Learning Machine learning is a subfield of artificial intelligence concerned with techniques that allow computers to improve their outputs based on previous

### Evaluating Algorithms that Learn from Data Streams

João Gama LIAAD-INESC Porto, Portugal Pedro Pereira Rodrigues LIAAD-INESC Porto & Faculty of Sciences, University of Porto, Portugal Gladys Castillo University Aveiro, Portugal jgama@liaad.up.pt pprodrigues@fc.up.pt

### 1 Portfolio Selection

COS 5: Theoretical Machine Learning Lecturer: Rob Schapire Lecture # Scribe: Nadia Heninger April 8, 008 Portfolio Selection Last time we discussed our model of the stock market N stocks start on day with

### 6. If there is no improvement of the categories after several steps, then choose new seeds using another criterion (e.g. the objects near the edge of

Clustering Clustering is an unsupervised learning method: there is no target value (class label) to be predicted, the goal is finding common patterns or grouping similar examples. Differences between models/algorithms

### Forecasting Hospital Bed Availability Using Simulation and Neural Networks

Forecasting Hospital Bed Availability Using Simulation and Neural Networks Matthew J. Daniels Michael E. Kuhl Industrial & Systems Engineering Department Rochester Institute of Technology Rochester, NY

### princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 6: Provable Approximation via Linear Programming Lecturer: Sanjeev Arora

princeton univ. F 13 cos 521: Advanced Algorithm Design Lecture 6: Provable Approximation via Linear Programming Lecturer: Sanjeev Arora Scribe: One of the running themes in this course is the notion of

### Computer and Network Security

MIT 6.857 Computer and Networ Security Class Notes 1 File: http://theory.lcs.mit.edu/ rivest/notes/notes.pdf Revision: December 2, 2002 Computer and Networ Security MIT 6.857 Class Notes by Ronald L. Rivest

### A Framework for Statistical Clustering with a Constant Time Approximation Algorithms for K-Median Clustering

A Framework for Statistical Clustering with a Constant Time Approximation Algorithms for K-Median Clustering Shai Ben-David Department of Computer Science Technion, Haifa 32000, Israel and School of ECE

### 3. INNER PRODUCT SPACES

. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.

### Sequences and Convergence in Metric Spaces

Sequences and Convergence in Metric Spaces Definition: A sequence in a set X (a sequence of elements of X) is a function s : N X. We usually denote s(n) by s n, called the n-th term of s, and write {s

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. !-approximation algorithm.

Approximation Algorithms Chapter Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of

### Making Sense of the Mayhem: Machine Learning and March Madness

Making Sense of the Mayhem: Machine Learning and March Madness Alex Tran and Adam Ginzberg Stanford University atran3@stanford.edu ginzberg@stanford.edu I. Introduction III. Model The goal of our research

### K-means Clustering with Feature Hashing

K-means Clustering with Feature Hashing Hajime Senuma Department of Computer Science University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan hajime.senuma@gmail.com Abstract One of the major

### THE FUNDAMENTAL THEOREM OF ARBITRAGE PRICING

THE FUNDAMENTAL THEOREM OF ARBITRAGE PRICING 1. Introduction The Black-Scholes theory, which is the main subject of this course and its sequel, is based on the Efficient Market Hypothesis, that arbitrages

### Draft Martin Doerr ICS-FORTH, Heraklion, Crete Oct 4, 2001

A comparison of the OpenGIS TM Abstract Specification with the CIDOC CRM 3.2 Draft Martin Doerr ICS-FORTH, Heraklion, Crete Oct 4, 2001 1 Introduction This Mapping has the purpose to identify, if the OpenGIS

### Decision Support System Methodology Using a Visual Approach for Cluster Analysis Problems

Decision Support System Methodology Using a Visual Approach for Cluster Analysis Problems Ran M. Bittmann School of Business Administration Ph.D. Thesis Submitted to the Senate of Bar-Ilan University Ramat-Gan,

### Practical Data Science with Azure Machine Learning, SQL Data Mining, and R

Practical Data Science with Azure Machine Learning, SQL Data Mining, and R Overview This 4-day class is the first of the two data science courses taught by Rafal Lukawiecki. Some of the topics will be

### DATA MINING CLUSTER ANALYSIS: BASIC CONCEPTS

DATA MINING CLUSTER ANALYSIS: BASIC CONCEPTS 1 AND ALGORITHMS Chiara Renso KDD-LAB ISTI- CNR, Pisa, Italy WHAT IS CLUSTER ANALYSIS? Finding groups of objects such that the objects in a group will be similar

### Rafael Witten Yuze Huang Haithem Turki. Playing Strong Poker. 1. Why Poker?

Rafael Witten Yuze Huang Haithem Turki Playing Strong Poker 1. Why Poker? Chess, checkers and Othello have been conquered by machine learning - chess computers are vastly superior to humans and checkers

### The Planar k-means Problem is NP-hard

The Planar k-means Problem is NP-hard Meena Mahajan a, Prajakta Nimbhorkar a, Kasturi Varadarajan b a The Institute of Mathematical Sciences, Chennai 600 113, India. b The University of Iowa, Iowa City,

### Steven C.H. Hoi. School of Computer Engineering Nanyang Technological University Singapore

Steven C.H. Hoi School of Computer Engineering Nanyang Technological University Singapore Acknowledgments: Peilin Zhao, Jialei Wang, Hao Xia, Jing Lu, Rong Jin, Pengcheng Wu, Dayong Wang, etc. 2 Agenda

### ! Solve problem to optimality. ! Solve problem in poly-time. ! Solve arbitrary instances of the problem. #-approximation algorithm.

Approximation Algorithms 11 Approximation Algorithms Q Suppose I need to solve an NP-hard problem What should I do? A Theory says you're unlikely to find a poly-time algorithm Must sacrifice one of three

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### COMP 250 Fall 2012 lecture 2 binary representations Sept. 11, 2012

Binary numbers The reason humans represent numbers using decimal (the ten digits from 0,1,... 9) is that we have ten fingers. There is no other reason than that. There is nothing special otherwise about

### 2 The Euclidean algorithm

2 The Euclidean algorithm Do you understand the number 5? 6? 7? At some point our level of comfort with individual numbers goes down as the numbers get large For some it may be at 43, for others, 4 In

### IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 7, JULY 2009 1181

IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 20, NO. 7, JULY 2009 1181 The Global Kernel k-means Algorithm for Clustering in Feature Space Grigorios F. Tzortzis and Aristidis C. Likas, Senior Member, IEEE

### 2.5 Gaussian Elimination

page 150 150 CHAPTER 2 Matrices and Systems of Linear Equations 37 10 the linear algebra package of Maple, the three elementary 20 23 1 row operations are 12 1 swaprow(a,i,j): permute rows i and j 3 3

### Stress Recovery 28 1

. 8 Stress Recovery 8 Chapter 8: STRESS RECOVERY 8 TABLE OF CONTENTS Page 8.. Introduction 8 8.. Calculation of Element Strains and Stresses 8 8.. Direct Stress Evaluation at Nodes 8 8.. Extrapolation

### LS.6 Solution Matrices

LS.6 Solution Matrices In the literature, solutions to linear systems often are expressed using square matrices rather than vectors. You need to get used to the terminology. As before, we state the definitions

### CS Master Level Courses and Areas COURSE DESCRIPTIONS. CSCI 521 Real-Time Systems. CSCI 522 High Performance Computing

CS Master Level Courses and Areas The graduate courses offered may change over time, in response to new developments in computer science and the interests of faculty and students; the list of graduate

### An Analysis on Density Based Clustering of Multi Dimensional Spatial Data

An Analysis on Density Based Clustering of Multi Dimensional Spatial Data K. Mumtaz 1 Assistant Professor, Department of MCA Vivekanandha Institute of Information and Management Studies, Tiruchengode,