Data visualization and clustering. Genomics is to no small extend a data science


 Laurence Turner
 1 years ago
 Views:
Transcription
1 Data visualization and clustering Genomics is to no small extend a data science [www.data2discovery.org]
2 Data visualization and clustering Genomics is to no small extend a data science [Andersson et al., Nature 2015]
3 Data visualization and clustering Data visualization: Look at the data. Why?  Quality control did a experiment work?  Exploratory data analysis what does the data say?  Sanity checks does my code work?  Interpreting data making a point. CAGE signature correlates with other enhancer marks. Fraction enhancers Mean signal [Andersson et al., Nature 2015]
4 Data visualization and clustering 1.Tools for data visualization. How to:  Visualize distributions correlations  Visualize group structure: clustering  Visualize data along genomic coordinates  Visualize dependencies and interactions: Graphs/networks and layouts 2. Examples
5 Visualizing distributions Discrete RV Continuous RV X! {x 1,x 2,...,x n } Random variable Observed realizations (n data points)
6 Visualizing distributions: histogram X! {x 1,x 2,...,x n } {x 1,x 2,...,x n }! Distribution? Number of realizations in bin Bins, e.g. [0.5,0.6)
7 Visualizing distributions: density ESTIMATE of continuous p.d.f The sale of xaxis matters! [Gentleman et al. 2006] 2. mode
8 Visualizing distributions: 2d histograms Binned realizations of RV Y Contour lines Binned realizations of RV X Information about DEPENDENCE between X and Y: P (X, Y )=P (X Y )P (Y ) Joint distribution Conditional distribution Marginal distribution Independence: P (X Y )=P (X) Are the rows in the plot similar?
9 Correlation {(x 1,y 1 ), (x 2,y 2 ),...,(x n,y n )} Joint realizations of X and Y Scatterplot Scatterplot with regression line Linear regression: E[Y X] =f(x) = + x Model the conditional expectation as linear function
10 Correlation Linear regression: E[Y X] =f(x) = + x How well does this work? Coefficient of determination. SS = X i ~Variance of Y Linear relation good: R 2 1 Linear relation bad: R 2 0 Can do the same for nonlinear f (y i ȳ) 2 = X (f i ȳ) 2 i {z } SSreg ~Variance of regression line R 2 =1 SSres SS tot + X (y i f i ) 2 i {z } SSres ~Variance not explained by f
11 Correlation Linear regression: E[Y X] =f(x) = + x How well does this work? Correlation coefficient Pearson s correlation coefficient: ˆ = r = X i r 2 = R 2 = Cov(X, Y ) X Y Cov(X, Y )=E[(X E[X])(Y E[y])] (x i x)(y i ȳ). Xi (x i x) X 2 (y i ȳ) i Coefficient of determination for linear model
12 Correlation coefficient Pearson s correlation coefficient: [1,1] and measures linear dependence
13 Correlation coefficient Pearson s correlation coefficient: [1,1] and measures linear dependence There are measures that capture nonlinear correlations.
14 Bar and Boxplots: comparing distributions [Spitzer et al., Nature Methods 2014]
15 Data visualization and clustering 1.Tools for data visualization. How to:  Visualize distributions correlations Histogram, density estimate, 2d histogram, coefficient of determination, correlation coefficient, scatterplot, boxplot (and variations thereof)  Visualize group structure: clustering  Visualize data along genomic coordinates  Visualize dependencies and interactions: Graphs/networks and layouts 2. Examples
16 Clustering: Grouping data Measurement of Y Measurement of X
17 Clustering: Grouping data 1. Organize data into clusters 2. No prior information (unsupervised) 3. Need some notion of distance/similarity
18 Hierarchical clustering  Euclidean distance  Agglomerative scheme  Average linkage Dendrogram Leafs are data points
19 Hierarchical clustering: distance Euclidean distance  Distance: D(a, b) =D(b, a) D(a, b) =0 iif a = b D(a, b) 0 D(a, b) apple D(a, c)+d(c, b)  Euclidean: D(x, y) = s X (x i y i ) 2 i
20 Hierarchical clustering: distance matrix Euclidean distance D(x, y) = s X (x i y i ) 2 i Heatmap of distances All pairs of data points  Magnitude is color coded  Matrix is symmetric  No apparent order
21 Hierarchical clustering: linkage Agglomerative scheme Start with each data point as its own cluster. Repeat until done: Merge the closest clusters.  Need closeness between points: Euclidean distance.  Need closeness between clusters (sets of points)  Average linkage: Average similarity points.  Single linkage: Take closest pair.  Complete linkage: Take furthest pair.
22 Hierarchical clustering  Euclidean distance  Agglomerative scheme  Average linkage Dendrogram
23 Hierarchical clustering  Euclidean distance  Agglomerative scheme  Linkage method matters Average linkage Single linkage (nearest neighbor)
24 Hierarchical clustering Distance matrix (heatmap) Ordered examples Internal nodes (not all are highlighted) Subtrees can rotate around nodes Ordering of leafs only partially defined
25 Hierarchical clustering Cutting the dendrogram defines clusters Distance matrix (heatmap) Ordered examples Cluster A Cluster B but it is often not clear how many to choose.
26 Prominent example: Clustering gene expression data Samples Genes Group [Gentleman et al. 2006] Group Group Group
27 Clustering: disclaimer There are a lot more clustering methods:  Partition clustering: No hierarchy, just disjoint clusters. Example: kmeans.  Modelbased clustering: Mixture distributions.  Others. P (X, Y )=P (X, Y cluster 1)P (cluster 1) + P (X, Y cluster 2)P (cluster 2) +... P (X, Y )=P (X, Y, Z) Unobserved cluster indicator random variable For each (xi,yi): Find the most likely zi Clustering.
28 Data visualization and clustering 1.Tools for data visualization. How to:  Visualize distributions correlations  Visualize group structure: clustering distance, distance matrix, heatmap, dendrogram, linkage (single, complete, average), partition clustering, modelbased clustering  Visualize data along genomic coordinates  Visualize dependencies and interactions: Graphs/networks and layouts 2. Examples
29 Plotting data along linear genomic coordinates UCSC genome browser
30 Plotting data along linear genomic coordinates UCSC genome browser [Rosenbloom et al., NAR 2015]
31 Circular arrangement Circular visualization: circos Enrichment analysis [Saben et al., Placenta, 2013]
32 Circular arrangement Circular visualization: circos [Zhang et al. 2013]
33 Data visualization and clustering 1.Tools for data visualization. How to:  Visualize distributions correlations  Visualize group structure: clustering  Visualize data along genomic coordinates UCSC browser, circular visualization  Visualize dependencies and interactions: Graphs/networks and layouts 2. Examples
34 Graphs Vertices v 2 V Edges e 2 E Can be directed or undirected (E,V): Graph. Entities and relations between entities [Gentleman et al. 2006] [dzone.com] Tree: acyclic and connected
35 Graphs Tree: acyclic and connected [dzone.com] Directed: edges/arcs have direction
36 Graphs [dzone.com]
37 Rooted trees and DAGs DAG: directed acyclic graph Rooted tree: DAG where each node has one parent. [dzone.com] [cs.cornell.edu]
38 Rooted trees and DAGs DAG: directed acyclic graph Rooted tree: DAG where each node has one parent. [dzone.com] Gene Ontology: heart development Phylogenetic tree
39 Plotting of graphs: layout Same graph, three pictures [Gentleman et al. 2006] dot: hierarchical neato: no edge crossing two: circular structure
40 Plotting of graphs: hairballs Different layout algorithms: an interaction network Gene A Interaction Gene B [http://www.hiveplot.net] Inferred by:  experimental assay  insilico analyses
41 Data visualization and clustering 1.Tools for data visualization. How to:  Visualize distributions correlations  Visualize group structure: clustering  Visualize data along genomic coordinates  Visualize dependencies and interactions 2. Examples
42 Quality control: Color Number of cells in a well: Handling problem [Gentleman et al. 2006]
43 Example figure Mean signal Fraction enhancers
44 Example figure
45 Example figure
46 Visualizing distributions: microarray probes Intensity stratified by G+C [Gentleman et al. 2006]
Clustering and Data Mining in R
Clustering and Data Mining in R Workshop Supplement Thomas Girke December 10, 2011 Introduction Data Preprocessing Data Transformations Distance Methods Cluster Linkage Hierarchical Clustering Approaches
More informationDefinition The covariance of X and Y, denoted by cov(x, Y ) is defined by. cov(x, Y ) = E(X µ 1 )(Y µ 2 ).
Correlation Regression Bivariate Normal Suppose that X and Y are r.v. s with joint density f(x y) and suppose that the means of X and Y are respectively µ 1 µ 2 and the variances are 1 2. Definition The
More informationClustering Hierarchical clustering and kmean clustering
Clustering Hierarchical clustering and kmean clustering Genome 373 Genomic Informatics Elhanan Borenstein The clustering problem: A quick review partition genes into distinct sets with high homogeneity
More informationDistances, Clustering, and Classification. Heatmaps
Distances, Clustering, and Classification Heatmaps 1 Distance Clustering organizes things that are close into groups What does it mean for two genes to be close? What does it mean for two samples to be
More informationData Mining Cluster Analysis: Advanced Concepts and Algorithms. ref. Chapter 9. Introduction to Data Mining
Data Mining Cluster Analysis: Advanced Concepts and Algorithms ref. Chapter 9 Introduction to Data Mining by Tan, Steinbach, Kumar 1 Outline Prototypebased Fuzzy cmeans Mixture Model Clustering Densitybased
More information6. If there is no improvement of the categories after several steps, then choose new seeds using another criterion (e.g. the objects near the edge of
Clustering Clustering is an unsupervised learning method: there is no target value (class label) to be predicted, the goal is finding common patterns or grouping similar examples. Differences between models/algorithms
More informationMachine Learning and Data Mining. Clustering. (adapted from) Prof. Alexander Ihler
Machine Learning and Data Mining Clustering (adapted from) Prof. Alexander Ihler Unsupervised learning Supervised learning Predict target value ( y ) given features ( x ) Unsupervised learning Understand
More informationNeural Networks Lesson 5  Cluster Analysis
Neural Networks Lesson 5  Cluster Analysis Prof. Michele Scarpiniti INFOCOM Dpt.  Sapienza University of Rome http://ispac.ing.uniroma1.it/scarpiniti/index.htm michele.scarpiniti@uniroma1.it Rome, 29
More informationClustering. Data Mining. Abraham Otero. Data Mining. Agenda
Clustering 1/46 Agenda Introduction Distance Knearest neighbors Hierarchical clustering Quick reference 2/46 1 Introduction It seems logical that in a new situation we should act in a similar way as in
More informationExploratory Data Analysis with MATLAB
Computer Science and Data Analysis Series Exploratory Data Analysis with MATLAB Second Edition Wendy L Martinez Angel R. Martinez Jeffrey L. Solka ( r ec) CRC Press VV J Taylor & Francis Group Boca Raton
More informationCluster Analysis: Advanced Concepts
Cluster Analysis: Advanced Concepts and dalgorithms Dr. Hui Xiong Rutgers University Introduction to Data Mining 08/06/2006 1 Introduction to Data Mining 08/06/2006 1 Outline Prototypebased Fuzzy cmeans
More informationData Mining. Cluster Analysis: Advanced Concepts and Algorithms
Data Mining Cluster Analysis: Advanced Concepts and Algorithms Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 More Clustering Methods Prototypebased clustering Densitybased clustering Graphbased
More informationClustering. 15381 Artificial Intelligence Henry Lin. Organizing data into clusters such that there is
Clustering 15381 Artificial Intelligence Henry Lin Modified from excellent slides of Eamonn Keogh, Ziv BarJoseph, and Andrew Moore What is Clustering? Organizing data into clusters such that there is
More informationMTH 140 Statistics Videos
MTH 140 Statistics Videos Chapter 1 Picturing Distributions with Graphs Individuals and Variables Categorical Variables: Pie Charts and Bar Graphs Categorical Variables: Pie Charts and Bar Graphs Quantitative
More informationCONTENTS PREFACE 1 INTRODUCTION 1 2 DATA VISUALIZATION 19
PREFACE xi 1 INTRODUCTION 1 1.1 Overview 1 1.2 Definition 1 1.3 Preparation 2 1.3.1 Overview 2 1.3.2 Accessing Tabular Data 3 1.3.3 Accessing Unstructured Data 3 1.3.4 Understanding the Variables and Observations
More informationCurriculum Map Statistics and Probability Honors (348) Saugus High School Saugus Public Schools 20092010
Curriculum Map Statistics and Probability Honors (348) Saugus High School Saugus Public Schools 20092010 Week 1 Week 2 14.0 Students organize and describe distributions of data by using a number of different
More informationBivariate Distributions
Chapter 4 Bivariate Distributions 4.1 Distributions of Two Random Variables In many practical cases it is desirable to take more than one measurement of a random observation: (brief examples) 1. What is
More informationData Mining and Visualization
Data Mining and Visualization Jeremy Walton NAG Ltd, Oxford Overview Data mining components Functionality Example application Quality control Visualization Use of 3D Example application Market research
More informationCluster Analysis. Isabel M. Rodrigues. Lisboa, 2014. Instituto Superior Técnico
Instituto Superior Técnico Lisboa, 2014 Introduction: Cluster analysis What is? Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from
More information10810 /02710 Computational Genomics. Clustering expression data
10810 /02710 Computational Genomics Clustering expression data What is Clustering? Organizing data into clusters such that there is high intracluster similarity low intercluster similarity Informally,
More informationClustering UE 141 Spring 2013
Clustering UE 141 Spring 013 Jing Gao SUNY Buffalo 1 Definition of Clustering Finding groups of obects such that the obects in a group will be similar (or related) to one another and different from (or
More informationData Mining Clustering (2) Sheets are based on the those provided by Tan, Steinbach, and Kumar. Introduction to Data Mining
Data Mining Clustering (2) Toon Calders Sheets are based on the those provided by Tan, Steinbach, and Kumar. Introduction to Data Mining Outline Partitional Clustering Distancebased Kmeans, Kmedoids,
More informationDistance based clustering
// Distance based clustering Chapter ² ² Clustering Clustering is the art of finding groups in data (Kaufman and Rousseeuw, 99). What is a cluster? Group of objects separated from other clusters Means
More informationData Mining Cluster Analysis: Basic Concepts and Algorithms. Lecture Notes for Chapter 8. Introduction to Data Mining
Data Mining Cluster Analysis: Basic Concepts and Algorithms Lecture Notes for Chapter 8 Introduction to Data Mining by Tan, Steinbach, Kumar Tan,Steinbach, Kumar Introduction to Data Mining 4/8/2004 Hierarchical
More informationChapter 5: Joint Probability Distributions. Chapter Learning Objectives. The Joint Probability Distribution for a Pair of Discrete Random
Chapter 5: Joint Probability Distributions 51 Two or More Random Variables 51.1 Joint Probability Distributions 51.2 Marginal Probability Distributions 51.3 Conditional Probability Distributions 51.4
More informationChapter ML:XI (continued)
Chapter ML:XI (continued) XI. Cluster Analysis Data Mining Overview Cluster Analysis Basics Hierarchical Cluster Analysis Iterative Cluster Analysis DensityBased Cluster Analysis Cluster Evaluation Constrained
More informationMedical Information Management & Mining. You Chen Jan,15, 2013 You.chen@vanderbilt.edu
Medical Information Management & Mining You Chen Jan,15, 2013 You.chen@vanderbilt.edu 1 Trees Building Materials Trees cannot be used to build a house directly. How can we transform trees to building materials?
More informationElementary Statistics. Scatter Plot, Regression Line, Linear Correlation Coefficient, and Coefficient of Determination
Scatter Plot, Regression Line, Linear Correlation Coefficient, and Coefficient of Determination What is a Scatter Plot? A Scatter Plot is a plot of ordered pairs (x, y) where the horizontal axis is used
More informationSocial Media Mining. Data Mining Essentials
Introduction Data production rate has been increased dramatically (Big Data) and we are able store much more data than before E.g., purchase data, social media data, mobile phone data Businesses and customers
More informationCELLULAR MANUFACTURING
CELLULAR MANUFACTURING Grouping Machines logically so that material handling (move time, wait time for moves and using smaller batch sizes) and setup (part family tooling and sequencing) can be minimized.
More informationJoint Probability Distributions and Random Samples. Week 5, 2011 Stat 4570/5570 Material from Devore s book (Ed 8), and Cengage
5 Joint Probability Distributions and Random Samples Week 5, 2011 Stat 4570/5570 Material from Devore s book (Ed 8), and Cengage Two Discrete Random Variables The probability mass function (pmf) of a single
More informationData Mining Cluster Analysis: Basic Concepts and Algorithms. Lecture Notes for Chapter 8. Introduction to Data Mining
Data Mining Cluster Analysis: Basic Concepts and Algorithms Lecture Notes for Chapter 8 by Tan, Steinbach, Kumar 1 What is Cluster Analysis? Finding groups of objects such that the objects in a group will
More informationThe Big 50 Revision Guidelines for S1
The Big 50 Revision Guidelines for S1 If you can understand all of these you ll do very well 1. Know what is meant by a statistical model and the Modelling cycle of continuous refinement 2. Understand
More informationL15: statistical clustering
Similarity measures Criterion functions Cluster validity Flat clustering algorithms kmeans ISODATA L15: statistical clustering Hierarchical clustering algorithms Divisive Agglomerative CSCE 666 Pattern
More informationAP STATISTICS REVIEW (YMS Chapters 18)
AP STATISTICS REVIEW (YMS Chapters 18) Exploring Data (Chapter 1) Categorical Data nominal scale, names e.g. male/female or eye color or breeds of dogs Quantitative Data rational scale (can +,,, with
More informationExercise 1.12 (Pg. 2223)
Individuals: The objects that are described by a set of data. They may be people, animals, things, etc. (Also referred to as Cases or Records) Variables: The characteristics recorded about each individual.
More informationChapter 7. Hierarchical cluster analysis. Contents 71
71 Chapter 7 Hierarchical cluster analysis In Part 2 (Chapters 4 to 6) we defined several different ways of measuring distance (or dissimilarity as the case may be) between the rows or between the columns
More informationAnalysing Questionnaires using Minitab (for SPSS queries contact ) Graham.Currell@uwe.ac.uk
Analysing Questionnaires using Minitab (for SPSS queries contact ) Graham.Currell@uwe.ac.uk Structure As a starting point it is useful to consider a basic questionnaire as containing three main sections:
More informationUnsupervised learning: Clustering
Unsupervised learning: Clustering Salissou Moutari Centre for Statistical Science and Operational Research CenSSOR 17 th September 2013 Unsupervised learning: Clustering 1/52 Outline 1 Introduction What
More informationSteven M. Ho!and. Department of Geology, University of Georgia, Athens, GA 306022501
CLUSTER ANALYSIS Steven M. Ho!and Department of Geology, University of Georgia, Athens, GA 306022501 January 2006 Introduction Cluster analysis includes a broad suite of techniques designed to find groups
More informationData Mining Cluster Analysis: Advanced Concepts and Algorithms. Lecture Notes for Chapter 9. Introduction to Data Mining
Data Mining Cluster Analysis: Advanced Concepts and Algorithms Lecture Notes for Chapter 9 Introduction to Data Mining by Tan, Steinbach, Kumar Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004
More informationDATA MINING CLUSTER ANALYSIS: BASIC CONCEPTS
DATA MINING CLUSTER ANALYSIS: BASIC CONCEPTS 1 AND ALGORITHMS Chiara Renso KDDLAB ISTI CNR, Pisa, Italy WHAT IS CLUSTER ANALYSIS? Finding groups of objects such that the objects in a group will be similar
More informationData Analysis Tools. Tools for Summarizing Data
Data Analysis Tools This section of the notes is meant to introduce you to many of the tools that are provided by Excel under the Tools/Data Analysis menu item. If your computer does not have that tool
More informationIntroduction to Statistical Machine Learning
CHAPTER Introduction to Statistical Machine Learning We start with a gentle introduction to statistical machine learning. Readers familiar with machine learning may wish to skip directly to Section 2,
More informationCLUSTERING (Segmentation)
CLUSTERING (Segmentation) Dr. Saed Sayad University of Toronto 2010 saed.sayad@utoronto.ca http://chemeng.utoronto.ca/~datamining/ 1 What is Clustering? Given a set of records, organize the records into
More information4. Joint Distributions of Two Random Variables
4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint
More informationStatistical Databases and Registers with some datamining
Unsupervised learning  Statistical Databases and Registers with some datamining a course in Survey Methodology and O cial Statistics Pages in the book: 501528 Department of Statistics Stockholm University
More informationHierarchical Clustering Analysis
Hierarchical Clustering Analysis What is Hierarchical Clustering? Hierarchical clustering is used to group similar objects into clusters. In the beginning, each row and/or column is considered a cluster.
More informationHierarchical Data Visualization
Hierarchical Data Visualization 1 Hierarchical Data Hierarchical data emphasize the subordinate or membership relations between data items. Organizational Chart Classifications / Taxonomies (Species and
More informationExample: Document Clustering. Clustering: Definition. Notion of a Cluster can be Ambiguous. Types of Clusterings. Hierarchical Clustering
Overview Prognostic Models and Data Mining in Medicine, part I Cluster Analsis What is Cluster Analsis? KMeans Clustering Hierarchical Clustering Cluster Validit Eample: Microarra data analsis 6 Summar
More informationUNSUPERVISED MACHINE LEARNING TECHNIQUES IN GENOMICS
UNSUPERVISED MACHINE LEARNING TECHNIQUES IN GENOMICS Dwijesh C. Mishra I.A.S.R.I., Library Avenue, New Delhi110 012 dcmishra@iasri.res.in What is Learning? "Learning denotes changes in a system that enable
More informationMicroarray Data Analysis Using Partek Genomic Suite. Xiaowen Wang Field Application Specialist Partek Inc.
Microarray Data Analysis Using Partek Genomic Suite Xiaowen Wang Field Application Specialist Partek Inc. Who is Partek? Founded in 1993 Building tools for statistics & visualization Focused on genomics
More informationData Clustering. Dec 2nd, 2013 Kyrylo Bessonov
Data Clustering Dec 2nd, 2013 Kyrylo Bessonov Talk outline Introduction to clustering Types of clustering Supervised Unsupervised Similarity measures Main clustering algorithms kmeans Hierarchical Main
More informationData Mining Cluster Analysis: Advanced Concepts and Algorithms. Lecture Notes for Chapter 9. Introduction to Data Mining
Data Mining Cluster Analysis: Advanced Concepts and Algorithms Lecture Notes for Chapter 9 Introduction to Data Mining by Tan, Steinbach, Kumar Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004
More informationJoint Distributions. Tieming Ji. Fall 2012
Joint Distributions Tieming Ji Fall 2012 1 / 33 X : univariate random variable. (X, Y ): bivariate random variable. In this chapter, we are going to study the distributions of bivariate random variables
More informationARTIFICIAL INTELLIGENCE (CSCU9YE) LECTURE 6: MACHINE LEARNING 2: UNSUPERVISED LEARNING (CLUSTERING)
ARTIFICIAL INTELLIGENCE (CSCU9YE) LECTURE 6: MACHINE LEARNING 2: UNSUPERVISED LEARNING (CLUSTERING) Gabriela Ochoa http://www.cs.stir.ac.uk/~goc/ OUTLINE Preliminaries Classification and Clustering Applications
More informationLean Six Sigma Training/Certification Book: Volume 1
Lean Six Sigma Training/Certification Book: Volume 1 Six Sigma Quality: Concepts & Cases Volume I (Statistical Tools in Six Sigma DMAIC process with MINITAB Applications Chapter 1 Introduction to Six Sigma,
More informationChapter 2: Looking at Data Relationships (Part 1)
Chapter 2: Looking at Data Relationships (Part 1) Dr. Nahid Sultana Chapter 2: Looking at Data Relationships 2.1: Scatterplots 2.2: Correlation 2.3: LeastSquares Regression 2.5: Data Analysis for TwoWay
More informationLinearizing Data. Lesson3. United States Population
Lesson3 Linearizing Data You may have heard that the population of the United States is increasing exponentially. The table and plot below give the population of the United States in the census years 19
More informationBasics of microarrays. Petter Mostad 2003
Basics of microarrays Petter Mostad 2003 Why microarrays? Microarrays work by hybridizing strands of DNA in a sample against complementary DNA in spots on a chip. Expression analysis measure relative amounts
More informationSections 2.11 and 5.8
Sections 211 and 58 Timothy Hanson Department of Statistics, University of South Carolina Stat 704: Data Analysis I 1/25 Gesell data Let X be the age in in months a child speaks his/her first word and
More informationPCA, Clustering and Classification. By H. Bjørn Nielsen strongly inspired by Agnieszka S. Juncker
PCA, Clustering and Classification By H. Bjørn Nielsen strongly inspired by Agnieszka S. Juncker Motivation: Multidimensional data Pat1 Pat2 Pat3 Pat4 Pat5 Pat6 Pat7 Pat8 Pat9 209619_at 7758 4705 5342
More informationTechnology StepbyStep Using StatCrunch
Technology StepbyStep Using StatCrunch Section 1.3 Simple Random Sampling 1. Select Data, highlight Simulate Data, then highlight Discrete Uniform. 2. Fill in the following window with the appropriate
More informationJoint Probability Distributions and Random Samples (Devore Chapter Five)
Joint Probability Distributions and Random Samples (Devore Chapter Five) 101634501 Probability and Statistics for Engineers Winter 20102011 Contents 1 Joint Probability Distributions 1 1.1 Two Discrete
More informationHierarchical Cluster Analysis Some Basics and Algorithms
Hierarchical Cluster Analysis Some Basics and Algorithms Nethra Sambamoorthi CRMportals Inc., 11 Bartram Road, Englishtown, NJ 07726 (NOTE: Please use always the latest copy of the document. Click on this
More informationClass 6: Chapter 12. Key Ideas. Explanatory Design. Correlational Designs
Class 6: Chapter 12 Correlational Designs l 1 Key Ideas Explanatory and predictor designs Characteristics of correlational research Scatterplots and calculating associations Steps in conducting a correlational
More informationCLASSIFICATION AND CLUSTERING. Anveshi Charuvaka
CLASSIFICATION AND CLUSTERING Anveshi Charuvaka Learning from Data Classification Regression Clustering Anomaly Detection Contrast Set Mining Classification: Definition Given a collection of records (training
More informationHomework 11. Part 1. Name: Score: / null
Name: Score: / Homework 11 Part 1 null 1 For which of the following correlations would the data points be clustered most closely around a straight line? A. r = 0.50 B. r = 0.80 C. r = 0.10 D. There is
More informationSummary of Formulas and Concepts. Descriptive Statistics (Ch. 14)
Summary of Formulas and Concepts Descriptive Statistics (Ch. 14) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume
More informationAP Statistics 2001 Solutions and Scoring Guidelines
AP Statistics 2001 Solutions and Scoring Guidelines The materials included in these files are intended for noncommercial use by AP teachers for course and exam preparation; permission for any other use
More informationSPECIAL PERTURBATIONS UNCORRELATED TRACK PROCESSING
AAS 07228 SPECIAL PERTURBATIONS UNCORRELATED TRACK PROCESSING INTRODUCTION James G. Miller * Two historical uncorrelated track (UCT) processing approaches have been employed using general perturbations
More informationBNG 202 Biomechanics Lab. Descriptive statistics and probability distributions I
BNG 202 Biomechanics Lab Descriptive statistics and probability distributions I Overview The overall goal of this short course in statistics is to provide an introduction to descriptive and inferential
More informationPart 2: Community Detection
Chapter 8: Graph Data Part 2: Community Detection Based on Leskovec, Rajaraman, Ullman 2014: Mining of Massive Datasets Big Data Management and Analytics Outline Community Detection  Social networks 
More informationA Demonstration of Hierarchical Clustering
Recitation Supplement: Hierarchical Clustering and Principal Component Analysis in SAS November 18, 2002 The Methods In addition to Kmeans clustering, SAS provides several other types of unsupervised
More informationMining SocialNetwork Graphs
342 Chapter 10 Mining SocialNetwork Graphs There is much information to be gained by analyzing the largescale data that is derived from social networks. The bestknown example of a social network is
More informationInteractive Math Glossary Terms and Definitions
Terms and Definitions Absolute Value the magnitude of a number, or the distance from 0 on a real number line Additive Property of Area the process of finding an the area of a shape by totaling the areas
More informationProtein Protein Interaction Networks
Functional Pattern Mining from Genome Scale Protein Protein Interaction Networks YoungRae Cho, Ph.D. Assistant Professor Department of Computer Science Baylor University it My Definition of Bioinformatics
More informationLayout Based Visualization Techniques for Multi Dimensional Data
Layout Based Visualization Techniques for Multi Dimensional Data Wim de Leeuw Robert van Liere Center for Mathematics and Computer Science, CWI Amsterdam, the Netherlands wimc,robertl @cwi.nl October 27,
More informationSemester 2 Statistics Short courses
Semester 2 Statistics Short courses Course: STAA0001  Basic Statistics Blackboard Site: STAA0001 Dates: Sat 10 th Sept and 22 Oct 2016 (9 am 5 pm) Room EN409 Assumed Knowledge: None Day 1: Exploratory
More informationFig. 1 A typical Knowledge Discovery process [2]
Volume 4, Issue 7, July 2014 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com A Review on Clustering
More informationLecture 2: Descriptive Statistics and Exploratory Data Analysis
Lecture 2: Descriptive Statistics and Exploratory Data Analysis Further Thoughts on Experimental Design 16 Individuals (8 each from two populations) with replicates Pop 1 Pop 2 Randomly sample 4 individuals
More informationMinimum Spanning Trees
Minimum Spanning Trees Algorithms and 18.304 Presentation Outline 1 Graph Terminology Minimum Spanning Trees 2 3 Outline Graph Terminology Minimum Spanning Trees 1 Graph Terminology Minimum Spanning Trees
More informationClustering and Cluster Evaluation. Josh Stuart Tuesday, Feb 24, 2004 Read chap 4 in Causton
Clustering and Cluster Evaluation Josh Stuart Tuesday, Feb 24, 2004 Read chap 4 in Causton Clustering Methods Agglomerative Start with all separate, end with some connected Partitioning / Divisive Start
More informationUnsupervised Data Mining (Clustering)
Unsupervised Data Mining (Clustering) Javier Béjar KEMLG December 01 Javier Béjar (KEMLG) Unsupervised Data Mining (Clustering) December 01 1 / 51 Introduction Clustering in KDD One of the main tasks in
More informationData Mining for Model Creation. Presentation by Paul Below, EDS 2500 NE Plunkett Lane Poulsbo, WA USA 98370 paul.below@eds.
Sept 032305 22 2005 Data Mining for Model Creation Presentation by Paul Below, EDS 2500 NE Plunkett Lane Poulsbo, WA USA 98370 paul.below@eds.com page 1 Agenda Data Mining and Estimating Model Creation
More informationData Mining and Data Warehousing Henryk Maciejewski Data Mining Clustering
Data Mining and Data Warehousing Henryk Maciejewski Data Mining Clustering Clustering Algorithms Contents Kmeans Hierarchical algorithms Linkage functions Vector quantization Clustering Formulation Objects.................................
More informationSocial network analysis with R sna package
Social network analysis with R sna package George Zhang iresearch Consulting Group (China) bird@iresearch.com.cn birdzhangxiang@gmail.com Social network (graph) definition G = (V,E) Max edges = N All possible
More informationData Mining Cluster Analysis: Basic Concepts and Algorithms. Clustering Algorithms. Lecture Notes for Chapter 8. Introduction to Data Mining
Data Mining Cluster Analsis: Basic Concepts and Algorithms Lecture Notes for Chapter 8 Introduction to Data Mining b Tan, Steinbach, Kumar Clustering Algorithms Kmeans and its variants Hierarchical clustering
More informationJustClust User Manual
JustClust User Manual Contents 1. Installing JustClust 2. Running JustClust 3. Basic Usage of JustClust 3.1. Creating a Network 3.2. Clustering a Network 3.3. Applying a Layout 3.4. Saving and Loading
More informationDiagrams and Graphs of Statistical Data
Diagrams and Graphs of Statistical Data One of the most effective and interesting alternative way in which a statistical data may be presented is through diagrams and graphs. There are several ways in
More informationExploratory Data Analysis
Exploratory Data Analysis Johannes Schauer johannes.schauer@tugraz.at Institute of Statistics Graz University of Technology Steyrergasse 17/IV, 8010 Graz www.statistics.tugraz.at February 12, 2008 Introduction
More informationInteractive Exploration of Coherent Patterns in Timeseries Gene Expression Data
8.25.3 Interactive Exploration of Coherent Patterns in Timeseries Gene Expression Data Daxin Jiang Jian Pei Aidong Zhang Computer Science and Engineering Microarray Technology http://www.ipam.ucla.edu/programs/fg2/fgt_speed7.ppt
More informationClustering. Adrian Groza. Department of Computer Science Technical University of ClujNapoca
Clustering Adrian Groza Department of Computer Science Technical University of ClujNapoca Outline 1 Cluster Analysis What is Datamining? Cluster Analysis 2 Kmeans 3 Hierarchical Clustering What is Datamining?
More informationClustering. Danilo Croce Web Mining & Retrieval a.a. 2015/201 16/03/2016
Clustering Danilo Croce Web Mining & Retrieval a.a. 2015/201 16/03/2016 1 Supervised learning vs. unsupervised learning Supervised learning: discover patterns in the data that relate data attributes with
More informationExploratory data analysis (Chapter 2) Fall 2011
Exploratory data analysis (Chapter 2) Fall 2011 Data Examples Example 1: Survey Data 1 Data collected from a Stat 371 class in Fall 2005 2 They answered questions about their: gender, major, year in school,
More informationLecture 20: Clustering
Lecture 20: Clustering Wrapup of neural nets (from last lecture Introduction to unsupervised learning Kmeans clustering COMP424, Lecture 20  April 3, 2013 1 Unsupervised learning In supervised learning,
More informationStatistical Foundations: Measures of Location and Central Tendency and Summation and Expectation
Statistical Foundations: and Central Tendency and and Lecture 4 September 5, 2006 Psychology 790 Lecture #49/05/2006 Slide 1 of 26 Today s Lecture Today s Lecture Where this Fits central tendency/location
More informationHow To Run Statistical Tests in Excel
How To Run Statistical Tests in Excel Microsoft Excel is your best tool for storing and manipulating data, calculating basic descriptive statistics such as means and standard deviations, and conducting
More informationHierarchical Clustering. Clustering Overview
lustering Overview Last lecture What is clustering Partitional algorithms: Kmeans Today s lecture Hierarchical algorithms ensitybased algorithms: SN Techniques for clustering large databases IRH UR ata
More informationChapter 7. Cluster Analysis
Chapter 7. Cluster Analysis. What is Cluster Analysis?. A Categorization of Major Clustering Methods. Partitioning Methods. Hierarchical Methods 5. DensityBased Methods 6. GridBased Methods 7. ModelBased
More information