Massive Labeled Solar Image Data Benchmarks for Automated Feature Recognition

Size: px
Start display at page:

Download "Massive Labeled Solar Image Data Benchmarks for Automated Feature Recognition"

Transcription

1 Massive Labeled Solar Image Data Benchmarks for Automated Feature Recognition Michael A. Schuh1, Rafal A. Angryk2 1 Montana State University, Bozeman, MT 2 Georgia State University, Atlanta, GA

2 Introduction Big Data in Solar Physics (SDO) SDO: ~70,000 high def images of Sun per day (1.5 TB of 4k resolution per day) Traditional manual (human-in-the-loop) analyses infeasible to cover this scale Future mission only getting larger (DKIST,...) Content-based Image Retrieval (CBIR) Similarity Search and Indexing 2 2 of 21

3 Overview Goal: creating large-scale solar image datasets for further scientific inquiry Combining generalized image parameters and automated event labels Today's talk: Domain-specific data analysis and validation Dataset curation and dissemination Preliminary feature recognition results Future directions 3 3 of 21

4 Background Solar Dynamics Observatory (SDO) NASA's Living with a Star (LWS) program AIA instrument: 10 wavelength channels from 94 Å to 4500 Å Dedicated downlink with near real time data stream processing and lossless archival to tape 4 4 of 21

5 Background Feature Finding Team (FFT) International consortium of 16 automated feature (event) recognition modules Direct access to SDO pipeline 5 5 of 21

6 Background FFT Computer Science Trainable Module Montana State University Data Mining Lab Ten generalized image parameters extracted from each cell over a 64 x 64 grid 4,096 cells per image 64 x 64 pixels per cell All AIA channels Example 6 6 of 21

7 The Data Combining raster image with vector objects to create labeled feature (parameter) vectors + Seven FFT event-specific modules + MSU Trainable Module Parameter Data 7 7 of 21

8 The Data Overview of the process Collection Validation Transformation and Labeling Dataset creation Example data range: Jan 2012 (one month) Currently processing 2012 present Highlighting the 5 V's of Big Data Volume, Velocity, Veracity, Variety, Value 8 8 of 21

9 Collection MSU FFT: 6 minute cadence ~ 240 images per day, per wave (x10) ~ 9 million image cells per day (~ 3.25 billion / yr) Stored at MSU, accessible thru web API (soon) FFT Event Modules Reported to and retrieved from the Heliophysics Event Knowledgebase (HEK) Reporting variabilities based on event types Cadence, Counts, Durations, etc. 9 9 of 21

10 Validation Ensuring quality of raw data products Sanity checks for human maintainers Crucial before further works (trusting results) Pipeline Checks Large-scale, scalable analyses Expected cadences Alerting of data gaps and possible outages Sensible event counts and parameter values of 21

11 Validation (cont.) Time difference (in minutes) between image parameter files for each AIA channel of 21

12 Validation (cont.) Time difference (in hours) between reports to the HEK for each event type of 21

13 Validation (cont.) Number of event reports for each unique timestamp for all event types of 21

14 Validation (cont.) 3-statistic (min, avg, max) of P2 (mean) over all AIA channels of 21

15 Data Transformation Two considerations Removing solar expertise Choosing appropriate labels Spatial: coordinate conversions HPC to pixel-space (requires solar metadata) Bounding Boxes vs. Chain Codes Temporal: timestamp locations Events have start and end time (duration) Instantaneous vs. Continuous labeling of 21

16 Dataset Creation Goal: event-specific benchmarking As clean as possible for a given event type Instance Generation Apply an event label to all image cells it covers Instance: <MSU parameters, event label> Considerations Cells vs Regions Multi-class labeling Unlabeled ( quiet ) areas of 21

17 Case Study Two event types from the SPoCA module Active Regions (AR) Coronal Holes (CH) Labels Instantaneous time (smallest, but cleanest set) Bounding boxes Chain codes Cells Region-based cells Regions Regions: 3-stat x 10 params (over all cells) Basic classification evaluation Comparing labeling methods Validating proof of concept for recognition of 21

18 Case Study of 21

19 Case Study Preliminary Machine Learning Benchmarks Algorithms: Naive Bayes (NB), Decision Tree (DT), Support Vector Machine (SVM), K-nearest neighbor (KNN), Random Forests (RF) Methods: Bounding Box (BB), Chain Code (CC) Instances: region-based cells (R-cells), region feature vectors (R-fvs) of 21

20 Conclusions and Future Work Foundational starting place for data use Validations and Expectations Dissemination of dataset(s) Each individual event type Monthly and Cumulatively Stats, Charts, Benchmarks, etc. Advanced Uses Spatiotemporal classification Event tracking and continous labeling Content-based image retrieval with region-based querying Many many more of 21

21 Questions? Thank you. Michael A. Schuh

Big Data New Frontiers: Mining, Search and Management of Massive Repositories of Solar Image Data and Solar Events

Big Data New Frontiers: Mining, Search and Management of Massive Repositories of Solar Image Data and Solar Events Big Data New Frontiers: Mining, Search and Management of Massive Repositories of Solar Image Data and Solar Events Juan M. Banda, Michael A. Schuh, Rafal A. Angryk, Karthik Ganesan Pillai, and Patrick

More information

Big Data: Image & Video Analytics

Big Data: Image & Video Analytics Big Data: Image & Video Analytics How it could support Archiving & Indexing & Searching Dieter Haas, IBM Deutschland GmbH The Big Data Wave 60% of internet traffic is multimedia content (images and videos)

More information

EHR CURATION FOR MEDICAL MINING

EHR CURATION FOR MEDICAL MINING EHR CURATION FOR MEDICAL MINING Ernestina Menasalvas Medical Mining Tutorial@KDD 2015 Sydney, AUSTRALIA 2 Ernestina Menasalvas "EHR Curation for Medical Mining" 08/2015 Agenda Motivation the potential

More information

Steven C.H. Hoi School of Information Systems Singapore Management University Email: chhoi@smu.edu.sg

Steven C.H. Hoi School of Information Systems Singapore Management University Email: chhoi@smu.edu.sg Steven C.H. Hoi School of Information Systems Singapore Management University Email: chhoi@smu.edu.sg Introduction http://stevenhoi.org/ Finance Recommender Systems Cyber Security Machine Learning Visual

More information

Information Management course

Information Management course Università degli Studi di Milano Master Degree in Computer Science Information Management course Teacher: Alberto Ceselli Lecture 01 : 06/10/2015 Practical informations: Teacher: Alberto Ceselli (alberto.ceselli@unimi.it)

More information

The Scientific Data Mining Process

The Scientific Data Mining Process Chapter 4 The Scientific Data Mining Process When I use a word, Humpty Dumpty said, in rather a scornful tone, it means just what I choose it to mean neither more nor less. Lewis Carroll [87, p. 214] In

More information

Big Data Analytics. Genoveva Vargas-Solar http://www.vargas-solar.com/big-data-analytics French Council of Scientific Research, LIG & LAFMIA Labs

Big Data Analytics. Genoveva Vargas-Solar http://www.vargas-solar.com/big-data-analytics French Council of Scientific Research, LIG & LAFMIA Labs 1 Big Data Analytics Genoveva Vargas-Solar http://www.vargas-solar.com/big-data-analytics French Council of Scientific Research, LIG & LAFMIA Labs Montevideo, 22 nd November 4 th December, 2015 INFORMATIQUE

More information

Feature Subset Selection in E-mail Spam Detection

Feature Subset Selection in E-mail Spam Detection Feature Subset Selection in E-mail Spam Detection Amir Rajabi Behjat, Universiti Technology MARA, Malaysia IT Security for the Next Generation Asia Pacific & MEA Cup, Hong Kong 14-16 March, 2012 Feature

More information

Advanced Analytics for Call Center Operations

Advanced Analytics for Call Center Operations Advanced Analytics for Call Center Operations Ali Cabukel, Senior Data Mining Specialist Global Bilgi Kubra Fenerci Canel, Big Data Solutions Lead Oracle Speaker Bio Ali Çabukel Graduated from Hacettepe

More information

COMP9321 Web Application Engineering

COMP9321 Web Application Engineering COMP9321 Web Application Engineering Semester 2, 2015 Dr. Amin Beheshti Service Oriented Computing Group, CSE, UNSW Australia Week 11 (Part II) http://webapps.cse.unsw.edu.au/webcms2/course/index.php?cid=2411

More information

Introduction to Data Mining

Introduction to Data Mining Introduction to Data Mining Jay Urbain Credits: Nazli Goharian & David Grossman @ IIT Outline Introduction Data Pre-processing Data Mining Algorithms Naïve Bayes Decision Tree Neural Network Association

More information

Environmental Remote Sensing GEOG 2021

Environmental Remote Sensing GEOG 2021 Environmental Remote Sensing GEOG 2021 Lecture 4 Image classification 2 Purpose categorising data data abstraction / simplification data interpretation mapping for land cover mapping use land cover class

More information

Active Learning SVM for Blogs recommendation

Active Learning SVM for Blogs recommendation Active Learning SVM for Blogs recommendation Xin Guan Computer Science, George Mason University Ⅰ.Introduction In the DH Now website, they try to review a big amount of blogs and articles and find the

More information

How to use Big Data in Industry 4.0 implementations. LAURI ILISON, PhD Head of Big Data and Machine Learning

How to use Big Data in Industry 4.0 implementations. LAURI ILISON, PhD Head of Big Data and Machine Learning How to use Big Data in Industry 4.0 implementations LAURI ILISON, PhD Head of Big Data and Machine Learning Big Data definition? Big Data is about structured vs unstructured data Big Data is about Volume

More information

Data, Measurements, Features

Data, Measurements, Features Data, Measurements, Features Middle East Technical University Dep. of Computer Engineering 2009 compiled by V. Atalay What do you think of when someone says Data? We might abstract the idea that data are

More information

lop Building Machine Learning Systems with Python en source

lop Building Machine Learning Systems with Python en source Building Machine Learning Systems with Python Master the art of machine learning with Python and build effective machine learning systems with this intensive handson guide Willi Richert Luis Pedro Coelho

More information

Knowledge Discovery and Data Mining. Bootstrap review. Bagging Important Concepts. Notes. Lecture 19 - Bagging. Tom Kelsey. Notes

Knowledge Discovery and Data Mining. Bootstrap review. Bagging Important Concepts. Notes. Lecture 19 - Bagging. Tom Kelsey. Notes Knowledge Discovery and Data Mining Lecture 19 - Bagging Tom Kelsey School of Computer Science University of St Andrews http://tom.host.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey ID5059-19-B &

More information

Coronal Hole Properties in Solar Cycle 24

Coronal Hole Properties in Solar Cycle 24 Coronal Hole Properties in Solar Cycle 24 Adrián Arteaga Harvard College Dr. Mari Paz Miralles Harvard-Smithsonian Center for Astrophysics What is a Coronal Hole? Regions of open magnetic flux in the solar

More information

Using D2K Data Mining Platform for Understanding the Dynamic Evolution of Land-Surface Variables

Using D2K Data Mining Platform for Understanding the Dynamic Evolution of Land-Surface Variables Using D2K Data Mining Platform for Understanding the Dynamic Evolution of Land-Surface Variables Praveen Kumar 1, Peter Bajcsy 2, David Tcheng 2, David Clutter 2, Vikas Mehra 1, Wei-Wen Feng 2, Pratyush

More information

ANALYTICS IN BIG DATA ERA

ANALYTICS IN BIG DATA ERA ANALYTICS IN BIG DATA ERA ANALYTICS TECHNOLOGY AND ARCHITECTURE TO MANAGE VELOCITY AND VARIETY, DISCOVER RELATIONSHIPS AND CLASSIFY HUGE AMOUNT OF DATA MAURIZIO SALUSTI SAS Copyr i g ht 2012, SAS Ins titut

More information

Advanced In-Database Analytics

Advanced In-Database Analytics Advanced In-Database Analytics Tallinn, Sept. 25th, 2012 Mikko-Pekka Bertling, BDM Greenplum EMEA 1 That sounds complicated? 2 Who can tell me how best to solve this 3 What are the main mathematical functions??

More information

Content-Based Recommendation

Content-Based Recommendation Content-Based Recommendation Content-based? Item descriptions to identify items that are of particular interest to the user Example Example Comparing with Noncontent based Items User-based CF Searches

More information

Big Data and Analytics: Challenges and Opportunities

Big Data and Analytics: Challenges and Opportunities Big Data and Analytics: Challenges and Opportunities Dr. Amin Beheshti Lecturer and Senior Research Associate University of New South Wales, Australia (Service Oriented Computing Group, CSE) Talk: Sharif

More information

GEOGRAPHIC CONTEXT ANALYSIS OF VOLUNTEERED INFORMATION

GEOGRAPHIC CONTEXT ANALYSIS OF VOLUNTEERED INFORMATION GEOGRAPHIC CONTEXT ANALYSIS OF VOLUNTEERED INFORMATION (GEOCONAVI) Frank O. Ostermann COST Energic Meeting 26.05.2014, Zürich GEOGRAPHIC CONTEXT ANALYSIS OF VOLUNTEERED INFORMATION PRESENTATION OVERVIEW

More information

Introduction to Data Mining

Introduction to Data Mining Introduction to Data Mining 1 Why Data Mining? Explosive Growth of Data Data collection and data availability Automated data collection tools, Internet, smartphones, Major sources of abundant data Business:

More information

ADVANCED MACHINE LEARNING. Introduction

ADVANCED MACHINE LEARNING. Introduction 1 1 Introduction Lecturer: Prof. Aude Billard (aude.billard@epfl.ch) Teaching Assistants: Guillaume de Chambrier, Nadia Figueroa, Denys Lamotte, Nicola Sommer 2 2 Course Format Alternate between: Lectures

More information

SIPAC. Signals and Data Identification, Processing, Analysis, and Classification

SIPAC. Signals and Data Identification, Processing, Analysis, and Classification SIPAC Signals and Data Identification, Processing, Analysis, and Classification Framework for Mass Data Processing with Modules for Data Storage, Production and Configuration SIPAC key features SIPAC is

More information

Knowledge Discovery and Data Mining

Knowledge Discovery and Data Mining Knowledge Discovery and Data Mining Unit # 11 Sajjad Haider Fall 2013 1 Supervised Learning Process Data Collection/Preparation Data Cleaning Discretization Supervised/Unuspervised Identification of right

More information

Advanced Image Management using the Mosaic Dataset

Advanced Image Management using the Mosaic Dataset Esri International User Conference San Diego, California Technical Workshops July 25, 2012 Advanced Image Management using the Mosaic Dataset Vinay Viswambharan, Mike Muller Agenda ArcGIS Image Management

More information

Sense Making in an IOT World: Sensor Data Analysis with Deep Learning

Sense Making in an IOT World: Sensor Data Analysis with Deep Learning Sense Making in an IOT World: Sensor Data Analysis with Deep Learning Natalia Vassilieva, PhD Senior Research Manager GTC 2016 Deep learning proof points as of today Vision Speech Text Other Search & information

More information

Let the data speak to you. Look Who s Peeking at Your Paycheck. Big Data. What is Big Data? The Artemis project: Saving preemies using Big Data

Let the data speak to you. Look Who s Peeking at Your Paycheck. Big Data. What is Big Data? The Artemis project: Saving preemies using Big Data CS535 Big Data W1.A.1 CS535 BIG DATA W1.A.2 Let the data speak to you Medication Adherence Score How likely people are to take their medication, based on: How long people have lived at the same address

More information

Data Mining. Nonlinear Classification

Data Mining. Nonlinear Classification Data Mining Unit # 6 Sajjad Haider Fall 2014 1 Nonlinear Classification Classes may not be separable by a linear boundary Suppose we randomly generate a data set as follows: X has range between 0 to 15

More information

Big Data and Analytics: Getting Started with ArcGIS. Mike Park Erik Hoel

Big Data and Analytics: Getting Started with ArcGIS. Mike Park Erik Hoel Big Data and Analytics: Getting Started with ArcGIS Mike Park Erik Hoel Agenda Overview of big data Distributed computation User experience Data management Big data What is it? Big Data is a loosely defined

More information

High Productivity Data Processing Analytics Methods with Applications

High Productivity Data Processing Analytics Methods with Applications High Productivity Data Processing Analytics Methods with Applications Dr. Ing. Morris Riedel et al. Adjunct Associate Professor School of Engineering and Natural Sciences, University of Iceland Research

More information

Conquering the Astronomical Data Flood through Machine

Conquering the Astronomical Data Flood through Machine Conquering the Astronomical Data Flood through Machine Learning and Citizen Science Kirk Borne George Mason University School of Physics, Astronomy, & Computational Sciences http://spacs.gmu.edu/ The Problem:

More information

Data Centric Systems (DCS)

Data Centric Systems (DCS) Data Centric Systems (DCS) Architecture and Solutions for High Performance Computing, Big Data and High Performance Analytics High Performance Computing with Data Centric Systems 1 Data Centric Systems

More information

CLASSIFYING NETWORK TRAFFIC IN THE BIG DATA ERA

CLASSIFYING NETWORK TRAFFIC IN THE BIG DATA ERA CLASSIFYING NETWORK TRAFFIC IN THE BIG DATA ERA Professor Yang Xiang Network Security and Computing Laboratory (NSCLab) School of Information Technology Deakin University, Melbourne, Australia http://anss.org.au/nsclab

More information

Deep Learning Meets Heterogeneous Computing. Dr. Ren Wu Distinguished Scientist, IDL, Baidu wuren@baidu.com

Deep Learning Meets Heterogeneous Computing. Dr. Ren Wu Distinguished Scientist, IDL, Baidu wuren@baidu.com Deep Learning Meets Heterogeneous Computing Dr. Ren Wu Distinguished Scientist, IDL, Baidu wuren@baidu.com Baidu Everyday 5b+ queries 500m+ users 100m+ mobile users 100m+ photos Big Data Storage Processing

More information

Biomedical Informatics Applications, Big Data, & Cloud Computing

Biomedical Informatics Applications, Big Data, & Cloud Computing Biomedical Informatics Applications, Big Data, & Cloud Computing Patrick Widener, PhD Assistant Professor, Biomedical Engineering Senior Research Scientist, Center for Comprehensive Informatics Emory University

More information

NAVIGATING SCIENTIFIC LITERATURE A HOLISTIC PERSPECTIVE. Venu Govindaraju

NAVIGATING SCIENTIFIC LITERATURE A HOLISTIC PERSPECTIVE. Venu Govindaraju NAVIGATING SCIENTIFIC LITERATURE A HOLISTIC PERSPECTIVE Venu Govindaraju BIOMETRICS DOCUMENT ANALYSIS PATTERN RECOGNITION 8/24/2015 ICDAR- 2015 2 Towards a Globally Optimal Approach for Learning Deep Unsupervised

More information

Exploring Big Data in Social Networks

Exploring Big Data in Social Networks Exploring Big Data in Social Networks virgilio@dcc.ufmg.br (meira@dcc.ufmg.br) INWEB National Science and Technology Institute for Web Federal University of Minas Gerais - UFMG May 2013 Some thoughts about

More information

Machine Learning: Overview

Machine Learning: Overview Machine Learning: Overview Why Learning? Learning is a core of property of being intelligent. Hence Machine learning is a core subarea of Artificial Intelligence. There is a need for programs to behave

More information

Scalable Developments for Big Data Analytics in Remote Sensing

Scalable Developments for Big Data Analytics in Remote Sensing Scalable Developments for Big Data Analytics in Remote Sensing Federated Systems and Data Division Research Group High Productivity Data Processing Dr.-Ing. Morris Riedel et al. Research Group Leader,

More information

Graph Database Performance: An Oracle Perspective

Graph Database Performance: An Oracle Perspective Graph Database Performance: An Oracle Perspective Xavier Lopez, Ph.D. Senior Director, Product Management 1 Copyright 2012, Oracle and/or its affiliates. All rights reserved. Program Agenda Broad Perspective

More information

Automatic land-cover map production of agricultural areas using supervised classification of SPOT4(Take5) and Landsat-8 image time series.

Automatic land-cover map production of agricultural areas using supervised classification of SPOT4(Take5) and Landsat-8 image time series. Automatic land-cover map production of agricultural areas using supervised classification of SPOT4(Take5) and Landsat-8 image time series. Jordi Inglada 2014/11/18 SPOT4/Take5 User Workshop 2014/11/18

More information

Introduction to Machine Learning. Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011

Introduction to Machine Learning. Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011 Introduction to Machine Learning Speaker: Harry Chao Advisor: J.J. Ding Date: 1/27/2011 1 Outline 1. What is machine learning? 2. The basic of machine learning 3. Principles and effects of machine learning

More information

Industrial Challenges for Content-Based Image Retrieval

Industrial Challenges for Content-Based Image Retrieval Title Slide Industrial Challenges for Content-Based Image Retrieval Chahab Nastar, CEO Vienna, 20 September 2005 www.ltutech.com LTU technologies Page 1 Agenda CBIR what is it good for? Technological challenges

More information

GPU Programming in Computer Vision

GPU Programming in Computer Vision Computer Vision Group Prof. Daniel Cremers GPU Programming in Computer Vision Preliminary Meeting Thomas Möllenhoff, Robert Maier, Caner Hazirbas What you will learn in the practical course Introduction

More information

Practical Data Science with Azure Machine Learning, SQL Data Mining, and R

Practical Data Science with Azure Machine Learning, SQL Data Mining, and R Practical Data Science with Azure Machine Learning, SQL Data Mining, and R Overview This 4-day class is the first of the two data science courses taught by Rafal Lukawiecki. Some of the topics will be

More information

Opportunities and Challenges in Big Data Neuroscience

Opportunities and Challenges in Big Data Neuroscience Opportunities and Challenges in Big Data Neuroscience Joshua T. Vogelstein {BME, ICM, CIS, IDIES}@JHU Co-founder and Director of the Open Connectome Project e: jovo@jhu.edu, w: http://ocp.me Why is it

More information

An Introduction to Data Mining. Big Data World. Related Fields and Disciplines. What is Data Mining? 2/12/2015

An Introduction to Data Mining. Big Data World. Related Fields and Disciplines. What is Data Mining? 2/12/2015 An Introduction to Data Mining for Wind Power Management Spring 2015 Big Data World Every minute: Google receives over 4 million search queries Facebook users share almost 2.5 million pieces of content

More information

Biomedical Big Data for Clinical Research and Patient Care: Role of Semantic Computing

Biomedical Big Data for Clinical Research and Patient Care: Role of Semantic Computing Biomedical Big Data for Clinical Research and Patient Care: Role of Semantic Computing Satya S. Sahoo Division Medical Informatics Case Western Reserve University June 17, 2014 1200 Signal Big Data: Research

More information

Distributed forests for MapReduce-based machine learning

Distributed forests for MapReduce-based machine learning Distributed forests for MapReduce-based machine learning Ryoji Wakayama, Ryuei Murata, Akisato Kimura, Takayoshi Yamashita, Yuji Yamauchi, Hironobu Fujiyoshi Chubu University, Japan. NTT Communication

More information

Making Sense of Physician Notes: A Big Data Approach. Anupam Joshi UMBC joshi@umbc.edu Joint work with students, UBMC Colleagues, and UMMS Colleagues

Making Sense of Physician Notes: A Big Data Approach. Anupam Joshi UMBC joshi@umbc.edu Joint work with students, UBMC Colleagues, and UMMS Colleagues Making Sense of Physician Notes: A Big Data Approach Anupam Joshi UMBC joshi@umbc.edu Joint work with students, UBMC Colleagues, and UMMS Colleagues Where we are Significant progress in applying NLP and

More information

Introduzione alle Biblioteche Digitali Audio/Video

Introduzione alle Biblioteche Digitali Audio/Video Introduzione alle Biblioteche Digitali Audio/Video Biblioteche Digitali 1 Gestione del video Perchè è importante poter gestire biblioteche digitali di audiovisivi Caratteristiche specifiche dell audio/video

More information

Workload Characterization and Analysis of Storage and Bandwidth Needs of LEAD Workspace

Workload Characterization and Analysis of Storage and Bandwidth Needs of LEAD Workspace Workload Characterization and Analysis of Storage and Bandwidth Needs of LEAD Workspace Beth Plale Indiana University plale@cs.indiana.edu LEAD TR 001, V3.0 V3.0 dated January 24, 2007 V2.0 dated August

More information

Canadian Astronomy Data Centre. Séverin Gaudet David Schade Canadian Astronomy Data Centre

Canadian Astronomy Data Centre. Séverin Gaudet David Schade Canadian Astronomy Data Centre Canadian Astronomy Data Centre Séverin Gaudet David Schade Canadian Astronomy Data Centre Data Activities in Astronomy Features of the astronomy data landscape Multi-wavelength datasets are increasingly

More information

Mining Big Data. Pang-Ning Tan. Associate Professor Dept of Computer Science & Engineering Michigan State University

Mining Big Data. Pang-Ning Tan. Associate Professor Dept of Computer Science & Engineering Michigan State University Mining Big Data Pang-Ning Tan Associate Professor Dept of Computer Science & Engineering Michigan State University Website: http://www.cse.msu.edu/~ptan Google Trends Big Data Smart Cities Big Data and

More information

Bing Liu. Web Data Mining. Exploring Hyperlinks, Contents, and Usage Data. With 177 Figures. ~ Spring~r

Bing Liu. Web Data Mining. Exploring Hyperlinks, Contents, and Usage Data. With 177 Figures. ~ Spring~r Bing Liu Web Data Mining Exploring Hyperlinks, Contents, and Usage Data With 177 Figures ~ Spring~r Table of Contents 1. Introduction.. 1 1.1. What is the World Wide Web? 1 1.2. ABrief History of the Web

More information

Azure Machine Learning, SQL Data Mining and R

Azure Machine Learning, SQL Data Mining and R Azure Machine Learning, SQL Data Mining and R Day-by-day Agenda Prerequisites No formal prerequisites. Basic knowledge of SQL Server Data Tools, Excel and any analytical experience helps. Best of all:

More information

Image Analytics on Big Data In Motion Implementation of Image Analytics CCL in Apache Kafka and Storm

Image Analytics on Big Data In Motion Implementation of Image Analytics CCL in Apache Kafka and Storm Image Analytics on Big Data In Motion Implementation of Image Analytics CCL in Apache Kafka and Storm Lokesh Babu Rao 1 C. Elayaraja 2 1PG Student, Dept. of ECE, Dhaanish Ahmed College of Engineering,

More information

Data Literacy For All: Astrophysics and Beyond (Astronomy is evidence-based forensic science, thus it is a data & information science)

Data Literacy For All: Astrophysics and Beyond (Astronomy is evidence-based forensic science, thus it is a data & information science) Data Literacy For All: Astrophysics and Beyond (Astronomy is evidence-based forensic science, thus it is a data & information science) Kirk Borne George Mason University, Fairfax, VA www.kirkborne.net

More information

Solar Irradiance Forecasting Using Multi-layer Cloud Tracking and Numerical Weather Prediction

Solar Irradiance Forecasting Using Multi-layer Cloud Tracking and Numerical Weather Prediction Solar Irradiance Forecasting Using Multi-layer Cloud Tracking and Numerical Weather Prediction Jin Xu, Shinjae Yoo, Dantong Yu, Dong Huang, John Heiser, Paul Kalb Solar Energy Abundant, clean, and secure

More information

HPC technology and future architecture

HPC technology and future architecture HPC technology and future architecture Visual Analysis for Extremely Large-Scale Scientific Computing KGT2 Internal Meeting INRIA France Benoit Lange benoit.lange@inria.fr Toàn Nguyên toan.nguyen@inria.fr

More information

Random forest algorithm in big data environment

Random forest algorithm in big data environment Random forest algorithm in big data environment Yingchun Liu * School of Economics and Management, Beihang University, Beijing 100191, China Received 1 September 2014, www.cmnt.lv Abstract Random forest

More information

Monday Morning Data Mining

Monday Morning Data Mining Monday Morning Data Mining Tim Ruhe Statistische Methoden der Datenanalyse Outline: - data mining - IceCube - Data mining in IceCube Computer Scientists are different... Fakultät Physik Fakultät Physik

More information

SeaCloudDM: Massive Heterogeneous Sensor Data Management in the Internet of Things

SeaCloudDM: Massive Heterogeneous Sensor Data Management in the Internet of Things SeaCloudDM: Massive Heterogeneous Sensor Data Management in the Internet of Things Jiajie Xu Institute of Software, Chinese Academy of Sciences (ISCAS) 2012-05-15 Outline 1. Challenges in IoT Data Management

More information

Mining a Corpus of Job Ads

Mining a Corpus of Job Ads Mining a Corpus of Job Ads Workshop Strings and Structures Computational Biology & Linguistics Jürgen Jürgen Hermes Hermes Sprachliche Linguistic Data Informationsverarbeitung Processing Institut Department

More information

Learning from Big Data in

Learning from Big Data in Learning from Big Data in Astronomy an overview Kirk Borne George Mason University School of Physics, Astronomy, & Computational Sciences http://spacs.gmu.edu/ From traditional astronomy 2 to Big Data

More information

Programme Specification

Programme Specification Programme Specification Awarding Body/Institution Teaching Institution Queen Mary, University of London Queen Mary, University of London Name of Final Award and Programme Title Master of Science (MSc)

More information

An intelligent tool for expediting and automating data mining steps. Ourania Hatzi, Nikolaos Zorbas, Mara Nikolaidou and Dimosthenis Anagnostopoulos

An intelligent tool for expediting and automating data mining steps. Ourania Hatzi, Nikolaos Zorbas, Mara Nikolaidou and Dimosthenis Anagnostopoulos An intelligent tool for expediting and automating data mining steps Ourania Hatzi, Nikolaos Zorbas, Mara Nikolaidou and Dimosthenis Anagnostopoulos Outline Data Mining, current tools An intelligent tool

More information

Data Mining. Dr. Saed Sayad. University of Toronto 2010 saed.sayad@utoronto.ca. http://chem-eng.utoronto.ca/~datamining/

Data Mining. Dr. Saed Sayad. University of Toronto 2010 saed.sayad@utoronto.ca. http://chem-eng.utoronto.ca/~datamining/ Data Mining Dr. Saed Sayad University of Toronto 2010 saed.sayad@utoronto.ca http://chem-eng.utoronto.ca/~datamining/ 1 Data Mining Data mining is about explaining the past and predicting the future by

More information

Data Mining for Customer Service Support. Senioritis Seminar Presentation Megan Boice Jay Carter Nick Linke KC Tobin

Data Mining for Customer Service Support. Senioritis Seminar Presentation Megan Boice Jay Carter Nick Linke KC Tobin Data Mining for Customer Service Support Senioritis Seminar Presentation Megan Boice Jay Carter Nick Linke KC Tobin Traditional Hotline Services Problem Traditional Customer Service Support (manufacturing)

More information

A Service for Data-Intensive Computations on Virtual Clusters

A Service for Data-Intensive Computations on Virtual Clusters A Service for Data-Intensive Computations on Virtual Clusters Executing Preservation Strategies at Scale Rainer Schmidt, Christian Sadilek, and Ross King rainer.schmidt@arcs.ac.at Planets Project Permanent

More information

INTERSEC BENCHMARK. High Performance for Fast Data & Real-Time Analytics Part I: Vs Hadoop

INTERSEC BENCHMARK. High Performance for Fast Data & Real-Time Analytics Part I: Vs Hadoop INTERSEC BENCHMARK High Performance for Fast Data & Real-Time Analytics Part I: Vs Hadoop BENCHMARK VS HADOOP (STAND ALONE OR COMBINED) Intersec solution in a Redhat Openstack NFV framework complements

More information

Software challenges in the implementation of large surveys: the case of J-PAS

Software challenges in the implementation of large surveys: the case of J-PAS Software challenges in the implementation of large surveys: the case of J-PAS 1/21 Paulo Penteado - IAG/USP pp.penteado@gmail.com http://www.ppenteado.net/ast/pp_lsst_201204.pdf (K. Taylor) (A. Fernández-Soto)

More information

Large Scale Mobility Analysis: Extracting Significant Places using Hadoop/Hive and Spatial Processing

Large Scale Mobility Analysis: Extracting Significant Places using Hadoop/Hive and Spatial Processing Large Scale Mobility Analysis: Extracting Significant Places using Hadoop/Hive and Spatial Processing Apichon Witayangkurn 1, Teerayut Horanont 2, Masahiko Nagai 1, Ryosuke Shibasaki 1 1 Institute of Industrial

More information

Machine learning for algo trading

Machine learning for algo trading Machine learning for algo trading An introduction for nonmathematicians Dr. Aly Kassam Overview High level introduction to machine learning A machine learning bestiary What has all this got to do with

More information

Search Taxonomy. Web Search. Search Engine Optimization. Information Retrieval

Search Taxonomy. Web Search. Search Engine Optimization. Information Retrieval Information Retrieval INFO 4300 / CS 4300! Retrieval models Older models» Boolean retrieval» Vector Space model Probabilistic Models» BM25» Language models Web search» Learning to Rank Search Taxonomy!

More information

Machine Learning Capacity and Performance Analysis and R

Machine Learning Capacity and Performance Analysis and R Machine Learning and R May 3, 11 30 25 15 10 5 25 15 10 5 30 25 15 10 5 0 2 4 6 8 101214161822 0 2 4 6 8 101214161822 0 2 4 6 8 101214161822 100 80 60 40 100 80 60 40 100 80 60 40 30 25 15 10 5 25 15 10

More information

WebFOCUS RStat. RStat. Predict the Future and Make Effective Decisions Today. WebFOCUS RStat

WebFOCUS RStat. RStat. Predict the Future and Make Effective Decisions Today. WebFOCUS RStat Information Builders enables agile information solutions with business intelligence (BI) and integration technologies. WebFOCUS the most widely utilized business intelligence platform connects to any enterprise

More information

Email: justinjia@ust.hk Office: LSK 5045 Begin subject: [ISOM3360]...

Email: justinjia@ust.hk Office: LSK 5045 Begin subject: [ISOM3360]... Business Intelligence and Data Mining ISOM 3360: Spring 2015 Instructor Contact Office Hours Course Schedule and Classroom Course Webpage Jia Jia, ISOM Email: justinjia@ust.hk Office: LSK 5045 Begin subject:

More information

Big Data Research in the AMPLab: BDAS and Beyond

Big Data Research in the AMPLab: BDAS and Beyond Big Data Research in the AMPLab: BDAS and Beyond Michael Franklin UC Berkeley 1 st Spark Summit December 2, 2013 UC BERKELEY AMPLab: Collaborative Big Data Research Launched: January 2011, 6 year planned

More information

Data Validation and Data Management Solutions

Data Validation and Data Management Solutions FRONTIER TECHNOLOGY, INC. Advanced Technology for Superior Solutions. and Solutions Abstract Within the performance evaluation and calibration communities, test programs are driven by requirements, test

More information

Text Analytics and Big Data

Text Analytics and Big Data Text Analytics and Big Data META-FORUM 2012 Brussels, 20 th June 2012 Atos Research & Innovation 1 Table of Contents 1. Atos and why we are here 2. Examples 3. BIG: Big Data Public Private Forum 2 2 Atos:

More information

ASKAP Science Data Archive: Users and Requirements CSIRO ASTRONOMY AND SPACE SCIENCE (CASS)

ASKAP Science Data Archive: Users and Requirements CSIRO ASTRONOMY AND SPACE SCIENCE (CASS) ASKAP Science Data Archive: Users and Requirements CSIRO ASTRONOMY AND SPACE SCIENCE (CASS) Jessica Chapman, Data Workshop March 2013 ASKAP Science Data Archive Talk outline Data flow in brief Some radio

More information

Data Integration and long-term planning of the Observing Systems as a cross-cutting process in a NMS

Data Integration and long-term planning of the Observing Systems as a cross-cutting process in a NMS Data Integration and long-term planning of the Observing Systems as a cross-cutting process in a NMS ECAC Zurich, Setpember 15 2020 Ch. Häberli Deputy Head Climate Division/Head Meteorological Data Coordination

More information

Progress Towards the Solar Dynamics Observatory

Progress Towards the Solar Dynamics Observatory Progress Towards the Solar Dynamics Observatory Barbara J. Thompson SDO Project Scientist W. Dean Pesnell SDO Assistant Project Scientist Page 1 SDO OVERVIEW Mission Science Objectives The primary goal

More information

Applying Data Science to Sales Pipelines for Fun and Profit

Applying Data Science to Sales Pipelines for Fun and Profit Applying Data Science to Sales Pipelines for Fun and Profit Andy Twigg, CTO, C9 @lambdatwigg Abstract Machine learning is now routinely applied to many areas of industry. At C9, we apply machine learning

More information

Copernicus and Big Data: Challenges and Opportunities

Copernicus and Big Data: Challenges and Opportunities Copernicus and Big Data: Challenges and Opportunities Alessandro Annoni European Commission Joint Research Centre www.jrc.ec.europa.eu Serving society Stimulating innovation Supporting legislation Big

More information

Is a Data Scientist the New Quant? Stuart Kozola MathWorks

Is a Data Scientist the New Quant? Stuart Kozola MathWorks Is a Data Scientist the New Quant? Stuart Kozola MathWorks 2015 The MathWorks, Inc. 1 Facts or information used usually to calculate, analyze, or plan something Information that is produced or stored by

More information

On Establishing Big Data Breakwaters

On Establishing Big Data Breakwaters On Establishing Big Data Breakwaters with Analytics Dr. - Ing. Morris Riedel Head of Research Group High Productivity Data Processing, Juelich Supercomputing Centre, Germany Adjunct Associated Professor,

More information

esdo Algorithms, Data Centre and Visualization Tools

esdo Algorithms, Data Centre and Visualization Tools Astron. Nachr. / AN 000, No. 00, 1 6 (0000) / DOI please set DOI! esdo Algorithms, Data Centre and Visualization Tools E. Auden 1, T. Toutain 2, and S. Zharkov 3 1 Mullard Space Science Laboratory, University

More information

CINECA Innovative Open Source Technologies for a CRIS: SURplus ~ www.cineca.it

CINECA Innovative Open Source Technologies for a CRIS: SURplus ~ www.cineca.it CINECA Innovative Open Source Technologies for a CRIS: SURplus ~ www.cineca.it Topics CINECA: a brief overview Solutions for Higher Education & Research Institutions Three innovative open-source technologies

More information

Objectives. Raster Data Discrete Classes. Spatial Information in Natural Resources FANR 3800. Review the raster data model

Objectives. Raster Data Discrete Classes. Spatial Information in Natural Resources FANR 3800. Review the raster data model Spatial Information in Natural Resources FANR 3800 Raster Analysis Objectives Review the raster data model Understand how raster analysis fundamentally differs from vector analysis Become familiar with

More information

A Novel Cloud Based Elastic Framework for Big Data Preprocessing

A Novel Cloud Based Elastic Framework for Big Data Preprocessing School of Systems Engineering A Novel Cloud Based Elastic Framework for Big Data Preprocessing Omer Dawelbeit and Rachel McCrindle October 21, 2014 University of Reading 2008 www.reading.ac.uk Overview

More information

Galaxy Morphological Classification

Galaxy Morphological Classification Galaxy Morphological Classification Jordan Duprey and James Kolano Abstract To solve the issue of galaxy morphological classification according to a classification scheme modelled off of the Hubble Sequence,

More information

Open Source UAS Software Toolkits. Keith Fieldhouse Technical Lead, Kitware Inc. keith.fieldhouse@kitware.com

Open Source UAS Software Toolkits. Keith Fieldhouse Technical Lead, Kitware Inc. keith.fieldhouse@kitware.com Open Source UAS Software Toolkits Keith Fieldhouse Technical Lead, Kitware Inc. keith.fieldhouse@kitware.com 1 Best known for open source toolkits and applications Collaborative software R&D: Algorithms

More information

Scalable End-User Access to Big Data http://www.optique-project.eu/ HELLENIC REPUBLIC National and Kapodistrian University of Athens

Scalable End-User Access to Big Data http://www.optique-project.eu/ HELLENIC REPUBLIC National and Kapodistrian University of Athens Scalable End-User Access to Big Data http://www.optique-project.eu/ HELLENIC REPUBLIC National and Kapodistrian University of Athens 1 Optique: Improving the competitiveness of European industry For many

More information