# Predicting the Stock Market with News Articles

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Predicting the Stock Market with News Articles Kari Lee and Ryan Timmons CS224N Final Project Introduction Stock market prediction is an area of extreme importance to an entire industry. Stock price is determined by the behavior of human investors, and the investors determine stock prices by using publicly available information to predict how the market will act or react. Financial news articles can thus play a large role in influencing the movement of a stock as humans react to the information. Previous research has suggested that there is a relationship between news articles and stock price movement, as there is some lag between when the news article is released and when the market has moved to reflect this information. We built a text classification system to categorize news articles containing references to a list of companies and predict the impact of the news on the stock price. Our trading system to act on these predictions outperformed a baseline strategy of simply holding on to equal amounts of the stocks in question for the test time period. Previous Research Gidofalvi trained a naive Bayesian text classifier by scoring news articles using a function combining the change in price with the β-value, which measures the volatility of the stock. The articles were given one of three labels based on the stock s movement compared to its expected movement. Using this method, the predictive power of the classifier was limited, but there was a strong correlation between the news article and the stock price behavior within a 20 minute window around the news article s release time. Fung, et al designed a system which used a t-test based split-and-merge piecewise linear approximation algorithm to filter out the noise of the movement over the stock time series. Features were the words in a document weighted using term frequency inverse document frequency, and the optimization problem was solved using a Support Vector Machine. The resulting system was most stable and appropriate for predictions within 3-5 days. Algorithms Bag of Words We generated four different word distributions for each company in question based on proximity to the company reference in the article - the paragraph containing the reference, the paragraph preceding the reference, the paragraph after the reference, and the rest of the article. The word counts for each distribution were incremented for every reference in each article. The word counts were weighted by the stock price change for the article date as well as by the weight of the paragraph (e.g. weighting the paragraph containing the reference higher).

2 Algorithm to compute relevance for each word in training: foreach (article in training set) foreach (word in article) word relevance = Sum over paragraphs (word count for paragraph * weight for paragraph) end foreach Total relevance = Sum over all words in article (word relevance) Multiple each word's relevance by (daily % change for stock / Total relevance) Add each word relevance to total score for word end foreach foreach (word in data) normalize - divide word's score by number of articles word has appeared in, to get an average score per article end foreach The output is a map of (words, scores). When making a prediction on a test article, the algorithm is: score = 0 foreach (word in article) score += (word score) * Sum over paragraphs (word count for paragraph * weight for paragraph) endforeach The returned score is the predicted daily percentage change for the stock in question. We chose this way of weighting the scores so that if a predictor was trained on exactly one word bag and then asked to predict based on that word bag, it would return the actual change for that day exactly. Implementation We used New York Times articles from the EnglishGigaWord corpus, which spans the timeframe July 1994 December There was no tag to mark financial articles specifically, but we separated out any articles of type story which contained a reference to a company name on a modified list of the Dow Jones Industrial Average (ignoring companies which were not publicly listed during the timeframe in question). We tokenized the articles by splitting on any non-word character and lowercased everything. Company name references were determined before the lowercasing to take advantage of proper-noun capitalization. Stock price information was downloaded from Yahoo!Finance, which gives the historical opening and closing prices (adjusted for splits). Because this was the finest-grained data easily available to us, we used a daily window in attempting to correlate news articles to stock prices. NewsReader primary class for the project, reads in the data and applies the appropriate filters to the filenames, creates an instance of the appropriate algorithm and runs the predictor NewsArticle converts an article in which all the paragraphs are single strings into usable form for the algorithms WordBag contains the relevant word distribution and stock information for a particular article

4 We experimented with different ways to extract word-label pairs from articles. One approach is to use only the paragraph in which the company appears. Using the entire article was generally not feasible as memory requirements ballooned, so efforts were focused more narrowly. Unlike the bag of words approach, precise weighting was not possible as the maximum entropy classifier treats all input pairs equally. However, a crude form of weighting is possible by repeating parts of the article; for example one could train and classify on the paragraph before the company reference and two copies of the paragraph with the reference. Due to time constraints involved in training about 25 classifiers, we limited the training to 40 iterations. To make a prediction, we combined all articles for a given stock and date, then label each word in this combined test data using the maxent classifiers. We first tried a simple method of taking a simple plurality vote; buying the stock if the positive label has the most occurrences, shorting if the negative label is most prominent, and doing nothing if neutral is most common. In this system we would need to use equal amounts of money for each stock (equivalent to testing method 1b above) since the predictions are discrete and have no weighting. A second method using weighting computes a score as (# positive labels - # negative labels) / (# total labels), which will be a value between -1 and 1. The absolute value of this was used as a measure of confidence in our prediction and allowed us to allocate our daily investment based on predicted profit (Testing Methodology 1c). However, in contrast to the bag of words method this score is an abstract prediction, not a predicted percentage change. We experimented with different features a bit on the smaller test sets and came up with the following list: the word itself, adjacent words up to two positions away, beginning and inner capitals in the word, and if the word contained a number/digit. Unfortunately time did not allow extensive testing of different feature combinations on the whole dataset. Initially the label of the previous word was included in the features but we removed it for reasons that will be discussed in the results section. Results Bag of Words We were able to run the bag of words on one year at a time (training on all train data for that year and test on the one-two months of test data) as memory constraints prevented training on the whole set simultaneously. Combining these results gave an average of 2.05 % return per test month for the bag of words, vs % per month for the baseline method. Fig.1 is a graph of the yearly comparison between the simulation and the baseline (see next page). It is interesting that while the average for the simulation is significantly better than the baseline, the year-to-year comparison is very inconsistent, with the simulation underperforming the baseline as often as not. Unfortunately, this makes experimenting with parameters such as the weights for the paragraphs difficult as the entire training and test sets must be tried; it would seem dangerous to tune parameters to a subset of the training set with validation data given that performance is so variable.

5 Fig. 1 Comparison of Bag of Words to Baseline 15.00% 10.00% Percent change 5.00% 0.00% -5.00% % Year Simulated Trading Baseline Trading Maximum Entropy Fig. 2 Comparison of MaxEnt vs. Baseline (omitting prevlabel, only one paragraph) 20.00% Percentage Change 15.00% 10.00% 5.00% 0.00% -5.00% Simulated Trading Baseline Trading % Test Month First, let us discuss one alternative to the features list for the maximum entropy classifier. We tried removing the label of the previous word. This was motivated by our training data labeling method; since every word in an article got the same label the previous label would be a 100% accuracy method of predicting the current label in the training data. This struck us as bad, and not surprisingly the results of the trading simulation were considerably better when this feature

6 was removed. When trained over the entire data set, the maxent system (using just the paragraph with the company reference as the data), averaged over all months in the test set, produced an average return of 0.63 % per month, slightly higher than the average baseline return of % per month. However, with the previous label removed from the features list, the same trial produced an average return of 2.77 %, quite an improvement (and better than the bag of words approach). Fig. 2 on the previous page shows comparisons of the simulated return vs. the baseline return for this version of the maxent classifier method. Notice that while omitting prevlabel makes it much better on a month-by-month basis to the bag of words, on certain months it still does worse than the baseline. Also notice that Fig.1 contains data averaged by year (years generally had 1 or 2 test months) so there are slightly fewer data points than Fig. 2. We also experimented with using more than just the paragraph surrounding the company reference. As previously mentioned, using the whole article was too memory intensive to be practical, but we were able to run with one copy of the paragraph before the reference, two copies of the paragraph with the reference and one copy of the paragraph after. However, even this greatly increased the memory requirements and it was no longer possible to run the whole set at once, so we tried and (time constraints prevented more complete testing). Here are the results: Average Performance per Month Year range MaxEnt, 1 paragraph MaxEnt, Baseline The variability curse strikes again, as we have one result where is significantly better and one where a single paragraph is. Conclusion Of the two methods we tried, the maximum entropy classifier was superior. It is unclear whether a single-paragraph method or method is better, more testing should be done. However, there are significant tradeoffs between the systems. As more paragraphs are used for the maxent method, the memory use goes up quite a bit and the convergence time does as well. In addition, to make a detailed comparison of single paragraphs vs. multiple paragraphs in the maxent method, it would be good to hold a constant-size memory comparison (so the single paragraph, which uses less memory and processing time per article, gets more training data). The bag of words is generally faster to run (primarily since it s training method does not need to iterate like the classifiers does) but takes more memory, so there is worse hard limit on the size of data that can be trained at once. Both methods give substantial improvements over the baseline trading system, but for reasons discussed in the trading methodology section and in the final section, that may not transfer to real-world success. Future Work Our research could almost certainly benefit from more work with the corpus. As was previously mentioned, our news corpus was not separated into financial news, and while we separated the articles based on our company names there are many types of articles that could get in without

7 actually relating to business at all (such as any article mentioning people drinking a Coke or eating McDonald's food). There s also the danger of picking up after-the-fact stories like Microsoft stock rose \$2.50 today. Also our assignment of the training and test sets were broken up by months; given more time we could break up the test set on a daily basis. Another improvement to the data would be to get price data for stocks at greater resolution than the daily opening and close along with articles with time stamps as well as dates. This would allow us to both measure and predict the price changes in the immediate vicinity of when a news article appears, like Gidofalvi's 20 minute window. In order to do this, it would probably be necessary to write scripts to pull articles from news websites and prices from financial websites as information with this time detail does not appear to be readily available. This also means waiting for a few months to build up data instead of accessing an existing corpus. All these limitations definitely limit the extent to which we think our system would be successful in the real world. Yet another thing to try would be to train the paragraph weights for the bag of words method; again due to the length of the training we had to experiment with a few different choices and couldn t try very many. The bag of words is pretty quick to train (the major program time is reading the files), so it may be possible to have a gradient descent on the 4 weights in a reasonable amount of time but we didn t have the programming time to try this. Other methods of baseline comparison could be used also, although the current one is a reasonable benchmark since many people invest in funds that simply index a major set of companies like the Dow. It would be nice to get the variety of the S&P 500, but given that we ran up against memory issues on the 30-company collection that s not likely to be practical. It could be possible if we didn t use a separate classifier for each company; a study would need to be done on the current data set to see how much performance is lost by using the same classifiers for multiple companies. On the other hand, it would be interesting to apply this to non-blue-chip companies, but selecting an appropriate set with enough news data to train on could also be problematic.

8 References Gidofalvi, Gyozo. Using News Articles to Predict Stock Price Movements. Department of Computer Science and Engineering, University of California, San Diego Fung, Gabriel, et. al. The Predicting Power of Textual Information on Financial Markets. IEEE Intelligent Informatics Bulletin. Vol. 5. No. 1. June 2005.

### Applying Machine Learning to Stock Market Trading Bryce Taylor

Applying Machine Learning to Stock Market Trading Bryce Taylor Abstract: In an effort to emulate human investors who read publicly available materials in order to make decisions about their investments,

### CS224n Final Project Stock Price Prediction Using News Articles

CS224n Final Project Stock Price Prediction Using News Articles Qicheng Ma June 10, 2008 1 Introduction The basic form of efficient market hypothesis postulates that publicly available information is incorporated

### Using News Articles to Predict Stock Price Movements

Using News Articles to Predict Stock Price Movements Győző Gidófalvi Department of Computer Science and Engineering University of California, San Diego La Jolla, CA 9237 gyozo@cs.ucsd.edu 21, June 15,

Recognizing Informed Option Trading Alex Bain, Prabal Tiwaree, Kari Okamoto 1 Abstract While equity (stock) markets are generally efficient in discounting public information into stock prices, we believe

### CS224N Final Project: Sentiment analysis of news articles for financial signal prediction

1 CS224N Final Project: Sentiment analysis of news articles for financial signal prediction Jinjian (James) Zhai (jameszjj@stanford.edu) Nicholas (Nick) Cohen (nick.cohen@gmail.com) Anand Atreya (aatreya@stanford.edu)

### II. RELATED WORK. Sentiment Mining

Sentiment Mining Using Ensemble Classification Models Matthew Whitehead and Larry Yaeger Indiana University School of Informatics 901 E. 10th St. Bloomington, IN 47408 {mewhiteh, larryy}@indiana.edu Abstract

### Neural Networks for Sentiment Detection in Financial Text

Neural Networks for Sentiment Detection in Financial Text Caslav Bozic* and Detlef Seese* With a rise of algorithmic trading volume in recent years, the need for automatic analysis of financial news emerged.

VCU-TSA at Semeval-2016 Task 4: Sentiment Analysis in Twitter Gerard Briones and Kasun Amarasinghe and Bridget T. McInnes, PhD. Department of Computer Science Virginia Commonwealth University Richmond,

### Machine Learning Final Project Spam Email Filtering

Machine Learning Final Project Spam Email Filtering March 2013 Shahar Yifrah Guy Lev Table of Content 1. OVERVIEW... 3 2. DATASET... 3 2.1 SOURCE... 3 2.2 CREATION OF TRAINING AND TEST SETS... 4 2.3 FEATURE

### Logistic Regression for Spam Filtering

Logistic Regression for Spam Filtering Nikhila Arkalgud February 14, 28 Abstract The goal of the spam filtering problem is to identify an email as a spam or not spam. One of the classic techniques used

### Social Media Mining. Data Mining Essentials

Introduction Data production rate has been increased dramatically (Big Data) and we are able store much more data than before E.g., purchase data, social media data, mobile phone data Businesses and customers

### Data Mining Project Report. Document Clustering. Meryem Uzun-Per

Data Mining Project Report Document Clustering Meryem Uzun-Per 504112506 Table of Content Table of Content... 2 1. Project Definition... 3 2. Literature Survey... 3 3. Methods... 4 3.1. K-means algorithm...

### Stock Market Forecasting Using Machine Learning Algorithms

Stock Market Forecasting Using Machine Learning Algorithms Shunrong Shen, Haomiao Jiang Department of Electrical Engineering Stanford University {conank,hjiang36}@stanford.edu Tongda Zhang Department of

### Three types of messages: A, B, C. Assume A is the oldest type, and C is the most recent type.

Chronological Sampling for Email Filtering Ching-Lung Fu 2, Daniel Silver 1, and James Blustein 2 1 Acadia University, Wolfville, Nova Scotia, Canada 2 Dalhousie University, Halifax, Nova Scotia, Canada

### Sentiment Analysis of Twitter Feeds for the Prediction of Stock Market Movement

Sentiment Analysis of Twitter Feeds for the Prediction of Stock Market Movement Ray Chen, Marius Lazer Abstract In this paper, we investigate the relationship between Twitter feed content and stock market

### Making Sense of the Mayhem: Machine Learning and March Madness

Making Sense of the Mayhem: Machine Learning and March Madness Alex Tran and Adam Ginzberg Stanford University atran3@stanford.edu ginzberg@stanford.edu I. Introduction III. Model The goal of our research

### Email Spam Detection A Machine Learning Approach

Email Spam Detection A Machine Learning Approach Ge Song, Lauren Steimle ABSTRACT Machine learning is a branch of artificial intelligence concerned with the creation and study of systems that can learn

### Twitter sentiment vs. Stock price!

Twitter sentiment vs. Stock price! Background! On April 24 th 2013, the Twitter account belonging to Associated Press was hacked. Fake posts about the Whitehouse being bombed and the President being injured

### Equity forecast: Predicting long term stock price movement using machine learning

Equity forecast: Predicting long term stock price movement using machine learning Nikola Milosevic School of Computer Science, University of Manchester, UK Nikola.milosevic@manchester.ac.uk Abstract Long

### Spam Filtering based on Naive Bayes Classification. Tianhao Sun

Spam Filtering based on Naive Bayes Classification Tianhao Sun May 1, 2009 Abstract This project discusses about the popular statistical spam filtering process: naive Bayes classification. A fairly famous

### FREQUENTLY ASKED QUESTIONS March 2015

FREQUENTLY ASKED QUESTIONS March 2015 Table of Contents I. Offering a Hedge Fund Strategy in a Mutual Fund Structure... 3 II. Fundamental Research... 4 III. Portfolio Construction... 6 IV. Fund Expenses

### Implementing Point and Figure RS Signals

Dorsey Wright Money Management 790 E. Colorado Blvd, Suite 808 Pasadena, CA 91101 626-535-0630 John Lewis, CMT August, 2014 Implementing Point and Figure RS Signals Relative Strength, also known as Momentum,

### Predicting borrowers chance of defaulting on credit loans

Predicting borrowers chance of defaulting on credit loans Junjie Liang (junjie87@stanford.edu) Abstract Credit score prediction is of great interests to banks as the outcome of the prediction algorithm

### Chapter 6: The Information Function 129. CHAPTER 7 Test Calibration

Chapter 6: The Information Function 129 CHAPTER 7 Test Calibration 130 Chapter 7: Test Calibration CHAPTER 7 Test Calibration For didactic purposes, all of the preceding chapters have assumed that the

### Homework 4 Statistics W4240: Data Mining Columbia University Due Tuesday, October 29 in Class

Problem 1. (10 Points) James 6.1 Problem 2. (10 Points) James 6.3 Problem 3. (10 Points) James 6.5 Problem 4. (15 Points) James 6.7 Problem 5. (15 Points) James 6.10 Homework 4 Statistics W4240: Data Mining

### ALGORITHMIC TRADING USING MACHINE LEARNING TECH-

ALGORITHMIC TRADING USING MACHINE LEARNING TECH- NIQUES: FINAL REPORT Chenxu Shao, Zheming Zheng Department of Management Science and Engineering December 12, 2013 ABSTRACT In this report, we present an

### Using Text and Data Mining Techniques to extract Stock Market Sentiment from Live News Streams

2012 International Conference on Computer Technology and Science (ICCTS 2012) IPCSIT vol. XX (2012) (2012) IACSIT Press, Singapore Using Text and Data Mining Techniques to extract Stock Market Sentiment

### ARTIFICIAL INTELLIGENCE METHODS IN STOCK INDEX PREDICTION WITH THE USE OF NEWSPAPER ARTICLES

FOUNDATION OF CONTROL AND MANAGEMENT SCIENCES No Year Manuscripts Mateusz, KOBOS * Jacek, MAŃDZIUK ** ARTIFICIAL INTELLIGENCE METHODS IN STOCK INDEX PREDICTION WITH THE USE OF NEWSPAPER ARTICLES Analysis

### CS 229, Autumn 2011 Modeling the Stock Market Using Twitter Sentiment Analysis

CS 229, Autumn 2011 Modeling the Stock Market Using Twitter Sentiment Analysis Team members: Daniel Debbini, Philippe Estin, Maxime Goutagny Supervisor: Mihai Surdeanu (with John Bauer) 1 Introduction

### How can we discover stocks that will

Algorithmic Trading Strategy Based On Massive Data Mining Haoming Li, Zhijun Yang and Tianlun Li Stanford University Abstract We believe that there is useful information hiding behind the noisy and massive

### Active Learning SVM for Blogs recommendation

Active Learning SVM for Blogs recommendation Xin Guan Computer Science, George Mason University Ⅰ.Introduction In the DH Now website, they try to review a big amount of blogs and articles and find the

### Data Mining. Nonlinear Classification

Data Mining Unit # 6 Sajjad Haider Fall 2014 1 Nonlinear Classification Classes may not be separable by a linear boundary Suppose we randomly generate a data set as follows: X has range between 0 to 15

### BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES

BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES 123 CHAPTER 7 BEHAVIOR BASED CREDIT CARD FRAUD DETECTION USING SUPPORT VECTOR MACHINES 7.1 Introduction Even though using SVM presents

### Detecting Email Spam. MGS 8040, Data Mining. Audrey Gies Matt Labbe Tatiana Restrepo

Detecting Email Spam MGS 8040, Data Mining Audrey Gies Matt Labbe Tatiana Restrepo 5 December 2011 INTRODUCTION This report describes a model that may be used to improve likelihood of recognizing undesirable

### Anti-Spam Filter Based on Naïve Bayes, SVM, and KNN model

AI TERM PROJECT GROUP 14 1 Anti-Spam Filter Based on,, and model Yun-Nung Chen, Che-An Lu, Chao-Yu Huang Abstract spam email filters are a well-known and powerful type of filters. We construct different

### Machine Learning using MapReduce

Machine Learning using MapReduce What is Machine Learning Machine learning is a subfield of artificial intelligence concerned with techniques that allow computers to improve their outputs based on previous

### Adaption of Statistical Email Filtering Techniques

Adaption of Statistical Email Filtering Techniques David Kohlbrenner IT.com Thomas Jefferson High School for Science and Technology January 25, 2007 Abstract With the rise of the levels of spam, new techniques

### Comparing Artificial Intelligence Systems for Stock Portfolio Selection

Abstract Comparing Artificial Intelligence Systems for Stock Portfolio Selection Extended Abstract Chiu-Che Tseng Texas A&M University Commerce P.O. BOX 3011 Commerce, Texas 75429 Tel: (903) 886-5497 Email:

### Sentiment Analysis on Twitter with Stock Price and Significant Keyword Correlation. Abstract

Sentiment Analysis on Twitter with Stock Price and Significant Keyword Correlation Linhao Zhang Department of Computer Science, The University of Texas at Austin (Dated: April 16, 2013) Abstract Though

### Decision Trees from large Databases: SLIQ

Decision Trees from large Databases: SLIQ C4.5 often iterates over the training set How often? If the training set does not fit into main memory, swapping makes C4.5 unpractical! SLIQ: Sort the values

### Automatic Inventory Control: A Neural Network Approach. Nicholas Hall

Automatic Inventory Control: A Neural Network Approach Nicholas Hall ECE 539 12/18/2003 TABLE OF CONTENTS INTRODUCTION...3 CHALLENGES...4 APPROACH...6 EXAMPLES...11 EXPERIMENTS... 13 RESULTS... 15 CONCLUSION...

### Building a Question Classifier for a TREC-Style Question Answering System

Building a Question Classifier for a TREC-Style Question Answering System Richard May & Ari Steinberg Topic: Question Classification We define Question Classification (QC) here to be the task that, given

### Predicting Student Performance by Using Data Mining Methods for Classification

BULGARIAN ACADEMY OF SCIENCES CYBERNETICS AND INFORMATION TECHNOLOGIES Volume 13, No 1 Sofia 2013 Print ISSN: 1311-9702; Online ISSN: 1314-4081 DOI: 10.2478/cait-2013-0006 Predicting Student Performance

### Real Stock Trading Using Soft Computing Models

Real Stock Trading Using Soft Computing Models Brent Doeksen 1, Ajith Abraham 2, Johnson Thomas 1 and Marcin Paprzycki 1 1 Computer Science Department, Oklahoma State University, OK 74106, USA, 2 School

### CI6227: Data Mining. Lesson 11b: Ensemble Learning. Data Analytics Department, Institute for Infocomm Research, A*STAR, Singapore.

CI6227: Data Mining Lesson 11b: Ensemble Learning Sinno Jialin PAN Data Analytics Department, Institute for Infocomm Research, A*STAR, Singapore Acknowledgements: slides are adapted from the lecture notes

### Classification of Bad Accounts in Credit Card Industry

Classification of Bad Accounts in Credit Card Industry Chengwei Yuan December 12, 2014 Introduction Risk management is critical for a credit card company to survive in such competing industry. In addition

### Analysis of Tweets for Prediction of Indian Stock Markets

Analysis of Tweets for Prediction of Indian Stock Markets Phillip Tichaona Sumbureru Department of Computer Science and Engineering, JNTU College of Engineering Hyderabad, Kukatpally, Hyderabad-500 085,

### Machine Learning for Naive Bayesian Spam Filter Tokenization

Machine Learning for Naive Bayesian Spam Filter Tokenization Michael Bevilacqua-Linn December 20, 2003 Abstract Background Traditional client level spam filters rely on rule based heuristics. While these

### Bayesian Spam Filtering

Bayesian Spam Filtering Ahmed Obied Department of Computer Science University of Calgary amaobied@ucalgary.ca http://www.cpsc.ucalgary.ca/~amaobied Abstract. With the enormous amount of spam messages propagating

### Chapter 20: Data Analysis

Chapter 20: Data Analysis Database System Concepts, 6 th Ed. See www.db-book.com for conditions on re-use Chapter 20: Data Analysis Decision Support Systems Data Warehousing Data Mining Classification

### Ensemble Methods. Knowledge Discovery and Data Mining 2 (VU) (707.004) Roman Kern. KTI, TU Graz 2015-03-05

Ensemble Methods Knowledge Discovery and Data Mining 2 (VU) (707004) Roman Kern KTI, TU Graz 2015-03-05 Roman Kern (KTI, TU Graz) Ensemble Methods 2015-03-05 1 / 38 Outline 1 Introduction 2 Classification

### Identifying SPAM with Predictive Models

Identifying SPAM with Predictive Models Dan Steinberg and Mikhaylo Golovnya Salford Systems 1 Introduction The ECML-PKDD 2006 Discovery Challenge posed a topical problem for predictive modelers: how to

### Supervised Learning (Big Data Analytics)

Supervised Learning (Big Data Analytics) Vibhav Gogate Department of Computer Science The University of Texas at Dallas Practical advice Goal of Big Data Analytics Uncover patterns in Data. Can be used

### Automated News Item Categorization

Automated News Item Categorization Hrvoje Bacan, Igor S. Pandzic* Department of Telecommunications, Faculty of Electrical Engineering and Computing, University of Zagreb, Croatia {Hrvoje.Bacan,Igor.Pandzic}@fer.hr

### Twitter Stock Bot. John Matthew Fong The University of Texas at Austin jmfong@cs.utexas.edu

Twitter Stock Bot John Matthew Fong The University of Texas at Austin jmfong@cs.utexas.edu Hassaan Markhiani The University of Texas at Austin hassaan@cs.utexas.edu Abstract The stock market is influenced

### Applied Data Mining Analysis: A Step-by-Step Introduction Using Real-World Data Sets

Applied Data Mining Analysis: A Step-by-Step Introduction Using Real-World Data Sets http://info.salford-systems.com/jsm-2015-ctw August 2015 Salford Systems Course Outline Demonstration of two classification

### FTS Real Time System Project: Portfolio Diversification Note: this project requires use of Excel s Solver

FTS Real Time System Project: Portfolio Diversification Note: this project requires use of Excel s Solver Question: How do you create a diversified stock portfolio? Advice given by most financial advisors

### Why is Internal Audit so Hard?

Why is Internal Audit so Hard? 2 2014 Why is Internal Audit so Hard? 3 2014 Why is Internal Audit so Hard? Waste Abuse Fraud 4 2014 Waves of Change 1 st Wave Personal Computers Electronic Spreadsheets

### Predicting Soccer Match Results in the English Premier League

Predicting Soccer Match Results in the English Premier League Ben Ulmer School of Computer Science Stanford University Email: ulmerb@stanford.edu Matthew Fernandez School of Computer Science Stanford University

### T O B C A T C A S E G E O V I S A T DETECTIE E N B L U R R I N G V A N P E R S O N E N IN P A N O R A MISCHE BEELDEN

T O B C A T C A S E G E O V I S A T DETECTIE E N B L U R R I N G V A N P E R S O N E N IN P A N O R A MISCHE BEELDEN Goal is to process 360 degree images and detect two object categories 1. Pedestrians,

### large-scale machine learning revisited Léon Bottou Microsoft Research (NYC)

large-scale machine learning revisited Léon Bottou Microsoft Research (NYC) 1 three frequent ideas in machine learning. independent and identically distributed data This experimental paradigm has driven

### Multiple Kernel Learning on the Limit Order Book

JMLR: Workshop and Conference Proceedings 11 (2010) 167 174 Workshop on Applications of Pattern Analysis Multiple Kernel Learning on the Limit Order Book Tristan Fletcher Zakria Hussain John Shawe-Taylor

### Chapter 12 INVESTING IN STOCKS

Chapter 12 INVESTING IN STOCKS CHAPTER 12 WHAT WE WILL LEARN Characteristics of Stock Classifications of Stock Investments Determining a Stock s Worth Calculating Stock Return on Investment (ROI) The Securities

### Hong Kong Stock Index Forecasting

Hong Kong Stock Index Forecasting Tong Fu Shuo Chen Chuanqi Wei tfu1@stanford.edu cslcb@stanford.edu chuanqi@stanford.edu Abstract Prediction of the movement of stock market is a long-time attractive topic

### Example: Credit card default, we may be more interested in predicting the probabilty of a default than classifying individuals as default or not.

Statistical Learning: Chapter 4 Classification 4.1 Introduction Supervised learning with a categorical (Qualitative) response Notation: - Feature vector X, - qualitative response Y, taking values in C

### Sentiment Score based Algorithmic Trading

Sentiment Score based Algorithmic Trading 643 1 Sukesh Kumar Ranjan, 2 Abhishek Trivedi, 3 Dharmveer Singh Rajpoot 1,2,3 Department of Computer Science and Engineering / Information Technology, Jaypee

### Simple Language Models for Spam Detection

Simple Language Models for Spam Detection Egidio Terra Faculty of Informatics PUC/RS - Brazil Abstract For this year s Spam track we used classifiers based on language models. These models are used to

### www.optionseducation.org OIC Options on ETFs

www.optionseducation.org Options on ETFs 1 The Options Industry Council For the sake of simplicity, the examples that follow do not take into consideration commissions and other transaction fees, tax considerations,

### Knowledge Discovery and Data Mining. Bootstrap review. Bagging Important Concepts. Notes. Lecture 19 - Bagging. Tom Kelsey. Notes

Knowledge Discovery and Data Mining Lecture 19 - Bagging Tom Kelsey School of Computer Science University of St Andrews http://tom.host.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey ID5059-19-B &

### Semi-Supervised Learning for Blog Classification

Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008) Semi-Supervised Learning for Blog Classification Daisuke Ikeda Department of Computational Intelligence and Systems Science,

### Keywords social media, internet, data, sentiment analysis, opinion mining, business

Volume 5, Issue 8, August 2015 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Real time Extraction

### Challenges of Cloud Scale Natural Language Processing

Challenges of Cloud Scale Natural Language Processing Mark Dredze Johns Hopkins University My Interests? Information Expressed in Human Language Machine Learning Natural Language Processing Intelligent

### Towards better accuracy for Spam predictions

Towards better accuracy for Spam predictions Chengyan Zhao Department of Computer Science University of Toronto Toronto, Ontario, Canada M5S 2E4 czhao@cs.toronto.edu Abstract Spam identification is crucial

### Chapter 8. Final Results on Dutch Senseval-2 Test Data

Chapter 8 Final Results on Dutch Senseval-2 Test Data The general idea of testing is to assess how well a given model works and that can only be done properly on data that has not been seen before. Supervised

### E-discovery Taking Predictive Coding Out of the Black Box

E-discovery Taking Predictive Coding Out of the Black Box Joseph H. Looby Senior Managing Director FTI TECHNOLOGY IN CASES OF COMMERCIAL LITIGATION, the process of discovery can place a huge burden on

### Can Twitter provide enough information for predicting the stock market?

Can Twitter provide enough information for predicting the stock market? Maria Dolores Priego Porcuna Introduction Nowadays a huge percentage of financial companies are investing a lot of money on Social

### Homework Margin Purchases. Dr. Patrick Toche

Homework Margin Purchases Dr. Patrick Toche A dagger indicates a possibly more challenging question. Maintenance Margin 1. You are bullish on Telecom stock. The current market price is \$50 per share, and

### Determining optimum insurance product portfolio through predictive analytics BADM Final Project Report

2012 Determining optimum insurance product portfolio through predictive analytics BADM Final Project Report Dinesh Ganti(61310071), Gauri Singh(61310560), Ravi Shankar(61310210), Shouri Kamtala(61310215),

### Identifying At-Risk Students Using Machine Learning Techniques: A Case Study with IS 100

Identifying At-Risk Students Using Machine Learning Techniques: A Case Study with IS 100 Erkan Er Abstract In this paper, a model for predicting students performance levels is proposed which employs three

### Black Box Trend Following Lifting the Veil

AlphaQuest CTA Research Series #1 The goal of this research series is to demystify specific black box CTA trend following strategies and to analyze their characteristics both as a stand-alone product as

### Web Document Clustering

Web Document Clustering Lab Project based on the MDL clustering suite http://www.cs.ccsu.edu/~markov/mdlclustering/ Zdravko Markov Computer Science Department Central Connecticut State University New Britain,

### The relation between news events and stock price jump: an analysis based on neural network

20th International Congress on Modelling and Simulation, Adelaide, Australia, 1 6 December 2013 www.mssanz.org.au/modsim2013 The relation between news events and stock price jump: an analysis based on

### Can the Content of Public News be used to Forecast Abnormal Stock Market Behaviour?

Seventh IEEE International Conference on Data Mining Can the Content of Public News be used to Forecast Abnormal Stock Market Behaviour? Calum Robertson Information Research Group Queensland University

### Issues in Information Systems Volume 16, Issue IV, pp. 30-36, 2015

DATA MINING ANALYSIS AND PREDICTIONS OF REAL ESTATE PRICES Victor Gan, Seattle University, gany@seattleu.edu Vaishali Agarwal, Seattle University, agarwal1@seattleu.edu Ben Kim, Seattle University, bkim@taseattleu.edu

### Section 1 - Dow Jones Index Options: Essential terms and definitions

1 of 17 Section 1 - Dow Jones Index Options: Essential terms and definitions Download this in PDF format. In many ways index options are similar to options on individual stocks, so it is relatively easy

### Club Accounts. 2011 Question 6.

Club Accounts. 2011 Question 6. Anyone familiar with Farm Accounts or Service Firms (notes for both topics are back on the webpage you found this on), will have no trouble with Club Accounts. Essentially

### 2014 S&P500 Bulletin Review

2015 I Am in Wall Street Ltd Registered in England No. 9440096 Director: Daniele Prandelli W: www.iaminwallstreet.com E: info@iaminwallstreet.com T: 0044 75 212 46 792 2014 S&P500 Bulletin Review +124

### The Artificial Prediction Market

The Artificial Prediction Market Adrian Barbu Department of Statistics Florida State University Joint work with Nathan Lay, Siemens Corporate Research 1 Overview Main Contributions A mathematical theory

### Pattern Recognition and Prediction in Equity Market

Pattern Recognition and Prediction in Equity Market Lang Lang, Kai Wang 1. Introduction In finance, technical analysis is a security analysis discipline used for forecasting the direction of prices through

### A Logistic Regression Approach to Ad Click Prediction

A Logistic Regression Approach to Ad Click Prediction Gouthami Kondakindi kondakin@usc.edu Satakshi Rana satakshr@usc.edu Aswin Rajkumar aswinraj@usc.edu Sai Kaushik Ponnekanti ponnekan@usc.edu Vinit Parakh

### UNDERSTANDING THE EFFECTIVENESS OF BANK DIRECT MARKETING Tarun Gupta, Tong Xia and Diana Lee

UNDERSTANDING THE EFFECTIVENESS OF BANK DIRECT MARKETING Tarun Gupta, Tong Xia and Diana Lee 1. Introduction There are two main approaches for companies to promote their products / services: through mass

### Machine Learning. CS 188: Artificial Intelligence Naïve Bayes. Example: Digit Recognition. Other Classification Tasks

CS 188: Artificial Intelligence Naïve Bayes Machine Learning Up until now: how use a model to make optimal decisions Machine learning: how to acquire a model from data / experience Learning parameters

### Part III: Machine Learning. CS 188: Artificial Intelligence. Machine Learning This Set of Slides. Parameter Estimation. Estimation: Smoothing

CS 188: Artificial Intelligence Lecture 20: Dynamic Bayes Nets, Naïve Bayes Pieter Abbeel UC Berkeley Slides adapted from Dan Klein. Part III: Machine Learning Up until now: how to reason in a model and

### Whether you re new to trading or an experienced investor, listed stock

Chapter 1 Options Trading and Investing In This Chapter Developing an appreciation for options Using option analysis with any market approach Focusing on limiting risk Capitalizing on advanced techniques

### Introduction to Machine Learning and Data Mining. Prof. Dr. Igor Trajkovski trajkovski@nyus.edu.mk

Introduction to Machine Learning and Data Mining Prof. Dr. Igor Trajkovski trajkovski@nyus.edu.mk Ensembles 2 Learning Ensembles Learn multiple alternative definitions of a concept using different training

### Weekly Sales Forecasting

Weekly Sales Forecasting! San Diego Data Science and R Users Group June 2014 Kevin Davenport! http://kldavenport.com kldavenportjr@gmail.com @KevinLDavenport Thank you to our sponsors: The competition