! E6893 Big Data Analytics Lecture 12:! Final Project Proposals

Transcription

1 E6893 Big Data Analytics Lecture 12: Final Project Proposals Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science Mgr., Dept. of Network Science and Big Data Analytics, IBM Watson Research Center November 20th, 2014

2 Course Structure Class Data Number Topics Covered 09/04/14 1 Introduction to Big Data Analytics 09/11/14 2 Big Data Analytics Platforms 09/18/14 3 Big Data Storage and Processing 09/25/14 4 Big Data Analytics Algorithms -- I 10/02/14 5 Big Data Analytics Algorithms -- II (recommendation) 10/09/14 6 Big Data Analytics Algorithms III (clustering) 10/16/14 7 Big Data Analytics Algorithms IV (classification) 10/23/14 8 Big Data Analytics Algorithms V (classification & clustering) 10/30/14 9 Linked Big Data Graph Computing I (Graph DB) 11/06/14 10 Linked Big Data Graph Computing II (Graph Analytics) 11/13/14 11 Linked Big Data Graph Computing III (Graphical Models & Platforms) 11/20/14 12 Final Project First Presentations 11/27/14 Thanksgiving Holiday 12/04/14 13 Next Stage of Big Data Analytics 12/11/14 14 Big Data Analytics Workshop Final Project Presentations 2

3 Proposal List (#1 - #17, Page 7-76) Industry Sector Finance Finance Finance Finance Finance Finance Finance Finance Finance Finance Finance Finance Finance Finance Finance Finance Finance Project name Exchange Rates Inquiry and Analysis Algorithm Trading Strategies Using Hadoop MapReduce Image Classification in the Cloud and GPU (H-Classification & G-Classification) Google-Analytics, Graph based Online Movie Recommendation System Currency Trend Analyzer Correlating Price / Volume of Low Volume Stocks with Social Media Trading Using Nonparametric Time Series Classification Models Stock Forecasting Using Hadoop Map-Reduce Real-time Risk Management System Stock Daily Price Predictions Based on News Stock Recommendation System Stock signal generation using real time news analysis Customer Complaint Analyses Financial Market Volatility Salary Engine Stock price Movement Prediction with Hadoop+Mahout & Pydoop+Scikit Sector-based Classification and Clustering of Financial News Articles 3

4 Proposal List (#18 - #33, Page ) Information Information Information Information Information Information Information Information Information Information Life Science Life Science Life Science Life Science Life Science Life Science TV Analytics Project Nova Game Outcome Analysis Exploring the Online and Offline Social World Yelp Review Analysis and Recommendation Sentiment Analysis on Movie Document Analysis with Latent Dirichlet Allocation Yelp Recommendation Analysis Using Big Data (Hadoop) for Identification of Aberrant Behavior Clusters in Server Performance Time Series Dat Crocuta: JavaScript Analytics System Learning Brain Activity From fmri Images Network Analysis on the Big Cancer Genome Data Reversal Prediction from Physiology Data EEGoVid: An EEG-Based Interest Level Video Recommendation Engine Brain Edge Detection BrainBow: Reconstruction of Neurons 4

5 Proposal List (#34 - #55, Page ) Retail Retail Retail Retail Retail Retail Retail Retail Retail Telecom Telecom Telecom Telecom Telecom Transportation and Energy Transportation and Energy Transportation and Energy Transportation and Energy Transportation and Energy Transportation and Energy Transportation and Energy Transportation and Energy Yelp Fake Review Detection Predicting Excitement at donorschoose.org Market strategy suggestions for B2C websites Résumé Category Classification TV Genome Project / Recommendation Engine Analytics Media Acquire Valued Shoppers Twitter-Based Product / Sales Events Recommender Yelp-er: Analyzing Yelp Data Big Mobile Data Network Congestion Analysis Comparison Analysis of Different Telecoms Operators User s Web Events Analysis Based on Browser Extension Analysis of telecom service in cellular networks Human Activity Monitoring and Prediction Minimizing Risk in Energy Arbitrage Best Transportation Choice Manage Energy Consumption By Smart Meters Location Specific Optimization of Taxi Efficiency in NYC Citi Bike System Data Analysis PeopleMaps Project Transeo: Making public buses more efficient and accessible Image Based Geo-localization 5

6 Proposal List (#56 - #77, Page ) Media Media Media Media Media Media Media Media Media Media Media Media Media Media Social Science-Government Social Science-Government Social Science-Government Social Science-Government Social Science-Government Social Science-Government Social Science-Government Social Science-Government HVision PlayPalate Movie Exploration Fantasy Basketball Fantasy Basketball Prediction Using Previous Season s Data Hunting for NBA players Music-Links Affective Computational Cinematography Improving Movie Recommender System with User Behavior c hanges and Demographics MOVIE RECOMMENDATION AND ANALYSIS OF THIS APPLICATION TV Genome Project / Recommendation Engine TrendyWriter Analysis on pricing strategy for sports team Spark NLP Predicting usefulness of restaurants reviews from subtopics using Yelp data Study Buddy Big Data Analysis on Log Data of Standardized IBT Test (TOEFL) Taker for Effects of Selection Changing Improving Education for At-risk Students Scratch Analyzer Oscar Award Analysis based on big data Error Correction in Large Volume OCR Datasets How to name your new born baby (babies)? 6

7 E6893 Big Data Analytics Finance Group Proposals 7 Nov 20, 2014

8 E6893 Big Data Analytics: Project Name Exchange Rates Inquiry and Analysis Team Members: Mengnan Wang(mw2969), Xiaomeng Zhang(xz2350), Jianze Wang(jw3127), Wanding Li(wl2501) 8 Nov 20, 2014

9 Motivation With the internatioanl trade and commerce being of more importance, it's necessary not only for a specific group of industries but also for all of us to gain daily access to the updated currency rates. With more convenience in getting the exchange rate information, which hopefully will be provided by our project, people are equipped with more related knowledge to make better decisions regarding to the currency market. 9

10 Dataset, Algorithm, Tools Dataset: Instant exchange rates from Bloomberg History exchange rates from Algorithm: We would forecast the exchange rate of currency in some targeted countries against US Dollar in a developed market, applying scalable model to forecast in real-time. And we would like to use RMSE to measure the reliability and accuracy of our prediction. Besides, by showing the statistical significance, such as P values, the repeatability of the outcome will then be proved. Tools: Eclipse Tomcat Apache 10

11 Progress and Expected Contributions Expected Contributions: Our project is initially designed to provide users with the following contents. Forward and cross exchange rates for most world currencies Both instant and history data Basic analysis of exchange rates including regression and K means Latest news about exchange rates Current Progress A sketch webpage design that looks something like Key Factors Affecting Exchange Rate PPP(Purchasing Power Parity) INT(Interest Rate Differential) Such as Libor GDP(The Difference in GDP Growth Rates) IGP(Income Growth Rate) Relative Economic Strength we may use factors like GDP and IGP to measure it quantitatively 11 E6893 Big Data Analytics Lecture 12: Final Project Proposal

12 E6893 Big Data Analytics: Project Name: Algorithm Trading Strategies Using Hadoop MapReduce Team Members: Yifan Wu, Meibin Chen 12 Nov 20, 2014

13 Motivation Improvement of internet speed and storage Data flooding everywhere Hadoop be able to do this & => MapReduce is the tool for parallel computing of the data Usage of Algorithm trading: Algorithm trading is the use of computer programs for entering trading orders, in which computer algorithms decide on every aspects of the order, such as the timing, price, and quantity of the order. 1. Back test the algorithm using enough historical price data to validate and optimize the algorithm in terms of profitability, stability, etc. 2. For a complex algorithm, there are many parameters that need to be optimized. 1. The system will builds and select most suitable strategy 20% faster than before 2. The enhanced platform doubled the number of strategy groups. 3. Last, the strategies can now be updated more frequently and can include more parameters in the analysis. 13

14 Dataset, Algorithm, Tools The dataset we choose is from stock future index of Shanghai Future Exchange (big bull market of A share ) or others Algorithm describes: The algorithm is given as combination of moving average convergence divergence(mcad) and Relative Strength Index(RSI) MCAD: long average period and the short average period RSI: the upper threshold, the lower threshold and the calculation period The project will divided into two part: First one is MapReduce platform----hadoop Second one is the trading strategies to test the data on this platform 14

15 Progress and Expected Contributions Inner MapReduce: Input: Daily price data. Each line contains 100 days price information Output: The performance of the parameters on the data Outside MapReduce: Input:Parameters combination: Each line contains one combination of parameters Ouput: The best parameters The contribution as Algorithm trading can provide many usage in investment strategy, including market marking, inter-market spreading, arbitrage, or pure speculation. With MapReduce, it can achieve faster, multi-tasks, and more real time updates. 15

16 E6893 Big Data Analytics: Image Classification in the Cloud and GPU (H-Classification & G-Classification) Team Members: Anand Rajan & Eric Johnson 16 Nov 20, 2014

17 Motivation - With the advent of social media the number of mobile pictures being taken and uploaded is increasing exponentially - Although most photos are uploaded with some basic metadata: date, time, camera model, and possibly geo-location - a great deal of details are missing when they enter the cloud. Unless users physically go through and tag each image this can create a search nightmare - Example: How do you find that picture you took a few years back while on vacation in Paris? It was under a bridge by the river right? Resorting to clicking through hundreds of photos or waiting for images to cache on your phone can take forever when you want to show someone in a pinch. Challenge In order to make more effective image search it will be important to develop and utilize advanced algorithms to help auto-tag images. Doing so can help narrow down image search and improve the quality of search results. - Leveraging the Yahoo Labs Flickr dataset we plan to test and develop upon feature extraction methods utilizing a parallelized computing system to efficiently extract image characteristics. - Using these image characteristics we will train and test the image classification of these images and evaluate them based on precision. Going beyond this step we also plan to experiment with a GPU powered processing system to evaluate added benefits and performance benchmarks that might be had during the image analysis stage over a standard distributed system. 17

18 Dataset, Algorithm, Tools Dataset: Yahoo/Flickr Image Dataset (83GB) 2 Million User Photos (200,000 x 10 categories) - Contains: photo_id, jpeg url, and some corresponding metadata such as the title, description, title, camera type, title, tags. Addition: Flickr API Details like comments, favorites, and social network data, can be queried. Data is broken into 10 categories - 1 nature - 6 sky - 2 food - 7 london - 3 people - 8 beach - 4 wedding music - 10 travel Toolset: R, Java, Hadoop/Mahout, NVidia CUDA - Linux based Cluster Array - 26x 2.7GHZ Intel Xeon CPUs - 64 GB of GDDR3 RAM - Windows 7 Desktop - 2.7GHZ Intel i7 CPU - 24 GB of GDDR3 RAM - 2GB 256-Bit GDDR5 w/ 1344 CUDA Cores Algorithms: Implemented by Stage - Feature Extraction / Reduction - SIFT - Hog - PCA (Map stage) Good for GPU - Encoding - KMeans Clustering (Reduce Stage) - Classification - Naïve Bayes - Log SGD Regression 18

19 Progress and Expected Contributions Progress - Conducted thorough research on opensource Image Classification packages as well as the tools required to perform feature extraction and analysis - Setup an environment for Hadoop distributed system - Acquired Yahoo Flickr dataset and begun initial testing - Acquired hardware and performed initial tests on a GPU based system Expected Contributions - Our goal is to research, implement, and build upon the current Open Source offerings for Image Classification to help improve the auto-tagging process of digital photos - If given the time and resources we aim develop a web based interface that will allow users to upload an image and perform a feature analysis and extraction to determine which tags and keywords are associated with it 19 Potential Challenges - Research for feature extract is very resource intensive so we have picked a challenging project for only two students - Conducting such a challenging project will involve many abstract mathematical models - Companies like Yahoo and Facebook invest millions in this field - *We anticipate this to be an ongoing project that we can continue well beyond the course and perhaps into the second semester

20 E6893 Big Data Analytics: Google-Analytics, Graph based Online Movie Recommendation System Team Members: Tian Han, Yifan Du, and Hang Guan 20 Nov 20, 2014

21 Motivation - To build our own movie website and implement the movie recommendation functionality. 21

22 Process Flow Recommendation Movie Website Users log files Graph Database Google Analytics Key words for query 22

23 Dataset, Algorithm, Tools - Dataset MovieLens 1M dataset, which contains 1 million ratings from 6000 users on 4000 movies - Algorithm Various collaborative filtering algorithms, e.g. user-based recommendation, item-based recommendation etc. - Tools Web design - Dreamweaver / CoffeeCup Users log file analysis - Google Analytics Graph Database Gremlin / Neo4j Others Mahout / Eclipse 23

24 Progress and Expected Contributions Timeline: Now -- 11/27/14 : Movie website design and publish 11/28 12/04/14: Plug in Google Analytics in website and Analyze user log file 12/05 12/11/14: Generate Query for the Graph Database and update recommended movies on website Expected Contributions: - To build our own online movie website. - To implement the movie recommendation functionality To to website analysis using Google Analytics

25 E6893 Big Data Analytics: Currency Trend Analyzer Team Members: Tim Paine, Mark Aligbe 25 Nov 20, 2014

26 Motivation Forex trading requires statistical insight into the exchange market. Large quantity of data, visualization only utilized at the day/week/ month level Difficult to see real time trends, analyze real time trends at granularity < 1 day Need to be able to collect, analyze, visualize data streaming in real time Solution Distributed Computing Real Time Computation Statistical Analysis Data Visualization 26

27 Datasets Large amounts of intraday/daily forex/equity/other data Algorithms Recommenders - suggesting trading prices and items to exchange Clustering - to analyze trends over a variable period of time Classifying - to classify trends into upward/downward movements, Tools momentum Java and Mahout for the analytics Javascript, Python, and R for data gathering, web server, and visualization 27

28 Progress and Expected Contributions Forex data acquired, sanitized, formatted, ready to 28 use Built system to batch collect data from multiple feeds when it becomes available Current stage: building design and field research Next steps: other distributed computing libraries End contribution: an extensible framework for collecting, analyzing, and visualizing real time data feeds

29 E6893 Big Data Analytics: Correlating Price / Volume of Low Volume Stocks with Social Media Jeff Ho, MS Statistics William Lee, MS Operations Research (CVN) 29 Oct 20, 2014

30 Hypothesis and Method Hypothesis Low volume stocks typically do not generate mainstream news coverage We hypothesize that social media could be a useful source of information Method Backtest different methods of using Big Data (specifically Twitter) to ultimately try to predict future price movements We will test various cases attempting to seek a correlation between tweets and movement of low volume stocks in price or volume We will verify whether these tweets are leading or lagging indicators of price or volume changes 30

31 Stock Criteria Stock Criteria Low volume stocks No / Low analyst coverage Stick with one industry Data Source: Yahoo Finance Big Data Platform Twitter Data Source: Twitter API 31

32 Predicting stock movements with tweet volume 1. Built list of stocks to test 2. Found cases where tweets can predict absolute movement in prices and stock volume 3. Can build a strategy around tweet volumes Absolute Percent Change in Stock Price 10.00% 7.50% 5.00% 2.50% 0.00% ALSK shows Rise in Tweet Volume 2 Days Prior to Significant Stock Movement y = x R² = Tweet Volume 2 Days Prior ALSK shows Rise in Tweet Volume 2 Days Prior to Signifcant Stock Volume Increase y = 22249x R² = Stock Volume Tweet Volume 2 Days Prior

33 E6893 Big Data Analytics: Trading Using Nonparametric Time Series Classification Models Team Members: Yufan Cai, Bowen Wang, Junchao Zhang 33 Nov 20, 2014

34 Motivation Traditional trading strategies usually involved with time series models such GARCH. It is difficult to incorporate categorized parameters such Twitter data. Using classification models, we can give a prediction on whether the asset price will go up or down by incorporating unstructured data stream. 34

35 Dataset, Algorithm, Tools Dataset: Stock live price data, order book data (Bloomberg API, Bitcoin/USD). Twitter (Optional) Algorithms: Mahout Algorithms such as Logistic Regression Others like Weighted Majority Voting and Nearest-Neighbor Classification Tools: Java, Mahout, Hadoop 35

36 Progress and Expected Contributions Progress: Researched on various recent time series classification models Have set up interface with data API Expected Contributions: Provide a Hadoop based classification model implementations on time series 36

37 E6893 Big Data Analytics: Project Name STOCK FORECASTING USING HADOOP MAP-REDUCE Team Members: Yi Yu, Yu Xia, Xiangliang Yang, Yumeng Xu 37 Nov 20, 2014

38 Motivation Stock Market has high profit and high risk features, on the stock market analysis and prediction research has been paid attention by people. The stock price trend is complex nonlinear function so the price has certain predictability Stock Market has high profit and high risk features, on the stock market analysis and prediction research has been paid attention by people. The stock price trend is complex nonlinear function so the price has certain predictability Hadoop MapReduce is a latest framework specially designed for processing large datasets on distributed sources. Apache s Hadoop is an implementation of MapReduce. 38

39 Dataset, Algorithm, Tools 39 Algorithms: Pearson Correlation Similarity Euclidean Distance Similarity Stochastic Gradient Descent (SDG) Tools: Hadoop Mahout Hbase

40 Progress and Expected Contributions Expected Contributions: We are going to analyze the dataset called Daily Holdings for All ProShares ETFs which contains tons of information collected from the stock exchange market. The first step is to scrutinize the data and provide the stock which may go up potentially. With these screened stocks, suggest a certain user a potential stock which she/he may be interested in. Progress: We have already got the dataset and analyzed the some similarities which could be useful in further steps. 40

41 E6893 Big Data Analytics: Real-time Risk Management System Team Members: Iljoon Hwang, Sungwoo Yoo, Sungjoon Huh 41 Nov 20, 2014

42 Motivation 1. Motivation - Objective: Developing the Real-time Risk Management System (Intraday Value at Risk) for large complex portfolio in an unified framework - Expected Outcome: The system which performs the calculation of stressed VaR, "what-if" scenarios, stresstesting on complex portfolio with large number of underlying risk factors and vectors in real-time. - Importance: Risk management is crucial to throughout the investment/trading activities from front trading desk to back office. However, because of the complexity of calculating VaR in large multi-asset portfolio, delivering the VaR in real-time is not available at legacy system. Big Data with in-memory multi-dimensional analytics can resolve this big issue. 42

43 Dataset, Algorithm, Tools 2. Dataset Yahoo Finance Tick data for S&P Algorithms Valut at Risk: Monte Carlo simulation, Parametric method Big Data: Map-Reduce, In-Memory 4. Tools / Languages Hadoop, Spark, Scala, R, Python, and Google Cloud 43

44 Progress and Expected Contributions 5. Current Status - Developed environment system construction in Google Cloud - Have been collecting tick data from Yahoo Finance - Have decided specific algorithms for calculating Value at Risk - Have been studying related papers and articles 6. Team members and Expected Contributions - Iljoon Hwang: Environment systems construction development, Storage server programming, Testbed - Sungwoo Yoo: Data collection, Research papers / articles - Sungjoon Huh: Batch processing / Real-time app programming 44

45 E6893 Big Data Analytics: Stock Daily Price Predictions Based on News Team Members: Jie Liu, Jingnan Li, Lu Qiu, Ruixin Tan 45 Nov 20, 2014

46 Motivation Traditional technical trading only take into account the quantitative but not qualitative factors that influence the stock prices. It is well know that news items have significant impact on stock indices and prices. To make a better prediction, we combine quantitative methods with headline NLP feature analysis in our model. 46

47 Dataset, Algorithm, Tools Dataset: ( Jan 2, 2013 Nov 17, 2014) Stock Prices Indices& Yield News Yahoo Finance/ Bloomberg Yahoo Fiance Text Algorithm: NLP, Lasso, SVR, ARIMAX Attributes Data Source Data Type Description News Articles Indices, Yield Historical Prices Yahoo Finance Numeric Google, Apple and IBM stock daily prices NLP Tools: Java, Python, Hadoop, etc. Numeric Nasdaq, S&P 500, Treasury Yield 5 Years NLP Features News articles related to Google, Apple and IBM Data collection: Yahoo Finance RSS Around 3000 articles Feature Selection: Lasso Prediction: ARIMAX, SVR Stock Prices 47

48 Progress and Expected Contributions 0.05 What we have done: Collected numerical data and parts of news. Built preliminary models, verifying our thoughts reasonable. Correlation (NLP features VS AAPL Stock Prices) R 2 of Regression Methods Method Train Test Baseline: Linear SVM with RBF kernel Day -3 Day -2 Day -1 Day 0 Day 1 Day 2 SVM with RBF of Indexes and Lag =1 on Stock What we will do: Implement the algorithm on the whole dataset. 48 Prove the positive impression of NLP on stock price prediction.

49 E6893 Big Data Analytics: Project Name Stock Recommendation System Team Members: Guangyang Zhang, Yuechen Qin, Bowen Dang, Zheng Fang 49 Nov 20, 2014

50 Motivation To help stock buyers to make wiser choices To find those who are very good at gaining profits from stocks (experts) Using user-based collaborative recommendation to find the similarity between the buyer and the expert Recommend the stock buyer some stocks from the most similar expert. To ease stock buyers from the heavy burden of looking through thousands of stocks and making a wise choice. 50

51 Dataset, Algorithm, Tools Dataset NASDAQ Stock Exchange Data Yahoo Finance dataset for historical prices Simulated trading records of users Algorithm User-based Collaborative Filtering with Inferred Tag Ratings Tools Eclipse J2EE, Mahout, Maven, MySQL, PHP server 51

52 Progress and Expected Contributions Progress Implemented UI and database Determine the algorithm User-based Collaborative Filtering with Inferred Tag Ratings. Expected Contributions Achieve improved Collaborative Filtering algorithm with Inferred Tag Ratings. Present clients with sound stock recommendations based on the analysis of experts buying choices. 52

53 E6893 Big Data Analytics: Project Name Stock signal generation using real time news analysis Team Members: Mandeep Singh, Mayank Misra, Rajesh Madala, Shreyas Shrivastava 53 Nov 20, 2014

54 Motivation Stock movement due to white house related news on twitter Twitter based hedge fund Algorithmic trading Twitter data traditionally used for Sentiment Analysis But now also a great source to consume News real time Stock prices have correlation with news By applying appropriate filters on tweets by news agencies, and then scoring the filtered tweets we aim to generate signals for stock prices that could be consumed by algorithms or traders to do better trades The framework we are building is scalable and could potentially be used to generate signals for a portfolio of heterogeneous stocks 54

55 Dataset, Algorithm, Tools Stream Filter Algo Filter Algo Filter Algo Scorer Algo Scorer Algo Scorer Algo Dataset: Twitter Stream Algorithms: Custom filtering and scoring algorithms with use of NLP, logistic regression Tools: Stream processing, Python, Json parser, Python ML libraries, d3.js 55

56 Progress and Expected Contributions So far we have completed the stream ingestion part and are working on refining our filtering and scoring algorithms to generate better correlation between generated signals and stock price movements. Below is a breakdown of work and contributions by team members 1. Ingesting data stream Mayank 2. Filtering Algo - Mandeep 3. Scoring Algo - Shreyas 4. Fetching real time stock data and plotting it together with stock signals generated in real time Rajesh 56

57 E6893 Big Data Analytics Project Name Customer Complaint Analyses Insights Into Issues plaguing the Banking Sector Team Members Abhaar Gupta, Avinash Sridhar, Nachiket Rau, Sankalp Singayapally 57 Nov 20, 2014

58 Motivation One of the biggest challenges for banks is minimizing customer attrition rate which is directly dependent on customer satisfaction. Customers are inclined to choose the banks who can be trusted for their services. Banks make their decisions based on a subset of data because of absence of scalable solutions. In this project, we propose a scalable design to counter the above problems 58

59 Dataset, Algorithm, Tools Consumer Complaints Database: The dataset contains retail consumer complaints with banks and financial institutions (provided by Consumer Financial Protection Bureau) Algorithms: Various Clustering and Classification Algorithms Tools and Languages: Hadoop, Mahout, Java, Python

60 Progress and Expected Contributions Major retail banking issues by state and match-analyze them based on geographic or socio-economic brackets. Top concerns of consumers in various states. Derive business impact of customer satisfaction or dissatisfaction with their complaints on the institution. Propose likely solutions that can be deemed as first response for future complaints of similar nature. Hypothesize a performance metric to apply to all complaints can be used to prioritize complaints based on resolution time. 60

61 E6893 Big Data Analytics: Financial Market Volatility Team Members: John Terzis Tim Wu Oliver Zhou Jimmy Zhong 61 Nov 20, 2014

62 Motivation Understanding volatility in financial markets has long been of interest to hedge and speculators. Empirical evidence has shown us that volatility is a highly nonlinear evolving process. Modeling this process using the Hadoop ecosystem can offer tremendous advantages over traditional econometric models that are limited to datasets which fit in main memory. 62

63 Dataset, Algorithm, Tools Dataset: We have procured a massive dataset of price quotes on equities, exchange traded futures, futures, and market indices over the span of the last ten to fifteen years at the one minute granularity level. In addition to price quotes on specific instruments, our dataset features derivative indicators of price and volume activity. Algorithm: We propose to train and test several scalable machine learning based regression models on our dataset with the goal of producing a functional form of future realized volatility at the symbol level that minimizes bias and variance and ultimately generalizes well to unforeseen data. Feature selection will be integral to the task given the likelihood that many of our input variables are highly correlated. We intend to build a framework on top of Apache Spark that can at a minimum perform an n-fold cross validation of a training model and use beam search or other established methods to calibrate the hyperparameters of our SVM, random forest, or regularized regression model in a reasonably fast time frame given the algorithmic complexity of the underlying routines employed. Tools: Hadoop Apache Spark Mahout AWS Git 63 R, Java, Python

64 Progress and Expected Contributions Progress: Set up AWS web server with Ubuntu, Hadoop, and Mahout environments. Purchased and uploaded dataset. Selected initial set of machine learning algorithms. Expected Contributions: Preprocessing: Jimmy/Tim/John Feature Selection: John/Oliver Spark SVM: John Mahout Random Forest: Oliver/Tim Evaluator: Jimmy/Tim R APIs: Tim Java APIs: Jimmy Forward Progress: Preprocessing & Setup: 11/22 Algorithm Application: 11/30 Evaluation: 12/6 Final Report: 12/11 64

65 E6893 Big Data Analytics: Project Name: Salary Engine Team Members: Lin Huang, Mingrui Xu, Wei Cao, Fan Ye 65 Nov 20, 2014

66 Motivation Main idea: job ad. Employers -would determine more reasonable salary for a position. Employees - could find more jobs match their background by using our recommendation system. We want to help employers and jobseekers figure out the market worth of different positions by building a prediction engine for the salary of any UK In this way, we would bring more transparency to this important market : Simple sample of our Salary Engine 66