Financial Data Access with SQL, Excel & VBA

Computational Finance and Risk Management Financial Data Access with SQL, Excel & VBA Guy Yollin Instructor, Applied Mathematics University of Washington Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 1 / 59

Outline 1 What is RExcel? 2 Installation and configuration 3 Getting help with RExcel 4 Getting started with RExcel 5 The RCommander 6 Basic data transfer 7 Scratchpad mode 8 Scratchpad example Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 2 / 59

Lecture references T. Baier, E. Neuwirth Using R from within Excel. 2007. T. Baier, E. Neuwirth Creating and Deploying an Application with (R)Excel and R. The R Journal Vol. 3/2, December 2011. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 3 / 59

Computational Finance and Risk Management Let s not kid ourselves: The most widely used piece of software for statistics is Excel. Brian D. Ripley Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 4 / 59

RExcel - an interface between R and Excel RExcel is an interface program that uses R as an add-in to Excel Transfers data between R and Excel, using Excel as a data editor for data to be analyzed with R Uses Excel as a container and simple code editor for R commands, and running R commands from Excel Runs R commands from Excel VBA macros allowing users to create Excel based applications and Excel add-ins using R Uses R functions in Excel worksheets functions, integrating R functionality into the automatic recalculation engine of Excel Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 6 / 59

RExcel modes of operation RExcel supports three modes of operation: Scratchpad and data transfer mode Menus control data transfer from R to Excel and back; commands can be executed immediately, either from Excel cells or from R command line Macro mode Macros, invisible to the user, control data transfer and R command execution Spreadsheet mode Formulas in Excel cells control data transfer and command execution, automatic recalculation is controlled by Excel Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 7 / 59

RExcel components statconndcom rcom rscproxy RExcel RAndFriends A DCOM server which allows to seamlessly integrate R or Scilab into other applications. An R package allowing to have R and another program running at the same time and let them communicate. An R package which is required for rcom and for the statconn (D)COM Server. An Excel addin using statconn (D)COM or rcom to allow Excel to call R from within Excel. An installation wrapper containing installation programs for R (including many packages), statconndcom, RExcel, and the example files for the book R through Excel. http://en.wikipedia.org/wiki/component_object_model Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 8 / 59

RAndFriends Download the latest version of RAndFriends from the statconn website. Download includes: Latest version of R (2.15.1) Package rscproxy 2.0-5 Package rcom 2.2-5 To install: Run the executable and accept default installation options http://rcom.univie.ac.at/download.html#randfriends Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 10 / 59

Excel add-ins and developer tabs Add-Ins Tab Contains RExcel add-in menu Developer Tab Contains access to macro creation and VBA development If the Add-Ins and/or Developer tab is not visible: Go to File Options Customize Ribbon. Activate the Add-Ins and Developer checkbox under "Main Tabs". Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 11 / 59

Establishing a reference to RExcelVBAlib To use RExcel VBA procedures, a reference must be established to the RExcelVBAlib object library From the VBE in Excel: Select the "Tools" tab Select "References" Find the object library labeled RExcelVBAlib and activate the checkbox Click "OK" and exit the VBA editor Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 12 / 59

Getting help To access the RExcel help click the RExcel add-in menu and select "RExcel Help" This help option opens "Using R from within Excel" in a web browser Contents "Using R from within Excel" is a quick reference guide For additional information, see "R Through Excel" Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 14 / 59

RExcel tutorial video "Combining R and Excel" tutorial video T. Baier, E. Neuwirth Provides RExcel walkthrough commentary Walkthrough material is available through the RExcel add-in menu http://rcom.univie.ac.at/rexceldemo/ It is highly recommended that students view this material before getting started with RExcel. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 15 / 59

Walkthroughs RExcel comes with walkthrough demo worksheets. To start working on the demo material click the RExcel add-in and select "Demo Worksheets" Demo worksheets include: Data transfer Writing macros Worksheet functions Graphics with sliders Interactive graphics Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 16 / 59

Starting RExcel To start RExcel: Navigate to the add-ins tab Click on the RExcel add-in to bring up the RExcel add-in menu Click "Start R" The RExcel splash screen should appear briefly Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 18 / 59

The RExcel add-in menu Once the R Server is started, the RExcel add-in menu will appear as follows : If foreground server is started, option will read Disconnect R rather then Close R Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 19 / 59

RExcel add-in menu commands RExcel add-in commands: Start R Close R Run Code Get R Value Put R Var Get R output Starting RExcel will begin a new R session. Closing RExcel will end your R session. Executes valid R code from a cell or a range of cells. Puts the value of an R expression into the active cell or range of cells. Puts a valid selection of cells into an array or dataframe object. Retrieves R output from the most recent command. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 20 / 59

RExcel add-in menu commands cont. Set R working dir Load R file Copy Code Debug R Error Log Options Set R server Allows the user to set a new working directory. Loads an.r file into the current R session from the working directory. Copies selected R code and inputs the selection into a VBA procedure. Displays the debug window in which all calls to R are displayed before execution. Displays an additional window with error messages. Displays an options menu. Allows the user to switch between a foreground server or background server. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 21 / 59

RExcel add-in menu commands cont. RExcel Help R Help RCommander Demo Worksheets RthroughExcel Worksheets Mark Calc cells About RExcel Opens "Using R from within Excel" in a browser window. Opens the R help documentation. Opens the RCommander with Excel or separate menu options. Opens the Demo walkthroughs in a new Excel spreadsheet. Opens RExcel projects with advance graphics and analysis. Marks cells containing calculated results. Opens the "About RExcel" menu. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 22 / 59

The RExcel RMB menu cont. The RExcel RMB menu adds the following options: Insert Current R Plot Name Range Prettyformat Numbers Inserts the current R Plot into a given range. Create a named range for easier selection. Aligns decimal points in a selected range. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 24 / 59

Setting the R Server type Setting the R Server Before starting RExcel, the "Set R server" option is available. There are two methods by which to run RExcel: Background server Foreground server Starting RExcel initiates a number of different processes at startup. These processes are explained at greater length in the Startup section of "Using R from within Excel". Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 25 / 59

Setting the R Server type Background Server The background server is totally hidden from the user, all interaction with R has to be done in Excel. Foreground Server The foreground server allows direct access to the R GUI command line while working in Excel. See Using R from with Excel, T. Baier and E. Neuwirth Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 26 / 59

Background server To initiate the background server: Note Open the RExcel add-in menu Select Set R server Under server type, click "Background" No visible R GUI All R commands must be sent from Excel Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 27 / 59

Foreground server To initiate the foreground server: Note Open the RExcel add-in menu Select Set R server Under server type, click "Foreground" R GUI is visible R commands can be typed directly into the command line or sent from Excel Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 28 / 59

Background vs. foreground Background advantages R operates in the background, invisible to front-end users Background disadvantages R operates in the background, invisible to front-end users Foreground advantages For those familiar with R, a foreground server provides a more convenient method by which to pass R commands A foreground server is simply supplemental, no RExcel functionality is lost by running a foreground server Foreground disadvantages Do not close the R GUI before disconnecting from the DCOM server. This may cause system instability. If R is upgraded but RExcel is not reinstalled, RExcel will not work with the foreground server Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 29 / 59

RCommander R Commander is a GUI for the R Developed by John Fox at McMaster University Licensed under GNU GPL Along with plug-ins, arguably the most fully featured GUI for R along the lines of S-PLUS http://socserv.mcmaster.ca/jfox/misc/rcmdr/ Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 31 / 59

The RCommander Window The RCommander windows can be thought of as a re-skinned version of the R GUI. RCommander contains three distinct sub-windows: Script Window Output Window Messages Type or open R scripts Displays output from submitted R commands Displays error messages and general information Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 32 / 59

The RCommander functions and methods RCommander functionality: File Edit Data Statistics Graph Models Distributions Tools Help Save and/or open scripts, set a new working directory Copy, paste, cut, undo, redo, etc. Import data from various sources and manage datasets Get statistics regarding the active dataset Plot the active dataset Create statistical models based upon the active dataset Plot distributions and compute relevant statistical values Load packages/plug-ins and access RCommander options Access R and RCommander Help Depending upon the nature of the active dataset, some options may not be available. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 33 / 59

Starting the RCommmander RExcel provides menu options, via the RCommander, for some data transfer, analysis and visualization. To start the RCommander: Start RExcel Open the RExcel add-in menu Click "RCommander" Select either "with Excel menus" or "with separate menus" Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 34 / 59

Activating a dataset In order create models and distributions using the RCommander, a dataset must be defined: To activate a dataset click "<No active dataset>" located below the tab bar Notice that the user must define at least one dataset in the current R session to activate a dataset (to be explained in the following sections) Select a dataset from the dropdown menu and click "OK" Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 35 / 59

Arrays and dataframes RExcel can handle two classes of data: Array RExcel uses the term array to mean R arrays, matrices, and vectors. Basic R data types where all elements are of the same class. Dataframe A dataframe is a two-dimensional matrix-like structure where each column can be of different classes. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 37 / 59

Using RExcel to create a data.frame in R There are various methods by which RExcel can create a data.frame in R. RExcel add-in menu options RMB menu options RCommander menu options R commands typed directly into the R GUI or the RCommander script window RExcel worksheet functions RExcel VBA procedures Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 38 / 59

Sending a data.frame from Excel via RExcel add-in menu For beginning users, the most convenient method is to use the RMB menu options. Note Start RExcel Highlight a selection of data Right-click the data selection Choose "Put R DataFrame" Create a dataframe name or accept default name RExcel assumes that the data selection contains column headers By default, RExcel names the dataframe after the spreadsheet title. This can be changed in the RExcel add-in menu under "Options". Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 39 / 59

Importing data via RCommander A dataframe can also be created through the RCommander menu options: Start RExcel and open the RCommander Open the "Data" tab Import data Note The RCommander accepts various data forms:.txt,.dat,.csv,.xls,.xlsx, etc. The RCommander can also load data from any attached R package To attach additional packages go to the "Tools" tab in the RCommander and select "Load package(s)" Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 40 / 59

Retrieving a data.frame from R There are a few methods by which to retrieve a dataframe from R. RExcel add-in menu options RMB menu options RExcel VBA procedures Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 42 / 59

Retrieving a data.frame from R via RExcel RMB RMB menu options Note Insure RExcel is running Right click the cell in which you want the dataframe to begin Choose "Get R DataFrame" Click "Get from R" Select a dataframe from the dropdown If you ended the connection to the DCOM server, previously created dataframes will have been deleted Sequential numbering will be included unless the "with rownames" checkbox is off Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 43 / 59

RExcel as code container and editor RExcel can run valid R code from a cell or selection of cells. Sending and receiving information between R and Excel is accessible through the RExcel add-in menu and the RMB menu. Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 45 / 59

The "Run code" option By interfacing R and Excel through RExcel, R code can be typed directly into a cell or a range of cells and be sent to R: There is no need to enclose the R expression in quotations Code can be run by accessing the RExcel add-in menu or the RMB menu If there is a coding issue, an pop-up will relay error messages directly from R Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 46 / 59

Special RExcel commands RExcel implements a few special commands that allow the automation of data exchanage without VBA. They can be run through the "Run Code" option. Command #!rput variable range #!rputdataframe variable range #!rputpivottable variable range #!rget r-expression range #!rgetdataframe r-expression range #!insertcurrentrplot cell-address Description store the value (contents) of a range in an R variable store the value of a range in an R data frame store the value of a range in an R variable store the value of the R expression in the range store the data frame of the R expression in the range insert the active plot into the worksheet Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 47 / 59

The quantmod package The quantmod package for R is designed to assist the quantitative trader in the development, testing, and deployment of statistically based trading models. Key functions: getsymbols chartseries Author: Jeffrey Ryan load or download price data Yahoo Finance / Google Finance FRED Oanda csv, RData MySQL, SQLite charting tool to create standard financial charts Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 49 / 59

The getsymbols function The getsymbols function loads (downloads) historic price data R Code: The getsymbols function > library(quantmod) > args(getsymbols) function (Symbols = NULL, env =.GlobalEnv, reload.symbols = FALSE, verbose = FALSE, warnings = TRUE, src = "yahoo", symbol.lookup = TRUE, auto.assign = TRUE,...) NULL Main arguments: Symbols symbols to be loaded src source of the data (Yahoo, Google, FRED, etc.) Return value: a time series object Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 50 / 59

Federal reserve economic data The function getsymbols can also be used to access data from the Federal Reserve Economic Data (FRED) database http://research.stlouisfed.org/fred2/ Guy Yollin (Copyright 2012) Data Access with SQL, Excel & VBA RExcel 51 / 59