MATLAB in Production Systems, Database Integration, and Big Data Eugene McGoldrick

Transcription

1 MATLAB in Production Systems, Database Integration, and Big Data Eugene McGoldrick 2013 The MathWorks, Inc. 1

2 Agenda MATLAB Production Server and Excel Integrating MATLAB Production Server into Database Servers MATLAB Production Server and Web Applications MATLAB Production Server based Trading System MATLAB, Big Data, & HADOOP 2

3 MATLAB in the Enterprise Goals: Enable customers to rapidly develop and deploy MATLAB applications onto the desktop. Seamlessly integrate MATLAB generated components into other languages, applications and enterprise production systems. 3

4 MATLAB Production Server Use Cases Big Data HADOOP Spreadsheet Applications Client Front End Applications MATLAB Production Server Request Broker & Program Manager Application Servers MSMQ, JMS, IBM MQ Web Applications Enterprise Service/Messaging Buses Database Servers 4

5 MATLAB Production Server and Excel 2013 The MathWorks, Inc. 5

6 Trader: Architecture VB to the.net client library Adds trade to portfolio New exposures & CVA Small data transfer 6

7 Integrating MATLAB Production Server into Database Servers 2013 The MathWorks, Inc. 7

8 Workflow for Embedding MATLAB Components in Production Systems The development to production process is a two step process Step 1: Bring Data to MATLAB Multiple data sources Build algorithms/models Test Compile to target platform component Step 2: Bring algorithm to the Data Install MATLAB component into the enterprise production applications. Same functionality/single source 8

9 MATLAB Components in Production Databases MATLAB Production Server can provide predictive analytics in the database Oracle (Java,.NET) Microsoft SQL Server (.NET) Microsoft Access (.NET) Netezza (JAVA) SAS (JAVA) Teradata (JAVA) Thin client with MPS Java and.net supported Database Server MATLAB Production Server Request Broker & Program Manager Central repository for models Simplifies change management 9

10 MATLAB Production Server and Web Applications 2013 The MathWorks, Inc. 10

11 Risk Manager: Architecture Architecture Web Server MATLAB Production Server CCRWeb Request Broker & Program Manager ZeroCurveServlet genscenservlet.... computecvaservlet Database Server 11

12 MATLAB Production Server based Trading System 2013 The MathWorks, Inc. 12

13 Messaging A very brief tutorial! Sending Application 1. Prepare Message 2. Send to Queue 3. Receive Message Back Message Bus/Queue Message 1 Message 2 Message 3... Message N Receiving Application 1. Receive Message 2. Process Message 3. Send Message Back 13

14 MATLAB Trading Application (1) Made up of many components Leverages MATLAB and Deployment Tools to integrate MATLAB analytics into 3 rd party applications Uses MSMQ as a messaging system to talk to other applications 14

15 MATLAB Trading Application (2) MATLAB Production Server Request Broker & Program Manager 15

16 MATLAB, Big Data, & HADOOP 2013 The MathWorks, Inc. 16

17 MATLAB HADOOP Integration 1. HADOOP Invoking Compiled MATLAB Java Components 2. HADOOP Invoking Compiled MATLAB Production Server Components 3. MATLAB Invoking HADOOP tasks/jobs 17

18 HADOOP calling MATLAB Analytic/Model http http MATLAB Production Server Request Broker & Program Manager Calculation Process Calculation Process Worker Pool 18

19 HADOOP calling MATLAB Analytic/Model Master Slave Slave Slave Slave http http MATLAB Production Server http Request Broker & Program Manager Calculation Process Calculation Process http Worker Pool 19

20 Hadoop calling MATLAB Analytic/Model Hadoop Cluster #1 Hadoop Cluster #2 Master Master Slave Slave Slave Slave Slave Slave Slave Slave http http http http MATLAB Production Server Request Broker & Program Manager Calculation Process Calculation Process Worker Pool 20

21 Workflow Build algorithm in MATLAB Execute Hadoop in pseudo mode enabling interactive debugging of MATLAB code Compile code into MATLAB Production Server component Integrate MATLAB Production Server into HADOOP cluster Execute HADOOP job and MATLAB Production Server component. 21

22 MATLAB Calling HADOOP MATLAB and JAVA are tightly integrated Users can instantiate JAVA classes in MATLAB Can invoke JAVA Methods from the command prompt HADOOP can be invoked directly from the MATLAB prompt Set HADOOP configuration files to local file system(for testing) or HDFS for production 22

23 MATLAB calling HADOOP 23

24 MATLAB HADOOP Applications 24