Session D15 Simple Visualization of your TimeSeries Data Shawn Moe IBM 1
Agenda IoT & Gateways Moving sensor data around jquery and Ajax Data Access Options Open Source Visualization packages 2
Acknowledgements and Disclaimers Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Copyright IBM Corporation 2015. All rights reserved. U.S. Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo, ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or TM), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml Other company, product, or service names may be trademarks or service marks of others. 3
Internet of Things Topography Deep Analytics Zone Sensor Analytics Zone Streams Smart Gateways Device/Sensors 4
What are we trying to do? Develop an IoT Gateway Kit to give various interested parties a framework to explore IoT Gateway benefits: Business Partners Hackathon participants Hobbyists Gateway kit will demonstrate: How to get sensor data into the Informix database How to visualize the data How to aggregate and establish alert conditions How to transport the data to the Cloud 5
Two key challenges for the Gateway 1. Working with sensors & getting data from sensor to the Gateway database ~200 different protocols used by sensors Established protocols for various industry devices 2. Visualizing the data in the gateway Simple current & historical status visualization Simple analytics acting on information stored on the gateway and from the cloud Aggregation where appropriate 6
Getting data into the Gateway Node-RED framework (nodered.org) used to wire things together visually IBM backed open source project also very visible on Bluemix Light-weight runtime is built on Node.js, taking full advantage of its event-driven, non-blocking model 120,000 modules in Node's package repository. Contributions help interactions with various devices, sensors, actions, analytics, and external SW packages We have developed & contributed timeseries node that knows how to insert & query Informix timeseries 7
Node-RED 8
Getting data into the Gateway Besides Node-RED, other methods we have experimented with: Python JavaScript This code is available at our GitHub project: https://github.com/ibm-iot/ibm-sample-code 9
Getting data to the Cloud MQTT (was Message Queue Telemetry Transport Machine-to-machine (M2M) IoT connectivity protocol Extremely lightweight publish/subscribe messaging transport Open source versions (Mosquitto) Hosted versions (IBM MessageSight appliance) 10
Visualizing Sensor Data in the Database No Node-RED options exist (yet) Our requirements: Graphs and/or charts Simple (follow the 80/20 rule, just a few knobs) Flexible No Informix timeseries complexity Can run on the gateway as well as in the cloud Open source with good license JavaScript and web based 11
jquery Cross-platform JavaScript library designed to simplify the client-side scripting of HTML Most popular JavaScript library in use today Free, open-source software licensed under the MIT License Syntax designed to make it easier to navigate a document, select DOM elements, create animations, handle events, and develop Ajax applications 12
Ajax Initially AJAX (asynchronous JavaScript and XML) Used on the Client side A is for asynchronous data can be sent to the server in the background without interfering with the loading and display of the web page Simple data access mechanism uses XMLHttpRequest object Despite the name, XML is not required JSON is often used instead PHP data access and manipulation functions also commonly used requests do not need to be asynchronous 13
Data Access 14
Data Access Options REST Client Mongo Client SQLI Client DRDA Client Informix Relational Table JSON Collection Time Series Spatial 15
Data Access Options REST Client MongoDB Client Informix Informix NoSQL Listener NoSQL SQL Translation REST, MongoDB Protocol Support SQLI Client DRDA Client Informix DBMS Relational Table JSON Collection Time Series Spatial SQLI, DRDA Protocol Support Relational, Collection, Time Series, and Spatial Data Support 16
A REST client is any program capable of making a HTTP request REST Client MongoDB Client Data Access Options Informix Informix NoSQL Listener NoSQL SQL Translation REST, MongoDB Protocol Support SQLI Client DRDA Client Informix DBMS Relational Table JSON Collection Time Series Spatial SQLI, DRDA Protocol Support Relational, Collection, Time Series, and Spatial Data Support 17
Data Access Options REST Client MongoDB Client Informix Informix NoSQL Listener NoSQL SQL Translation REST, MongoDB Protocol Support SQLI Client DRDA Client You can use the MongoDB PHP drivers are available for MongoDB with the NoSQL Listener Informix DBMS Relational Table JSON Collection Time Series Spatial SQLI, DRDA Protocol Support Relational, Collection, Time Series, and Spatial Data Support 18
Data Access Options REST Client MongoDB Client Informix Informix NoSQL Listener NoSQL SQL Translation REST, MongoDB Protocol Support SQLI Client DRDA Client You can also use the Informix SQL or DB2-Informix DRDA PHP driver via direct connection to IDS Informix DBMS Relational Table JSON Collection Time Series Spatial SQLI, DRDA Protocol Support Relational, Collection, Time Series, and Spatial Data Support 19
Informix REST - HTTP Method Overview Method Path Description POST / Create a new database POST /db Create a new collection POST /db/collection Creates a new document GET / Database listing GET /db Collection listing GET /db/collection Query the collection GET /db/collection?query={ } Query documents that satisfy query document DELETE / Drop all databases DELETE /db Drop a database DELETE /db/collection Drop a collection DELETE /db/collection?query={...} Delete documents that satisfy the query from a collection PUT /db/collection Update a document 20
Informix REST examples Query for the databases on localhost: http://localhost:27018/ Returns ["city_info","energy_demo","stores_demo","system","test","zipcodes"] Query for the people with first name of Han : http://localhost:27018/test_db/people?query={firstname: Han } Returns [{"_id":{"$oid":"553017a7e4b0fda5475a51a0"},"firstname": "Han","lastName":"Solo","age":36}] 21
Informix REST examples TimeSeries query using Virtual Table: http://localhost:27018/stores_demo/ts_data_v?query={loc_esi _id:"4727354321000111"}&fields={"loc_esi_id":0,"measure_uni t":0,"direction":0} Specific loc_esi_id = "4727354321000111" requested. 3 fields (loc_esi_id, measure_unit, direction) to be excluded from the projection list Returns [{"tstamp":{"$date":1289368800000},"value":0.092},{"tstamp" :{"$date":1289369700000},"value":0.084},{"tstamp":{"$date": 1289370600000},"value":0.09},{"tstamp":{"$date":12893715000 00},"value":0.085},{"tstamp":{"$date":1289372400000},"value ":0.088},{"tstamp":{"$date":1289373300000},"value":0.088},{ "tstamp":{"$date":1289374200000},"value":0.085},{"tstamp":{ "$date":1289375100000},"value":0.091},{"tstamp":{"$date":12 89376000000},"value":0.083},{"tstamp... 22
Informix REST with SQL pass through http://localhost:27018/stores_demo/system.sql?query= {"$sql":"select tstamp, value from ts_data_v where loc_esi_id=4727354321000111"} is equivalent to http://localhost:27018/stores_demo/ts_data_v?query={loc_es i_id:"4727354321000111"}&fields={"loc_esi_id":0,"measure_u nit":0,"direction":0} 23
Informix REST tips Data is returned as JSON Data is formatted as per MongoDB specs Why? Our REST interface is built on the NoSQL listener and is intended for compatibility with MongoDB. This can be a good thing, as JavaScript graphing packages (naturally) rely on JavaScript data types. Some Informix data types are non-standard 24
Informix REST tips Informix TimeSeries uses non standard DATETIME data type REST returns DATETIME YEAR TO SECOND (5) values as MongoDB Timestamps, which are 32-bit unsigned integers with seconds since the epoch (Jan 1, 1970 00:00) JavaScript timestamps are represented as milliseconds since the epoch Conversion between MongoDB timestamp and JavaScript timestamps is simple multiply by 1000. 25
jquery & Ajax Snippet $(function (){ $.ajax({ type: 'GET' url: '/api/orders', success: function(data) { console.log('success', data); } }); }); HTTP GET request URL /api/orders returns JSON document success function provided 26
Ajax Data Access Ajax URL can perform the Informix REST request using the MongoDB style query or the SQL pass through format TimeSeries SQL query can be run against the timeseries virtual table (VTI) or directly against the timeseries 27
Ajax URL can make Informix REST request function DoREST( collection, type, params, data, callback ) { type = type "GET"; if( params === undefined params === null ) params = ""; else params = "?" + params; } $.ajax( { url : "http://" + dbinfo.host + ":" + dbinfo.port + "/" + dbinfo.db + "/" + collection + "/" + params, type : type, data : data, contenttype : "text/plain; charset=utf-8", datatype : "json", success : callback, error : AJAXError, xhrfields: { withcredentials: true } } ); 28
Access your data with PHP d3.text("mapread.php", function(txt) { d3.selectall("#loading").remove(); txt.split("\n").foreach(function(line,i) { line.split(",").foreach(function(d,j) { data[i][j]=parsefloat(d); d3.selectall(".r"+i+".c"+j).style("fill",function() {return cscale(data[i][j]);}); }) }); }) Use MongoDB, Informix SQLI or DRDA PHP driver This example shows the JavaScript code calling the mapread.php function to perform some database activity and then the inline function operates on the data. 29
Open Source Visualization Packages 30
Some Open Source JavaScript Charting packages Package Link License dc.js - Dimensional Charting JavaScript Library http://dc-js.github.io/dc.js/ Apache 2.0 NVD3 Re-usable charts for d3.js http://nvd3.org/ Apache 2.0 Dojo GFX http://dojotoolkit.org/features/graphics-andcharting D3.js http://d3js.org/ BSD YUI Charts http://yuilibrary.com/yui/docs/charts/ BSD jqplot plugin for the jquery framework http://www.jqplot.com/ MIT Chart.js http://www.chartjs.org/ MIT dygraphs http://dygraphs.com/ MIT Flot for jquery http://www.flotcharts.org/ MIT Flotr2 for HTML5 charts and graphs http://humblesoftware.com/flotr2/documentation MIT graphaël http://g.raphaeljs.com/ MIT Rickshaw http://code.shutterstock.com/rickshaw/ MIT xcharts http://tenxer.github.io/xcharts/ MIT BSD 31
D3.js The most popular JavaScript library for producing dynamic, interactive data visualizations in web browsers (D3 = Data Driven Documents) Does not work real well with jquery right now, however. Most data access is via PHP program d3.text("mapread.php", function(txt) { d3.selectall("#loading").remove(); txt.split("\n").foreach(function(line,i) { line.split(",").foreach(function(d,j) { data[i][j]=parsefloat(d); d3.selectall(".r"+i+".c"+j).style("fill",function() {return cscale(data[i][j]);}); }) }); }) 32
D3 Example 33
Flot Flot is a pure JavaScript plotting library for jquery, with a focus on simple usage, attractive looks and interactive features Flot time series data is based on JavaScript timestamps, that is milliseconds, since January 1, 1970 00:00:00 UTC. Timestamps are interpreted according to UTC and, by default, displayed as such. You can set the axis "timezone" option to "browser" to display the timestamps in the user's timezone, or, if you use timezonejs, you can specify a time zone 34
Flot Example 35
jqplot jqplot is a plotting and charting plugin for the jquery jqplot produces line, bar and pie charts Computation and drawing of lines, axes, shadows even the grid itself is handled by pluggable "renderers Hooks into the core jqplot code allowing for custom event handlers, creation of new plot types, adding canvases to the plot, etc. Ajax data access 36
jqplot Example 37
Dygraphs Basic graph package intended for displaying time series data Data must be CSV format, even if returned from XMLHttpRequest function Your function would have to reformat the JSON returned from REST request OR PHP program to query Informix and format as CSV data Built in functionality to display plotted values when hovered over them in the browser Nice, built in, functionality to zoom into a range of selected data 38
Dygraphs Example 39
Home Grown Visualization Can use JavaScript, jquery and Ajax Design your own visualization components based on your requirements Determine how to correlate timeseries values with various widget states Visualization on Informix and Intel demos take this approach 40
Intel & Informix Demo 41
Home Grown Visualization Example Gauge shows the number of people on the floor Grid show the current location of each person Display refreshes each second as the count changes (corresponding to people leaving the floor) 42
Summary JavaScript & jquery/ajax, along with Informix REST interface can make visualization of your timeseries data relatively simple. We are working on adding this functionality to our in-progress Gateway Kit Look for updates on our GitHub site: https://github.com/ibm-iot/ 43
References (Node-RED) http://nodered.org/ (D3) http://d3js.org/ (Flot) http://www.flotcharts.org/ (jqplot) http://www.jqplot.com/ (Dygraphs) http://dygraphs.com/ 44
Questions? smoe@us.ibm.com 45