Session D15 Simple Visualization of your TimeSeries Data. Shawn Moe IBM

Transcription

1 Session D15 Simple Visualization of your TimeSeries Data Shawn Moe IBM 1

2 Agenda IoT & Gateways Moving sensor data around jquery and Ajax Data Access Options Open Source Visualization packages 2

3 Acknowledgements and Disclaimers Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Copyright IBM Corporation All rights reserved. U.S. Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo, ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or TM), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Other company, product, or service names may be trademarks or service marks of others. 3

4 Internet of Things Topography Deep Analytics Zone Sensor Analytics Zone Streams Smart Gateways Device/Sensors 4

5 What are we trying to do? Develop an IoT Gateway Kit to give various interested parties a framework to explore IoT Gateway benefits: Business Partners Hackathon participants Hobbyists Gateway kit will demonstrate: How to get sensor data into the Informix database How to visualize the data How to aggregate and establish alert conditions How to transport the data to the Cloud 5

6 Two key challenges for the Gateway 1. Working with sensors & getting data from sensor to the Gateway database ~200 different protocols used by sensors Established protocols for various industry devices 2. Visualizing the data in the gateway Simple current & historical status visualization Simple analytics acting on information stored on the gateway and from the cloud Aggregation where appropriate 6

7 Getting data into the Gateway Node-RED framework (nodered.org) used to wire things together visually IBM backed open source project also very visible on Bluemix Light-weight runtime is built on Node.js, taking full advantage of its event-driven, non-blocking model 120,000 modules in Node's package repository. Contributions help interactions with various devices, sensors, actions, analytics, and external SW packages We have developed & contributed timeseries node that knows how to insert & query Informix timeseries 7

8 Node-RED 8

9 Getting data into the Gateway Besides Node-RED, other methods we have experimented with: Python JavaScript This code is available at our GitHub project: 9

10 Getting data to the Cloud MQTT (was Message Queue Telemetry Transport Machine-to-machine (M2M) IoT connectivity protocol Extremely lightweight publish/subscribe messaging transport Open source versions (Mosquitto) Hosted versions (IBM MessageSight appliance) 10

11 Visualizing Sensor Data in the Database No Node-RED options exist (yet) Our requirements: Graphs and/or charts Simple (follow the 80/20 rule, just a few knobs) Flexible No Informix timeseries complexity Can run on the gateway as well as in the cloud Open source with good license JavaScript and web based 11

12 jquery Cross-platform JavaScript library designed to simplify the client-side scripting of HTML Most popular JavaScript library in use today Free, open-source software licensed under the MIT License Syntax designed to make it easier to navigate a document, select DOM elements, create animations, handle events, and develop Ajax applications 12

13 Ajax Initially AJAX (asynchronous JavaScript and XML) Used on the Client side A is for asynchronous data can be sent to the server in the background without interfering with the loading and display of the web page Simple data access mechanism uses XMLHttpRequest object Despite the name, XML is not required JSON is often used instead PHP data access and manipulation functions also commonly used requests do not need to be asynchronous 13

14 Data Access 14

15 Data Access Options REST Client Mongo Client SQLI Client DRDA Client Informix Relational Table JSON Collection Time Series Spatial 15

16 Data Access Options REST Client MongoDB Client Informix Informix NoSQL Listener NoSQL SQL Translation REST, MongoDB Protocol Support SQLI Client DRDA Client Informix DBMS Relational Table JSON Collection Time Series Spatial SQLI, DRDA Protocol Support Relational, Collection, Time Series, and Spatial Data Support 16

17 A REST client is any program capable of making a HTTP request REST Client MongoDB Client Data Access Options Informix Informix NoSQL Listener NoSQL SQL Translation REST, MongoDB Protocol Support SQLI Client DRDA Client Informix DBMS Relational Table JSON Collection Time Series Spatial SQLI, DRDA Protocol Support Relational, Collection, Time Series, and Spatial Data Support 17

18 Data Access Options REST Client MongoDB Client Informix Informix NoSQL Listener NoSQL SQL Translation REST, MongoDB Protocol Support SQLI Client DRDA Client You can use the MongoDB PHP drivers are available for MongoDB with the NoSQL Listener Informix DBMS Relational Table JSON Collection Time Series Spatial SQLI, DRDA Protocol Support Relational, Collection, Time Series, and Spatial Data Support 18

19 Data Access Options REST Client MongoDB Client Informix Informix NoSQL Listener NoSQL SQL Translation REST, MongoDB Protocol Support SQLI Client DRDA Client You can also use the Informix SQL or DB2-Informix DRDA PHP driver via direct connection to IDS Informix DBMS Relational Table JSON Collection Time Series Spatial SQLI, DRDA Protocol Support Relational, Collection, Time Series, and Spatial Data Support 19

20 Informix REST - HTTP Method Overview Method Path Description POST / Create a new database POST /db Create a new collection POST /db/collection Creates a new document GET / Database listing GET /db Collection listing GET /db/collection Query the collection GET /db/collection?query={ } Query documents that satisfy query document DELETE / Drop all databases DELETE /db Drop a database DELETE /db/collection Drop a collection DELETE /db/collection?query={...} Delete documents that satisfy the query from a collection PUT /db/collection Update a document 20

21 Informix REST examples Query for the databases on localhost: Returns ["city_info","energy_demo","stores_demo","system","test","zipcodes"] Query for the people with first name of Han : Han } Returns [{"_id":{"$oid":"553017a7e4b0fda5475a51a0"},"firstname": "Han","lastName":"Solo","age":36}] 21

22 Informix REST examples TimeSeries query using Virtual Table: _id:" "}&fields={"loc_esi_id":0,"measure_uni t":0,"direction":0} Specific loc_esi_id = " " requested. 3 fields (loc_esi_id, measure_unit, direction) to be excluded from the projection list Returns [{"tstamp":{"$date": },"value":0.092},{"tstamp" :{"$date": },"value":0.084},{"tstamp":{"$date": },"value":0.09},{"tstamp":{"$date": },"value":0.085},{"tstamp":{"$date": },"value ":0.088},{"tstamp":{"$date": },"value":0.088},{ "tstamp":{"$date": },"value":0.085},{"tstamp":{ "$date": },"value":0.091},{"tstamp":{"$date": },"value":0.083},{"tstamp... 22

23 Informix REST with SQL pass through {"$sql":"select tstamp, value from ts_data_v where loc_esi_id= "} is equivalent to i_id:" "}&fields={"loc_esi_id":0,"measure_u nit":0,"direction":0} 23

24 Informix REST tips Data is returned as JSON Data is formatted as per MongoDB specs Why? Our REST interface is built on the NoSQL listener and is intended for compatibility with MongoDB. This can be a good thing, as JavaScript graphing packages (naturally) rely on JavaScript data types. Some Informix data types are non-standard 24

25 Informix REST tips Informix TimeSeries uses non standard DATETIME data type REST returns DATETIME YEAR TO SECOND (5) values as MongoDB Timestamps, which are 32-bit unsigned integers with seconds since the epoch (Jan 1, :00) JavaScript timestamps are represented as milliseconds since the epoch Conversion between MongoDB timestamp and JavaScript timestamps is simple multiply by

26 jquery & Ajax Snippet $(function (){ $.ajax({ type: 'GET' url: '/api/orders', success: function(data) { console.log('success', data); } }); }); HTTP GET request URL /api/orders returns JSON document success function provided 26

27 Ajax Data Access Ajax URL can perform the Informix REST request using the MongoDB style query or the SQL pass through format TimeSeries SQL query can be run against the timeseries virtual table (VTI) or directly against the timeseries 27

28 Ajax URL can make Informix REST request function DoREST( collection, type, params, data, callback ) { type = type "GET"; if( params === undefined params === null ) params = ""; else params = "?" + params; } $.ajax( { url : " + dbinfo.host + ":" + dbinfo.port + "/" + dbinfo.db + "/" + collection + "/" + params, type : type, data : data, contenttype : "text/plain; charset=utf-8", datatype : "json", success : callback, error : AJAXError, xhrfields: { withcredentials: true } } ); 28

29 Access your data with PHP d3.text("mapread.php", function(txt) { d3.selectall("#loading").remove(); txt.split("\n").foreach(function(line,i) { line.split(",").foreach(function(d,j) { data[i][j]=parsefloat(d); d3.selectall(".r"+i+".c"+j).style("fill",function() {return cscale(data[i][j]);}); }) }); }) Use MongoDB, Informix SQLI or DRDA PHP driver This example shows the JavaScript code calling the mapread.php function to perform some database activity and then the inline function operates on the data. 29

30 Open Source Visualization Packages 30

31 Some Open Source JavaScript Charting packages Package Link License dc.js - Dimensional Charting JavaScript Library Apache 2.0 NVD3 Re-usable charts for d3.js Apache 2.0 Dojo GFX D3.js BSD YUI Charts BSD jqplot plugin for the jquery framework MIT Chart.js MIT dygraphs MIT Flot for jquery MIT Flotr2 for HTML5 charts and graphs MIT graphaël MIT Rickshaw MIT xcharts MIT BSD 31

32 D3.js The most popular JavaScript library for producing dynamic, interactive data visualizations in web browsers (D3 = Data Driven Documents) Does not work real well with jquery right now, however. Most data access is via PHP program d3.text("mapread.php", function(txt) { d3.selectall("#loading").remove(); txt.split("\n").foreach(function(line,i) { line.split(",").foreach(function(d,j) { data[i][j]=parsefloat(d); d3.selectall(".r"+i+".c"+j).style("fill",function() {return cscale(data[i][j]);}); }) }); }) 32

33 D3 Example 33

34 Flot Flot is a pure JavaScript plotting library for jquery, with a focus on simple usage, attractive looks and interactive features Flot time series data is based on JavaScript timestamps, that is milliseconds, since January 1, :00:00 UTC. Timestamps are interpreted according to UTC and, by default, displayed as such. You can set the axis "timezone" option to "browser" to display the timestamps in the user's timezone, or, if you use timezonejs, you can specify a time zone 34

35 Flot Example 35

36 jqplot jqplot is a plotting and charting plugin for the jquery jqplot produces line, bar and pie charts Computation and drawing of lines, axes, shadows even the grid itself is handled by pluggable "renderers Hooks into the core jqplot code allowing for custom event handlers, creation of new plot types, adding canvases to the plot, etc. Ajax data access 36

37 jqplot Example 37

38 Dygraphs Basic graph package intended for displaying time series data Data must be CSV format, even if returned from XMLHttpRequest function Your function would have to reformat the JSON returned from REST request OR PHP program to query Informix and format as CSV data Built in functionality to display plotted values when hovered over them in the browser Nice, built in, functionality to zoom into a range of selected data 38

39 Dygraphs Example 39

40 Home Grown Visualization Can use JavaScript, jquery and Ajax Design your own visualization components based on your requirements Determine how to correlate timeseries values with various widget states Visualization on Informix and Intel demos take this approach 40

41 Intel & Informix Demo 41

42 Home Grown Visualization Example Gauge shows the number of people on the floor Grid show the current location of each person Display refreshes each second as the count changes (corresponding to people leaving the floor) 42

43 Summary JavaScript & jquery/ajax, along with Informix REST interface can make visualization of your timeseries data relatively simple. We are working on adding this functionality to our in-progress Gateway Kit Look for updates on our GitHub site: 43

44 References (Node-RED) (D3) (Flot) (jqplot) (Dygraphs) 44

45 Questions? 45