Analysing open data with Cloudant and dashdb on Bluemix. Glynn Bird Developer Advocate Cloud Data Services October 2015

Transcription

1 Analysing open data with Cloudant and dashdb on Bluemix Glynn Bird Developer Advocate Cloud Data Services October 2015

2 Glynn Bird Developer Advocate IBM Cloud Data Services

3 Agenda What is Open Data? What can we do with it? Where can we get it? Technology Cloudant Bluemix dashdb

4 Open Data

5 5

6 World Wide Web URLs Attributable Machine Readable (Editable) 6

7 Clinical Trials

8 Sources of Open Data

9 9

10 10

11 11

12 Selecting a template 12

13 Data Formats

14 CSV comma separated values id,price,date,postcode 1,195000, :00,"EN8 7EG" 2,60000, :00,"S2 5FW" 3,248000, :00,"BR5 3BQ" 14

15 TSV tab separated values Id price date postcode :00 "EN8 7EG" :00 "S2 5FW" :00 "BR5 3BQ" 15

16 XML Extensible Markup Language <?xml version="1.0"?> <housesales> <housesale Id="1"> <price>195000</price> <date> :00</date> <postcode>en8 7EG</postcode> </housesale> <housesale> 16

17 JSON JavaScript Object Notation [ { "Id": 1, "price": , "date": " :00", "postcode": "EN8 7EG" }, { 17

18 Other formats WMS Map data for Geographical Information Systems Excel PDF! 18

19 House Prices data id,price,date,postcode 1,195000, :00,"EN8 7EG" 2,60000, :00,"S2 5FW" 3,248000, :00,"BR5 3BQ" IBM Cloudant dashdb CouchDB 19

20 IBM Cloudant Scalable, NoSQL DB-a-a-S

21 IBM Cloudant Operational JSON document store Web service + Sync clients Advanced APIs Replication & Sync Full-text Search Geospatial Incremental MapReduce Scalable, Highly Available Performance Cross-data center data distribution & failover Geo load balancing Shared or dedicated clusters Managed 24x7 by experts

22 IBM Cloudant CRUD Document Primary Index Secondary Index (view) Search Index GeoSpatial Index Cloudant Query Direct document look up by _id Exists OOTB stored in a b-tree Primary key à doc._id Built using MapReduce stored in a b-tree Key à userdefined field(s) Built using Lucene FTI: Any or all fields can be indexed stored in R*, TPR, KD tree Lat/Long coorindates in GeoJSON Mongo-style querying Built natively in erlang Use when you want a single document and can find by its _id Use when you can find documents based on their _id Pull back a range of keys Use when you need to analyze data or get a range of keys Ex: count data fields, sum/average numeric results, advanced stats, group by date, etc. Ad-hoc queries Find documents based on their contents Can do groups, facets, and basic geo queries (bbox & sort by distance) Complex geometries (polygon, circularstring, etc.) Advanced relations (intersect, overlaps, etc.) Ad-hoc queries Lots of operators (>, <, IN, OR, AND, etc.) Intuitive for people coming from Mongo or SQL backgrounds

23 IBM Cloudant Visualization Geo-Load Balancing IOQ GeoSpatial Indexing Dashboards: Monitoring, Admin, Development Connects users to closest copy of data Developer APIs Prioritizing IO types; prevents noisy neighbors in multi-tenancy Horizontal Clustering Framework Apache CouchDB Clustering API, Sharding, Intracluster messaging JSON Doc Store, HTTP API, GET/PUT docs, Views, Replication Docs: JSON, Attachments

24 dashdb

25 SDP Schema Discovery! Process! Write Read/Write (HTTP) Read/Write Read/Write DataWorks Data Refinery! Services! Read/Write (On/Off Prem)

26 Bluemix

27

28

29

30

31

32

33 Importing data More text on one line in this location if needed

34 couchimport Command-line tool Imports CSV/TSV/JSON into CouchDB/Cloudant Uses bulk import API 34

35 couchimport - architecture 35

36 36 Demo

37 Importing data into Cloudant/CouchDB # store connections details in an environment variable export COUCH_URL= # create a database called "hp" ccurl X PUT /hp # import the contents of hp2.csv into database "hp" cat hp2.csv couchimport --db hp --delimiter "," 37

38 Web App 19m documents in database MapReduce used to build materialized views Supply JavaScript "map" function Cloudant passes each document to the function Emitted key/values stored in a B-Tree "_stats" reducer used to aggregate search results Queries can be used to extract data matching keys or key ranges 38

39 Example Queries /houseprices/_design/fetch/_view/bytime /houseprices/_design/fetch/_view/bytime? startkey=[2000,1,1]&endkey=[2001,1,1] { "rows": [ { "key": null, "value": { "sum": , "count": , "min": 5050, "max": , "sumsqr": } } ] } 39

40 Houseprices website 40

41 Export to dashdb

42 Export to dashdb Schema Discovery Automatic export of data to dashdb table(s) Query data in SQL or R 42

43 Run SQL 43

44 Example Query SELECT "town", AVG("price") as p FROM DASH "houseprices" GROUP BY "town" ORDER BY p DESC; 44

45 Example Query SELECT DASH HOUSEPRICES.POSTCODE,PRICE, LATITUDE, LONGITUDE FROM DASH HOUSEPRICES LEFT JOIN DASH UKPOSTCODES ON REPLACE(DASH HOUSEPRICES.POSTCODE,' ','') = DASH UKPOSTCODES.POSTCODE 45

46 R 46

47 Summary Lots of free open-source data available Import simply into Cloudant for web-applications and operational data Import into dashdb for data warehousing and analysis