Time series IoT data ingestion into Cassandra using Kaa

Size: px

Start display at page:

Download "Time series IoT data ingestion into Cassandra using Kaa"

Earl Powers
10 years ago
Views:

1 Time series IoT data ingestion into Cassandra using Kaa Andrew Shvayka

2 Agenda Data ingestion challenges Why Kaa? Why Cassandra? Reference architecture overview Hands-on Sandbox setup Raspberry PI application code walkthrough Cassandra appender configuration Live demo Q&A

3 Data ingestion requirements/challenges Must have: Guaranteed data delivery Scalability Security Performance Low latency Nice to have: Built-in data structure validation Device platform independent Low footprint Low bandwidth support

Low latency Nice to have: Built-in data structure

5 Why Kaa? Fully-featured IoT middleware platform 10 Kb RAM footprint (with C SDK) Guaranteed data delivery and reliable local storage Built-in transport security Efficient data serialization Horizontally scalable and fault tolerant 100% open-source (Apache license 2.0) Rapid application development using C / C++ / Java SDKs Integration with popular device platforms

delivery and reliable local storage Built-in transport security Efficient data

6 Why Cassandra? Fault tolerant Performant Horizontally scalable Easy deployment Integration with popular analytics platforms

7 Problem description Region 1 Region 2 Region 3 Region 4

8 Reference architecture Raspberry Pi Temperature sensor (DHT11) Client application Kaa SDK Kaa cluster/sandbox Kaa node... Raw data Structured Data Client application Cassandra Kaa node Raspberry Pi Temperature sensor (DHT11) Cassandra appender Kaa SDK Cassandra appender

.. Raw data Structured Data Client application Cassandra Kaa node

9 Development environment setup Sample project repository: Kaa Sandbox: Raspberry Pi:

com/kaaproject/kaa-cassandrasample Kaa Sandbox:

10 Data modeling Kaa data collection schema: { "type" : "record", "name" : "SensorData", "namespace" : "org.kaaproject.kaa.sample", "fields" : [ { "name" : "sensorid", "type" : "string"}, { "name" : "model", "type" : "string"}, { "name" : "region", "type" : "string"}, { "name" : "value", "type" : "float"}] } Single sensor per row Single sensor per row, with date partitions and TTL Sensors per region and model, with date partitions

: "string"}, { "name" : "region", "type" : "string"}, { "name" : "value", "type" : "float"}] } Single

11 Single sensor per row Partition Key: Sensor ID Clustering Key: Timestamp Fields: Region, Model, Value, JSON, BLOB Timestamp 1 Timestamp 2 Fields Fields Sensor ID... Timestamp N Fields Query: select * from sensor_per_row where sensor_id = Sensor 1 and ts > 42

12 Single sensor per row, with date partitions and TTL Partition Key: Sensor ID, Date Clustering Key: Timestamp Fields: Region, Model, Value TTL: 60 sec Sensor ID, Date Timestamp N Timestamp N-1 Fields Fields... Timestamp 1 Fields Query: select * from sensor_per_date where sensor_id = Sensor 1 and date = 2015/09/10 and ts > 42

ID, Date Timestamp N Timestamp N-1 Fields Fields.

13 Sensors per region and model, with date partitions Partition Key: Region, Model, Date Clustering Key: Timestamp, Sensor ID Fields: Value Region, Model Date Timestamp 1, Sensor A Timestamp 1, Sensor B Value Value... Timestamp N, Sensor A Value Query: select * from sensor_per_region where region = Region A and model = DHT11 and date = 2015/09/10 17:10

Sensor A Timestamp 1, Sensor B Value Value.

14 THANK YOU FOR YOUR ATTENTION QUESTIONS? Andrew Shvayka cybervisiontech.com

15 Fault-tolerance and horizontal scalability Zookeeper quorum Control servers standby active Bootstrap servers Operations servers Endpoints

Case Study: Real-time Analytics With Druid. Salil Kalia, Tech Lead, TO THE NEW Digital

Case Study: Real-time Analytics With Druid Salil Kalia, Tech Lead, TO THE NEW Digital Agenda Understanding the use-case Ad workflow Our use case Experiments with technologies Redis Cassandra Introduction