Introduction to Apache Kafka And Real-Time ETL. for Oracle DBAs and Data Analysts

Transcription

1 Introduction to Apache Kafka And Real-Time ETL for Oracle DBAs and Data Analysts 1

2 About Myself Gwen Shapira System Apache Kafka, Apache Sqoop Author of Hadoop Application Architectures, Kafka The Definitive Guide Previously: Software Cloudera Oracle ACE Director Senior Pythian Mercury Interactive Find me:

3 There s a Book on That!

4 An Optical Illusion Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. turned into a stream data platform

5 We ll talk about: Redo Logs Pub-sub message queues So What is Kafka? Awesome use-cases for Kafka Data streams and real-time ETL Where can you learn more

6 Transaction Log: The most crucial structure for recovery operations store all changes made to the database as they occur.

7 Important Point The redo log is the only reliable source of information about current state of the database.

8 Redo Log is used for Recover consistent state of a database Replicate the database (Dataguard, Streams, GoldenGate ) If you look far enough into archive logs you can reconstruct the entire database

9

10 Publish-Subcribe Messaging

11 The Problem We have more user requests than database connections

12 What do we do? 1. Your call is important to us 2. Show them static content 3. Distract them with funny cat photos 4. Prioritize them 5. Acknowledge them and handle the request later 6. Jump them to the head of line 7. Tell them how long the wait is 12

13 Solution Message Queue Msg 1 Msg 2 Msg 3... Msg N

14 Application Business Layer Message Queue Application Data Layer DataSource Interface DataSource JDBC Driver Connection Pool 1 4

15 Queues decouple systems: Database is protected from user load

16 Raise your hand if this sounds familiar My next project was to get a working Hadoop setup Having little experience in this area, we naturally budgeted a few weeks for getting data in and out, and the rest of our time for implementing fancy algorithms. --Jay Kreps, Kafka PMC

17 Client Source Data Pipelines Start like this. 17

18 Client Source Client Client Client Then we reuse them 18

19 Client Backend Client Another Backend Client Client Then we add consumers to the existing sources 19

20 Client Backend Client Another Backend Client Another Backend Client Another Backend Then it starts to look like this 20

21 Client Backend Client Another Backend Client Another Backend Client Another Backend With maybe some of this 21

22 Queues decouple systems: Adding new systems doesn t require changing Existing systems

23 This is where we are trying to get Producers Source System Source System Source System Source System Brokers Kafka Consumers Hadoop Security Systems Real-time monitoring Data Warehouse Kafka decouples Data Pipelines Kafka decouples Data Pipelines 23

24 Important notes: Producers and Consumers dont need to know about each other Performance issues on Consumers dont impact Producers Consumers are protected from herds of Producers Lots of flexibility in handling load Messages are available for anyone lots of new use cases, monitoring, audit, troubleshooting

25 That s nice, but what is Kafka?

26 Kafka provides a fast, distributed, highly scalable, highly available, publish-subscribe messaging system. In turn this solves part of a much harder problem: Communication and integration between components of large software systems Click to enter confidentiality

27 The Basics Messages are organized into topics Producers push messages Consumers pull messages Kafka runs in a cluster. Nodes are called brokers

28 Topics, Partitions and Logs

29 Each partition is a log

30 Each Broker has many partitions Partition 0 Partition 0 Partition 0 Partition 1 Partition 1 Partition 1 Partition 2 Partition 2 Partion 2

31 Producers load balance between partitions Client Partition 0 Partition 0 Partition 0 Partition 1 Partition 1 Partition 1 Partition 2 Partition 2 Partion 2

32 Producers load balance between partitions Client Partition 0 Partition 0 Partition 0 Partition 1 Partition 1 Partition 1 Partition 2 Partition 2 Partion 2

33 Off Set X Off Set X Off Set X Off Set Y Off Set Y Off Set Y Consumers Kafka Cluster Consumer Group X Order retained with in partition Topic Consumer Partition A (File) Consumer Consumer Partition B (File) Consumer Group Y Order retained with in partition but not over partitions Partition C (File) Consumer Off sets are kept per consumer group

34 Kafka Magic Why is it so fast? 250M Events per sec on one node at 3ms latency Scales to any number of consumers Stores data for set amount of time Without tracking who read what data Replicates but no need to sync to disk Zero-copy writes from memory / disk to network

35 How do people use Kafka? As a message bus As a buffer for replication systems (Like AdvancedQueue in Streams) As reliable feed for event processing As a buffer for event processing Decouple apps from database (both OLTP and DWH)

36 My Favorite Use Cases Shops consume inventory updates Clicking around an online shop? Your clicks go to Kafka and recommendations come back. Flagging credit card transactions as fraudulant Flagging game interactions as abuse Least favorite: Surge pricing in Uber Huge list of users at kafka.apache.org 36

37 Got it! But what about real-time ETL?

38 Remember This? Producers Source System Source System Source System Source System Brokers Kafka Consumers Hadoop Security Systems Real-time monitoring Data Warehouse Kafka is smack in middle of all Data Pipelines Kafka decouples Data Pipelines 38

39 If data flies into Kafka in real time Why wait 24h before pulling it into a DWH? 39

40 40

41 Why Kafka makes real-time ETL better? Can integrate with any data source RDBMS, NoSQL, Applications, web applications, logs Consumers can be real-time But they do not have to Reading and writing to/from Kafka is cheap So this is a great place to store intermediate state You can fix mistakes by rereading some of the data again Same data in same order Adding more pipelines / aggregations has no impact on source systems = low risk 41

42 It is all valuable data Raw data Clean data Aggregated data Enriched data Raw data Clean data Filtered data Dash board Report Data scientist Alerts OMG

43 But wait, how do we process the data? However you want: You just consume data, modify it, and produce it back Built into Kafka: Kprocessor Kstream Popular choices: Storm SparkStreaming

44 Need More Kafka? My video tutorial: Our website: Oracle guide to real-time ETL: