DATA INTEGRATION. in the world of microservices

Transcription

1 DATA INTEGRATION in the world of microservices

2 About me Valentine Gogichashvili Head of Data google+: +valgog

3

4 One of Europe's largest online fashion retailers 15 countries 4 fulfillment centers 15+ million active customers 2.9 billion revenue ,000+ products 9,000+ employees

5 Zalando Technology BERLIN DORTMUND DUBLIN ERFURT HELSINKI MÖNCHENGLADBACH

6 Zalando Technology 900+ TECHNOLOGISTS Rapidly growing international team

7 Good old small world

8 Once upon a time... Web Application Started as a tiny online shop Backend Prototyped on Magento (PHP) Used MySQL as a database Database

9 REBOOT

10 REBOOT 5½ years ago Java macro service architecture with SOAP as RPC layer PostgreSQL Heavy usage of Stored Procedures 4 databases + 1 sharded database on 2 shards Python for tooling (i.e code deploy automation)

11 REBOOT "macro" services Java Web Frontend Java Backend PostgreSQL Java Backend PostgreSQL 9.0 PostgreSQL 9.0 RC1 PostgreSQL 9.0 RC1 RC1 PostgreSQL Java Backend PostgreSQL

12 REBOOT PostgreSQL Heavy usage of Stored Procedures clean transaction scope very clean data processing close to data

13 REBOOT PostgreSQL Java Sproc Wrapper complex type mapping transparent sharding

14 REBOOT PostgreSQL introduced DBDIFF database schema management schema based Stored Procedure versioning

15 Live long and prosper... Very stable architecture that is still in use in the oldest (vintage) components We implemented everything ourselves starting from warehouse and order management and finishing with Web Shop and Mobile Applications

16 Live long and prosper... "I want to code in Scala/Clojure/Haskell because it is cool and compact" "But nobody will be able to support your code if you leave the company, everybody should use Java, learn SQL and write Stored Procedures" "SQL is cool but f*ck you, I am moving on to another company where I can use cool technologies!"

17 RADICAL AGILITY

18 Radical Agility AUTONOMY PURPOSE MASTERY

19 Autonomy Autonomous teams can choose own technology stack including persistence layer are responsible for operations should use isolated AWS accounts

20 Autonomy is not an Anarchy

21 Supporting autonomy STUPS

22 Supporting autonomy STUPS

23 Supporting autonomy Microservices Business Logic Business Logic Database public Internet Team B REST API REST API Team A Database Applications communicate using REST APIs Databases hidden behind the walls of AWS VPC

24 Supporting autonomy Microservices Business Logic Business Logic Database public Internet Team B REST API REST API Team A Database Database team is consulting autonomous teams Spilo as STUPS service for PostgreSQL

25 Supporting autonomy Microservices Business Logic Business Logic Database public Internet REST API REST API Team A Database Classical ETL process is impossible! Team B

26 Data integration in the classical world

27 Data integration in the classical world

28 Data integration in the classical world Developers DBA Business Intelligence

29 Data integration in the classical world Dev Business Logic DBA Database ETL process BI Data Warehouse (DWH) DBA Business Intelligence

30 Data integration in the classical world Dev Business Logic Business Logic Business Logic Business Logic DBA Database Database Database Database Classical ETL process BI Data Warehouse (DWH)

31 Data integration in the classical world Classical ETL process Use-case specific a lot of manual work Usually outputs data into a Data Warehouse well structured easy to use by the end user (SQL)

32 Data integration in the world of microservices

33 Data integration in the world of microservices

34 Database Business Logic Database REST API Database Business Logic Database REST API Business Logic Business Logic Database REST API Database REST API Database Business Logic REST API Business Logic REST API Business Logic REST API Data integration in the world of microservices

35 Data integration in the world of microservices Lawang Data Collector REST API Business Intelligence Kafka Queue (Buku) / Nakadi Queue Broker Stream Processing Engine (Flink/Spark) Automated Entity Materialization Engine DWH Data Lake

36 Data integration in the world of microservices Team A Business Logic REST API Database Business Intelligence

37 Data integration in the world of microservices manually push data changes to BI Team A Business Logic Error prone 2PC is actually needed Very difficult to implement REST API Database Business Intelligence

38 Data integration in the world of microservices Team A Business Logic automatically push data changes to BI You cannot miss anything No additional work needed on the business logic side REST API Database Business Intelligence

39 Data integration in the world of microservices PostgreSQL Logical Replication Enables automatic Data Change Event Extraction pglogical by 2nd Quadrant (streaming plugin) BottledWater by Confluent (Avro to Kafka streaming) Tawon by Zalando (parallel snapshotting)

40 Data integration in the world of microservices PostgreSQL 9.4+ Streaming Plugin Snapshot Exporting App (Tawon) Stream Processing App Queue

41 Try it yourself

42 Python support for replication protocol is the best to make you do experiments psycopg2 PR for replication protocol psycopg2 replication protocol usage Tawon initial snapshot export tool streaming initial database snapshot

43 Open Source at Zalando Technology Tech Blog of Zalando Technology - Open Source Projects pg_view PGObserver Java Sproc Wrapper Patroni Spilo...

44 Questions?