UBER RUSH AND REBUILDING UBER S DISPATCHING PLATFORM

Transcription

1 UBER RUSH AND REBUILDING UBER S DISPATCHING PLATFORM

2 motivation CHAPTER 1 OF 8

3 MOTIVATION TOWARDS MICROSERVICES STUNTS AND EXPERIMENTS

4 UBER RUSH, DELIVERY SERVICE MOTIVATION TOWARDS MICROSERVICES

5 UBER RUSH REQUIREMENTS MOTIVATION TOWARDS MICROSERVICES MULTI-PICKUP BULK DELIVERIES MULTI-DISPATCH SOPHISTICATED MATCHING CAPACITY MANAGEMENT

6 UBER EATS REQUIREMENTS MOTIVATION TOWARDS MICROSERVICES NO PICKUP LOCATION TEMPERATURE REGULATION INVENTORY MANAGEMENT RE-SUPPLY STATIONS CHECKOUT FLOW

7 evolution CHAPTER 2 OF 8

8 MONOSERVICE TO MICROSERVICES MONOLITHIC ARCHITECTURE CONFIG USER CACHE CARS POSTPROCESSOR DISPATCH MONOSERVICE MONOSERVICE ETA SURGE GEOCODE

9 1ST GENERATION MICROSERVICES MONOSERVICE TO MICROSERVICES LOGISTICS SERVICES DISPATCH FOUNDATIONAL SERVICES PLATFORM SERVICES DEPENDENCIES SUPPLY ONEDIRECTION FLIPR GOLDETA DEMAND INV. MGMT ARBITER GEOCOD GEOSPATIAL OPTIC RAMEN CEREBRO SYNC LOCKET MONOSERVICE

10 2ND GENERATION MICROSERVICES MONOSERVICE TO MICROSERVICES LOGISTICS SERVICES DISPATCH FOUNDATIONAL SERVICES PLATFORM SERVICES DEPENDENCIES SUPPLY ONEDIRECTION FLIPR GOLDETA DEMAND INV. MGMT ARBITER GEOCOD GEOSPATIAL OPTIC RAMEN CEREBRO DISCO GEOFENCE SYNC LOCKET RTTR SCAVENGER UDESTROY LUMBERGH

11 A MICROSERVICE GATEWAY MONOSERVICE TO MICROSERVICES LOGISTICS SERVICES DISPATCH GATEWAY FOUNDATIONAL SERVICES PLATFORM SERVICES DEPENDENCIES SUPPLY ONEDIRECTION FLIPR GOLDETA DEMAND INV. MGMT ARBITER GEOCOD GEOSPATIAL OPTIC RAMEN CEREBRO DISCO GEOFENCE SYNC LOCKET RTTR SCAVENGER UDESTROY LUMBERGH

12 MOTIVATION TOWARDS MICROSERVICES THE TRADE-OFFS UPGRADES ARE PAINFUL MONOSERVICE MICROSERVICE vs. TEST SUITE IS SLOW FAILURE IS CATASTROPHIC CODE IS BRITTLE DEPLOYS ARE SLOW

13 topologies CHAPTER 3 OF 8

14 MICROSERVICE LAYOUT INDEPENDENT, INDIVIDUALLY ADDRESSABLE SERVERS HOST SERVICE MICROSERVICE LOAD AVERAGE DEMAND SERVICE WORKERS

15 ARRANGEMENT OF MICROSERVICES MULTI-TENANT OR DEDICATED HOSTS? DEMAND SUPPLY DISCO OPTIC MULTI-TENANT HOSTS OR DEDICATED DEMAND HOST

16 communications and fault tolerance CHAPTER 4 OF 8

17 MANAGING MICROSERVICE DEPENDENCIES AUTO-GENERATED CLIENTS JSON OVER HTTP DISCO MICROSERVICE DEMAND MICROSERVICE THRIFT OVER HTTP LUMBERGH

18 LOAD-BALANCING MICROSERVICES WITH CLIENT-SIDE LOAD-BALANCING B HOT HOST FILE RELOADING HEALTH CHECKING A B PERSISTENT CONNECTIONS CONNECTION POOLING B A TALKS DIRECTLY TO B RETRIES

19 COOPERATIVE MICROSERVICE INSTANCES FROM INDEPENDENT WORKERS TO COOPERATIVE 232 INDEPENDENT DEMAND HOSTS AND WORKERS COOPERATIVE DEMAND WORKERS ACROSS MANY HOSTS GOSSIP WITH ONE ANOTHER AND MAINTAIN A HASH RING OF EACH WORKER

20 COOPERATIVE MICROSERVICE INSTANCES WITH GITHUB.COM/UBER/RINGPOP 232 HASH WORKER ADDRESSES > hash(' :9000') > hash(' :9000') HASH APPLICATION IDS > hash('33e2dc8c-16fd-4a19-9fad-4ebfc76c66c9') EACH DEMAND WORKER OWNS A PORTION OF THE KEYSPACE > hash(' c-69c5-4b79-ae5e-6204c5f615ff')

21 RELIABLE BACKGROUND OPERATIONS WITH HASH RING TECHNOLOGY VNODE KEYSPACE (OUTER RING) FIXED AND SMALLER ENTITY KEYSPACE (INNER RING) DYNAMIC AND LARGER

22 RELIABLE BACKGROUND OPERATIONS WITH HASH RING TECHNOLOGY Riak DEMAND A WORKER RECEIVES DELIVERY & INITIATES DISPATCH DEMAND A WORKER WRITES UUID TO VNODE SET IN THE DB AND STARTS TIMER POST /jobs hash(uuid) % 1024 Demand A DEMAND A WORKER CRASHES BEFORE IT EXPIRES DISPATCH

23 RELIABLE BACKGROUND OPERATIONS WITH HASH RING TECHNOLOGY Riak 0 DEMAND B DETECTS MEMBERSHIP CHANGE IN RING 1023 DEMAND B SCANS ENTIRE VNODE KEYSPACE for vnode in range(0, 1023) if lookup(vnode) == whoami() restore(load_uuids(vnode)) DEMAND B LOADS VNODE SET FROM DB AND RESTORES BACKGROUND TIMERS

24 failure, monitoring and alerting CHAPTER 5 OF 8

25 FAILURE TESTING MICROSERVICES WITH REPEATABLE FAILURE SCENARIOS

26 FAULT ISOLATION IN MICROSERVICES WITH DEPLOYMENT PODS POD 1 GEOFENCE MICROSERVICE FLIPR MICROSERVICE , POD 2 DEMAND v1 DISPATCH GATEWAY SUPPLY v1 POD 2 DEMAND v2 SUPPLY v2

27 MICROSERVICE ALERTING WITH GRAPHITE/NAGIOS INTEGRATED THRESHOLD CHECKS PER REPO THRESHOLDS IMPORTED PYTHON BUILT AGAINST GRAPHITE ALERTS THROUGH NAGIOS

28 scalability and sharding CHAPTER 6 OF 8

29 PARTITIONING A MICROSERVICE A SCALABLE GEOSPATIAL INDEX c244 3 > convert(40.645, ) 864c244 > hash( 864c244 ) READ OR WRITE > lookup( ) :9000 EARTH IS BROKEN UP INTO CELLS. EACH CELL HAS AN ID. GEOSPATIAL READS/WRITES CONVERTS LAT/LNG TO CELL ID. CELL ID IS THEN HASHED ALONG RING. REQUEST IS EITHER HANDLED OR FORWARDED BY ONE OF THE 1300 GEOSPATIAL INDEX WORKERS.

30 LOCATION-AWARE MICROSERVICES WITH CONTEXT-SPECIFIC METADATA POST /pickup POST /pickup?lat=40.70&lng= X-Uber-City-ID: 1 X-Uber-City-Name: New York 2 DISPATCH GATEWAY DEVICE SENDS PICKUP REQUEST TO DISPATCH GATEWAY 3 DISPATCH GATEWAY DISPATCH GATEWAY RESOLVES LAT/LNG AGAINST GEOFENCE SERVICE SERVICE PROCESSES REQUEST WITHIN CONTEXT OF CITY

31 performance and diagnostics CHAPTER 7 OF 8

32 HIGH-PERFORMANCE MICROSERVICES WITH GITHUB.COM/UBER/TCHANNEL PERFORMANT MULTIPLEXING STREAMING RETRIES + CIRCUIT BREAKING POWERS RINGPOP

33 HIGH-PERFORMANCE MICROSERVICES WITH GITHUB.COM/UBER/NODE-STAP 1 TORCH LIVE PROCESS 2 OPEN FLAMEGRAPH IN BROWSER

34 DEBUGGING MICROSERVICES INSPECT INTERNALS WITH NODE REPL 1 CURL REPL ENDPOINT FOR REPL PORT 2 TELNET INTO REPL 3 INSPECT THE STATE OF YOUR WORKER

35 the next generation CHAPTER 8 OF 8

36 NEXT GENERATION MICROSERVICES A OVERLAY NETWORK FOR MICROSERVICE ROUTING REGISTER REGISTER DISCO MICROSERVICE OPTIC MICROSERVICE FORWARD SEND SEND SUPPLY MICROSERVICE DEMAND MICROSERVICE ROUTING SERVICE

37 THANKS! Presented by Jeff Wolski Uber is hiring!