Attila Szegedi, Software Wednesday, November 23, 11

Size: px

Start display at page:

Download "Attila Szegedi, Software Engineer @asz. Wednesday, November 23, 11"

Noel Green
8 years ago
Views:

1 Attila Szegedi, Software 1

2 Twitter s Open Source Involvements 2

3 Both users and producers Twitter s systems are almost completely based on Open Source software our finance department runs Windows and Outlook, though 3

4 4

5 Contributor agreements Twitter has signed a companywide contributor agreements with: Oracle (OpenJDK) Eclipse Foundation Apache Software Foundation Our employees can contribute to these projects automatically, no further red tape involved. 5

6 Twitter s own Open Source projects 6

7 Twitter s own Open Source projects We use these projects internally, and either: develop on GitHub, or frequently sync to GitHub You get access to same versions that we use. Lots of things for both front-end presentation and back-end capacity and scalability. 7

8 Hearst Castle 8

9 Hearst Castle William Randolph Hearst had it built between The estate is a pastiche of historic architectural styles that its owner admired in his travels around Europe. Hearst was an omnivorous buyer who did not so much purchase art and antiques to furnish his home as built his home to get his bulging collection out of warehouses The floor plan of the Main Building is chaotic due to his habit of buying centuries-old ceilings, which dictated the proportions and decor of various rooms. --Wikipedia 9

10 10

11 Bootstrap twitter.github.com/ bootstrap 11

12 Bootstrap Bootstrap is Twitter's frontend HTML, CSS and JavaScript toolkit for kickstarting websites. It includes base CSS styles for typography, forms, buttons, tables, grids, navigation, alerts, and more. Supports IE7 and up Very small (CSS is ~7Kb) 12

13 Incredibly popular 3rd most watched Github project (after Ruby on Rails and Node.js) 13

14 14

15 15

16 16

17 17

18 18

19 Built around a complete styleguide Scaffolding grid, fixed-width and variable width Typography headings, body text, quotes, lists, code, labels Navigation fixed topbar, tab and pill navigation, breadcrumbs, pagination Alerts, dialogs Media thumbnails, tables, forms, buttons 19

20 20

21 21

22 22

23 Bootstrap Lets you build websites that have consistent, beautiful look, quickly. 23

24 Finagle 24

25 Finagle Switching gears from front end to back end now Finagle is a library for building asynchronous RPC servers and clients on JVM. 25

26 Finagle Built on top of Netty Supports request-response, streaming, pipelining. Supports stateful RPC styles. 26

27 Client features Connection pooling Load balancing Failure detection Failover/retry Distributed tracing Service discovery Sharding Native OpenSSL support Rich statistics 27

28 Server features Backpressure (against abusive clients) Service registration Native OpenSSL bindings 28

29 Protocol support HTTP Streaming HTTP ( Comet ) Thrift Memcached Kestrel In no way limited to these only 29

30 Minimal HTTP server: val service: Service[HttpRequest, HttpResponse] = new Service[HttpRequest, HttpResponse] { def apply(request: HttpRequest) = Future(new DefaultHttpResponse(HTTP_1_1, OK)) } val server: Server[HttpRequest, HttpResponse] = ServerBuilder().codec(Http).bindTo(new InetSocketAddress(10000)).name("HttpServer").build(service) same in Java: Service<HttpRequest, HttpResponse> service = new Service<HttpRequest, HttpResponse>() { public Future<HttpResponse> apply(httprequest request) { return Future.value( new DefaultHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK)); } }; Server server = ServerBuilder.safeBuild(service, ServerBuilder.get().codec(Http.get()).name("HttpServer").bindTo(new InetSocketAddress("localhost", 10000))); 30

31 Minimal HTTP client val client: Service[HttpRequest, HttpResponse] = ClientBuilder().codec(Http).hosts(address).hostConnectionLimit(1).build() // Issue a request, get a response: val request: HttpRequest = new DefaultHttpRequest(HTTP_1_1, GET, "/") val responsefuture: Future[HttpResponse] = client(request) onsuccess { response => println("received response: " + response) } 31

32 Robust client val client = ClientBuilder().codec(Http).hosts("localhost:10000,localhost:10001,localhost:10003").hostConnectionLimit(1) // max num of connections at a time to a host.connectiontimeout(1.second) // max time to spend establishing a conn.retries(2) // (1) per-request retries.reportto(new OstrichStatsReceiver) // export host-level load data.logger(logger.getlogger("http")).build() 32

33 Architecture 33

34 Architecture 34

35 Futures Unifying abstraction for asynchronous computation A computation that has not yet completed can succeed or fail Either block and wait for it to return, or register a completion callback. completion callbacks provide scaling, timeouts, scatter-gather, etc. 35

36 Futures Socket handler is not blocked while the response is being generated. Socket handler can time out if the operation takes too long. Response generator can scatter its operation, and return once every sub-operation completed or timed out. 36

37 Futures Blocking style val future = dispatch(request) val response = future() // blocks Event handler style Non-blocking style val future = dispatch(request) future onsuccess { value => // do something asynchronously } val future = dispatch(request) if (future.isdefined()) { val response = future() } else { // do something - timeout? } 37

38 Cassandra 38

39 Cassandra Onto distributed storage... Cassandra is a decentralized, fault tolerant, highly scalable distributed database Multi-master, multi-datacenter Linearly scalable High performance 39

40 Project Multiple committers at Twitter Twitter is one of the largest users Has contributed major patches in performance, scalability, and operational efficiency. Hundreds of nodes in production Serving millions of reads/writes per second! 40

41 Use Cases Spiderduck (real-time crawler) Cuckoo (real-time monitoring/alerting engine for Twitter infrastructure) Tweet button Geolocation Distributed RPC tracing store Real-time spam/ip store and more! 41

42 Features Supports eventual AND strong consistency! Distributed counters CQL (SQL like interface - select * from table) Secondary Indexing Hadoop support Compression 42

43 Twitter at Scale Add capacity by racks not servers Measure everything in percentiles (p95,p99,p999) Tune Cassandra to better integrate with the kernel and our hardware platforms Profile, profile and profile! Agile build deployment processes (jenkins, bittorrent) Automated performance and distributed testing 43

44 FlockDb 44

45 FlockDb Distributed graph database for storing adjacency lists 45

46 FlockDb goals Support a high rate of add/update/remove operations Support potientially complex set arithmetic queries Support paging through query result sets containing millions of entries Ability to archive and later restore archived edges Horizontal scaling including replication Online data migration 46

47 FlockDb Simpler, because it solves fewer problems than generic graph databases. Scales horizontally, and is designed for low-latency, high-throughput environments. Twitter uses it to store its social graph ( follows and blocks relations). 47

48 Gizzard twitter.github.com/ gizzard 48

49 Gizzard A Scala framework for creating fault-tolerant distributed databases. 49

50 Gizzard Lots of Open Source eventually-consistent distributed databases lately. Gizzard turns it around: it s a middleware framework sitting between clients and your replicated/partitioned storage but it doesn t tell you or limit you in your: data storage choice sharding and replication strategy choice 50

51 Gizzard Any storage backend: MySQL, Redis, Lucene, Stateless: run as many instances as necessary 51

52 Gizzard partitioning Hash function + forwarding table Not consistent hashing Allows heterogeneously sized partitions easy hotspot management 52

53 Gizzard replication Each shard is either physical or logical Logical shards are trees of other shards, with propagation strategies for reads and writes. You can code your own strategies for transaction coordination, quorum, etc., or use standard ones. Standard: Replicate (write to all children, load balance reads), Write-Only, Read-Only, Blocked. 53

54 Gizzard replication Replication topologies can vary per partition, i.e.: higher level for hotter partitions backends can mirror each other, or stripe partitions across machines better fault tolerance, higher configuration complexity 54

55 Gizzard fault tolerance If a partition replica crashes, request are rerouted to healthy ones. If all replicas of a partition crash, then reads from that partition are unavailable, but the other partitions are unaffected. writes can be buffered in durable journal with error queue. Requires writes to be idempotent and commutative. Data modeling needs to account for write conflicts. 55

56 The rest 56

57 The rest Kestrel: queueing system Fault tolerant, robust Not JMS compliant! Loosely ordered; no cross communication in cluster. 57

58 The rest Commons: Java libraries augmenting Google Guava Unified configuration & service launcher Closures, codecs, memcache client, Thrift server and client, load balancer, Zookeeper serversets Stats collection and publishing (non-jmx) Twitter-text included 58

59 The rest Kiji: Twitter s fork of REE (Ruby Enterprise Edition) Used for all Ruby runtimes at Twitter Has memory management and garbage collector radically rewritten Before, Twitter frontend REE runtimes spent 33% of CPU in GC Now they spend 5% 59

60 Attila Szegedi, Software 60

GigaSpaces Real-Time Analytics for Big Data

GigaSpaces Real-Time Analytics for Big Data GigaSpaces makes it easy to build and deploy large-scale real-time analytics systems Rapidly increasing use of large-scale and location-aware social media and