Removing Failure Points and Increasing Scalability for the Engine that Drives webmd.com

Transcription

1 Removing Failure Points and Increasing Scalability for the Engine that Drives webmd.com Matt Wilson Director, Consumer Web Operations, 9/12/2013

2

3 About this talk Go over original site architecture and challenges How a request goes through the system Caching DB / NAS How were the challenges addressed Reasons why we picked the technology we did

4 About WebMD Technology WebMD, Medscape, MedicineNet, emedicine, UK cobrand Serving nearly 1 Billion Pageviews a month, 132 million unique visitors Running over 200 separate applications, vast majority in-house developed Environments: Dev/Devint, QA01/02, QA00, Production/DR Two main data centers, geographically diverse OS: mix of Linux and Windows Datastores: Sql Server, Oracle, Mongo, Vertica and mysql Web: mix of Apache and IIS App: mix of Tomcat, ASP,.Net 2.x - 4.x Service: ActiveMQ, Memcache,

5 Anatomy of a Request WebMD User Layer 7 Switch Load Balancer WebMD Runtime Server Runtime DB Server (Clustered) NAS

6 Anatomy of a Request What assets/xml/xsl are associated with this URL? Runtime DB Server (Clustered) WebMD User Layer 7 Switch Load Balancer WebMD Runtime Server NAS

7 Anatomy of a Request Here you go sir. \\nasserver\blah\blah\ blah.xml \\nasserver\blah\blah\blah.xsl Runtime DB Server (Clustered) WebMD User Layer 7 Switch Load Balancer WebMD Runtime Server NAS

8 Anatomy of a Request Runtime DB Server (Clustered) WebMD User Layer 7 Switch Load Balancer WebMD Runtime Server Fetch Content NAS

9 Anatomy of a Request Runtime DB Server (Clustered) WebMD User Layer 7 Switch Load Balancer Return content Blah.xml Blah.xsl WebMD Runtime Server NAS

10 Anatomy of a Request Server processes XML/ XSL and returns content to user Runtime DB Server (Clustered) WebMD User Layer 7 Switch Load Balancer WebMD Runtime Server NAS

11 Where s the cache bro? Page object cached in server s memory for 5 min

12 Where s the cache bro? Page object cached in server s memory for 5 min Widgets code snippets on page are cached at different variables. 60 min, 24 hours, 3 hours.

13 Where s the cache bro? Page object cached in server s memory for 5 min Widgets code snippets on page are cached at different variables. 60 min, 24 hours, 3 hours. Widget caching is determined at page design time in the content publishing system

14 Where s the cache bro? Page object cached in server s memory for 5 min Widgets code snippets on page are cached at different variables. 60 min, 24 hours, 3 hours. Widget caching is determined at page design time in the content publishing system Runtime system caches widgets on disk and/or in memory which is configurable in the publishing system

15 Caching, Caching, Caching How are existing cached objects updated?

16 Caching, Caching, Caching How are existing cached objects updated? How are in-memory page objects updated?

17 Caching, Caching, Caching Background thread calls NAS/DB for data and replaces object for widget and page cache pulls content Background thread works from a queue like data structure

18 This is all fine and dandy until

19 Caching, Caching, Caching What s the problem with this method? What does this method protect? Is it good enough?

20 Caching, Caching, Caching The background thread queues calls to the NAS/DB for update requests which creates a natural barrier to new content herding problems

21 Caching, Caching, Caching The background thread queues calls to the NAS/DB for update requests which creates a natural barrier to new content herding problems Not all web servers will get the updated content at the same time

22 Caching, Caching, Caching The background thread queues calls to the NAS/DB for update requests which creates a natural barrier to new content herding problems Not all web servers will get the updated content at the same time Additional web servers means more calls to the NAS/DB

23 Caching, Caching, Caching The background thread queues calls to the NAS/DB for update requests which creates a natural barrier to new content herding problems Not all web servers will get the updated content at the same time Additional web servers means more calls to the NAS/DB Individual Web Servers do not have the same cache

24 Caching, Caching, Caching The background thread queues calls to the NAS/DB for update requests which creates a natural barrier to new content herding problems Not all web servers will get the updated content at the same time Additional web servers means more calls to the NAS/DB Individual Web Servers do not have the same cache Publishing event could take up to an hour to refresh content

25

26 NAS Problem Problem Constraint: Still need a proven storage method Ubiquitous protocol Solution Constraint: 200 apps to update or not Use NAS as a backup method

27 What can replace a NAS Filestore? Does the solution need to provide SMB / NFS interface? Can we use something else

28 What can replace a NAS Filestore? Does the solution need to provide SMB / NFS interface? Can we use something else Remember, we have 200 apps to update

29 What can replace a NAS Filestore? Does the solution need to provide SMB / NFS interface? Can we use something else Remember, we have 200 apps to update Looked at Scality, Cassandra, MongoDB, Couchbase

30 What can replace a NAS Filestore? Does the solution need to provide SMB / NFS interface? Can we use something else Remember, we have 200 apps to update Looked at Scality, Cassandra, MongoDB, Couchbase

31 Why Couchbase? Memcached protocol already in use at WebMD Add servers to cluster without client reconfiguration Support for hundreds of thousands of transactions per second Content stored in memory fast set/get

32 We have to fail over the DB Cluster

33

34

35 Problem Constraints: Improve Availability Improve Scalability DB Problem Solution Constraints: No code updates. Needs to just work

36 Data Access Pattern narrow-read workload many copies on many nodes (above line borrowed from Theo) Database workload does not exceed server hardware All data is read only Load balancing works better than clustering in this pattern

37 DB Solution Read/only DB Calls Peer to Peer Replication Publishing System Writes to one SQL server

38 DB Solution Read/only DB Calls Peer to Peer Replication Publishing System Writes to one SQL server

39

40 Putting it all together Couchbase Persistent Data Store Read Only DB Requests WebMD Web Servers ActiveMQ NAS Publishing System

41 Putting it all together Content is written to DB / NAS and Couchbase. Couchbase gets same content as NAS. DB has metadata about the content. All are part of a transaction Couchbase Persistent Data Store WebMD Web Servers ActiveMQ NAS Publishing System

42 Putting it all together Couchbase Persistent Data Store WebMD Web Servers ActiveMQ NAS Publish Object IDs Publishing System

43 Putting it all together Couchbase Persistent Data Store Web Server Gets Object ID s off the Queue WebMD Web Servers ActiveMQ NAS Publishing System

44 Putting it all together Fetches xml/xsl content from Couchbase Couchbase Persistent Data Store WebMD Web Servers ActiveMQ NAS Publishing System

45 Putting it all together Web Server compiles page and stores cache objects on disk and in Couchbase Cache Object Couchbase Persistent Data Store WebMD Web Servers ActiveMQ NAS Publishing System

46 Add New Server Cache Object New Web Server gets cache objects from couchbase All Web Servers have the same cache Couchbase Persistent Data Store WebMD Web Servers ActiveMQ NAS Publishing System

47 Was it worth it? Fixed DB problem by using Peer to Peer replication and Load Balancing no code changes Fixed NAS problem by adding caching layer to reduce calls to NAS Fixed cache pull model with push model for the content publishing system reduces publishing times to all web servers in seconds and all web servers have the same cached content Serving content is now faster less latency Able to virtualize the web servers

48 Why was this Successful? Multi-Disciplined Team Operations, Development, QA and Project Management Buy-in from Senior Management Creative solutions within constraints resources, time, problem and solution Phased Implementation

49 Questions?