[Hadoop, Storm and Couchbase: Faster Big Data]

Transcription

1 [Hadoop, Storm and Couchbase: Faster Big Data] With over 8,500 clients, LivePerson is the global leader in intelligent online customer engagement. With an increasing amount of agent/customer engagements, LivePerson required a next generation architecture designed to meet today s real time big data requirements. After internal benchmarking and testing, the platform engineering team chose to integrate Apache Kafka, Apache Hadoop, Apache Storm, and Couchbase Server to enable Faster Big Data. LivePerson by the Numbers Over 8,500 customers; 8 of the top 10 Fortune billion visits/month 20 million engagements/month 13 TB of compressed data/month Four data centers worldwide

2 LivePerson s Technology Challenge With billions of visits and over 20 million engagements made via the LivePerson platform every month, the company needed a scalable backend capable of streaming, processing, and storing big data in real time. LivePerson s legacy platform was somewhat monolithic. The web tier collected visitor information in the form of events and stored them in memory. It was impossible to scale out, and the ability to scale up was limited. With the massive amount of events being generated, LivePerson needed to create a distributed, service- oriented platform. This platform would rely on multiple deployments in a multi- master topology allowing events to be processed as messages in multiple data centers.

3 All of the messages had to be available to web agents, but the messages were being stored and processed in multiple data centers. Looking for a solution, LivePerson looked to NoSQL. LivePerson s Database Requirements The LivePerson team evaluated three NoSQL databases: Cassandra, MongoDB, and Couchbase Server. After internal testing and benchmarking, LivePerson found that Couchbase Server was the only NoSQL database that met all of its requirements: Always On: With Couchbase Server, there was no downtime; not even during SW/HW upgrades, backups, etc. Linear Scale: With Couchbase Server, it was easy for LivePerson to add additional nodes without interrupting production Searchable Document Database and Key / Value Store

4 High Throughput: LivePerson s application required very high read/write throughput with most writes being updates, a scenario which Couchbase Server supports XDCR: With an active- active architecture, Couchbase Server s cross data center replication was an important component of LivePerson s application success Moving to Couchbase Server and a full data stack In order to stream, process, and store messages at this scale, the team had to design a real time big data architecture based on Apache Kafka, Apache Hadoop, Apache Storm, and Couchbase Server. The monitoring and communication services respond to customer events by publishing messages to Apache Kafka. The messages are consumed by offline and real time systems for processing and analysis. Batch Track Apache Hadoop stores historical data with both HP Vertica and Couchbase Server deployed to analyze it. For example, LivePerson uses map / reduce tasks within Couchbase Server to index and aggregate data. In addition, MicroStrategy is used for business intelligence (BI).

5 Real Time Track Apache Storm processes streams of events by consuming the messages published to Apache Kafka with data now being persisted to Couchbase Server instead of an Apache Cassandra distribution. This enables web agents to access the data via a dashboard. The End Result For LivePerson, Couchbase Server is a strategic choice. In addition to their current usage of the technology, LivePerson plans to leverage Couchbase Server in future applications: Visitor Session State Cross Session State Server Side Cookies Fast Read/Write Persistent Store Caching See the LivePerson video presentation at About LivePerson At LivePerson, our mission is to Create Meaningful Connections. Connection at the workplace inspires us to serve our clients exceptionally, to work together efficiently, and to get personally involved in our local communities. Our emphasis on connection leads to more opportunity whether it s increased value for our customers, product innovation, or community growth. With over 8,500 clients, LivePerson is the global leader in intelligent online customer engagement. Our solutions empower clients to engage customers with the most appropriate, most cost- effective type of interaction such as chat, voice, targeted content, and video delivering rich, personalized, cross- channel experiences. We deliver the right experience at the right moment, on mobile, website or tablet, based on customer and business intelligence. From the largest global enterprises to small businesses, our customers trust us to deliver increased conversions, higher order values, and lower support costs.