CC 2.0 by William Brawley

Transcription

1 CC 2.0 by William Brawley

2 Why Hadoop and HBase? Social Media Monitoring Prospective Search and Coprocessors Challenges & Lessons Learned Resources to get started 2 Agenda

3 Software sentric 3 Co-founder and organizer of the Swiss HUG Contact: [email protected] About me

4 Spin-off of MeMo News AG, the leading provider for Social Media Monitoring & Analytics in Switzerland Big Data expert, focused on Hadoop, HBase and Solr Objective: Transforming data into insights 4 About sentric

5 CC 2.0 by Pete Reed h"p://flic.kr/p/ks9kf

6 6 Information Gathering Information Processing Analysis & Interpretation Insight Presentation Why Hadoop and HBase? Social Media Monitoring Process

7 7 Cost effective Reliable SMM High scalable Analytical capabilities RT Alerting Why Hadoop and HBase? Requirements

8 Storage HBase /HDFS 8 Search Solr Analytics Hadoop Mahout Event mechanism (MQ) HBase RowLog Real-time alerting Prospective search Why Hadoop and HBase? Technology Stack

9 CC 2.0 by nolifebeforecoffee

10 Downloaded Articles 10 match? Search Agents Output Web-UI Reports RT Alerts Icons by Social Media Monitoring Overview

11 n Crawler 11 REST HBase Web-UI RowLog Coprocessor MySQL Solr RT Alerts Social Media Monitoring Solution Architecture Icons by

12 Inspired by Google Bigtable coprocessors HBase version 0.92 Embed code directly into server processes High-level call interface for clients Automatic scaling, load balancing, request routing 12 Short Primer on Coprocessors Overview

13 Like a database trigger Provides event based hooks Concrete Implementations RegionObserver CRUD or DML type operations MasterObserver DDL or metadata operations and cluster administration WALObserver Write-ahead-log appending and restoration 13 Short Primer on Coprocessors Observer Classes

14 Client:Get() 14 CP1:preGet() CP2:preGet() CP3:preGet() Hregion:Get() CP1:postGet() CP2:postGet() CP3:postGet() RegionServer client response Short Primer on Coprocessors Observer Execution

15 Comparable to stored procedures Custom RPC protocol, used between client and region server Loaded in region server Client call APIs over single row or a row range Framework translates row keys to region location Parallel execution 15 Short Primer on Coprocessors Endpoint Classes

16 16 Client code Batch.Call<CountProtocol,int> int call(countprotocol p) { return p.getrowcount(); }. Region Server 1 table,, table,bbb, CountProtocol CountProtocol HTable coprocessorexec() Region Server 2 table,ccc, CountProtocol Map<byte[], Integer> countsbyregion table,ddd, CountProtocol Short Primer on Coprocessors Endpoint Call Routine

17 HBase Security (Version 0.94) Aggregate operations avg(), sum() AggregatorProtocol HBASE-3529: Embedded search 17 Short Primer on Coprocessors Use Cases

18 18 Processing Put operations HRegion HRegionServer Prospective Search RT Alerts Social Media Monitoring Icons by Prospective Search with Coprocessors

19 Standard, virtualized test cluster: 4RS/DN, 1HM, 1NN, 3ZK Test dataset created from 2h of live index (1GB) Drive load on RS/DN 19 Social Media Monitoring Testing Setup

20 Writes/sec # of agents Social Media Monitoring Test Results

21 CC 2.0 by Sean Maurik h"p://flic.kr/p/juduu

22 Everyone is still learning Some issues only appear at scale Production cluster configuration Hardware issues Tuning cluster configuration to our work loads HBase stability Monitoring health of HBase 22 Challenges & Lessons Learned Challenges

23 Be careful with expensive operations in coprocessors At scale, nothing works as advertised Monitoring/Operational tooling is most important Play with all the configurations and benchmark for tuning 23 Challenges & Lessons Learned Lessons

24 entry/coprocessor_introduction index.html playground/hbaserowlog.html HBasePS 24 Resources to get started

25 25 Questions? Christian Gügi Berlin Buzzwords Thank you!