A Scalable Data Transformation Framework using the Hadoop Ecosystem

Transcription

1 A Scalable Data Transformation Framework using the Hadoop Ecosystem Raj Nair Director Data Platform Kiru Pakkirisamy CTO

2 AGENDA About Penton and Serendio Inc Data Processing at Penton PoC Use Case Functional Aspects of the Use Case Big Data Architecture, Design and Implementation Lessons Learned Conclusion Questions

3 About Penton Professional information services company Provide actionable information to five core markets Agriculture Transportation Natural Products Infrastructure Industrial Design & Manufacturing Success Stories EquipmentWatch.com Prices, Specs, Costs, Rental Govalytics.com Analytics around Gov tcapital spending down to county level SourceESB NextTrend.com Vertical Directory, electronic parts Identify new product trends in the natural products industry

4 About Serendio Serendio provides Big Data Science Solutions & Services for Data-Driven Enterprises.

5 Data Processing at Penton

6 What got us thinking? Business units process data in silos Heavy ETL Hours to process, in some cases days Not even using all the data we want Not logging what we needed to Can t scale for future requirements

7 The Data Processing Pipeline Assembly Line processing Biz Value Data Processing Pipeline New features New New Insights Products Data Processing Pipeline

8 Penton examples Daily Inventory data, ingested throughout the day (tens of thousands of parts) Auction and survey data gathered daily Aviation Fleet data, varying frequency Ingest, store Clean, validate Apply Business Rules Map Analyze Report Distribute Various data formats, mostly unstructured Slow Extract, Transform and Load = Frustration + missed business SLAs Won t scale for future

9 Current Design Survey data loaded as CSV files Data needs to be scrubbed/mapped All CSV rows loaded into one table Once scrubbed/mapped data is loaded into main tables Not all rows are loaded, some may be used in the future

10 What were our options? Expand RDBMS options -Expensive -Complex Adopt Hadoop Ecosystem - M/R: Ideal for Batch Processing - Flexible for storage - NoSQL: scale, usability and flexibility HBASE Oracle Drools SQL Server

11 POC Use Case

12 Primary Use Case Daily model data upload and map Ingest data, build buckets Map data (batch and interactive) Build Aggregates (dynamic) Issue: Mapping time

13 Functional Aspects

14 Data Scrubbing Standardized names for fields/columns Example - Country Unites States of America -> USA United States -> USA

15 Data Mapping Converting Fields - > Ids Manufacturer - Caterpillar -> 25 Model - Caterpillar/Front Loader -> 300 Requires the use of lookup tables and partial/fuzzy matching strings

16 Data Exporting Move scrubbed/mapped data to main RDBMS

17 Key Pain Points CSV data table continues to grow Large size of the table impacts operations on rows in a single file CSV data could grow rapidly in the future

18 Criteria for New Design Ability to store an individual file and manipulate it easily No join/relationships across CSV files Solution should have good integration with RDBMS Could possibly host the complete application in future Technology stack should possibly have advanced analytics capabilities NoSQLmodel would allow to quickly retrieve/address individual file and manipulate it

19 Big Data Architecture

20 Solution Architecture RDB-> Data Upload UI CSV Files Data manipulation APIs exposed through REST layer Drools for rule based data scrubbing Use HBase as a store for CSV files REST API CSV and Rule Management Endpoints API Calls Drools HBASE HADOOP HDFS Launch MR Jobs Operations on all/groups of files using MR jobs Insert Accepted Data Push Updates Master database of Products/ Parts Survey REST Operations on individual files in UI through Hbase Get/Put Existing Business Applications Current Oracle Schema

21 Hbase Schema Design One row per HBase row One file per HBase row One cell per column qualifier (simple and started the development with this approach) One row per column qualifier (more performant approach)

22 Hbase Rowkey Design Row Key Composite Created Date (YYYYMMDD) User FileType GUID Salting for better region splitting One byte

23 Hbase Column Family Design Column Family Data separated from Metadata into two or more column families One cf for mapping data (more later) One cf for analytics data (used by analytics coprocessors)

24 M/R Jobs Jobs Scrubbing Mapping Export Schedule Manually from UI On schedule using Oozie

25 Sqoop Jobs One time FileDetailExport (current CSV) RuleImport (all current rules) Periodic Lookup Table Data import Manufacture Model State Country Currency Condition Participant

26 Application Integration - REST Hide HBase AP/Java APIs from rest of application Language independence for PHP front-end REST APIs for CSV Management Drools Rule Management

27 Lessons Learned

28 Performance Benefits Mapping csv files, 20 million records Time taken 1/3rd of RDBMS processing Metrics < 10 secs vs (Oracle Materialized View) Upload a file < 10 secs Delete a file < 10 secs

29 Hbase Tuning Heap Size for RegionServer MapReduce Tasks Table Compression SNAPPY for Column Family holding csv data Table data caching IN_MEMORY for lookup tables

30 Application Design Challenges Pagination implemented using intermediate REST layer and scan.setstartrow. Translating SQL queries Used Scan/Filter and Java (especially on coprocessor) No secondary indexes - used FuzzyRowFilter Maybe something like Phoenix would have helped Some issues in mixed mode. Want to move to for Some issues in mixed mode. Want to move to for better/individual column family flushing but needed to 'port' coprocessors (to protobuf)

31 Hbase Value Proposition Better response in UI for CSV file operations - Operations within a file (map, import, reject etc) not dependent on the db size Relieve load on RDBMS -no more CSV data tables Scale out batch processing performance on the cheap (vs vertical RDBMS upgrade) Redundant store for CSV files Versioning to track data cleansing

32 Roadmap Benchmark with 0.96 Retire Coprocessors in favor of Phoenix (?) Lookup Data tables are small. Need to find a better alternative than HBase Design UI for a more Big Data appropriate model Search oriented paradigm, than exploratory/ paginative Add REST endpoints to support such UI

33 Wrap-Up

34 Conclusion PoC demonstrated value of the Hadoop ecosystem Co-existence of Big data technologies with current solutions Adoption can significantly improve scale New skill requirements

35 Thank You