Presented By Smit Ambalia

Transcription

1 Presented By Smit Ambalia

2 Map Reduce Background In-Database Map-Reduce Why Previous initiatives and Limitation Oracle In-Database HADOOP Integration with Oracle s Big-Data Solution Conclusion

3

4 Avoid bringing data residing in ORACLE to a separate infrastructure Issues in HADOOP at enterprise level Lake of development Lake of enterprise-class security

5 Incompatible with HADOOP, so need to rewrite HADOOP programs in a different language Loose integration with HADOOP Dependency on HADOOP infrastructure in addition to RDBMS

6 It s an prototype not feature of Oracle No need to bring data in RDBMS to a separate infrastructure Preserve HADOOP Programming model Minimal dependency on HADOOP infrastructure, query executions are managed by the Oracle Parallel query engine Bring high level security to MapReduce SQL interface :Seamless integration of Map Reduce steps in SQL statements

7

8 Individual Map task are associated with InputSplit which represent the input data that this map task is to process TaskTracker forks the work process to process and call the map and reduce method

9

10 Configure parameters for MapReduce job and actions of those jobs Configuration parameters stored in the job Class before the HADOOP is job executed job.setmapperclass(tokenizermapper.class) job.setreducerclass(intsumreducer.class) job.setoutputkeyclass(text.class) job.setoutputvalueclass(intwritable.class)

11

12 Map and Reduce is performed by table functions, take stream of row as input and generate stream of row as output Select * from TABLE (Word_Count_Mapper(:confKey, Cursor(select * from InTable))) Need to configure parameters in conf files and drivers that are used by table Function The retrieval key is provided to the table function allowing the configuration parameters to be accessed from java domin These parameters are stored in java domain. On the other hand table function are called from within the SQL domain

13 Insert into OutTable select * from Table (Word_Count_Reduce(:confkey, Cursor(Select * from TABLE (Word_Count_Map (:confkey, Cursor(Select * from InTABLE)))))) The output from Word_count_Map is partitioned by Key and streamed to Word_count_Reducer whose output is inserted into OutTable The PQ engine and pipelined table function allows partial results from previous round to be streamed into the next which increases the parallelism

14 Software Component

15

16 Oracle in-database HADOOP accepts configuration parameters in a job class similar to the original HADOOP, this class provides API to accept values from function arguments Instead of passing the value for each conf parameter directly, we will store values in java Conf object, available for the process We use ConfManager to pass parameters among components We use ConfDBStore Class to manage storage of a set of configuration object inside a user table

17 Type MapReduceInputType IS Record ( KEYIN Sys.ANYDATA VALUE Sys.ANYDATA ); Type inputcur IS REF CURSOR RETURN MapReduceInputType; JDBC Driver will generate the matching of input types when making call to DataTypeManager In-Database HADOOP reads data from database table Unique KEY is generated from Oracle pseudo column ROWID

18 We allow users to any legal DataTypes instead of ANYDATA type job.setoutputkeydbtype( VARCHAR2(100) ) job.setoutputvaluedbtype( Number )

19 Useful for Data Type Mapping between SQl and Hadoop SQL queries reads SQL data Type from database and convert it into corresponding Hadoop data type DataTypeManager uses JDBC Driver to provide Data type conversion

20 ODCITableStart : initiates mapper/reducer Classes and gains access rights via Java reflection, set conf parameters ODCITableFetch : repeatedly invokes TableReader to read the data, and TableWriter to write the result back to DB ODCITableClose : Clean up the environment and returns to the caller

21

22 Auditing and monitoring Db activity auditing Db firewall monitoring Centralized audit data Warehouse Encryption and Masking Privilege use Access Control Separation of Duties for DBA s

23 Major Advantage is HADOOP Source compatibility, no need to rewrite into different language to execute in Database Customer can execute Hadoop job within SQl Database without investment to HADOOP cluster Easily integrate HADOOP functionality with SQL

24