BDA Technologies & Selected Case Studies

Transcription

1 BDA Technologies & Selected Case Studies Ettikan Kandasamy Karuppiah (Ph.D), Principal Researcher & Director of Accelerative Technologies Lab MIMOS Berhad SEMINAR INTERNET COMPUTING TECHNOLOGY Theme: Delivering Values From Hyperconnectivities Serbaguna 1, MAMPU 19 th January 2015

2 Big Analytics in a Glance Big data is defined by the high volume, velocity, variety, veracity and value of data which are generated every second, minute, hour, day.by device, human etc Growing data VOLUME 90% of world s data generated over last 2 years Turning big data into Value Broadening data VARIETY 80% of the world s data is unstructured (text, geospatial, audio, video) Increasing data VELOCITY 175,000 tweets per second ECONOMIC BENEFITS GOVERNMENT BENEFITS SOCIETAL BENEFITS Establishing the VERACITY of big data sources Big technology allows us to establish quality and accuracy especially in unstructured data 2

3 B u s i n e s s V a l u e Big Computing in ICT Sector The Malaysian ICT services sub-sector has huge potential growth, with a projected share of 35% in the nation s Digital Economy in Requires Transformative Platform Source: MDEC, as taken from APeJ Big MaturityScape Assessment 2013 by IDC Software Solutions and Support is the Key GDP Contributor

4 Conducted Workshop, Hadoop Programming training to Malaysian Research Community MIMOS Big Technologies R&D MultiCore Java Compiler Acquire Train R&D Intel Malaysia /US MoU Establish work on General Purpose Graphics Processing Unit for text manipulation, Hadoop Trainings nvidia COE for GPGPU Established Cleansing Engine for PERKESO & Warehouse for PERKESO R&D Collaboration MiAccLib Cleansing MiAccLib Cleansing RM10 -> Foundation & Early Adaptation for Heterogenetic Computing AMD Malaysia /US/Europe MoU GE 13 Electoral Roll Analysis with Hadoop & GPU GPU Accelerated Libraries for Cleansing & Financial Risk Modeling ESRI Inc/US Mou Established MiAccLib Finance Acquire Train Sentiment Analysis Model & Modeling & Warehouse for PIK MOH & GPGPU Video Analytics Library MiAccLib Video Modeling & Visualization for PDRM Workforce Planning & GPGPU Security Library Accelerated Libraries for base Accelerator Library (Galactica) MiAccLib Algo/Map High Risk Profiling, Illicit, Taxable & Drugs Detection (PoC) MiAccLib Image MiAccLib Big 2014 MIMOS Berhad. All Rights Reserved. 4 Encryption/Decr yption for National Protection MiAccLib Crypto RM11 -> Maturation & Progressive Deployment of Scalable Heterogenetic Computing

5 Assisting Both Government & Private Sector Needs Private Sector to Go Global National Public Sector DECISIONS REQUESTED FCC is requested to: 1. Take note of data science upskilling for civil servants 2. Take note of MAMPU developing the Government Open framework by Endorse the DG Lab on BDA to identify use cases and pilot projects that address societal wellbeing 4. Take note of MIMOS defining and developing the Big technology platform for Government by Mandate opening up of all relevant data (Open/Non-Open) to the DG Lab on BDA for the pilot projects Source : MDeC Rahsia Besar Rahsia Sulit Terhad Terbuka Opening Up Non-Sensitive Government Policy for all government agencies to open up data categorised under terbuka o Policy Technology E.g. - non-sensitive data like meteorology, transport timetables and pricing of essential goods based on Open criteria +

6 Developing BDA Open Innovation Platform An open-innovation platform between Government, businesses and Rakyat to improve e- participation and user satisfaction. Prioritization through the development of high impact, lowcost, demand driven life-event solutions Secure environment (sandbox) for Government DATA - Community - Government BDA DG (Digital Government) LAB Project Sponsor Sector-specific use cases /life-events: eg. Welfare, Education, Healthcare, Transportation BDA Technology Platform Expertise OUTCOMES POCs, pilots & apps Open.gov.my

7 BDA Technology Platform Strategy Research & Development on KEY Extraction, Processing & Analytics Components DATA Community Government Secured Cloud Services Extraction Security DB Store Infrastructure Management Accelerated Staging Computing Cleansing Harmonisation Anonymisation Machine Learning - Malaysian Model & Analytics Context - (BM, English, Chinese, Tamil) Visualization Visualization - Malaysian Traceability Perspective Key Values i. National Sovereignty ii. Trusted iii. Secured Localized Entity (ie. MIMOS, Cybersecurity)

8 BDA Technology Platform Strategy Applications Source Mi-CLIP Extraction Mi-Harvester Staging Mi-Morphe Cleansing Mi-Harmony Customization Mi-Helio Visualization Mi-BIS Mi-UAP Security Mi-ARMC Structured + Open Linked Harmonisation Mi-Doc Mi-Scrambler Mi-Portal Mi-Trust Mi-DW Management Anonymisation Mi-AccLib Mi-DSS Mi-SP (Video Analytics) Mi-Market Galactica DB Store Traceability Mi-Trace Mi-ROSS Mi-AccLytics Model & Analytics Mi-STP Mi-HPDW Mi-Target Unstructured Infrastructure Mi-Cloud Management Mi-Mobile Mi-MOCHA 3 rd Party Systems & Hardware 8

9 Extracting Value from Unstructured Collector Mi-Clip Harvesting Cleansing Harmonisation Anonymisation Sharing Scrambled database & marts Staging UnStructured Sources Structured Sources Knowledge Harvester (LOD) Mi-Harvester Authentication & Authorization Mi-UAP Mi-ARMC Cleansing Correction Detect Correction Exception Mi-Morphe + Mi-AccLib Harmonisation Harmonisation Terminologies Harmonisation Mi-Harmony + Mi-Semantics Warehouse Platform (Mi-Galactica, Mi-AccConnect, Mi-HPDW) Anonymisation Mi-Scramble + Mi-Crypto + MiAccLib Granular Primary base Published Marts Modeling Visualization Mi-HELIO; Mi-BIS Analytics Mi-HPDW Statistics Mi-AccStat Visualization Social Network Analytics Mi-Visualitic Analytics Mi-Portal Analytics Sentiment Analytics Mi-Intelligence; Mi-NLP Analytics Mi-Target Virtualized Platform & Integrity Manager Mi-CLOUD + Mi-Mocha 2014 MIMOS Berhad. All Rights Reserved. 9

10 Technology Push Technology Challenges Ahead (11 th Malaysia Plan) NEWER Sources of (eg. high speed streams) Technology Pull NEWER Channels of Consumption (eg. Omni channel data market) New Platforms & Revisions NEWER Methods of Visualization (eg. Multi dimensional view) Mi-CLIP Mi-Morphe Mi-Helio Mi-UAP Mi-Harvester Mi-Harmony Mi-BIS Mi-ARMC Mi-Doc Mi-Scrambler Mi-Portal Mi-Trust Mi-DW Mi-AccLib Mi-DSS Mi-SP (Video Analytics) Mi-Market Mi-Trace Mi-AccLytics Mi-STP Galactica Mi-ROSS Mi-HPDW Mi-Target Mi-Cloud Mi-Mobile Mi-MOCHA NEWER Paradigms on Computing (eg. Dockers) 10

11 Big Moving Forward IoA Internet of Anything II Industrial Internet IoE Internet of Everything IoT Internet of Things 11

12 Big Moving Forward IoA Internet of Anything Big Processing Software Defined Network II Industrial Internet IoE Internet of Everything Mobile Systems Wearables Cloud Computing IoT Internet of Things Cyberphysical systems Cyberbiological systems Internet of Humans 12

13 Open Platform & BDA Middleware Architecture Source Structured, Semistructured & Un-structured Sources Web & Social Media RDBM S Extraction Mi-Clip Mi-Harvester Mi-Morphe Flume Sqoop Kafka Model Staging Cleansing Mi-Morphe Mi-AccLib Harmonization Mi-Harmony Anonymisation Mi-Scramble Mi-BIS Mi- HPDW Mi-AccConnect Mi-HPDW Mi-Portal Visualisation Mi-Helio Analytics Tools (Machine Learning) Mi-Intelligence R Mahout ML-Lib (Spark) Mi-NLP Cloudera Search & Solr Galactica Connector Mi-Visualitics Mi-AccStat GIS Mi-Target Apache Drill Spark/Shark Hue Pig Hive Impala Shark Security Mi-UAP Mi- Trust Open Linked Mi-HPDW Mi-Crypto Mi-AccLib Mi-HPDW Galactica YARN Management Files Mi-HPDW STORAGE Storage Galactica FS HDFS, NoSQL RDBMS Galactica Hadoop warehouse / mart Infrastructure RDF Graph DB Cloudera Manager/ Falcon Zoo Keeper Oozie Mi-Cloud Mi-Mocha Sentry MIMOS Solution 3 rd Party Solution

14 Security and Authentication Sentry Mi-UAP Mi-ARMC Mi-Trust MIMOS Big Stack With Reference to Hadoop Stack Visualization Mi-Helio Mi-Portal Mi-BIS (Mi-AccConnect) 3 rd Party Apps Batch Query MapReduce v2 Pig Hive Processing Mi-Morphe Morphlines Mi-Acclib MapReduce v2 (Accelerated ETL) HPDW Model Plugin (For MiMorphe v3 /Pentaho) Machine Learning Mi-BIS (Weka) Accstats (R and Cloudera C++) ML-LIB (Spark) Revolution R, Weka Real Time Query Mi-BIS with Impala through Mi-AccConnect Hue Galactica Apache Drill Spark/Shark HPDW-Big DB Analytics Simulator Planning Tool Predictive Prescriptive Prediction Algorithm Mi-BIS (Mi-Accstats) Mi-BIS ( Mining) Revolution R 3 rd Party GIS 3rd party Management YARN (resource management) Big Orchestration Engine/Layer Zookeeper (configuration and synchronization) Oozie (work flow scheduler) Cloudera Manager Management for Luster Management Sqoop Flume Storage Application Program Interface Thrift REST Java API AVRO Stream Spark Kafka Spring XD & Storm HDFS HPDW-Storage Galactica FS NoSQL (Hbase) Distributed base (Cassandra) RDBMS (Postgress, MySQL) Multi & Many Cores Processors (CPU + GPU) Search Cloudera Search & Solr Streaming (twitter, logs, etc) RDBMS ( Sources Type) NoSQL Type Legend: Complete 3 rd Party 3 rd Party & MIMOS Offering MIMOS Technologies 3 rd Party Technologies

15 Proof of Concepts Selected Use Cases 15

16 Proof of Concepts -Mixed Scenario- (Technology Capabilities) 16

17 Challenges to be Addressed During Initial Roll-Outs 17

18 Challenges (Stage 1) is stored in partial & distributed locations Format of data both in digital & non digital while some are in paper based format Incomplete data set (Q issues) Cleanliness of the data Missing values, Random, Non-Random, CR, Noise Cleaning while maintaining integrity & value Extracting the features in plural languages (at least English & Malay) Structured has longer historical value to be acquired storage media & format for extraction and usage How to authenticate the key values? Where is the reference point? As for unstructured data (e.g social media), current technology is adequate to support the pre-processing, analytics With some local challenges Who are the data owner? How to ensure the security level of the data for sharing? PDP compliance confusion. More to be share by visiting MIMOS Lab

19 Analytics Challenges (Stage 2) Tools are available but right approach is still critical for evaluation Which are the best/right algorithms to be used? Can you identify the right domain expert within the organization? Who are the local domain experts to be consulted for the methods/algorithms selection? You may not have data scientist in specific gov. organization, but how to form one (external + internal) -> analytics team What exactly are the data owners business needs? Why do they need to do this? Headache for them best to leave the data to rest in peace!! Which data to be included and which to be excluded, what to be anonymized? concern of meaning/trend extraction Plurality of languages & interpretation accuracy Semantification of the language specific analytics Bottlenecks to be identified and accelerated approach required for the specific processing Agile is the best way

20 Results Challenges (Stage 3) Visualization of the results in simple, action-able and communicable how to handle continuously changing analytics (and the results) due to New data inclusion New domain expert inclusion New additional factors to be considered Who validates the results? How to translate results to value for (gov) organization How to translate the value to actions? How to follow-up on 2nd cycle of activities?

21 Benefiting Humanity Through Technology Thank You