A RESTful, pluggable architecture to tackle Big Data in the Cloud. Marco Massenzio, Director Engineering (Server & Cloud)

Transcription

1 A RESTful, pluggable architecture to tackle Big Data in the Cloud Marco Massenzio, Director Engineering (Server & Cloud)

2 SnapLogic Overview 100% Internet Technology for Integration and SOA in the Age of the Internet Premiere Investors Gaurav Dhillon Founder & CEO Co-Founder & Former CEO Informatica Select Customers Global Operations Founded 2006 Headquarters: San Mateo, California Global Offices: Chicago, New York City, London, Munich Key Partnerships: salesforce.com, SAP, Box, Infosys & Orchestra Networks

3 New(ish) Trends Volume NoSQL Variety Hadoop BIG DATA REST Velocity and then there s this thing called the Cloud

4 Deploy On-Premises, In the Cloud or Both REST based distributed architecture execute integration pipelines (workflows) in the cloud, onpremises or both Corporate Firewall Secure Bridge securely orchestrate business processes that traverse cloud and ground assets

5 Select Customer Implementations Complete virtual cloud based system integration Marketing automation, HR, Finance Complex integration between NetSuite and salesforce.com Swapping out NetSuite for Microsoft Dynamics AX UCSF Medical Center Largest Salesforce installation globally Aggregating social media and news feeds to serve Fox News staff Platform to replace hand coded integration If every change, customization or refinement becomes a project, we can't keep up. SnapLogic changes all that elevating us out of code level and putting the power directly into our hands. Mark Brennan, Director Business Applications Pandora SAP, Salesforce and Lotus Notes Integration Customer Master Sync and Order to Cash Near real time calendar sync

6 Shift to Cloud and Eliminate Waste at OSI Business Challenge Help OSI, a global portfolio of over 1,200 restaurant brands such as Outback Steakhouse, realize its vision of a cloud-centric IT portfolio that: Integrates disparate human resources, sales, supply chain, financial, social media and business intelligence systems (many of which were acquired during previous corporate mergers) Minimizes person-hours required to integrate each new application Leverages data trapped in legacy apps with cloud-based applications Increases profit margins by eliminating waste in the supply chain and human resources management SnapLogic Solution Lightweight Integration Hub to orchestrate simple to complex business processes across approximately 20 Cloud and on-premise systems Internal SnapStore to wrap legacy apps with Private Snaps in order to leverage and extend legacy data into a cloud future 6

7 HRIS Modernization at OSI Challenge OSI has 80,000 employees The restaurant business has high employee turnover Eliminate waste in the on-boarding, transfer and termination of OSI employees within the HRIS by activating and deactivating accounts as appropriate to each employee s profile and employment status Decrease the high number of daily support calls (~500) related to HRIS Requirements Replace hand-coded cron jobs (all business logic and field mapping was buried in the code) Eliminate manual data entry and processes Near real time < 10 Minutes Data movement redundancy Failover mechanism for every pipeline Reduce the number of service calls!

8 Architecture Ultipro OSI Integration Employe e (XML) Event Pipeline SnapLogic Integration Hub Excha ReSyn c AD ReSyn c Exchang e Pipeline Exchange XML Ultipro OSI FTP site CSV file Queue the event, save details to DB, Set destination flags Look at event type Call appropriate destination pipelines POSi ReSyn c AD Pipeline POSi Pipeline LMS Pipeline Travel Leader Pipeline AD POSi LMS TL CSV listener / resync Pipeline Database, All Employees latest data, event Queue, Logs, Business rules Expense wire Pipeline Comp Card Pipeline EW CC Admin Tool

9 Close-up of Architecture of OSI Active Directory SnapLogic Integration Hub AD Create Pipeline AD Resync Pipeline AD Activate Pipeline AD update Pipeline AD Determine Action based on event / Flag in Database Get Employee Details Get the business rules and determine eligibility Create / In Activate / Modify the Account in destination system. Call other dependent pipes Database, All Employees latest data, event Queue, Logs, Business rules Admin Tool to modify business rules

10 HRIS Modernization at OSI Results Replaced all hand-coded scripts with reusable, transparent Integration Pipelines Reduced the number of HRIS internal customer support calls to less than 100 per day (and falling) Increased company morale by providing a smooth on-boarding process for new employees Re-deployed the equivalent of 5 FTE towards more strategic projects Sources Ultipro (Cloud HCM) MS Exchange POSi (Legacy POS/HCM App) Destinations Active Directory ADP MS Exchange Travel Leader ExpenseWire Comp Card Lawson Financials POSi LMS and more

11 The O(N 2 ) Complexity of Hand Coding AS/400 Connecting 2 data sources is easy. Connecting 15 requires 105 unique integrations Enterprises are struggling to move to the cloud b/c of legacy integration

12 The O(N 2 ) Complexity of Hand Coding 1 4 AS/ Connecting 2 data sources is easy. Connecting 15 requires 105 unique integrations Enterprises are struggling to move to the cloud b/c of legacy integration

13 Distributed Architecture FTW Not only there is a lot of data, but it s also all over the place: Inside the Enterprise, behind a firewall Provided, managed and stored in multiple SaaS offerings In third party systems, mediated by a variety of vendors

14 SnapLogic s SOA Shopping List SOA via REST Universally Accessible Endpoints API Layer Mediation and Orchestration across Public and Private clouds Extensible Plug-in Architecture for Rapid Development Browser-Based Visual Designer No Coding Required

15 SnapStore: A New Business Model for Integration SnapStore Ecosystem Multiple providers keep the price low Provides for better domain expertise Helps with volume, availability and updating Snaps are securely sandboxed and certified by SnapLogic

16 Remember the O(N 2 ) thing?

17 Pre-Packaged Functionality to Give You a Huge Head Start Database Functions DB Analyzer Resource to DB Table Wizard DB Lookup DB Stored Procedure DB Reader DB Writer DB Lookup DB Upsert DB Bulk Loader Database Connectivity Amazon simpledb DB2 CouchDB EMC Greenplum Hive Informix MarkLogic MySQL MS Access Oracle ODBC PostresSQL SQLite SQL Server Transformations Aggregate Compute Join Outer Join Math Operations MergeSorter (Streaming) Mixer Sort String Operations Data Quality & Standardization Filter Constant Date Dimension Date Operation Format Fields HTML Formatter Regex Sequence Type Converter Unique Read & Write Atom/RSS Reader Atom/RSS Writer CSV Read CSV Write Fixed-Width Reader Fixed-Width Writer FTP SFTP HTML Read HTTP Put/Post JSON Reader JSON Writer RSS Read RSS Write Spreadsheet Reader XML Read XML Read by XSD XML Write & More

18 And You Can Build Your Own Private Snaps Private Snaps Reduce the time and cost to create robust connectors by 80% Same Snap Development Kit used by Commercial SnapStore Developers & ISVs Enable lines of business to access core systems in a controlled and standardized environment

19 Modern Usability: Friendly to Non-Coders 19

20 Modern Usability: Smart Linking 20

21 Simple to test on a sample dataset 21

22 Instant Gratification: run in Designer mode 22

23 Deploy in Production (1/2) 23

24 Deploy in Production (2/2) 24

25 25 Use of HTML5 to Enable Non-Desktop Clients

26 26 Monitoring API

27 JOINs? Seriously, dude We re no longer trying to `join` our SALES table with our CUSTOMERS table, to find out in which ZIP code the most soap powder was sold Our problem is now to figure out what can we learn from last month s SALES, cross-reference it with Google s Search API, augment it with Facebook Sentiment Analysis and then integrate the finding with our vendors system, hosted on AWS, Rackspace and whatever else and next month, the APIs will change

28 (further) Fragmentation of Enterprise Systems Products Assets Employees Organization Accounts Vendors Locations

29 SnapLogic and Cloud MDM Master data governance features for business users Data Models Flexible data models and business rules Data Authoring CRUD, hierarchy management Data Workflow Collaborative data management Data Versions Past, present and future versions Data Integration Via SnapLogic

30 SNAP Cloud MDM Full access to master data in SnapLogic Designer Select records Insert records Update records Delete records Count records Create DataSpace Get DataSpace list Get DataSet list Get Table list Start Workflow Get WF status

31 FMI PLACE HOLDER