The Data Reservoir. 10 th September Mandy Chessell FREng CEng FBCS Dis4nguished Engineer, Master Inventor Chief Architect, Informa4on Solu4ons

Transcription

1 Mandy Chessell FREng CEng FBCS Dis4nguished Engineer, Master Inventor Chief Architect, Solu4ons The Reservoir 10 th September 2014

2 A growing demand Business Teams want Open access to more informa4on More powerful analysis and visualiza4on tools IT Teams are Concerned about cost. Concerned about governance and regulatory requirements. both the CMO and CIO are on the hook for turning all that data into above-market growth. Call it a shotgun marriage, but it s one that CMOs and CIOs both need to make work CMOs and CIOs Need to Get Along to Make Big Work (HBR ) 2

3 The reality of information management today 3

4 Business scenarios we see Subject maier experts want access to their organiza4on s data to explore the content, select, control, annotate and access informa4on using their terminology with an underpinning of protec4on and governance. Scien4sts seeking data for new analy4cs models. Marketeer seeking data for new campaigns. Fraud inves4gator seeking data to understand the details of suspicious ac4vity. Day- to- day acevity. Requiring ad hoc access to a wide variety of data sources. SupporEng analysis and decision making. Using the subject maper experts terminology. Providing the flexibility of spreadsheets that can scale to large volumes, a wide variety of informa4on types whilst protec4ng sensi4ve informa4on and op4mizing data storage and provisioning. 4

5 blues & skills issues A dispropor4onate por4on of the 4me spent in analy4cs project is about data prepara4on: acquiring/preparing/formarng/normalizing the data In addi4on to raw data, augmented data/analy4cal assets can significantly speed up the analy4cs process and par4ally bridge the talent gap 5

6 Initial Big efforts are focused on gaining insights from existing and new sources Big data sources +,-./-0123./ 89:.1/ 8;-2</ =:./3,/ =302-<6>:?2- BAC76=0-./63,6DE=67-1- H:3/I-12-< JK?23 =12<<6C;-5:/6L6M2?:3/ '# %!# $'" '$" "&# '(" '(" ')" ')" '*" $!" "%# ""# *(# *"# %&" %#" +&# Respondents with active big data efforts were asked which data sources are currently being collected and analyzed as part of active big data efforts within their organization. #$" ()# &'#!!" $%#!"# Global respondents Banking & Fin Mgmt respondents 6 6 Source: The real world use of Big, IBM & University of Oxford

7 The true value of Big is in context Ownership? Quality? Security? Privacy? Actuarial data Government statistics Epidemic data Patient records Weather history Location risk... Occupational risk Family history Raw data Travel history Feature extraction metadata Dietary risk... Personal financial situation Chemical exposure Domain linkages Social relationships Full contextual analytics 7

8 The interesting dilemma A man goes into a jewellers and buys an expensive watch Is it fraud in which case the bank must stop it Threat Is it money- laundering in which case the bank must report it Obliga4on Does he have an expensive trophy wife in which case perhaps he would be interested in a loan? Has he just won the loiery should the bank improve the services offered? Opportunity The same event is of interest by different departments. There is major overlap in the data required to answer the ques4on. It may not be possible to determine the answer with just the informa4on in the channel - Previous or subsequent ac4vity is required It is all a maier of coordina4on and 4ming 8

9 Big Lakes or Swamps? As we collect data Can we preserve clarity? Do we know what we are collecting? Can we find the data we need? Are we creating a data swamp? How do we build trust in big data? Do we know what data is being used for? 9

10 Organizations expect information governance will deliver Understanding of the informa4on they have, Confidence to share and reuse informa4on, Protec4on from unauthorised use of informa4on, Monitoring of ac4vity around the informa4on, Implementa4on of key business processes that manage informa4on, Tracking the provenance of informa4on and Management of the growth and distribu4on of their informa4on. 10

11 The Reservoir Reservoir = Efficient Management, Governance, ProtecEon and Access. As organiza4ons experiment with analy4cs they discover: Crea4ng new analy4cs requires access to historical data from many systems. This data includes valuable and sensi4ve data that is core to the organiza4on s opera4on. Hadoop is a flexible plazorm for storing many types of data but is not necessarily fast enough for the produc4on deployment of some analy4cs. needs to be reformaied and copied onto a specialist analy4cs plazorms such as Netezza. A data reservoir provides: Single extrac4on of data from opera4onal systems and distribu4on to mul4ple analy4cs plazorms. Cataloguing and governance of the data in the analy4cs plazorms Simple interfaces for the line of business to access the informa4on they need. 11

12 How does the data reservoir operate? Discover Explore Access 1 3 Adver4se Catalog 4 Reservoir Repositories Provision 12

13 reservoir system context diagram Decision Model Management Enterprise IT Deploy Real- 4me Decision Models Deploy Real- 4me Decision Models Service Calls Access Deposit Deploy Decision Models Line of Business Applications Systems of Engagement Events to Evaluate Search Requests System of Record Applications New Sources Enterprise Service Bus Third Party Feeds No4fica4ons Service Calls Reservoir Cura4on Interac4on Service Calls Deposit Simple, Ad Hoc Discovery and Analysis Out Third Party APIs Internal Sources In Access Report Requests Reporting Other Systems Other Of Insight Systems Of Insight Management Reservoir Operations 13

14 reservoir system context diagram Decision Model Management Enterprise IT Deploy Real- 4me Decision Models Deploy Real- 4me Decision Models Service Calls Access Deposit Deploy Decision Models Line of Business Applications Systems of Engagement System of Record Applications New Sources Enterprise Service Bus Third Party Feeds Third Party APIs Internal Sources Events to Evaluate No4fica4ons Service Calls Out In Descriptive Deposited Shared Operational Historical Harvested Content Hub Catalog Operational History Information Warehouse Deep Information Views Audit Code Hub Reservoir Repositories Search Requests Cura4on Interac4on Service Calls Deposit Access Report Requests Simple, Ad Hoc Discovery and Analysis Reporting Other Systems Other Of Insight Systems Of Insight 14 Reservoir Management Reservoir Operations

15 reservoir system context diagram Decision Model Management Information Curator Governance, Risk and Compliance Team Enterprise IT Systems of Engagement System of Record Applications New Sources Enterprise Service Bus Third Party Feeds Third Party APIs Internal Sources Deploy Real- 4me Decision Models Events to Evaluate No4fica4ons Service Calls Out In Deploy Real- 4me Decision Models Service Calls Access Descriptive Deposited Shared Operational Historical Harvested Deposit Content Hub Catalog Operational History Deploy Decision Models Information Warehouse Deep Information Views Audit Understand Sources Code Hub Catalog Interfaces Reservoir Repositories Adver4se Source Understand Compliance Understand Sources Search Requests Cura4on Interac4on Service Calls Deposit Access Report Requests Report Compliance Line of Business Applications Simple, Ad Hoc Discovery and Analysis Reporting Other Systems Other Of Insight Systems Of Insight 15 Reservoir Management Reservoir Operations

16 Differing user perspectives Provision Sand Boxes. Sand Box Search for, locate and download data and related ar4facts. Define governance policies, rules and classifica4ons. Monitor compliance. Governance Catalogue View lineage (business and technical) and perform impact analysis. Add addi4onal insight into data sources through automated analysis. 16 Develop data management models and implementa4ons. Cura4on of Metadata about Stores, Models, Defini4ons Stores Stores Stores

17 reservoir system context diagram Decision Model Management Information Curator Governance, Risk and Compliance Team Enterprise IT Systems of Engagement System of Record Applications New Sources Enterprise Service Bus Third Party Feeds Third Party APIs Internal Sources Deploy Real- 4me Decision Models Events to Evaluate No4fica4ons Service Calls Out In Deploy Real- 4me Decision Models Enterprise IT Interaction Real-time Analyics Streaming Analytics Real-time Interfaces Publishing Feeds Ingestion Service Calls Access Descriptive Deposited Shared Operational Historical Harvested Deposit Content Hub Catalog Operational History Deploy Decision Models Information Warehouse Deep Information Views Audit Understand Sources Code Hub Catalog Interfaces Reservoir Repositories Adver4se Source Understand Compliance Understand Sources Search Requests Cura4on Interac4on Service Calls Deposit Access Report Requests Report Compliance Line of Business Applications Simple, Ad Hoc Discovery and Analysis Reporting Other Systems Other Of Insight Systems Of Insight 17 Information Integration & Governance Information Broker Code Hub Staging Areas Operational Governance Hub Reservoir Monitor Workflow Guards Management Reservoir Operations

18 reservoir system context diagram Decision Model Management Information Curator Governance, Risk and Compliance Team Enterprise IT Systems of Engagement System of Record Applications New Sources Enterprise Service Bus Third Party Feeds Third Party APIs Internal Sources Deploy Real- 4me Decision Models Events to Evaluate No4fica4ons Service Calls Out In Deploy Real- 4me Decision Models Enterprise IT Interaction Real-time Analyics Streaming Analytics Real-time Interfaces Publishing Feeds Ingestion Service Calls Access Sand boxes Descriptive Deposited Shared Operational Historical Harvested Deposit Content Hub Catalog Operational History Deploy Decision Models Information Warehouse Deep Raw Interaction Information Views Audit Understand Sources Code Hub Catalog Interfaces Reservoir Repositories Adver4se Source View-based Interaction Information Access and Feedback Sand boxes Reporting Marts Understand Compliance Understand Sources Search Requests Cura4on Interac4on Service Calls Deposit Access Report Requests Report Compliance Line of Business Applications Simple, Ad Hoc Discovery and Analysis Reporting Other Systems Other Of Insight Systems Of Insight 18 Information Integration & Governance Information Broker Code Hub Staging Areas Operational Governance Hub Reservoir Monitor Workflow Guards Management Reservoir Operations

19 Classification Schemes Classifica4on is at the heart of informa4on governance. It characterizes the type, value and cost of informa4on, or the mechanism that manage it. The design of the classifica4on schemes is key to controlling the cost and effec4veness of the informa4on governance program. Business Classifica-ons Business classifica4ons characterize informa4on from a business perspec4ve. This captures its value, how it is used, and the impact to the business if it is misused. Resource Classifica-ons Resource classifica4ons characterize the capability of the IT infrastructure that supports the management of informa4on. A resource's capability is partly due to its innate func4ons and partly controlled by the way it has been configured. Ac-vity Classifica-ons Ac4vity classifica4ons help to characterize procedures, ac4ons and automated processes. Seman-c Classifica-on Seman4c classifica4on iden4fies the meaning of an informa4on element. The classifica4on scheme is a glossary of concepts from relevant subject areas. These glossaries are industry specific and they are shipped with our industry models. The seman4c classifica4ons are defined at two levels: Subject area classifica4on Business term classifica4on 19

20 Governance Rules Defined for each classification for each situation Sensi4ve informa4on masked here Personal informa4on masked here 20

21 New policy support inside the Information Governance Catalogue Principle Principle Policy Principle Policy Policy Implica4ons Implica4ons Ac4oned by Governance Rule Governs Classifica4on Classifica4on Classified by Metadata Descrip4on Implica4ons Implemented by Describes Governance Rule Governance Rule Implementa4ons Modelled Metadata Implementa4ons Asset Deployed to, Executed by, Monitored by Asset 21

22 Integrated Metadata Lineage (Traceability) Where does this data come from? Why is this data incorrect? Why is this data incomplete? Can I trust this value? Impact Analysis Where is this element used? What happens if I change this? Op4miza4on Where is the redundancy? How can I make this run more efficiently? Understanding What does this mean? How is this used? Control Why is this parameter set to this value? Who made this change? I can change this to meet new business requirements 22

23 Information Governance People, Process and Technology Successful Governance is implemented with a combina4on of: Skilled people, correct roles and organiza4on Processes that create a pragma4c, targeted and agile work environment. Standards, templates and assets that improve consistency between implementa4ons. Technology that automates classifica4on, enforcement valida4on, and correc4on of data. Governance Activity Type Information Exchange Role that technology can play Communication Policies & Metrics Delivering education, best practices, assessments, templates. Compliance Design Changes Implementing control points and enforcement points. Support for design and code reviews. Test Management. Exception Exception Requests Exception process management, incident reporting. Feedback Measurements Dashboards and reports on compliance. Vitality New Requirements Change process management 23

24 Example: Supporting the Use of Authoritative Sources Defini4ons of policies, rules and processes around the use of authorita4ve sources Enforcement in build, automated test cases and configura4on management Defini4ons of authorita4ve sources and where they are used. Make it Clear Make it Unavoidable Control Points in Sprint and Design Review Processes Make it Visible Make it Easy to do the right thing Easy access to compliant test data. Pre- built data structures implemen4ng standards. refinery services for common re- engineering tasks. 24

25 Three interlocking lifecycles of information governance Audit Define Policy Classify Monitor Roll out Metadata Design Execute Opera4ons Policy Policy Development Deploy Detect Remediate Develop 25

26 Summary Increasing use of analy4cs requires business to have greater access to more informa4on governance needs to adapt so that the business takes reasonability for governance The governance program should educate, trust implicitly that people will do the right thing - and check - and follow up People are at the heart of governance - processes help them collabora4on and build trust - technology supports their work 26

27 zz z z z z z Questions? 27

28 New Information Architectures and Capabilities Reference Material 28

29 Governing and managing Big for Analytics and Decision Makers An introduc4on to the Reservoir solu4on 29

30 Ethics for Big and Analytics Context for what purpose was the data originally surrendered? For what purpose is the data now being used? How far removed from the original context is its new use? Consent & Choice What are the choices given to an affected party? Do they know they are making a choice? Do they really understand what they are agreeing to? Do they really have an opportunity to decline? What alternatives are offered? Reasonable is the depth and breadth of the data used and the relationships derived reasonable for the application it is used for? Substantiated Are the sources of data used appropriate, authoritative, complete and timely for the application? Owned Who owns the resulting insight? What are their responsibilities towards it in terms of its protection and the obligation to act? Fair How equitable are the results of the application to all parties? Is everyone properly compensated? Considered What are the consequences of the data collection and analysis? Access What access to data is given to the data subject? Accountable How are mistakes and unintended consequences detected and repaired? Can the interested parties check the results that affect them? com/whitepaper/ethics-bigdata-and-analytics 30

31 PaIerns of Management Vocabulary for design discussions. Icons used in white- boarding and design documents. Design guidance for specific solu4ons. A catalogue of design choices. Founda4on for serng architecture standards and reference architectures. Material for educa4on and training in informa4on architecture ?WT.mc_id=Author_Chessell_PoIM 31

32 Staying Ahead in the Cyber Security Game hip://www- 01.ibm.com/common/ssi/cgi- bin/ssialias?subtype=wh&infotyp e=sa&appname=swge_ti_se_us EN&htmlfid=TIL14103USEN&aIac hment=til14103usen.pdf#loade d 32

33 33