IBM Analytics The fluid data layer: The future of data management Why flexibility and adaptability are crucial in the hybrid cloud world
1 2 3 4 5 6 The new world vision for data architects Why the fluid data layer matters Must-have capabilities The fluid data layer in the real world Benefits for the Resources
The new world vision The world of the data architect is getting more complex. New technologies such as clustered databases, cloud platforms and mobile data synchronization have become today s reality. Now more than ever, data architects must change the way they look at the data under their purview. First, there must be freedom of movement. Data must be able to move where users need it, when they need it, even in real time. There must be freedom of location. Data should live where its management, storage and analytics are most economical and logical. This could apply to any number of places, including data centers, private clouds, public clouds, on-premises storage or user devices. Finally, there must be freedom of expression and assembly. Data must be available and accessible so it can be combined with other sources for more detailed insights and analysis. 3
But for this vision to come to fruition, the traditional boundaries and data management structures have to change. Data type and location no longer matter; what is important is data access, responsiveness, accuracy and analytic power. This e-book explores a new approach to making data available and accessible when and where it s needed most: the fluid data layer. Creating a fluid data layer enables a business to move with unprecedented flexibility. By enhancing a hybrid cloud environment with data refinement, integration, management and more, data architects can free data from location restrictions and capitalize on its full value. 4
Why the fluid The fluid data layer offers numerous advantages : Being able to offload processing quickly to a hybrid cloud improves uptime Organizations can adjust seasonal or daily spikes without overprovisioning equipment Understanding the shortest path of movement between data source and user ensures lower latency and faster access Teams can try short-term projects that may have been expensive to attempt previously, because data architects can quickly stitch together systems without worrying about intensive data movement efforts If business users have an idea for a new application, data architects can get prototypes up and running quickly to prove or disprove concepts What s a data layer? In today s cloud and mobile environments, data management is often distributed across data centers and mobile devices. The modern data management component of an application is more aptly termed a data layer. More than just a database, it refers to the database management system and the distributed infrastructure that runs, scales and secures it. 5
On an organizational level, the fluid data layer helps data architects keep the data warehouse infrastructure out of the users way, while still providing tremendous business benefits (see Figure 1). Data architects can more easily move not just data, but databases themselves. For example, a music-sharing service with a data center on the eastern seaboard of the US suddenly discovers it s a big hit in Asia. Its data architects can set up a cloud-based data center in Asia that improves performance and reduces latency. If either data center goes down, the other will act as a backup. IBM dashdb portable analytics Private cloud Analytics Data center Automatic synchronization Acquisition Cleansing Public cloud Applications Always available IBM DataWorks Refinement Security Cloud data center Mobile Scalable DBaaS Users IBM Cloudant Locationindependent access Devices Offices Off-site/remote locations Figure 1. A fluid data layer distributes data management across data sources and allows simple data access and movement. 6
The fluid data layer: The future of data management That fluidity extends further. One of enterprises big concerns with as-a-service offerings is lock-in and inflexibility. A fluid data layer eliminates that concern, letting data architects shift data from one cloudbased system to another as necessary, changing the architecture without recoding or interrupting data accessibility. Flexibility extends to data sources as well. Data doesn t just live on servers in a database, although that is where it has been easiest to control. Today, data lives on users mobile devices. It s in videos, call logs, transaction details and social network posts. It streams in from Internet-enabled vehicles and embedded sensors. The fluid data layer allows architects to grab data from wherever it resides and push data to where it is needed, while ensuring it is secure and accessible. 7 1 The new world vision 2 Why the fluid 3 Must-have capabilities 4 The fluid data layer 5 Benefits for the 6 Resources
Must-have capabilities To create a fluid data layer, architects need a full portfolio of management capabilities that can be used together in various combinations to meet business needs, and easily integrated and reused as those needs change. That means being able to: Move between internal data centers and cloud-based systems Handle structured and unstructured data Scale up and down based on traffic, time of day and other parameters Integrate existing tools, systems and data sources into the cloud data layer Applications must all work together so data architects have the tools to orchestrate, manage, analyze and reconfigure the fluid data layer. The requirements break down into three basic categories: data management, data refinement and data analytics. 8
Data management At the heart of the data layer is a database management system and the supporting distributed infrastructure. Data architects rely on this common foundation for data sourcing and targets, using their experience to build on other technologies in the portfolio. But a data management strategy should also let data architects create a database in the cloud that can be quickly populated for a short-term application, whether for a pilot project or a seasonal need. Using a cloud database helps reduce costs because developers can provision environments quickly, but also provides agility though rapid project testing. Data architects should also have a way to build portable analytics that flow from where data is collected to where it s analyzed. Taking advantage of both in-memory and columnar technology enables them to deliver insights faster. Features that allow integration between the data layer and current tools and systems increase reuse opportunities and reduce the steep learning curves on new techniques. Data refinement Data architects need to deal with extract, transform, load (ETL) quickly, to identify the source and the target of data and automate the exchange of data between the two. That means helping data architects classify, profile, cleanse and qualify data to ensure consistent accessibility and usability across and beyond the enterprise. Plus, while enterprises are becoming more confident about the cloud s ability to secure data when it s in transit, protecting sensitive data remains critical. The fluid data layer must automatically encrypt data both at rest and in transit. 9 3 Must-have capabilities 4 The fluid data layer
Data analytics Data architects must be able to analyze streaming data sensor data, telematics, GPS information, regional sales data and more with as little delay as possible. Ideally, they should be able to design systems so that analytics can be performed in near-real time as data enters the system, regardless of the number of sources. Processing data from multiple sources offers the ability to add context to insights for additional understanding. The same need for speed applies to unstructured data. With the fluid data layer, data architects can assemble output from unstructured databases ranging from Apache Hadoop systems to NoSQL and JSON data flowing into an in-memory database. They don t have to force data into a schema that s not appropriate, which makes development more agile. There should also be no restrictions on tools so data architects can use SQL to craft queries of unstructured databases or take advantage of high-level statistical analysis. 10
Why IBM for cloud data management? 1 2 IBM is renowned for its data management intellectual property, now available through the cloud. IBM integrates its cloud data services to make them easier to use, avoiding the integrate-and-provision-it-yourself burden imposed by other standalone services. 3 4 IBM cloud data services run on the IBM SoftLayer bare-metal cloud platform, which enables reliability, performance and a flexible global footprint. IBM gives you the flexibility to deploy your cloud data on public, private or hybrid cloud platforms to maximize cost efficiency, control and performance. The IBM advantage: Integrated data services IBM offers a variety of powerful data services, many of which are available to developers and data architects on IBM Bluemix. Integrated together, they help make the fluid data layer encompassing all the applications simple to build and simple to configure. To provide the highest level of development flexibility and the lowest amount of lock-in, IBM builds many of its offerings using open source technologies like Hadoop, Spark and CouchDB. Learn more about the IBM data service portfolio with these resources. 11
The fluid data layer Any industry that relies on data for insights and efficiency will benefit from the fluid data layer. Even more important, this architecture can break down barriers for industries that traditionally store lots of data but have been unable to share it or analyze all of it. What could you do if data location was irrelevant? IBM customer Physion has created a fluid data layer for its cloud-based Ovation application, 1 which makes it easier for scientists to store, organize, manage and share experimental research data. By breaking down barriers to scientific collaboration, Ovation maximizes the value of data otherwise trapped on individual researcher desktops or within organizational silos. Data architects for a retailer could craft a cloud-based data warehouse that combines structured data such as inventory with unstructured data from social media or weather services to anticipate upcoming adverse weather conditions. The result: shipping tools or food to stores whose shelves need replenishment fast. In healthcare, the ability to incorporate data from multiple systems can help providers and public health organizations share data about illness outbreaks. Analyzing social media for incorrect healthcare information can help identify where and how accurate information should be disseminated for maximum reach. 12
Or consider the potential benefit to a ridesharing service, aggregating data from a variety of different data sources to improve the customer experience and ensure its drivers are in the right place at the right time. Data architects could take historical data showing when its service is in highest demand, and aggregate it with real-time information about current demands. They could also load additional weather and traffic data to provide more complete insight. With this information delivered in near-real time, the ridesharing company can inform its drivers of the best places for pickups and provide customers with more accurate travel times. Customers have shorter wait times, and drivers get more business. Mobile rideshare app running on IBM Cloudant Weather data Traffic patterns Road construction Local event schedules Location Time of request Wait times dashdb Data warehouse services DataWorks Refined, cleansed, secure data Analytics Pre-route drivers to heavy use locations at peak request times Advise drivers of routes around construction, flooding or accidents Proactively send more cars to areas with historically high volumes and long wait times Figure 2. Combining multiple data sources and types, analyzing all of the data and delivering it in near-real time enables a ridesharing service to reduce customer wait times and keep drivers busy. 13
Benefits for the The fluid data layer provides a foundation for accessibility, agility and adaptability. Data architects can now enable self-service access to trusted data. They can give developers the ability to embed data service into new applications and give business analysts the ability to find, use and contribute data for analysis. Data redundancy across data sources mobile, on-premises, in the cloud or elsewhere offers non-stop data access for all users. By expanding data layers on demand, they can hold more information for data scientists to analyze. And perhaps most important they can use the inherent scalability to expand and relocate the data layer as necessary, allowing it to adapt to future needs without a massive replacement or upgrade effort. The vast IBM portfolio of data management, refinement and analytics products and services gives data architects mix-andmatch capabilities to build better systems faster. There s no requirement to use all the products in the portfolio; architects can start with a single project, expand it, and add and change capabilities as necessary. This flexibility provides seamless data integration and governance for the creation of applications, with easy-to-use, simple and secure services running on cloud, on-premises or hybrid deployments. Exactly what data architects need to deal with today s ever-more-complex and competitive business world. 14
Resources To learn more about IBM services that can help you build and enhance the fluid data layer, visit these resources: Bluemix: ibm.com/software/bluemix IBM Cloud Marketplace: ibm.com/marketplace/cloud IBM cloud computing: ibm.com/cloud-computing IBM Cloudant: https://cloudant.com IBM DataWorks: ibm.com/software/data/make-data-work IBM dashdb: ibm.com/software/data/dashdb 15
Copyright IBM Corporation 2015 IBM Analytics Route 100 Somers, NY 10589 Produced in the United States of America May 2015 IBM, the IBM logo, ibm.com, Bluemix, Cloudant, dashdb, and DataWorks are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at Copyright and trademark information at ibm.com/legal/ copytrade.shtml SoftLayer is a trademark or registered trademark of SoftLayer, Inc., an IBM Company. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. This document is current as of the initial date of publication and may be changed by IBM at any time. Not all offerings are available in every country in which IBM operates. The client examples cited are presented for illustrative purposes only. Actual performance results may vary depending on specific configurations and operating conditions. THE INFORMATION IN THIS DOCUMENT IS PROVIDED AS IS WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED, INCLUDING WITHOUT ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OR CONDITION OF NON-INFRINGEMENT. IBM products are warranted according to the terms and conditions of the agreements under which they are provided. 1 Physion life science app: IBM Cloudant case study. https://cloudant.com/resources/case-studies Please Recycle IMM14178-USEN-00