DATA MANAGEMENT FOR THE INTERNET OF THINGS February, 2015 Peter Krensky, Research Analyst, Analytics & Business Intelligence Report Highlights p2 p4 p6 p7 Data challenges Managing data at the edge Time series, geospatial, and unstructured data Target areas for improvement This report examines organizations leveraging data generated by the Internet of Things and managing a constant flow of information.
2 As organizations embark on new IoT initiatives and work to attract more insight from swelling data volumes, a new data management approach is called for. Aberdeen s 2014 Big Data survey collected responses from 205 organizations in a variety of industries. From this respondent pool, 68 organizations reported their use of sensor and machine-tomachine data to develop an Internet of Things analytical infrastructure. The Internet of Things (IoT) has made the leap to become a mainstream topic. This growing recognition is due to the impact the IoT has had on business analytics and the potential that still remains untapped. Each day, new machines, sensors, and devices come online and feed information into data systems. As organizations embark on new IoT initiatives and work to extract more insight from swelling data volumes, a new data management approach is called for. Traditional databases and analytics architectures will always be vital, but the IoT calls for specific capabilities to handle diverse data constantly streaming from untold numbers of sources. IoT data is complex, vast, and fast-moving. This report examines the current state of data management and details the capabilities needed to manage IoT data and maximize value. - RT Adapt or Drown in Data Aberdeen examined organizations with the ability to collect, integrate, and analyze data generated by the Internet of Things. These IoT organizations seek to leverage the glut of information generated by disparate devices, systems, and other sources to better understand operations and overall performance. Past Aberdeen research has called for companies to invest in improving infrastructure and data management capabilities to handle the challenges and opportunities presented by the IoT. Aberdeen s survey of 68 IoT organizations revealed the areas where organizations struggle and hope to improve: The average IoT organization s total volume of data grew by 30% over the past year. 54% of IoT organizations reported that their current data analysis capabilities are insufficient. 50% of IoT organizations failed to improve time-todecision over the past year.
3 Sensors are more affordable than ever before and prices on connected devices continue to drop. New devices and machines that transmit data come online every day and full-scale data generation from the IoT is now economically feasible. Organizations that previously derived the majority of their insight from transactional data are shifting their focus to IoT data. All of this analytical development generates swelling data volumes, with IoT organizations averaging 30% data growth year-over-year. Other estimates put data growth rates higher across all industries. Even conservatively, enterprise data will double within three years. Not only is data growing, it is also diversifying. More than half of IoT organizations are concerned that their analytical tools and infrastructure are not equal to modern data demands. Many organizations lack the tools and infrastructure needed to leverage non-traditional data formats, such as unstructured and geospatial data. Decision makers know they have the data they need, but they cannot yet convert it into insight. Organizations staring down the daunting task of IoT data management seek features that enable them to process, store, and analyze the crush of information they can now generate. Finally, many IoT organizations cannot currently react fast enough to streaming data and have failed to improve time-to-decision over the past year. As data accelerates, so does the pace of business. Even as analysts work with huge data volumes and perform more complex analysis, data-driven decisions need to be made faster. IoT organizations need data management solutions that facilitate rapid decisions, no matter how many end points are involved. In an Internet of Things (IoT) environment, nearly every object, device, and consumer good is connected to networks and / or the public internet. These things or smart objects can be individually identified, tracked, and managed, and can be connected to networks through a variety of methods. In an Internet of Things enabled business, everything is connected, creating new capabilities and increased data awareness. IoT organizations need data management solutions that facilitate rapid decisions, no matter how many end points are involved. - Data growth
4 Managing Data at the Edge Data management is moving from the central data repository towards the edge of the network. As devices and sensors multiply and data volumes swell, legacy data management infrastructure and techniques will no longer be sufficient to fully leverage the IoT. IoT organizations demonstrate the direction that data management needs to take. Traditional centralized databases will always have a role to play in analytics. However, as IoT initiatives continue to gain momentum, data management is moving from the central data repository towards the edge of the network (Figure 1). IoT organizations are nearly twice as likely as all other organizations to have automated data capture. These organizations embed data management into the devices and sensors generating data to facilitate a smooth and steady stream of information. Data is managed as soon as it is generated. This is especially important for real-time data feeds. Streaming data can be aggregated at the edge and delivered to the central databases as averages of manageable periods of time. As data constantly pours in from sensors at the edge, the central database only has to handle the influx at a controlled rate, such as once per minute. Also, organizations still have the opportunity to employ tools that perform real-time analytics on the comprehensive volume of data in motion. Figure 1: IoT Data Management Capabilities n=205 Source: Aberdeen Group, September 2014
5 Many IoT organizations add intelligence to the edge to streamline data processing and management. Thirty-nine percent (39%) of IoT organizations have automated data indexing and classification, compared to 19% of all other organizations. These organizations deploy algorithms to filter data before it reaches a centralized database. Automated indexing at the edge ensures that important data and timely alerts are processed and delivered to the right decision makers. Properly classified information is also easier for users to find and incorporate into analysis, whether they are working with streaming data or pulling historical data. Additionally, impertinent and superfluous information is left at the edge and kept from clogging up the central system. Organizations with IoT aspirations should follow this example and invest in analytics at the edge to enable better data management. Automated aggregation and classification at the edge accelerates the generation of insight from IoT data and protects databases from overwhelming data volume and velocity. Properly managed IoT data is most valuable when it is connected and complemented with other data sources. IoT organizations are 58% more likely than all others to have realtime data integration tools. Integrating IoT data in real-time offers analysts immediate context and helps them uncover correlations between current sensor data and historical data. Analysts can ingest information as it arrives from any number of sources. IoT data also becomes instantly accessible via multiple interfaces. Aberdeen s report, Big Data Becomes Fast Data with Accelerated Integration, details the best practices of organizations that have invested to get information to the right people at the right time. Managing data at the edge does not mean eliminating current database architecture. IT decision makers should find a way to incorporate IoT-optimized data Related Research: Big Data Becomes Fast Data with Accelerated Integration"
6 management into their current systems. IoT data should gel with cloud databases and all backend analytical platforms. Handling Data Variety Related Research: The Analytical Eye in the Sky: Tracking Location- Based Insight" Aberdeen examined the types of data that organizations are able to manage and analyze. All of the survey respondents defined as IoT organizations capture time series data from sensors and other smart devices. Many IoT organizations have developed data infrastructure flexible enough to handle additional data types (Figure 2). The vast majority of IoT organizations work with geospatial data, whereas just 27% of all other organizations incorporate location information into their analysis. Geospatial data can be integrated with time series data to source problems and differentiate trends from localized anomalies. Time series data provides the what and the when, but users need to know the where to develop comprehensive insights in an IoT environment. Aberdeen s report, The Analytical Eye in the Sky: Tracking Location-Based Insight, found that users with access to geospatial data were more satisfied with the relevance of their analytical capabilities to their job role. Users can focus on the locations that fall under their purview rather than facing the entire stream of IoT data. Figure 2: Going Beyond Time Series Data n=205 Source: Aberdeen Group, September 2014
7 As IoT organizations mature, they will be able to handle more complex data types all within a cohesive data management system. To gain the full value of IoT analysis, organizations need to go beyond the standardized data formats they have handled in the past. Many IoT databases are flexible enough to enable analysis of unstructured data. The majority of IoT organizations are able to analyze both internally and externally generated unstructured data. Unstructured data offers intelligence that traditional data formats cannot. Organizations must manage this data so it can smoothly flow through analytical infrastructure and be combined with structured data to provide deeper intelligence. Aberdeen s report, Unstructured Data and the New Frontier of Fact-Based Insight, details improvements in data visibility and time-to-decision achieved by organizations using unstructured data. User Demand for Better Data Management Related Research: Unstructured Data and the New Frontier of Fact- Based Insight" All of this work and investment around the Internet of Things has its benefits. IoT organizations are still in the early stages of proving out the value of their analytical efforts. Although IoT organizations offer a glimpse into the best practices of the future, they retain plenty of room for improvement. Users and IT personnel at IoT organizations are generally dissatisfied with the current state of analytics (Figure 3). Ease-of-use should be a major area of focus for developing IoT organizations. Data processes (such as data capture and indexing at the edge) should be automated wherever possible. For example, in manufacturing, IoT data is used for predictive maintenance. Machines and sensors produce data that indicates where plant floor managers should direct their attention and resources. Users need to be able to easily access and analyze that data in order to keep operations running smoothly. Also, facile integration of machine performance data with geospatial data
8 Related Research: Data Visualization for the Internet of Things" can alert decision makers if the same problems are all tied to a single location. Intelligent data management around the IoT ensures that new information is available at all times. This steady stream of potential insights leads to optimized business decisions. Recent Aberdeen research on the IoT revealed that organizations with strong data interfaces and easy access to streaming data markedly improved time-to-decision over the past year. Figure 3: Satisfied or Very Satisfied Users n=205 Source: Aberdeen Group, September 2014 Less than a third of users and IT personnel are satisfied with their data access. Users especially feel that there are troves of IoT data they cannot get at. IoT organizations must invest to make data available in multiple interfaces for easy access and analysis. Aberdeen s report, The State of Data Availability: All the Right Data in all the Right Places, provides additional insights on how best to manage and deliver large data volumes. Finally, IoT organizations need to improve the speed of information delivery. The faster analytical minds can spot issues and opportunities, the faster the appropriate individuals can respond intelligently. Mature IoT organizations manage data so that users can read only pertinent information directly from the stream.
9 Organizations should strive for real-time analytics that enable users to work with data as it arrives from any number of sources. For example, in telecommunications, IoT data from smart grids will instantly reveal outages. The responsible parties need that information immediately so they can minimize downtime and keep customers happy. Better yet, rapid IoT data can reveal early signs of lagging performance and the problem can be fixed before it becomes a major issue that impacts customer experience. Key Takeaways The current capabilities and analytical achievements of IoT organizations are just the beginning. Enlightened organizations will continue to invest to improve the processing, storage, and querying of IoT data. Decision makers should consider the challenges and best practices of data management for the IoT: Data is swelling and analytical demands are mounting. The average organization s data will double within three years. As the flood of information deepens, analytical minds call for additional capabilities and faster access. IoT organizations are concerned that their current analysis is insufficient and time-to-decision is not improving. Live on the edge. IoT organizations are leading the way in managing data before it is stored in a centralized database. More mature organizations automatically filter and classify data at the edge. This ensures that users get pertinent information and prevents overwhelming databases. Embrace diverse data types. IoT organizations are significantly more likely than all others to have analytical capabilities for geospatial and unstructured data.
10 Combined with time series data, these data types offer valuable context. Data management should be flexible enough to handle diverse data and integrate with more traditional information formats. Invest to satisfy users and IT. Currently, the majority of users and IT personnel at IoT organizations are not satisfied with their data access, the speed of information delivery, and the ease-of-use of their data systems. Demonstrated improvement in these three areas is an excellent measure of value for potential data management investments. Organizations should consider automation initiatives to hasten data processes and simplify the user experience. The potential of the IoT is only as great as an organization s ability to manage data and fully harness the constant flow of information. For more information on this or other research topics, please visit. Data Visualization for the Internet of Things; December 2014 The State of Data Availability: All the Right Data in All the Right Places; November 2014 Unstructured Data and the New Frontier of Fact- Based Insight; November 2014 Related Research The Internet of Things: Connecting the Enterprise and the Customer; October 2014 Big Data Becomes Fast Data with Accelerated Integration; August 2014 The Analytical Eye in the Sky: Tracking Location- Based Insight; July 2014 Author: Peter Krensky, Research Analyst, Analytics & Business Intelligence (peter.krensky@aberdeen.com)
11 About Aberdeen Group For 26 years, Aberdeen Group has published research that helps businesses worldwide improve performance. We identify Best-in-Class organizations by conducting primary research with industry practitioners. Our team of analysts derives fact-based, vendor-agnostic insights from a proprietary analytical framework independent of outside influence. The resulting research content is used by hundreds of thousands of business professionals to drive smarter decision making and improve business strategy. Aberdeen's content marketing solutions help B2B organizations take control of the Hidden Sales Cycle through content licensing, speaking engagements, custom research, and content creation services. Located in Boston, MA, Aberdeen Group is a Harte Hanks Company. This document is the result of primary research performed by Aberdeen Group. Aberdeen Group's methodologies provide for objective fact-based research and represent the best analysis available at the time of publication. Unless otherwise noted, the entire contents of this publication are copyrighted by Aberdeen Group, Inc. and may not be reproduced, distributed, archived, or transmitted in any form or by any means without prior written consent by Aberdeen Group, Inc.