The Use and Performance of Big Data in the Energy Sector

Transcription

1 SURVEY REPORT The Use and Performance of Big Data in the Energy Sector A PennEnergy Report On the Use and Performance Of Big Data in the Energy Sector SPONSORED BY: 2 Forward 6 Key Findings of the Survey 16 Appendix Reprinted with revisions to format from PennEnergy. Copyright 2015 by PennWell Corporation

2 Forward WHILE GLOBAL ENERGY demand continues to grow, energy firms confront a range of daunting challenges. For oil and gas companies, exploration, development and production are more challenging than ever. Prices are fluctuating, regulations are more stringent, and forecasting and budgeting are more difficult than ever. Power sector companies must develop smarter energy capabilities to improve reliability, efficiency and return on their assets. At the same time, public utilities and private sector power companies are working to reduce costs, to better engage their customers, and to manage the distributed energy resources of the 21 st century. Not surprisingly, organizations across the energy sector continue to seek smarter and better ways to meet those demands and are increasingly deploying advanced data and analytics in pursuit of those objectives. In partnership with PennEnergy Research, IBM Systems & Technology Group conducted a survey to explore the use and performance of Big Data in the energy industry. The survey also sought to examine awareness among energy professionals of how Big Data will affect modeling, simulation, and analytics in the oil and gas sector. The study also sought to identify challenges the energy sector faces in acquiring the data needed to conduct advanced analytics. Out of 38,167 potential participants invited to complete and online survey, completed responses were received from 209 individuals. The sample includes responses from participants in North America, Europe, the Middle East and Africa. The confidence level for the survey is 95%, with a margin of error of 6.76%. Responses represented a variety of locations, types of businesses, and personnel. 2

3 Forward Responses by region and nature of business Responses by current role/position When asked if they understood the Survey s definition of Big Data: 3

4 Introduction THE ENERGY BUSINESS has never been easy, and the challenges continue. Economic, regulatory, and competitive pressures continue to grow. Consumers now expect more sophisticated and personalized services. At the most basic level, companies in the power sector and in oil and gas are pressed to deliver safe, reliable, affordable and sustainable energy. Oil and gas In the petroleum industry, exploration, development and production activities are shifting to more challenging sites. Regulations are more stringent, forecasting more complex, and it is harder than ever to retain key technical personnel. Oil and gas firms are pressed to find new processes and technologies and to capture and manage new kinds of information. They must also learn how to transform data into actionable insights to improve asset values, to drive yields, and to enhance safety and environmental protections. They need to make faster, smarter decisions that keep them ahead of the competition. The power sector In today s increasingly consumer-centric marketplace, power companies must optimize generation, transmission and distribution capabilities. They need intelligent, two-way energy and information networks capable of providing critical feedback on energy demand and usage. Forward-looking utilities are integrating new plants and energy sources into their portfolios. They are modernizing their customer service systems to better engage consumers and businesses. Many are leveraging in new ways to drive efficiencies and profit in a more challenging utility marketplace. A data challenge 4 Yet for many in the petroleum and power sectors, those positive outcomes are out of reach. While energy executives now recognize data and analytics as crucial enterprise resources, existing IT infrastructures were not designed to support Big Data and Analytics.

5 Introduction Many companies have access to vast troves of data and content, yet key decisions makers often lack the insights needed to make better business decisions. As the types and volume of data continues to grow, and as analytics becomes a critical competitive factor, a growing number of energy companies now realize they are not fully prepared. Being Prepared by Asking the questions To position themselves to realize the potential of the data revolution, energy companies must first ask the right questions. Those conversations should ideally include leaders from both Information Technology and the Lines of Business and must address the key challenges and concerns related to data and analytics in an energy context. To facilitate those conversations, IBM and PennEnergy surveyed perceptions of Big Data and Analytics in the energy sector. The following results reflect input from participants in the oil and gas and power sectors. 5

6 Key Findings of the Survey Data-related challenges Respondents identified key data-related challenges in the energy industry. Sector-specific perceptions. Oil and gas respondents ranked accessibility, volumes, and latency as their primary data-related challenges, in that order. In the power sector, respondents ranked latency, volumes, and accessibility as their top datarelated challenges. Power respondents indicated a higher level of data-related challenges than those in the oil and gas sector. Power Oil & Gas 6

7 Key Findings of the Survey Data types and location. Respondents identified two key challenges relating to Big Data Analytics: integrating different data types, and accessing physically-remote data. Data speed. Overall, 49.3% of respondents say the speed of analyzing, simulating, and modeling data is the most important requirement for a Bid Data Analytics application. Respondents identified speed of gathering data as the least important requirements for Big Data Analytics in the energy sector. Overall Industry Breakdown How data is being used Complex modeling. Perhaps not surprisingly, modeling and simulations dominate data and analytic use cases. Some 61.2% of respondents use data to perform complex modeling, simulations and analytics. Modeling satisfaction. Only 18.5% of power respondents say they are completely satisfied with the processing time needed to conduct modeling, simulation and analysis, while just 10.2% of oil and gas respondents are completely satisfied with modeling performance. 7 Data types. The types of data collected by energy firms suggest operational data and feeds are important, and show an emerging need to support operational, near/real-time activities in the future. Power respondents identified their most

8 Key Findings of the Survey critical performance-related data types as meter (51.1%), power (48.9%), load/ voltage (44.7%), and grid and forecasting (tied at 40.4%). Oil and gas respondents identified most critical performance-related data types as oil and gas data (67.2%), field operations (63.3%) well (59.4%), and asset (43.8%). Operational decisions. A majority of respondents (65.6% in oil and gas, and 56.4% in power) say providing data and insights to large numbers of operational decision makers is a mission-critical activity. Power Oil & Gas Data processing methods. In the Power sector, respondents identified periodic process update, near real-time, and end-of-day as the methods most often used to quickly process data and analytics. Respondents from the Oil and Gas sector named those same three methods in the same ranked order. 8

9 Key Findings of the Survey Power Oil & Gas Where the industry is today IT availability. Just 15% of respondents are confident they have the IT resources needed to support Big Data and analytics while 85% say they wait for IT resources at some level. This key survey finding speaks directly to the view that oil and gas firms are not fully prepared to address and leverage data and analytics. The availability of IT resources, reflecting the strategic view of data and analytics across the enterprise, is a key concern for oil and gas leaders. Deploying analytics. In both the power and oil and gas sectors, the majority of respondents say their organization is at parity or somewhat behind in the use of advanced analytic and simulation technologies. Power Oil & Gas 9

10 Key Findings of the Survey Key IT gaps. Respondents also identified their greatest data- and analytics-related IT gaps. In the power sector, 57.4% indicate power generation is their primary data/analytics related IT gap. In oil and gas, 62.5% say upstream activities (where most use cases reside) are their main data/analytics IT gap. Power Oil & Gas Engaging IT. A slight majority (51.7%) of respondents engage their IT organization to define requirements at the business planning/scope phase of a technology initiative. Advanced analytics. While 61.2% of respondents are performing complex modeling, simulations, and analytics with their data, 38.8% are not. Almost one in ten respondents did not know if their data is used for complex modeling, simulations, or analytics. A third expects a significant increase in advanced analytic activities, and almost a half expects a modest increase in modeling, simulation and analytics. 10

11 Key Findings of the Survey Overall Industry Breakdown Infrastructure awareness. Fully 87.1% of respondents are somewhat aware of the of the hardware and software choices available to them, and the rationale for those infrastructure resource decisions. Looking to the future Platform effectiveness. Less than 20% of respondents believe their current IT platform will be very effective in meeting their Big Data Analytics requirements in the next months. Overall Industry Breakdown 11

12 Key Findings of the Survey Analytic outlook. Over 80% of respondents expect a modest-to-significant increase in their use of complex modeling, simulation and analytics. Overall Industry Breakdown Future analytic models. A clear majority of respondents expect the predictive model to be the most important analytic approach in the future 78.7% of power respondents and 79.7% of oil and gas respondents. Slightly smaller majorities in both sectors also expect descriptive, prescriptive, and historical analytic models to be important tools going forward. Power Oil & Gas 12

13 Key Findings of the Survey Start the conversation As confirmed in this collaborative PennEnergy and IBM Survey Report, Big Data and Analytics offer significant benefits to organizations across the oil and gas sector. Yet for energy firms to fully realize the promise of these still-emerging capabilities, a serious conversation must begin between the technology and business leaders in those organizations. Specifically, IT managers and Line of Business (LoB) executives may have broadly differing perceptions of the uses, challenges, and benefits of modern data and analytics. The following key questions, derived from previously-published IBM resources, offer guidance on how IT and business leaders can start a Big Data discussion. IT might ask the Lines of Business: :: Do you recognize the benefits of Big Data and Analytics? Use cases across exploration, production, and operations demonstrate the benefits of data and analytics. Metrics are available to demonstrate benefits in a range of oil and gas situations. :: Have you calculated ROI for data and analytics? When compared to alternative investments, Big Data and Analytic infrastructure provides very positive returns. Substantial reductions in the cost of computer processing and data storage have greatly enhanced IT-related ROI. A cost/benefit analysis can reveal precisely how data and analytics can improve exploration, production, marketing and other activities. :: What are your performance-related data issues? Have the business units considered the challenges related to data speeds and volumes, who gathers and uses information, where it is stored, and how data is analyzed to yield insights into upstream or downstream activities? 13 :: What are your quality and service level needs? Oil and gas firms rely on software applications for a growing range of mission-critical activities. By deploying a robust Big Data and Analytics architecture, energy organizations can ensure they meet Quality of Service (QoS) and Service Level Agreement (SLA) objectives.

14 Key Findings of the Survey :: Are you concerned about security? Energy firms routinely manage data with proprietary, safety and environmental, and even national security implications. Any workable data and analytics platform must incorporate enterprise-class security protections. :: Have you thought about resiliency? Mission-critical applications must be protected against failures, shutdowns, and natural or man-made disasters. :: Is growth a consideration? As energy-oriented firms confront globalized challenges, they need flexible, scalable data and analytic systems. A robust platform should address changes in application types, data speeds and volumes, and the analytic needs of an evolving oil and gas sector. Business leaders might ask IT: :: How do data and analytics drive bottom-line business results? When correctly planned and deployed, a Big Data and Analytics solution can help energy firms spot emerging opportunities in exploration, production, and distribution. Information is the key to controlling costs. Good analytics provide fast, global insights into what may or may not be financially beneficial for a firm. :: Should we be on the cloud? Cloud does offer advantages, but many energy firms can gain significant advantages through robust data and analytics whether they are cloud-based or not. Whichever delivery model an organization pursues, both business and IT should address infrastructure requirements early in the process. :: Will we face obsolescence? When implemented on a flexible, agile infrastructure, a modern Big Data and Analytics approach provides greater insights, lower costs, and the ability to more quickly and easily adjust to changing conditions. Conclusion As continued change sweeps the energy sector, oil and gas companies and power providers face a classic conundrum: are Information Technology and the Lines of Business asking the right questions and pursuing the right answers? 14 In this collaborative industry survey, IBM and PennEnergy explored perceptions of Big Data and Analytics in the energy sector.

15 Key Findings of the Survey The key take-away was this: Responders indicate that in terms of using data and analytics the sector is moving from simulation and modeling and research and exploration, and towards more use of these technologies in operational decisionmaking. This insight suggests a very real need for a high performance, more modern, IT infrastructure that is designed for big data The question then is not if Big Data, but when. As a logical starting point, this Survey indicates energy firms should engage IT early and often, and start by asking the right questions. 15

16 Appendix Included below is a comprehensive list of all the questions asked in the survey. 1. Big Data is the concept of analyzing ALL relevant data in a way to provide the most timely, complete, and confident picture of a historical, current or predicted condition, process, or situation through the use of advanced algorithms and analytics. Based on this definition, how well do you understand the use of Big Data in your company? 2. Please indicate the primary region you operate in 3. Please indicate your current role in your company Please indicate what Industry in the Energy Sector you work in Oil & Gas (All areas) Power (All areas) 5. Please indicate in which areas there are plans to leverage big data approaches and/or advanced simulation, modeling or analytics. Please select all that apply Power Generation Smart Meter Customer Operations Grid Operations None 6. Thinking about the big data areas selected, please select the top 3 methods used to rapidly process data and analytics in this / these areas. Near real-time, less than 60 minute delay Periodic process update, within several hours In real-time, as it happens, with zero latency delay End-of-day for historical reporting 7. Thinking of the big data areas selected, please select the top 5 data types that are the most critical for ensuring consistent and manageable performance Field Service data Field Operation Data Grid Operation Data Load Forecasting Data Grid Data Meter Data System Control Data Asset Performance Data Customer Data Asset Condition Data Power Data Load/Voltage Data Customer Profile Data Other Data (please specify)

17 Appendix 8. Thinking of the data types selected in the previous question, please rate, on a scale of 1 to 5, where 1 is very challenging and 5 is not challenging at all, how challenging you find the following data characteristics: Data latency (need for real time view) Data formats (many different formats) Need to retain data (how long to keep data) Data volumes (very large and growing rapidly v stable) Data accessibility (due to many varied physical data locations) 1 Very Challenging Not Challenging At All 9. Thinking of the big data areas selected, please indicate the three most important types of analytics models or simulation techniques planned to be used in the future. Predictive (what is likely to happen) Descriptive (analysis of what happened) Prescriptive (optimal recommendation of best action)historical (analysis of past performance) 10. Thinking of the big data areas selected, please self assess your firm s deployment of advanced analytics or simulation techniques. Moderately advanced (more than industry average) State of the art Somewhat behind industry average Same as general industry 11. In what area do you have the greatest Information Technology (hardware and software) gaps to address Big Data Analytics opportunities (Please select all that apply): Power Generation Grid Operations Smart Metering Customer Operations None 12. Thinking of the big data areas selected, is the need to provide data for operational planning and decision making to large number of users or processes concurrently mission critical? Yes No I don t know N/A 17

18 Appendix 13. Thinking of the big data areas selected, is the need to provide data for operational planning and decision making on a 5 9 s reliable and consistent basis mission critical? Yes No I don t know 14. Please indicate in which areas there are plans to leverage big data approaches and/or advanced simulation, modeling or analytics. Please select all that apply Upstream (E&P, Unconventionals) Midstream (Pipelines, FPSOs, Transportation) Downstream None 15. Thinking about the big data areas selected, please select the top 3 methods used to rapidly process data and analytics in this / these areas. Periodic process update, within several hours Near real-time, less than 60 minute delay In real-time, as it happens, with zero latency delay End-of-day for historical reporting 16. Thinking of the big data areas selected, please select the top 5 data types that are the most critical for ensuring consistent and manageable performance System Control Data Oil and Gas Data Drilling Data Refining Data Asset Condition Data Well Data Pipeline Data Field Operation Data Asset Performance Data Transportation Service data Other Data (please specify) 18

19 Appendix 17. Thinking of the data types selected in the previous question, please rate, on a scale of 1 to 5, where 1 is very challenging and 5 is not challenging at all, how challenging you find the following data characteristics: Data formats (many different formats) Data latency (need for real time view) Data volumes (very large and growing rapidly v stable) Data accessibility (due to many varied physical data locations) Need to retain data (how long to keep data) 1 Very Challenging Not Challenging At All N/A Thinking of the big data areas selected, please indicate the three most important types of analytics models or simulation techniques planned to be used in the future. Predictive (what is likely to happen) Descriptive (analysis of what happened) Historical (analysis of past performance) Prescriptive (optimal recommendation of best action) 19. Thinking of the big data areas selected, please self assess your firm s deployment of advanced analytics or simulation techniques. Same as general industry Moderately advanced (more than industry average) Somewhat behind industry average State of the art 20. In what area do you have the greatest Information Technology (hardware and software) gaps to address Big Data Analytics opportunities (Please select all that apply): Upstream Midstream Downstream None 21. Thinking of the big data areas selected, is the need to provide data for operational planning and decision making to large number of users or processes concurrently mission critical? Yes No I don t know

20 Appendix 22. Thinking of the big data areas selected, is the need to provide data for operational planning and decision making on a 5 9 s reliable and consistent basis mission critical? Yes No I don t know 23. When do you typically engage your IT organization in helping to define requirements? During the business/process planning/scoping phase Once there is business-line approval for budget and resources At the time detailed requirements are prepared When ready to procure necessary hardware and software 24. Generally speaking, when thinking about the application of Big Data Analytics for the Energy Industry, which of the following do you believe to be the most important requirement? Speed to gather the data Speed to analyze/simulate/model the data Speed to take the output of the analysis and take action (either people or machine process) 25. How would you assess the IT resources available to meet your Big Data Analytics Requirements? More than sufficient resources or access to resources, no concern. Good access to resources in general, but sometimes have to wait I can get most of what I need for my projects, but can be time delayed Often have to wait for resources 26. On a scale of 1 to 5, where 1 is very challenging and 5 is not challenging at all, please rate the following challenges in getting the data you need to do Big Data Analytics 1 Very Challenging Not Challenging At All 20 Takes too long to get the data No efficient manner to store the data Data in too many different places making access difficult Integrating the different data types Data is physically remote Data volumes are huge causing manageability issues Data security and governance mandates Data is in too many different formats

21 Appendix 27. Do you perform complex modeling, simulations and analytics with your data? Yes No I don t know 28. How satisfied are you with the processing time to conduct the modeling / simulation / analysis? Completely Satisfied Somewhat Satisfied Somewhat Dissatisfied Completely Dissatisfied I don t know 29. What is your outlook in doing more modeling/simulation/analytics? Significant increase Modest increase No change Modest decline Significant decline 30. In working with your Information Technology Organization, are you aware of the IT software and hardware choices and the rationale why those choices are made? Yes Somewhat No 31. How effective will your firm s Information Technology platform (hardware and software) be in meeting your Big Data Analytics requirements in months? Very Adequate Somewhat Adequate Somewhat Inadequate Very Inadequate I don t know 32. Do you apply visualization tools and techniques to aid in advanced analytics and big data analysis? Yes No 21

22 About IBM IBM offers deep experience in the oil and gas sector, including tested solutions for both the upstream and downstream segments, from exploration to distribution. As a trusted partner, the company s solutions apply instrumentation, interconnectedness, and advanced intelligence to the physical and financial challenges of the energy industry. IBM s big data infrastructure and platform provides a scalable, easy-to-use, secure information management and analytics environment for complex, large-scale information storage and analysis. Those big data solutions support enhanced exploration and production, improved refining and manufacturing efficiencies, and optimized global operations. About PennEnergy PennEnergy serves global energy professionals with the broadest, most complete coverage of industry-related information, with resources to help effectively perform crucial job functions. PennEnergy.com delivers original industry news, financial market data, in-depth research, maps, surveys, statistical data, and equipment/service information. For more information on PennEnergy s resources for energy professionals, and to subscribe to our free enewsletters, visit For more information on IBM Big Data solutions, please visit: 22