Final Report. Duke Energy Emerging Technologies Organization. Data Modeling and Analytics Initiative

Size: px
Start display at page:

Download "Final Report. Duke Energy Emerging Technologies Organization. Data Modeling and Analytics Initiative"

Transcription

1 Final Report Duke Energy Emerging Technologies Organization Data Modeling and Analytics Initiative April 2014

2

3 Duke Energy Data Modeling and Analytics Initiative Table of Contents Table of Contents List of Tables List of Figures Executive Summary Section 1 OVERVIEW OF THE DATA MODELING AND ANALYTICS INITIATIVE Initiative Overview Dataset Overview Participation Opportunities during the Initiative Information Included in Participant Responses Vendors Invited to the DMAI Vendor Responses Overview Data Quality Issues Section 2 USE CASES OVERVIEW AND VALUE ESTIMATION Vendor Responses Overview Observations and Insights Section 3 TOOLS, SYSTEMS AND ANALYTICS USED Overview Data Analytics Tools used by Vendors Analytics Used by Vendors Insights and Observations Section 4 RECOMMENDATIONS AND NEXT STEPS Next Steps List of Appendices A List of Use Cases Data Modeling and Analytics Final Report

4 Table of Contents List of Tables Table 1-1 DMAI Core Team Members Table 1-3 Data Quality Issues Table 2-1 Value Estimates for Use Case Categories Table 2-2 Use Case Value-Consolidated Responses Table 3-1 Summary List of Tools by Big Data Component List of Figures Figure 1-1. Use Cases by Category and Vendor Coverage ii Duke Energy Data Modeling and Analytics Final Report 6/16/14

5 EXECUTIVE SUMMARY Duke Energy has been involved in data mining its smart grid system and device data for the last two years. One major issue that emerged through this work was extracting data from the various siloes it resided in for analysis and new value development. Therefore Duke Energy developed a Sandbox a data model and dataset that combined a finite set of data elements from a variety of systems to analyze and identify new value opportunities from utilizing the smart grid system. To accelerate this process, Duke Energy sponsored the Data Modeling and Analytics Initiative (DMAI), an innovative forum by which big data experts were given a slice of the dataset to analyze for opportunities and insights that it could incorporate into its big data analytics strategy and activities. Seventeen vendors submitted final reports that discussed data issues, models, and tools used to analyze the data, and use cases that could be developed for new value opportunities. Responses varied considerably based on the skills and expertise of each vendor. Vendors provided over 150 unique use cases for consideration. Duke Energy asked for general financial information regarding the potential benefits of the use cases; however, these results were limited. Therefore, a follow-up exercise and interview was implemented that allowed the vendors to provide value scores to major use case categories, and provided qualitative input as to where the value may be discovered within each category. Significant insights were generated from information on how vendors constructed and applied systems and analytics to develop and model the use cases. This information can be incorporated into Duke Energy s big data and analytics strategy activities. Below are key observations and insights that came out of the DMAI: There is significant potential value that can come from implementing a big data platform across a variety of use cases areas. We encountered a variety of problems extracting data from Duke Energy s systems. Consolidation and integration of data elements are required to perform the analytics necessary to identify and realize the value identified above. Issues with data include missing data, no common information model, problems linking data sets from different systems, and challenges from extracting data. Vendors had few problems ingesting Duke Energy data. While the systems and tools for data ingestion varied among vendors, the results for ingestion across vendor platforms was generally consistent. To understand big data implications for other areas of Duke Energy, more data is needed than what was included in the initial data set. The vendors provided numerous examples of use cases that could be implemented with the inclusion of additional data. Social media, asset attributes (age, type) and event alerts were the three most common data elements most often identified by vendors. Data Modeling and Analytics Final Report

6 EXECUTIVE SUMMARY The initiative provided significant insights into the different tools and systems used to manage big data. Development of technical and functional specifications, along with development of overall solution architecture should be Duke Energy s first priorities upon finalizing its strategy. Many new models and analytics were introduced to Duke Energy by the vendors in their final reports. Furthermore, they demonstrated the use of these models using the Duke Energy dataset. Vendor responses and capabilities typically fell into three categories: (1) industry experience, (2) analytics experience, and (3) IT systems. No one vendor has all three areas completely covered. There are resource and skill gaps between what is available at Duke Energy today versus the potential capabilities identified from the vendor final reports. The most important issues to address are: availability of data, a comprehensive analytics strategy, and overcoming the silobased structure of our data and systems. The following are recommendations and next steps for Duke Energy to implement in 2014: Expand the list of collaborative vendors beyond the initial participants, as they continue to provide valuable insights and feedback with respect to their offerings and experiences. Meet with them individually to discuss the technical and analytic results of their research. Continue work on refining and forecasting the potential value of having a big data and analytics platform at Duke Energy. Prioritize the use case categories to pursue, identify the top use cases, and develop a base case forecast for realizing the value. Utilize the results of the IT systems review to start development of technical and functional specifications that will become part of Sandbox 2.0 and the big data platform. Present final results to Analytics Leadership Team and make the resources, research, and documentation available to the team for input into their 2014 activities. Prepare an big data analytics organizational structure, resource, and skills gap analysis to identify areas that Duke Energy may want to supplement with future employees or external resources. Develop job descriptions for the key positions. Address the issue that consolidation and integration of data elements are required to perform the analytics necessary to identify and realize the value identified above. Issues with data include missing data, no common information model, problems linking data sets from different systems, and challenges from extracting data. ES-2 Duke Energy Data Modeling and Analytics Final Report 6/16/14

7 Section 1 OVERVIEW OF THE DATA MODELING AND ANALYTICS INITIATIVE 1.1 Initiative Overview Duke Energy has undertaken a series of data mining and analytics projects to reduce costs and improve operational efficiencies by monitoring, collecting, and cleansing data from its smart grid test area. As a result, these inter-related projects were launched to construct a data-rich environment of sensors, implement a dedicated data warehouse (known colloquially as the Sandbox ), mine the data for new insights, create new analytical tools, and develop a comprehensive big data strategy for the corporation. The Sandbox is a stable IT environment used to collect a variety of data beyond metering-based information from legacy and new systems, as it ensures development activities do not disturb production systems. These coordinated activities capture a wide range of data from the smart grid that can be translated into a variety of new, value-creating activities that reduce costs such as operations and maintenance as well as capital expenses, improve reliability, and anticipate hardware and software requirements for future technology development. Additionally, this project mapped each of the traditional and non-traditional indicators of distribution grid health, reliability, and connectivity; which can provide insights on how to correlate, aggregate, and scale data elements more effectively across all operational systems. This process was extremely complex and required inter-departmental collaboration across a variety of traditional utility silos. The first project that utilized this data was the Duke Energy Modeling and Analytics Initiative (DMAI). After internal collaboration, we invited qualified analytics vendors to participate in the initiative, and gave them the opportunity to analyze a cross section of data and provide Duke Energy with insights and recommendations for further analysis. We believed that there was significant value in working with vendors on this initiative as it would help Duke Energy accelerate its big data analytics process and capabilities. In summary, this project would be used to identify and quantify the value to Duke of a variety of use cases at each stage of a smart grid rollout. This would also allow us to quantify the value of these projects to the company, prioritize the work, and determine the value for building out smart grid-enabled infrastructure. The DMAI was undertaken in a collaborative spirit, with several departments within Duke participating. Weekly update calls were held, and the team was involved in all aspects of the process, from selection of vendors to participate, to review and sign off of the data elements and files that would be given to the participants, to ongoing review of information coming in during the Initiative. Table 1-1 below lists the members of the core team. Data Modeling and Analytics Final Report

8 Section 1 Table 1-1 DMAI Core Team Members Department Product and Program Development Metering Services Enterprise IT Grid Modernization Customer Architecture and Data Management Emerging Technologies Market Research/Customer Analytics Information Management Environmental Services IT Architecture and Security Environmental Services Emerging Technologies Emerging Technologies Revenue Services Emerging Technologies Load Forecasting Emerging Technologies Emerging Technologies Grid Modernization Emerging Technologies Supply Chain Grid Modernization Grid Modernization Technology Planning, Projects and Reporting 1.2 Dataset Overview Each participating vendor received a one-week comprehensive dataset of anonymized Duke Energy customer, weather, and grid-related data to mine for value and to demonstrate analytical capabilities to turn the data into valuable insights. The goal of the DMAI was to use data to identify, prioritize, and utilize common use cases that determine the analytics value from smart grid technology deployment. Upon completing several information security data requests and agreements, vendors received access to a secure FTP site to download files for analysis. Below is a summary of what they received: Dataset: Information in the Sandbox consists of data from: AMI systems Transformers 1-2 Duke Energy Data Modeling and Analytics Final Report

9 OVERVIEW OF THE DATA MODELING AND ANALYTICS INITIATIVE Distribution sensors Distribution grid devices (capacitor banks, reclosers) Outage Management Systems Smart grid communications nodes Weather stations Billing systems Socioeconomic databases For the Data Mining and Analytics Initiative, Duke Energy compiled a dataset of one week s worth of data during the summer coincident peak of This was comprised of eighteen (18) different files of data in either the CSV or XLS format, which allowed the vendors to import information into their own structural format, data warehouse, or platform. We were interested in learning in general about the vendors experiences with data ingestion and model structure, as Duke Energy is investigating different ways it will manage its big data in the future. Data Element Definitions: This file contains a list of all the files that comprise the above dataset, along with a list of the data elements within each file. Moreover, the data elements included a short description or definition. Data Map: This file contains a map of the major data elements as they were set up in the Duke Energy Sandbox. This allowed vendors to understand how each of the files may be connected with each other. Vendors were able to access this information via a secure FTP site once they completed and returned the Data Security Agreement. 1.3 Participation Opportunities during the Initiative There were several opportunities for participation and feedback during the DMAI. First, vendors had the ability to submit questions, clarifications, or issues encountered using the Power Advocate tool. Duke Energy made every effort to address these as quickly as possible. All questions and answers were made available to the rest of the vendors. This was to help create a collaborative learning process between Duke Energy and the participants as it learned more about how to manage and analyze big data sets. Duke Energy also hosted two conference calls/webinars during the DMAI. These were used to answer questions and clarify information that may be distributed throughout the Initiative. Duke Energy also used this time to present information on its analytics activities, and to introduce new ideas or use cases for consideration and discussion. Duke Energy learned a significant lesson with the conference calls. The first conference call was held jointly with all the participating vendors; over 70 people were in attendance. Even though we stressed that this was to be a collaborative initiative, vendors were reluctant to discuss issues and questions in front of their potential competitors. Therefore, the second conference calls were a series of one-on-one calls with each vendor team. This represented a significant investment of time on Duke Energy s part. However, it yielded substantial results in terms of feedback and Data Modeling and Analytics Final Report Duke Energy 1-3

10 Section 1 participation from the vendors. Coincident with the calls was a request by Duke Energy for vendors to provide an interim report on the progress being made on data analytics and use case development. The interim reports gave us insights into what was being considered, allowed Duke to provide input to each vendor process, and gave us a preview of what we could expect to receive at the end of the project. All responses submitted to Duke Energy at the end of the Initiative are considered confidential information. However, Duke Energy is aggregating ideas and common themes for use in analyses, presentations, or papers that may be made public. For example, it recently presented a paper at the 2014 Distributech conference on the DMAI. 1.4 Information Included in Participant Responses Duke Energy was looking to accelerate its big data analytics activities, and it hoped the DMAI would enable this process. As a result, we invited a diverse mix of participants to help us understand the major issues associated with big data analytics. The Duke Energy dataset was the starting point for participants to learn more about what type of information may be available at Duke Energy and other utilities. Therefore, we were very interested in not only analytic solutions, but also insights vendors might have with respect to data mining and analytics. Examples of information asked for inclusion into the responses were: New use case ideas Examples of analytics to investigate further Insights into what other types of data should be collected Data warehouse, models and structures Applications for modeling Data extraction and ingestion insights Insights and lessons learned from your data mining and analytics experiences The vendors provided over 150 unique use case ideas. These are discussed in more detail in Section 3, and are included in detail in Appendix A. 1.5 Vendors Invited to the DMAI The DMAI was an invitation-only event. The names of the participants were provided by members of the core team, and represent a diverse set of experiences and skills. Although twenty-eight vendors were invited to participate, seventeen completed the DMAI by submitting final reports. 1.6 Vendor Responses Overview Figure 1-1 summarizes the number of use cases submitted by the vendors. For each category it shows the number of use cases (with duplicates omitted), along with the number of vendors that provided use cases in this category (the total vendors = 17). This figure illustrates how popular the category might be with the vendors. For examples, most of the vendors provided use cases in 1-4 Duke Energy Data Modeling and Analytics Final Report

11 OVERVIEW OF THE DATA MODELING AND ANALYTICS INITIATIVE the Meter Analytics and Customer Analysis categories. In short, Duke Energy has many vendor options to select when conducting further investigations into these categories. Conversely, although there were many use cases in the Distribution Grid Analysis category, only half the vendors provided examples of these. Selection of vendors for future discussions may therefore depend on what use case categories are given the highest priority Use Cases Vendor Coverage Figure 1-1. Use Cases by Category and Vendor Coverage 1.7 Data Quality Issues The DMAI represented an opportunity to receive external feedback with respect to the new dataset developed from the integration of data from various systems. Duke Energy wanted to use the Initiative to understand what issues may have arisen from developing models and analytics with the data. Most vendors provided insights throughout their final reports. Table 1-3 below contains examples of the major data issues identified by the vendors. Table 1-3 Data Quality Issues Lack of data history Data Quality Issue No meter event information was provided. This is critical for event processing analytics. Network model and line GPS coordinates Description More historical data would allow for better model results. At least two years of data is required. In fact, all asset event information is important, especially if there are ways to link the events across the different assets using key variables like timestamps, GPS location, etc. Provide connectivity information between devices. Data Modeling and Analytics Final Report Duke Energy 1-5

12 Section 1 Data Quality Issue Line sensor device/line information missing No Substation or Circuit SCADA Data No customer billing information No Capacitor Banks and Recloser location information Missing/Null GPS coordinates for transformers Asset attribute data was not provided Social media datasets were missing No firmographic information for non-residential customers One-minute granularity data for line sensor events not enough for fault detection Line sensor dataset did not include circuit ID and threshold value of the current data collected Transformer ratings (kva) were not included Socioeconomic dataset not complete Customer/Transformer relationship Limited outage information available for review Description Line sensor dataset did not have Device IDs or associated Circuit IDs. Therefore, you cannot relate these to other devices or events (meters, reclosers, outage events, etc.). This information could be compared with sum of meter readings for a way to calculate losses. Need rate and billing information to understand how analytics may segment or predict customer usage based on price. This is needed to be able to relate events stored in Pi Historian with communication node, line sensor and meter events. This is needed to help identify asset locations. In order to do asset failure analysis, asset information (type, age, etc.) is required. These could provide significant customer sentiment information that could be used for segmentation and targeting opportunities. Twitter feeds are the most common example cited. For non-residential customers, information on square footage, number of employees, and NAICS (North American Industrial Classification System) or SIC (Standard Industrial Classification) code would also be useful to aid development of load forecasting models as well as segments of customers. Seconds or milliseconds are required; or set up Sandbox 2.0 to export data directly from the line sensor system. As a result, they could not be used to verify/correlate outage information. This is a key piece of data - representing the size of the transformer is required for monitoring, overload and failure analysis. Additional demographic data would make analytics more insightful. Substantial number of records did not have this. No customers mapped to several transformers, yet these transformers have load on them The Outage_Info file contained a limited set of events that were confirmed in the AMI data. However, that AMI data revealed much more extensive and complete outage information when analyzed on its own 1-6 Duke Energy Data Modeling and Analytics Final Report

13 Section 2 USE CASES OVERVIEW AND VALUE ESTIMATION 2.1 Vendor Responses Overview Duke Energy received over 150 use cases as part of the DMAI. These were grouped into major categories for additional analysis and prioritization. As part of the DMAI, we asked the vendors to provide business case value estimates for the use cases they provided. Whereas the vendors spent a significant amount of effort developing and analyzing use cases, limited quantifiable financial information was provided on the benefits. As the core team reviewed the vendor final reports, this issue was raised again. Given the significant number of use cases that were provided, Duke Energy wanted some initial indication of value it could use to help it develop short-term prioritization of activities. Therefore, Duke Energy asked the participants for insights as to where it should refine its focus to extract value from the use cases through advanced analytics. Duke Energy provided a table with the different use case categories, along with some examples of the use cases that were provided in the final report. Duke Energy then asked the participants to provide an estimate of what the value may be from pursuing use cases within the category. Value could come in the form of reduced operational expenses; increased revenues from existing or new product or service ideas; or protection of existing revenues from increased reliability, customer satisfaction, theft reduction, etc. Understanding that providing a financial metric would be difficult, we asked instead for them to use one of the codes below: Value Code Value Estimate $ Thousands in value $$ Millions in value $$$ Tens of millions in value $$$$ Hundreds of millions in value $$$$$ Billions in value To discourage guessing, we stressed that we were looking for the vendor experiences and insights with providing analytics under each category; if they did not have experience in a particular category, they were to just put an N/A there. The value estimates were based on a wide range of assumptions. Some vendors developed value potential estimates. These represented the lifetime value of the use case. Lifetimes ranged from 5 to 20 years. Others developed a value per year estimate. We will be working to convert the estimates to a common denominator for use in planning and prioritization of 2014 activities. Table 2-1 contains the value estimates by Vendor. Table 2-2 consolidates all the comments received by the vendors regarding their value estimates. Data Modeling and Analytics Final Report

14 Section Observations and Insights The following insights came from review of the vendor use cases and value estimate responses: Over 150 unique use cases were provided by the participating vendors; these were grouped into 12 macro categories for value analysis and prioritization. The average score for each category ranged from 2.6 to 3.6. While this was expected, what is more interesting is that for most categories, value scores varied widely. All categories had the highest score (5 - $$$$$) by at least one vendor. Time should be spent reviewing individual vendor scores and comments, to gain a better understanding of what motivated their responses. In every category, at least one financial metric or rule of thumb was given for the basis for value estimation. These need to be reviewed and standardized before a consolidated business case is developed. Many examples of value cited come from applications that cross more than one use case category. Therefore, as estimates are refined, care should be taken to ensure we are not double-counting the value of a particular use case. Some examples of cross value include: Smart Grid Monitoring and Analysis Systems: these incorporate meter and communications infrastructure data and analytics Program Development: incorporates customer analytics, energy efficiency and demand response information, and potential benefits Load Forecasting: could be considered a building block for other categories, such as demand response, energy efficiency, or distribution grid analysis Significant potential value may come from the incorporation of social media, socioeconomic, and interval data to segment new customers and identify new revenue opportunities. Duke Energy has significant new revenue goals over the next five years; knowing its customers on a more granular and segmented level will be required to meet this goal. 2-2 Duke Energy Data Modeling and Analytics Final Report

15 Table 2-1 Value Estimates for Use Case Categories Use Case Category Vendor 1 Vendor 2 Vendor 3 Vendor 4 Vendor 5 Vendor 6 Vendor 7 Vendor 8 Vendor 9 Vendor 10 Vendor 11 Vendor 12 Vendor 13 Vendor 14 Vendor 15 Vendor 16 Vendor 17 Average Standard Deviation # Vendor Responses Meter Analytics N/A Customer Analysis N/A Distribution Grid Analytics N/A 3 4 N/A N/A N/A N/A Outage Analysis N/A N/A N/A N/A Revenue Protection N/A N/A N/A Demand Response N/A N/A Asset Management Analysis N/A N/A N/A N/A Load Forecasting N/A N/A 4 N/A N/A Distributed Generation N/A N/A N/A N/A 2 N/A N/A 3 N/A N/A Energy Efficiency N/A N/A N/A Communications N/A 3 3 N/A N/A 3 2 N/A N/A N/A N/A 2 N/A 2 N/A 3 N/A Security N/A 3 N/A N/A N/A 4 4 N/A N/A N/A N/A 3 N/A 4 N/A N/A N/A Value Estimate Value Code Thousands in value $ - 1 Millions in value $$ - 2 Tens of millions in value $$$ - 3 Hundreds of millions in value $$$$ - 4 Billions in value $$$$$ - 5

16 2-4 Duke Energy Data Modeling and Analytics Final Report Use Case Category (with select use case examples) Meter Analytics Meter event analysis Meter interval usage outlier analysis Meter operations monitoring Customer Analysis Segment customers for utility program participation using socioeconomic data Customer load pattern analysis Develop rate class profiles for demand and voltage Table 2-2 Use Case Value-Consolidated Responses Participant Responses Value Opportunities Efficiency benefits though optimizing work order deployment and identifying and analyzing problems that may suggest infrastructure operations issues Analyzing operations data, identifying outliers, and responding to operations events will increase operational efficiency Better operational response time to identify and repair the estimated 1-2% of meter failure events that occur each year 1-2% savings ($20/customer) from billing system improvements that utilize meter data Significant value overlap with this category and Communications when utilizing data via a smart grid monitoring and analysis system (SGMAS) to operation the smart grid more efficiently Profiles from AMI data will support more exact rate design calculations and will also support more detailed rates. This has significant revenue enhancement and cost reduction potential. AMI data will allow more careful targeting of program marketing, which will lead to higher program participation or lower program costs or both. Customer targeting and segmentation analytics can lead to major improvements in customer satisfaction. $2/meter/year opportunity. Huge opportunity to look at customers the way on-line advertisers do. Social media data, when combined with socioeconomic and interval usage data, will allow Duke new visibility into customer actions and sentiments. Value may vary across service territories depending on the value Duke Energy places on increasing customer satisfaction scores. Customer and segmentation analysis can lead to new program and product ideas, which are new sources of value. Section 2

17 Data Modeling and Analytics Final Report Duke Energy 2-5 Use Case Category (with select use case examples) Distribution Grid Analysis Feeder/phase load prediction, monitoring and management Identify and manage voltage variations by device along the feeder Implement condition-based vegetation management Outage Analysis Reduce SAIFI and CAIDI through faster restoration management Automate outage notification and restoration documentation process Identify and resolve momentary outage events Participant Responses Value Opportunities Customer Volt/Var applications could save up to 3% of energy savings with optimization analytics. Typical Volt/Var applications save % of energy savings, or around $100M a year for the complete Duke Energy service territory. Reduction of vegetation management costs through optimization and forecasting of operations. Annual vegetation management spending is 1% of revenues; savings could be up to 20% of this. Another vendor estimates this at $0.5/meter/year, although the value may be farther out. Understanding device activities, operations, and events along the entire feeder can provide optimization value of that feeder, from a voltage and loading perspective. Load balancing will help enhance asset lives. Value can come from using asset analysis to deploy and/or enhance condition-based maintenance (CBM) programs. Visualization applications will aid operators to make quicker and better decisions to reduce technical losses and increase operations efficiency. Reduction in capital budgeting costs is achievable through better asset management. Development of locational marginal pricing (LMP) forecasts will provide significant value if Duke Energy joins an RTO in the South. Customer attrition may be affected by outages. This could be in retail competition states like Ohio, or by the increased installation of DG and microgrids ($$$$). Using analytics to respond faster to outages will lead to reliability index (SAIDI, SAIFI, CAIDI) improvements. This is an excellent example of direct benefits from complex event processing analytics in grid operations. Visualization and event processing analytics can also be used to identify, locate, and resolve momentary outages, improving the MAIFI score. Better analytics may help utilities manage mutual assistance crews (from other utilities), and predict how many resources they may need for impending storms. Using analytics just for increasing the efficiency of automated outage notifications is not that valuable. Value is higher in states where regulators penalize utilities for excess outages. USE CASES OVERVIEW AND VALUE ESTIMATION

18 2-6 Duke Energy Data Modeling and Analytics Final Report Use Case Category (with select use case examples) Revenue Protection Implement revenue protection alerts using anomaly/outlier analysis Develop billing and rate analysis to estimate, track, and recover lost revenues due to theft Predict delinquencies using billing data and socioeconomic information Demand Response (DR) Develop demand response impact evaluation metrics Identify potential DR participants Predict system load curtailment from DR programs using real time meter data Participant Responses Value Opportunities Accepted theft rates for North American utilities is 2-4% of annual revenues. Advanced algorithms that detect anomalies can identify where this may occur, allowing for quicker resolution. Energy theft is the third largest form of theft, behind credit card and autos, at around $6 billion a year; other estimates cited are $5/meter/year. There are some reality checks associated with identifying theft that will temper the results: You may never get all the money stolen back, since there was no formal way of estimating the theft in the past You also may not have protected any future revenues from those caught stealing, as they probably reduce their overall use (one way or the other ) Revenue protection can also come from prediction and proactive resolution of impending customer delinquencies. Revenue protection also should emphasize protecting base revenues from future competitive and disruptive influences (like what is happening with SDG&E and RWE concerning renewables). These influences include: Retail competitors Distributed Generation/Renewables Customer Flight due to cost or reliability issues Unwanted reduction of use by customers fuel switching or implementing energy efficiency projects Remote disconnect switches will allow for faster responses to event alerts. This has shown significant reductions in slow/non-pay customers, increasing cash flow. Social media could result in increased customer response in DR programs, resulting in a 3-4% increase in peak reduction. Analytics can help optimally forecast and dispatch DR, maximizing the ROI of the system. Identify potential DR participants Growing a DR program with targeted participants can mean the difference between a successful program and a complete failure. Actual value will depend upon the program delivered. Analysis of interval data can help determine peak period usage by customer, allowing Duke to prioritize where it would be able to reduce load most effectively. This would also help identify new DR participants. The ability to accurately document savings from DR programs will allow for more value to be recovered in ISO DR offerings. Real future value of DR will come from decentralized, localized, and optimized activities. Analytics will help achieve this. Development of metrics does not inherently provide direct value, but can attempt to measure the value realized. Section 2

19 Data Modeling and Analytics Final Report Duke Energy 2-7 Use Case Category (with select use case examples) Asset Management Analysis Implement condition-based asset management for distribution devices Identify and replace overloaded transformers Identify and monitor abnormal device activity or loading Load Forecasting Predict customer energy usage using local weather station information Develop load forecasts at the meter, transformer, phase, circuit, and substation level Develop optimization strategies to maximize operational value of distribution assets Participant Responses Value Opportunities Value will come from using analytics to operate conditioned and predictive based maintenance programs. Reduction of catastrophic asset failures and loss of energy sales from effective asset management. Reducing costs of maintaining assets. Analytics can lead to better management of assets to increase reliability and reduce outages. More precise operation of assets can extend their lives and allow for capital expenditure deferral. More granular information will help make forecasts more accurate, and will allow for the use of different forecast horizons. Robust forecast models can improve accuracies, leading to more cost effective decisions. Accurate load modeling/forecasting short and long term allows for better resource planning. New models can be applied to the big data to help forecasting accuracy. Examples of these models include nonlinear regression, neural networks, and random forecast models. Forecasts can be key inputs into DMS and DR systems. These systems continue to evolve; benefits will depend on the capabilities of these systems to translate forecasts into actionable and automated strategies in response to changing conditions. There is value from using forecasts to predict equipment overloading or system congestion issues. Load forecasting benefits are often interrelated with other use cases, such as asset management, energy efficiency, demand response, customer analytics. These forecasts are key inputs to distribution automation and distribution management systems as well as demand response management systems. These systems are evolving and the benefits will depend on the capabilities of these systems to translate forecasts into actionable and automated strategies in response to evolving conditions. USE CASES OVERVIEW AND VALUE ESTIMATION

20 2-8 Duke Energy Data Modeling and Analytics Final Report Use Case Category (with select use case examples) Distributed Generation (DG) Forecast/estimate distribution grid impacts from customer DG operations Develop PV adoption and usage models Identify customers with DG who consistently produce more electricity than they consume Energy Efficiency (EE) Develop/implement an energy efficiency impact evaluation system Identify potential energy efficiency program participants using interval data and socioeconomic information Create new programs targeted at specific customer segments Communications Implement a smart grid communications monitoring and analysis system Implement mobile communications and fleet telematics system Participant Responses Value Opportunities DG has the potential of being a huge negative value if not managed. Therefore, the more knowledge Duke Energy has as to where these are located and how they are operated, the better. Duke also has to learn how to profit from them, since they are already here. Monitoring weather variables against various renewable generation (wind, solar) could help manage and/or predict grid impacts from DG assets. Developing various DG scenarios will help Duke Energy understand what the different operational impacts they will have on the grid. Advanced analytics may help identify where newly installed DG is located, and how it may be affecting the grid. Understanding how DG is being used, and its impacts on the grid will help Duke Energy refine its tariffs accordingly. Modeling and analyzing DG will help Duke understand its variability, and how much spinning reserves it may need to have ready. Identification of variability in operations: need for additional spinning reserves (assume 20% of photovoltaic (PV) capacity must be met by GT), then assume value is 2% of this number. Cost of developing grid distribution models to help identify and integrate intermittent PV. Customer Plug-in Electric Vehicle (PEV) usage analytics will help Duke understand how vehicle-to-grid (V2G) programs may affect the grid operations at the edge. 1.5% -2.5% of additional EE savings can come from the strategic use of social media. Using Signal Analysis to deconstruct interval data into specific end uses could be used for better energy recommendations to customers. Combining this with socioeconomic information may allow for better segmentation and target marketing. Analytics will help better run and segment customers for EE programs. The issue is how to calculate value for programs that essentially pay customers to use less energy. Analytics brings limited additional value to EE efforts if it is only used for impact evaluation activities. More effective management of communications infrastructure can occur from the use of complex event processing. Using analytics to optimize bandwidth and costs of using public communications networks (cellular). Effective use of analytics with network management systems could help resolve issues and outages more quickly, reducing costs. Analytics will help manage service level agreements, with better crew response times. Section 2

I. TODAY S UTILITY INFRASTRUCTURE vs. FUTURE USE CASES...1 II. MARKET & PLATFORM REQUIREMENTS...2

I. TODAY S UTILITY INFRASTRUCTURE vs. FUTURE USE CASES...1 II. MARKET & PLATFORM REQUIREMENTS...2 www.vitria.com TABLE OF CONTENTS I. TODAY S UTILITY INFRASTRUCTURE vs. FUTURE USE CASES...1 II. MARKET & PLATFORM REQUIREMENTS...2 III. COMPLEMENTING UTILITY IT ARCHITECTURES WITH THE VITRIA PLATFORM FOR

More information

Big Data and Advanced Analytics Technologies for the Smart Grid

Big Data and Advanced Analytics Technologies for the Smart Grid 1 Big Data and Advanced Analytics Technologies for the Smart Grid Arnie de Castro, PhD SAS Institute IEEE PES 2014 General Meeting July 27-31, 2014 Panel Session: Using Smart Grid Data to Improve Planning,

More information

Integrating a Big Data Platform into Government:

Integrating a Big Data Platform into Government: Integrating a Big Data Platform into Government: Drive Better Decisions for Policy and Program Outcomes John Haddad, Senior Director Product Marketing, Informatica Digital Government Institute s Government

More information

Big Data for Investment Research Management

Big Data for Investment Research Management IDT Partners www.idtpartners.com Big Data for Investment Research Management Discover how IDT Partners helps Financial Services, Market Research, and Investment Management firms turn big data into actionable

More information

Big Data and Data Science: Behind the Buzz Words

Big Data and Data Science: Behind the Buzz Words Big Data and Data Science: Behind the Buzz Words Peggy Brinkmann, FCAS, MAAA Actuary Milliman, Inc. April 1, 2014 Contents Big data: from hype to value Deconstructing data science Managing big data Analyzing

More information

Hadoop Beyond Hype: Complex Adaptive Systems Conference Nov 16, 2012. Viswa Sharma Solutions Architect Tata Consultancy Services

Hadoop Beyond Hype: Complex Adaptive Systems Conference Nov 16, 2012. Viswa Sharma Solutions Architect Tata Consultancy Services Hadoop Beyond Hype: Complex Adaptive Systems Conference Nov 16, 2012 Viswa Sharma Solutions Architect Tata Consultancy Services 1 Agenda What is Hadoop Why Hadoop? The Net Generation is here Sizing the

More information

ANALYTICS CENTER LEARNING PROGRAM

ANALYTICS CENTER LEARNING PROGRAM Overview of Curriculum ANALYTICS CENTER LEARNING PROGRAM The following courses are offered by Analytics Center as part of its learning program: Course Duration Prerequisites 1- Math and Theory 101 - Fundamentals

More information

Architecting for Big Data Analytics and Beyond: A New Framework for Business Intelligence and Data Warehousing

Architecting for Big Data Analytics and Beyond: A New Framework for Business Intelligence and Data Warehousing Architecting for Big Data Analytics and Beyond: A New Framework for Business Intelligence and Data Warehousing Wayne W. Eckerson Director of Research, TechTarget Founder, BI Leadership Forum Business Analytics

More information

SAP Solution Brief SAP HANA. Transform Your Future with Better Business Insight Using Predictive Analytics

SAP Solution Brief SAP HANA. Transform Your Future with Better Business Insight Using Predictive Analytics SAP Brief SAP HANA Objectives Transform Your Future with Better Business Insight Using Predictive Analytics Dealing with the new reality Dealing with the new reality Organizations like yours can identify

More information

This Symposium brought to you by www.ttcus.com

This Symposium brought to you by www.ttcus.com This Symposium brought to you by www.ttcus.com Linkedin/Group: Technology Training Corporation @Techtrain Technology Training Corporation www.ttcus.com Big Data Analytics as a Service (BDAaaS) Big Data

More information

The 4 Pillars of Technosoft s Big Data Practice

The 4 Pillars of Technosoft s Big Data Practice beyond possible Big Use End-user applications Big Analytics Visualisation tools Big Analytical tools Big management systems The 4 Pillars of Technosoft s Big Practice Overview Businesses have long managed

More information

ORACLE UTILITIES ANALYTICS

ORACLE UTILITIES ANALYTICS ORACLE UTILITIES ANALYTICS TRANSFORMING COMPLEX DATA INTO BUSINESS VALUE UTILITIES FOCUS ON ANALYTICS Aging infrastructure. Escalating customer expectations. Demand growth. The challenges are many. And

More information

Native Connectivity to Big Data Sources in MSTR 10

Native Connectivity to Big Data Sources in MSTR 10 Native Connectivity to Big Data Sources in MSTR 10 Bring All Relevant Data to Decision Makers Support for More Big Data Sources Optimized Access to Your Entire Big Data Ecosystem as If It Were a Single

More information

End to End Solution to Accelerate Data Warehouse Optimization. Franco Flore Alliance Sales Director - APJ

End to End Solution to Accelerate Data Warehouse Optimization. Franco Flore Alliance Sales Director - APJ End to End Solution to Accelerate Data Warehouse Optimization Franco Flore Alliance Sales Director - APJ Big Data Is Driving Key Business Initiatives Increase profitability, innovation, customer satisfaction,

More information

Active Smart Grid Analytics Maximizing Your Smart Grid Investment

Active Smart Grid Analytics Maximizing Your Smart Grid Investment Itron White Paper Itron Enterprise Edition Meter Data Management Active Smart Grid Analytics Maximizing Your Smart Grid Investment Sharelynn Moore Director, Product Marketing Itron Stephen Butler Managing

More information

Testing Big data is one of the biggest

Testing Big data is one of the biggest Infosys Labs Briefings VOL 11 NO 1 2013 Big Data: Testing Approach to Overcome Quality Challenges By Mahesh Gudipati, Shanthi Rao, Naju D. Mohan and Naveen Kumar Gajja Validate data quality by employing

More information

White Paper. How Streaming Data Analytics Enables Real-Time Decisions

White Paper. How Streaming Data Analytics Enables Real-Time Decisions White Paper How Streaming Data Analytics Enables Real-Time Decisions Contents Introduction... 1 What Is Streaming Analytics?... 1 How Does SAS Event Stream Processing Work?... 2 Overview...2 Event Stream

More information

Data Science & Big Data Practice

Data Science & Big Data Practice INSIGHTS ANALYTICS INNOVATIONS Data Science & Big Data Practice Manufacturing Internet of Things (IoT) Amplify Serviceability and Productivity by integrating machine /sensor data with Data Science What

More information

HDP Hadoop From concept to deployment.

HDP Hadoop From concept to deployment. HDP Hadoop From concept to deployment. Ankur Gupta Senior Solutions Engineer Rackspace: Page 41 27 th Jan 2015 Where are you in your Hadoop Journey? A. Researching our options B. Currently evaluating some

More information

Real-Time Big Data Analytics + Internet of Things (IoT) = Value Creation

Real-Time Big Data Analytics + Internet of Things (IoT) = Value Creation Real-Time Big Data Analytics + Internet of Things (IoT) = Value Creation January 2015 Market Insights Report Executive Summary According to a recent customer survey by Vitria, executives across the consumer,

More information

NTT DATA Big Data Reference Architecture Ver. 1.0

NTT DATA Big Data Reference Architecture Ver. 1.0 NTT DATA Big Data Reference Architecture Ver. 1.0 Big Data Reference Architecture is a joint work of NTT DATA and EVERIS SPAIN, S.L.U. Table of Contents Chap.1 Advance of Big Data Utilization... 2 Chap.2

More information

Amplify Serviceability and Productivity by integrating machine /sensor data with Data Science

Amplify Serviceability and Productivity by integrating machine /sensor data with Data Science Data Science & Big Data Practice INSIGHTS ANALYTICS INNOVATIONS Manufacturing IoT Amplify Serviceability and Productivity by integrating machine /sensor data with Data Science What is Internet of Things

More information

Utility Analytics, Challenges & Solutions. Session Three September 24, 2014

Utility Analytics, Challenges & Solutions. Session Three September 24, 2014 The Place Analytics Leaders Turn to for Answers Member.UtilityAnalytics.com Utility Analytics, Challenges & Solutions Session Three September 24, 2014 The Place Analytics Leaders Turn to for Answers Member.UtilityAnalytics.com

More information

VIEWPOINT. High Performance Analytics. Industry Context and Trends

VIEWPOINT. High Performance Analytics. Industry Context and Trends VIEWPOINT High Performance Analytics Industry Context and Trends In the digital age of social media and connected devices, enterprises have a plethora of data that they can mine, to discover hidden correlations

More information

Introduction to Big Data! with Apache Spark" UC#BERKELEY#

Introduction to Big Data! with Apache Spark UC#BERKELEY# Introduction to Big Data! with Apache Spark" UC#BERKELEY# So What is Data Science?" Doing Data Science" Data Preparation" Roles" This Lecture" What is Data Science?" Data Science aims to derive knowledge!

More information

Big Data Analytics. Copyright 2011 EMC Corporation. All rights reserved.

Big Data Analytics. Copyright 2011 EMC Corporation. All rights reserved. Big Data Analytics 1 Priority Discussion Topics What are the most compelling business drivers behind big data analytics? Do you have or expect to have data scientists on your staff, and what will be their

More information

Bringing the Power of SAS to Hadoop. White Paper

Bringing the Power of SAS to Hadoop. White Paper White Paper Bringing the Power of SAS to Hadoop Combine SAS World-Class Analytic Strength with Hadoop s Low-Cost, Distributed Data Storage to Uncover Hidden Opportunities Contents Introduction... 1 What

More information

How to use Big Data in Industry 4.0 implementations. LAURI ILISON, PhD Head of Big Data and Machine Learning

How to use Big Data in Industry 4.0 implementations. LAURI ILISON, PhD Head of Big Data and Machine Learning How to use Big Data in Industry 4.0 implementations LAURI ILISON, PhD Head of Big Data and Machine Learning Big Data definition? Big Data is about structured vs unstructured data Big Data is about Volume

More information

A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani

A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani Technical Architect - Big Data Syntel Agenda Welcome to the Zoo! Evolution Timeline Traditional BI/DW Architecture Where Hadoop Fits In 2 Welcome to

More information

Business Analytics and Data Visualization. Decision Support Systems Chattrakul Sombattheera

Business Analytics and Data Visualization. Decision Support Systems Chattrakul Sombattheera Business Analytics and Data Visualization Decision Support Systems Chattrakul Sombattheera Agenda Business Analytics (BA): Overview Online Analytical Processing (OLAP) Reports and Queries Multidimensionality

More information

Demand Response Management System ABB Smart Grid solution for demand response programs, distributed energy management and commercial operations

Demand Response Management System ABB Smart Grid solution for demand response programs, distributed energy management and commercial operations Demand Response Management System ABB Smart Grid solution for demand response programs, distributed energy management and commercial operations Utility Smart Grid programs seek to increase operational

More information

Big Data & QlikView. Democratizing Big Data Analytics. David Freriks Principal Solution Architect

Big Data & QlikView. Democratizing Big Data Analytics. David Freriks Principal Solution Architect Big Data & QlikView Democratizing Big Data Analytics David Freriks Principal Solution Architect TDWI Vancouver Agenda What really is Big Data? How do we separate hype from reality? How does that relate

More information

Business Intelligence. Advanced visualization. Reporting & dashboards. Mobile BI. Packaged BI

Business Intelligence. Advanced visualization. Reporting & dashboards. Mobile BI. Packaged BI Data & Analytics 1 Data & Analytics Solutions - Overview Information Management Business Intelligence Advanced Analytics Data governance Data modeling & architecture Master data management Enterprise data

More information

ANALYTICS STRATEGY: creating a roadmap for success

ANALYTICS STRATEGY: creating a roadmap for success ANALYTICS STRATEGY: creating a roadmap for success Companies in the capital and commodity markets are looking at analytics for opportunities to improve revenue and cost savings. Yet, many firms are struggling

More information

Big Data for Investment Research Management

Big Data for Investment Research Management IDT Partners www.idtpartners.com Big Data for Investment Research Management Discover how IDT Partners helps Financial Services, Market Research, and Investment firms turn big data into actionable research

More information

Big Data Services From Hitachi Data Systems

Big Data Services From Hitachi Data Systems SOLUTION PROFILE Big Data Services From Hitachi Data Systems Create Strategy, Implement and Manage a Solution for Big Data for Your Organization Big Data Consulting Services and Big Data Transition Services

More information

Mike Maxey. Senior Director Product Marketing Greenplum A Division of EMC. Copyright 2011 EMC Corporation. All rights reserved.

Mike Maxey. Senior Director Product Marketing Greenplum A Division of EMC. Copyright 2011 EMC Corporation. All rights reserved. Mike Maxey Senior Director Product Marketing Greenplum A Division of EMC 1 Greenplum Becomes the Foundation of EMC s Big Data Analytics (July 2010) E M C A C Q U I R E S G R E E N P L U M For three years,

More information

The Future of Data Management

The Future of Data Management The Future of Data Management with Hadoop and the Enterprise Data Hub Amr Awadallah (@awadallah) Cofounder and CTO Cloudera Snapshot Founded 2008, by former employees of Employees Today ~ 800 World Class

More information

Sunnie Chung. Cleveland State University

Sunnie Chung. Cleveland State University Sunnie Chung Cleveland State University Data Scientist Big Data Processing Data Mining 2 INTERSECT of Computer Scientists and Statisticians with Knowledge of Data Mining AND Big data Processing Skills:

More information

Online Content Optimization Using Hadoop. Jyoti Ahuja Dec 20 2011

Online Content Optimization Using Hadoop. Jyoti Ahuja Dec 20 2011 Online Content Optimization Using Hadoop Jyoti Ahuja Dec 20 2011 What do we do? Deliver right CONTENT to the right USER at the right TIME o Effectively and pro-actively learn from user interactions with

More information

DATA VISUALIZATION: When Data Speaks Business PRODUCT ANALYSIS REPORT IBM COGNOS BUSINESS INTELLIGENCE. Technology Evaluation Centers

DATA VISUALIZATION: When Data Speaks Business PRODUCT ANALYSIS REPORT IBM COGNOS BUSINESS INTELLIGENCE. Technology Evaluation Centers PRODUCT ANALYSIS REPORT IBM COGNOS BUSINESS INTELLIGENCE DATA VISUALIZATION: When Data Speaks Business Jorge García, TEC Senior BI and Data Management Analyst Technology Evaluation Centers Contents About

More information

BEYOND BI: Big Data Analytic Use Cases

BEYOND BI: Big Data Analytic Use Cases BEYOND BI: Big Data Analytic Use Cases Big Data Analytics Use Cases This white paper discusses the types and characteristics of big data analytics use cases, how they differ from traditional business intelligence

More information

Big Data and Your Data Warehouse Philip Russom

Big Data and Your Data Warehouse Philip Russom Big Data and Your Data Warehouse Philip Russom TDWI Research Director for Data Management April 5, 2012 Sponsor Speakers Philip Russom Research Director, Data Management, TDWI Peter Jeffcock Director,

More information

JAVASCRIPT CHARTING. Scaling for the Enterprise with Metric Insights. 2013 Copyright Metric insights, Inc.

JAVASCRIPT CHARTING. Scaling for the Enterprise with Metric Insights. 2013 Copyright Metric insights, Inc. JAVASCRIPT CHARTING Scaling for the Enterprise with Metric Insights 2013 Copyright Metric insights, Inc. A REVOLUTION IS HAPPENING... 3! Challenges... 3! Borrowing From The Enterprise BI Stack... 4! Visualization

More information

2015 Analyst and Advisor Summit. Advanced Data Analytics Dr. Rod Fontecilla Vice President, Application Services, Chief Data Scientist

2015 Analyst and Advisor Summit. Advanced Data Analytics Dr. Rod Fontecilla Vice President, Application Services, Chief Data Scientist 2015 Analyst and Advisor Summit Advanced Data Analytics Dr. Rod Fontecilla Vice President, Application Services, Chief Data Scientist Agenda Key Facts Offerings and Capabilities Case Studies When to Engage

More information

W H I T E P A P E R. Deriving Intelligence from Large Data Using Hadoop and Applying Analytics. Abstract

W H I T E P A P E R. Deriving Intelligence from Large Data Using Hadoop and Applying Analytics. Abstract W H I T E P A P E R Deriving Intelligence from Large Data Using Hadoop and Applying Analytics Abstract This white paper is focused on discussing the challenges facing large scale data processing and the

More information

SQL Server 2012 Business Intelligence Boot Camp

SQL Server 2012 Business Intelligence Boot Camp SQL Server 2012 Business Intelligence Boot Camp Length: 5 Days Technology: Microsoft SQL Server 2012 Delivery Method: Instructor-led (classroom) About this Course Data warehousing is a solution organizations

More information

White Paper. Convergence of Information and Operation Technologies (IT & OT) to Build a Successful Smart Grid

White Paper. Convergence of Information and Operation Technologies (IT & OT) to Build a Successful Smart Grid White Paper Convergence of Information and Operation Technologies (IT & OT) to Build a Successful Smart Grid Contents Executive Summary... 3 Integration of IT and OT... 4 Smarter Grid using Integrated

More information

TRENDS IN THE DEVELOPMENT OF BUSINESS INTELLIGENCE SYSTEMS

TRENDS IN THE DEVELOPMENT OF BUSINESS INTELLIGENCE SYSTEMS 9 8 TRENDS IN THE DEVELOPMENT OF BUSINESS INTELLIGENCE SYSTEMS Assist. Prof. Latinka Todoranova Econ Lit C 810 Information technology is a highly dynamic field of research. As part of it, business intelligence

More information

Converged, Real-time Analytics Enabling Faster Decision Making and New Business Opportunities

Converged, Real-time Analytics Enabling Faster Decision Making and New Business Opportunities Technology Insight Paper Converged, Real-time Analytics Enabling Faster Decision Making and New Business Opportunities By John Webster February 2015 Enabling you to make the best technology decisions Enabling

More information

EVERYTHING THAT MATTERS IN ADVANCED ANALYTICS

EVERYTHING THAT MATTERS IN ADVANCED ANALYTICS EVERYTHING THAT MATTERS IN ADVANCED ANALYTICS Marcia Kaufman, Principal Analyst, Hurwitz & Associates Dan Kirsch, Senior Analyst, Hurwitz & Associates Steve Stover, Sr. Director, Product Management, Predixion

More information

PDF PREVIEW EMERGING TECHNOLOGIES. Applying Technologies for Social Media Data Analysis

PDF PREVIEW EMERGING TECHNOLOGIES. Applying Technologies for Social Media Data Analysis VOLUME 34 BEST PRACTICES IN BUSINESS INTELLIGENCE AND DATA WAREHOUSING FROM LEADING SOLUTION PROVIDERS AND EXPERTS PDF PREVIEW IN EMERGING TECHNOLOGIES POWERFUL CASE STUDIES AND LESSONS LEARNED FOCUSING

More information

DECISYON 360 ASSET OPTIMIZATION SOLUTION FOR U.S. ELECTRICAL ENERGY SUPPLIER MAY 2015

DECISYON 360 ASSET OPTIMIZATION SOLUTION FOR U.S. ELECTRICAL ENERGY SUPPLIER MAY 2015 Unifying People, Process, Data & Things CASE STUDY DECISYON 360 ASSET OPTIMIZATION SOLUTION FOR U.S. ELECTRICAL ENERGY SUPPLIER MAY 2015 Decisyon, Inc. 2015 All Rights Reserved TABLE OF CONTENTS THE BOTTOM

More information

Advanced Big Data Analytics with R and Hadoop

Advanced Big Data Analytics with R and Hadoop REVOLUTION ANALYTICS WHITE PAPER Advanced Big Data Analytics with R and Hadoop 'Big Data' Analytics as a Competitive Advantage Big Analytics delivers competitive advantage in two ways compared to the traditional

More information

Big Data on Microsoft Platform

Big Data on Microsoft Platform Big Data on Microsoft Platform Prepared by GJ Srinivas Corporate TEG - Microsoft Page 1 Contents 1. What is Big Data?...3 2. Characteristics of Big Data...3 3. Enter Hadoop...3 4. Microsoft Big Data Solutions...4

More information

How to Enhance Traditional BI Architecture to Leverage Big Data

How to Enhance Traditional BI Architecture to Leverage Big Data B I G D ATA How to Enhance Traditional BI Architecture to Leverage Big Data Contents Executive Summary... 1 Traditional BI - DataStack 2.0 Architecture... 2 Benefits of Traditional BI - DataStack 2.0...

More information

The Rise of Industrial Big Data

The Rise of Industrial Big Data GE Intelligent Platforms The Rise of Industrial Big Data Leveraging large time-series data sets to drive innovation, competitiveness and growth capitalizing on the big data opportunity The Rise of Industrial

More information

Next-Generation Cloud Analytics with Amazon Redshift

Next-Generation Cloud Analytics with Amazon Redshift Next-Generation Cloud Analytics with Amazon Redshift What s inside Introduction Why Amazon Redshift is Great for Analytics Cloud Data Warehousing Strategies for Relational Databases Analyzing Fast, Transactional

More information

BIG DATA What it is and how to use?

BIG DATA What it is and how to use? BIG DATA What it is and how to use? Lauri Ilison, PhD Data Scientist 21.11.2014 Big Data definition? There is no clear definition for BIG DATA BIG DATA is more of a concept than precise term 1 21.11.14

More information

DATAMEER WHITE PAPER. Beyond BI. Big Data Analytic Use Cases

DATAMEER WHITE PAPER. Beyond BI. Big Data Analytic Use Cases DATAMEER WHITE PAPER Beyond BI Big Data Analytic Use Cases This white paper discusses the types and characteristics of big data analytics use cases, how they differ from traditional business intelligence

More information

The BIg Picture. Dinsdag 17 september 2013

The BIg Picture. Dinsdag 17 september 2013 The BIg Picture Dinsdag 17 september 2013 2 Agenda A short historical overview on BI Current Issues Current trends Future architecture First steps to this architecture 3 MIS/EIS Data Warehouse BI Multidimensional

More information

CA Technologies Big Data Infrastructure Management Unified Management and Visibility of Big Data

CA Technologies Big Data Infrastructure Management Unified Management and Visibility of Big Data Research Report CA Technologies Big Data Infrastructure Management Executive Summary CA Technologies recently exhibited new technology innovations, marking its entry into the Big Data marketplace with

More information

Turnkey Hardware, Software and Cash Flow / Operational Analytics Framework

Turnkey Hardware, Software and Cash Flow / Operational Analytics Framework Turnkey Hardware, Software and Cash Flow / Operational Analytics Framework With relevant, up to date cash flow and operations optimization reporting at your fingertips, you re positioned to take advantage

More information

HDP Enabling the Modern Data Architecture

HDP Enabling the Modern Data Architecture HDP Enabling the Modern Data Architecture Herb Cunitz President, Hortonworks Page 1 Hortonworks enables adoption of Apache Hadoop through HDP (Hortonworks Data Platform) Founded in 2011 Original 24 architects,

More information

Big Data Architecture & Analytics A comprehensive approach to harness big data architecture and analytics for growth

Big Data Architecture & Analytics A comprehensive approach to harness big data architecture and analytics for growth MAKING BIG DATA COME ALIVE Big Data Architecture & Analytics A comprehensive approach to harness big data architecture and analytics for growth Steve Gonzales, Principal Manager steve.gonzales@thinkbiganalytics.com

More information

Enterprise Solutions. Data Warehouse & Business Intelligence Chapter-8

Enterprise Solutions. Data Warehouse & Business Intelligence Chapter-8 Enterprise Solutions Data Warehouse & Business Intelligence Chapter-8 Learning Objectives Concepts of Data Warehouse Business Intelligence, Analytics & Big Data Tools for DWH & BI Concepts of Data Warehouse

More information

Enable your Modern Data Architecture by delivering Enterprise Apache Hadoop

Enable your Modern Data Architecture by delivering Enterprise Apache Hadoop Modern Data Architecture with Enterprise Apache Hadoop Hortonworks. We do Hadoop. Jeff Markham Technical Director, APAC jmarkham@hortonworks.com Page 1 Our Mission: Enable your Modern Data Architecture

More information

KnowledgeSTUDIO HIGH-PERFORMANCE PREDICTIVE ANALYTICS USING ADVANCED MODELING TECHNIQUES

KnowledgeSTUDIO HIGH-PERFORMANCE PREDICTIVE ANALYTICS USING ADVANCED MODELING TECHNIQUES HIGH-PERFORMANCE PREDICTIVE ANALYTICS USING ADVANCED MODELING TECHNIQUES Translating data into business value requires the right data mining and modeling techniques which uncover important patterns within

More information

Transforming insight into action with business event processing

Transforming insight into action with business event processing IBM Software WebSphere IBM Business Process Management December 2009 Transforming insight into action with business event processing Using specialized tools and frameworks to generate smart grid solutions

More information

SOLVING REAL AND BIG (DATA) PROBLEMS USING HADOOP. Eva Andreasson Cloudera

SOLVING REAL AND BIG (DATA) PROBLEMS USING HADOOP. Eva Andreasson Cloudera SOLVING REAL AND BIG (DATA) PROBLEMS USING HADOOP Eva Andreasson Cloudera Most FAQ: Super-Quick Overview! The Apache Hadoop Ecosystem a Zoo! Oozie ZooKeeper Hue Impala Solr Hive Pig Mahout HBase MapReduce

More information

Achieving Business Value through Big Data Analytics Philip Russom

Achieving Business Value through Big Data Analytics Philip Russom Achieving Business Value through Big Data Analytics Philip Russom TDWI Research Director for Data Management October 3, 2012 Sponsor 2 Speakers Philip Russom Research Director, Data Management, TDWI Brian

More information

AGENDA. What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story. Our BIG DATA Roadmap. Hadoop PDW

AGENDA. What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story. Our BIG DATA Roadmap. Hadoop PDW AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap BIG DATA? Volume 59% growth in annual WW information 1.2M Zetabytes (10 21 bytes) this

More information

III Big Data Technologies

III Big Data Technologies III Big Data Technologies Today, new technologies make it possible to realize value from Big Data. Big data technologies can replace highly customized, expensive legacy systems with a standard solution

More information

Understanding the Value of In-Memory in the IT Landscape

Understanding the Value of In-Memory in the IT Landscape February 2012 Understing the Value of In-Memory in Sponsored by QlikView Contents The Many Faces of In-Memory 1 The Meaning of In-Memory 2 The Data Analysis Value Chain Your Goals 3 Mapping Vendors to

More information

In-Memory Analytics for Big Data

In-Memory Analytics for Big Data In-Memory Analytics for Big Data Game-changing technology for faster, better insights WHITE PAPER SAS White Paper Table of Contents Introduction: A New Breed of Analytics... 1 SAS In-Memory Overview...

More information

Zero-in on business decisions through innovation solutions for smart big data management. How to turn volume, variety and velocity into value

Zero-in on business decisions through innovation solutions for smart big data management. How to turn volume, variety and velocity into value Zero-in on business decisions through innovation solutions for smart big data management How to turn volume, variety and velocity into value ON THE LOOKOUT FOR NEW SOURCES OF VALUE CREATION WHAT WILL DRIVE

More information

Big Data. Fast Forward. Putting data to productive use

Big Data. Fast Forward. Putting data to productive use Big Data Putting data to productive use Fast Forward What is big data, and why should you care? Get familiar with big data terminology, technologies, and techniques. Getting started with big data to realize

More information

Maximizing Returns through Advanced Analytics in Transportation

Maximizing Returns through Advanced Analytics in Transportation Maximizing Returns through Advanced Analytics in Transportation Table of contents Industry Challenges 1 Use Cases for High Performance Analytics 1 Fleet Optimization / Predictive Maintenance 1 Network

More information

Leveraging Information to Drive Insurer Growth

Leveraging Information to Drive Insurer Growth KNOWLEDGENT INSIGHTS volume 2 no. 5 March 27, 2012 Leveraging Information to Drive Insurer Growth In the highly competitive insurance industry, leveraging information has become a key focal point for driving

More information

BIG DATA AND THE ENTERPRISE DATA WAREHOUSE WORKSHOP

BIG DATA AND THE ENTERPRISE DATA WAREHOUSE WORKSHOP BIG DATA AND THE ENTERPRISE DATA WAREHOUSE WORKSHOP Business Analytics for All Amsterdam - 2015 Value of Big Data is Being Recognized Executives beginning to see the path from data insights to revenue

More information

The State of Real-Time Big Data Analytics & the Internet of Things (IoT) January 2015 Survey Report

The State of Real-Time Big Data Analytics & the Internet of Things (IoT) January 2015 Survey Report The State of Real-Time Big Data Analytics & the Internet of Things (IoT) January 2015 Survey Report Executive Summary Much of the value from the Internet of Things (IoT) will come from data, making Big

More information

Data Warehouse design

Data Warehouse design Data Warehouse design Design of Enterprise Systems University of Pavia 10/12/2013 2h for the first; 2h for hadoop - 1- Table of Contents Big Data Overview Big Data DW & BI Big Data Market Hadoop & Mahout

More information

Extend your analytic capabilities with SAP Predictive Analysis

Extend your analytic capabilities with SAP Predictive Analysis September 9 11, 2013 Anaheim, California Extend your analytic capabilities with SAP Predictive Analysis Charles Gadalla Learning Points Advanced analytics strategy at SAP Simplifying predictive analytics

More information

A financial software company

A financial software company A financial software company Projecting USD10 million revenue lift with the IBM Netezza data warehouse appliance Overview The need A financial software company sought to analyze customer engagements to

More information

Predictive Analytics: Too Important to Ignore The six secrets to success with predictive analytics

Predictive Analytics: Too Important to Ignore The six secrets to success with predictive analytics Predictive Analytics: Too Important to Ignore The six secrets to success with predictive analytics Webinar December 18, 2013 Sponsored by: Tony Cosentino VP & Research Director, Business Analytics Ventana

More information

Data Science and Business Analytics Certificate Data Science and Business Intelligence Certificate

Data Science and Business Analytics Certificate Data Science and Business Intelligence Certificate Data Science and Business Analytics Certificate Data Science and Business Intelligence Certificate Description The Helzberg School of Management has launched two graduate-level certificates: one in Data

More information

Establishing the Scope for The Business Case Structure to Evaluate Advanced Metering

Establishing the Scope for The Business Case Structure to Evaluate Advanced Metering Establishing the Scope for The Business Case Structure to Evaluate Advanced Metering What factors should be considered when determining whether to invest in an advanced metering system? How can a business

More information

I N D U S T R Y S P O T L I G H T. T h e Grow i n g Appeal of Ad va n c e d a n d P r e d i c ti ve Analytics f o r the Utility I n d u s t r y

I N D U S T R Y S P O T L I G H T. T h e Grow i n g Appeal of Ad va n c e d a n d P r e d i c ti ve Analytics f o r the Utility I n d u s t r y (% of respondents) I N D U S T R Y S P O T L I G H T T h e Grow i n g Appeal of Ad va n c e d a n d P r e d i c ti ve Analytics f o r the Utility I n d u s t r y October 2014 Sponsored by SAP Advanced

More information

Apigee Insights Increase marketing effectiveness and customer satisfaction with API-driven adaptive apps

Apigee Insights Increase marketing effectiveness and customer satisfaction with API-driven adaptive apps White provides GRASP-powered big data predictive analytics that increases marketing effectiveness and customer satisfaction with API-driven adaptive apps that anticipate, learn, and adapt to deliver contextual,

More information

5 Keys to Unlocking the Big Data Analytics Puzzle. Anurag Tandon Director, Product Marketing March 26, 2014

5 Keys to Unlocking the Big Data Analytics Puzzle. Anurag Tandon Director, Product Marketing March 26, 2014 5 Keys to Unlocking the Big Data Analytics Puzzle Anurag Tandon Director, Product Marketing March 26, 2014 1 A Little About Us A global footprint. A proven innovator. A leader in enterprise analytics for

More information

Global outlook on the perspectives of technologies like Power Hub

Global outlook on the perspectives of technologies like Power Hub Power Hub January 2013 Global outlook on the perspectives of technologies like Power Hub Larry Cochrane, Microsoft Utilities Industry Technology Strategist & Architect Global outlook on the perspectives

More information

Utilities the way we see it

Utilities the way we see it Utilities the way we see it Advanced Distribution Management Systems How to choose the right solution to improve your utility s safety, reliability, asset protection and quality of service Photo provided

More information

Big Data Executive Survey

Big Data Executive Survey Big Data Executive Full Questionnaire Big Date Executive Full Questionnaire Appendix B Questionnaire Welcome The survey has been designed to provide a benchmark for enterprises seeking to understand the

More information

Cisco Data Preparation

Cisco Data Preparation Data Sheet Cisco Data Preparation Unleash your business analysts to develop the insights that drive better business outcomes, sooner, from all your data. As self-service business intelligence (BI) and

More information

SAP and Hortonworks Reference Architecture

SAP and Hortonworks Reference Architecture SAP and Hortonworks Reference Architecture Hortonworks. We Do Hadoop. June Page 1 2014 Hortonworks Inc. 2011 2014. All Rights Reserved A Modern Data Architecture With SAP DATA SYSTEMS APPLICATIO NS Statistical

More information

Delivering new insights and value to consumer products companies through big data

Delivering new insights and value to consumer products companies through big data IBM Software White Paper Consumer Products Delivering new insights and value to consumer products companies through big data 2 Delivering new insights and value to consumer products companies through big

More information

Top 5 Transformative Analytics Applications in Retail

Top 5 Transformative Analytics Applications in Retail Top 5 Transformative Analytics Applications in Retail Learn how you can boost your bottom line and acquire engaged, happy customers with actionable insight from the world s most comprehensive analytics

More information

In-Database Analytics

In-Database Analytics Embedding Analytics in Decision Management Systems In-database analytics offer a powerful tool for embedding advanced analytics in a critical component of IT infrastructure. James Taylor CEO CONTENTS Introducing

More information

SCADA Systems Automate Electrical Distribution

SCADA Systems Automate Electrical Distribution SCADA Systems Automate Electrical Distribution PC-based supervisory control and data acquisition systems increase uptime, cut costs and improve utilization. A White Paper from InduSoft Fabio Terezinho

More information