Data and Interfaces for Advanced Building Operations and Maintenance - RP 1633 Final Report

Transcription

1 Data and Interfaces for Advanced Building Operations and Maintenance - RP 1633 Final Report Submitted to: ASHRAE 1791 Tullie Circle, N.E. Atlanta, GA Contributors: Nicholas Gayeski, PhD Sian Kleindienst, PhD Jaime Gagne, PhD Bradley Werntz Ryan Cruz Stephen Samouhos, PhD KGS Buildings, LLC 66 Union Square Suite 300 Somerville, MA June

2 Acknowledgements Thank you to the project monitoring subcommittee including Reinhard Seidl, Steve Taylor, Chariti Young, Jim Kelsey, and Kristin Heinemeier for their guidance, feedback, and patience throughout this research project. Thank you to all of our participants for committing time to participate in interviews, review information, and share their experiences. Thank you to the sponsoring technical committees, ASHRAE staff, and ASHRAE membership for their ongoing efforts to advance the state of the industry. 2

3 Executive Summary Analyzing and interpreting building performance data to inform operations and maintenance is critical to the realization of energy efficient, high performance buildings. With the advance of technology hardware and software for buildings, there is an increasing amount of available data to inform building operations, maintenance and management. However, facility management personnel have limited time and resources and need concise metrics, visualizations, and information in order to support their daily operations and decision-making. Recent works, such as ASHRAE s Performance Measurement Protocols in Commercial Buildings, have focused attention on the metrics relevant to tracking building performance. The research described in this report seeks to expand such investigations to consider visualization of operational metrics focused on an audience including facility managers, control technicians, heating ventilation and air conditioning (HVAC) technicians, facilities service providers, and commissioning engineers. The ultimate goal is to provide recommendations about data-driven metrics and interfaces so that they clearly quantify and communicate building operational performance for a diverse set of building stakeholders. This report provides these recommendations and summarizes the activities conducted to arrive at them. These activities included: surveying relevant metrics, visualizations and software interfaces; interviewing building operations staff and supporting personnel; and creating mock up interfaces that research participants reviewed. The body of the report goes into detail about each task and how these tasks informed the recommendations. This executive summary describes the core recommendations of the research with only a brief overview of how these recommendations were arrived at through the project tasks. Before presenting recommendations, notable resources available through this project include the following: A compendium of available metrics and interfaces examples is included in Appendix A. Use this database to review operational metric options and to see examples of visualizations from real applications. Feedback from interview participants and survey respondents are included in Appendices B, C, and E. This includes anecdotal feedback, such as anonymous comments from participants about what they want in an operational interface, and survey feedback with statistics about interviewee preferences. Mock-up visualizations of metrics are available through the mock interface site, and screen shots are available in Appendix D. For the reader interested in scanning the example interfaces reviewed in this research, we recommend jumping ahead to Section 5, Appendix A, or the website listed above. Advanced Operations and Maintenance Interface Recommendations The old adage attributed to Henry Ford, If I had asked people what they wanted, they would have said faster horses applies to this research in that building operations and maintenance personnel do not 3

4 necessarily know what to ask for to get better metrics and visualizations through which to manage, operate and maintain buildings. We have condensed the preferences expressed by interviewees and best practices found in the industry into a set of recommendations that reflect the predominant needs underlying the expressed preferences. Specifying engineers, product designers, and facilities personnel may consider these recommendations as they specify, design or adopt operations and maintenance interfaces. The feedback we collected was from a diverse set of stakeholders which was at the same time broad - in that we talked to many different people, in different roles, and in different types of facilities - and limited in that the stakeholders represent only a tiny portion of the industry. We recognize that the feedback we gathered does not constitute a statistically significant sample from which to claim, definitively, that these recommendations are precisely what every operations and maintenance stakeholder wants in an advanced interface. With such caveats in mind, our recommendations are presented below. At the most basic level, we recommend the ability to view and drill down into different scales of information because facility management and operations personnel need the ability to assess performance at multiple scales. These scales include the following: Enterprise or portfolio scale, presenting performance of multiple facilities, Building scale, presenting overall building performance information, System scale, at which systems like heating, cooling, ventilation, lighting, generation and others may be drilled into and assessed from a systems perspective, and at Equipment and Zone scale, at which specific equipment like an air handler, pump, boiler, Fan Coil Unit, VAV box, or others may be assessed, and finally Project scale, at which the performance of the building or systems related to specific projects, such as a re-commissioning project or a chiller replacement, can be assessed. Many research participants indicated a strong need to be able to view information at this scale in order to assess the effectiveness of their investments and initiatives. We recommend including certain types of information across all scales, including the following: Cost information, such as how much energy cost a building or equipment consuming. Utility information, such as how much electric, gas or water a building or equipment consuming, their carbon equivalents, progress related to utility consumptions goals. Operating characteristics, such as visualizations and graphics of how buildings, systems or equipment are performing now or over time. This can include characteristics such as runtimes, expected occupied hours, average operating temperatures, pressures, flows or other characteristics indicative of performance. Diagnostic information, such as automated fault detection and diagnostic outputs which can detect when buildings, systems or equipment have faults or opportunities for higher efficiency. This might include, for example, when mechanical or control faults such as valves leaking by, but also opportunities for more efficient operation such as installing variable speed drives, cleaning heat exchangers, programming reset schedules, or optimizing a chilled water loop. 4

5 Data visualization tools. This spans all scales and reflects an underlying need for the ability to create charts, scatter plots, and other views in multiple formats using any data from any scale. It also presumes data is gathered and stored for later use. We recommend software interfaces allow users to navigate from each of these scales into each of these types of information, with associated metrics and visualizations for each category. The specific design, user experience, or workflows within these interfaces is a product design and user experience challenge outside the scope of this research. Within each of these scales certain metrics or visualizations stand out based on the interviewees responses, and these are listed below. Enterprise scale Metrics o Daily and monthly operating costs for utilities, like energy and water o Daily, monthly and real time consumption for utilities, like energy and water o Utility peak demand use and time o Greenhouse gas emissions in carbon equivalents o Current and recent whole building operating modes for heating, cooling, or both o Diagnostic metrics including number of faults, rise or fall in fault counts, avoidable cost associated with faults and opportunities, and savings achieved o Normalization of all metrics by building area and weather conditions Visualizations o Maps allowing users to compare and select buildings for deeper investigation, with multiple layers to display the metrics listed above o Line charts to view portfolio performance over time o Bar charts to compare buildings, benchmarks, and goals o Pie charts to show building or utility contributions to overall use o Tabular views of buildings, sortable by the metrics above Building scale Metrics o All of the metrics at the enterprise scale listed above, but for each specific building o End-use breakdowns presented both by utility type, for example by electric, gas, steam, and chilled water consumption, as well as by end use type, for example by cooling, heating, ventilation, lighting, plug loads, and other uses o For demand response applications, projected future consumption and the timing of demand response events o Operating characteristics such as building expected occupancy, measured occupancy, and whole building comfort indices o Major system and equipment operating characteristics such as major equipment run time hours or overall plant performance o Major system and equipment diagnostic metrics rollups such as total avoidable cost associated with faults, impact of faults on occupant comfort, and fault severity Visualizations o All visualizations listed at the enterprise scale o Calendar plots or time series overlays to compare performance under similar conditions or day types over time 5

6 o Tabular views of operating characteristics and diagnostic information System scale Metrics o Current operating conditions for key variables, such as supply temperatures or pressures relative to setpoints for major systems, temperature differences on major hydronic loops, statistics on valve positions served by loops or damper positions served by ventilation systems o Run-time hours for major systems and equipment o Fault indicators showing system-level faults such as simultaneous heating and cooling or competing systems or suboptimal controls like lack of staging or suboptimal start/stop o Fault metrics for each system such as the number of faults, the avoidable cost associated with faults, and the impact of faults on occupant comfort conditions. Visualizations o Time series plots of conditions for each system with representations of allowable operating ranges and setpoints o Tables showing statistics about major equipment, such as run time hours, current operating o conditions, fault counts, fault impacts, and cost impact. Color coded graphics illustrating systems deviating from expected performance, in alarm, or with diagnostic faults, with multiple layers of information overlayed in a systems diagram. Layers may include, for example, deviations from setpoints, alarms, and fault severity measured by cost impact, comfort impact, or equipment maintenance priority o Drill down capabilities into textual and graphical information about a fault describing and illustrating the nature of the fault, root causes of the fault, suggested resolution, and impact of the fault on operating cost, energy consumption, occupant comfort, or equipment lifetime o o Co-presented graphs of supply side and load side conditions, such as a time series of hydronic loop temperature differences over time relative to the mean, minimum and maximum hydronic loop load side valve positions over time. Similar visualizations can be created for ventilation system dampers and air handler supply air conditions. Histograms for major point compliance deviations (e.g. number of hours deviating from setpoint by one, two, or three degrees) or damper and valve positions (e.g. number of hours during each valves or dampers were positioned at 10%, 20%, 30%, etc. open) Equipment and zone scale Metrics o Equipment and zone deviations from setpoints or thermal comfort conditions, related to for example temperature, humidity, carbon dioxide, and light levels o Fault information such as equipment operating off schedule, stuck dampers, leak-by on valves, simultaneous heating and cooling, or suboptimal equipment controls o Fault metrics such as the number of faults, the avoidable cost associated with each fault, the impact of faults on zone comfort conditions or equipment lifetime, and duration of faults Visualizations o Time series plots of conditions for each equipment with representations of allowable operating ranges and setpoints o Color coded equipment graphics illustrating equipment deviating from expected performance with multiple layers such as deviations from setpoints, alarms, component 6

7 o o o o faults, or fault severity measured by cost impact, comfort impact, or equipment maintenance priority Color-coded floorplans with multiple layers representing metrics, such as deviation from comfort or supply conditions, and faults, such as zones or components with specific faults and their fault metrics above Animations of floorplans or equipment graphics illustrating performance metrics over time Floorplans illustrating groups of zones served by common plant or ventilation systems Drill down capabilities into textual and graphical information about a fault describing and illustrating the nature of the fault, root causes of the fault, suggested resolution, and impact of the fault on cost, energy, comfort, and equipment lifetime Project scale Metrics o Expected project cost o Expected energy and cost savings and projected payback o Actual project cost o Achieved energy and cost savings and payback Visualizations o Time series, such as line or bar charts, of project-related utility consumption with an indication of project start date and completion date o Tabular views of all projects, with the ability to sort projects by the metrics listed above Here are a few considerations for consulting engineers: Many of the metrics and visualizations above presume the underlying data is available from sensors and systems in the building, that the building automation and metering systems capabilities are sufficient to collect this data, and that the data is trended somewhere in a scalable database. Many of the metrics and visualizations demonstrate the need to be able to represent data in many ways, such as time series, bar charts, or scatter plots and with the flexibility to allow users to create their own views of the data. Do not specify a fixed set of graphics or metrics, but rather the ability to represent data and metrics at different scales and for different purpose and stakeholders. This requires flexible tools and configurability of components or interfaces for different stakeholders. Anticipate the need to integrate building data with other data sets and systems by specifying integration capabilities such as webservices. Common systems with other relevant data include maintenance management systems, integrated workplace management systems, complaints software, space management software, and accounting tools. For graphical system representations, where they exist, enforce accurate representations of systems, e.g. heating plants, air handlers, in automation system graphics or other representations It is unlikely that a single software package will provide all of the recommended functionality, because the metrics and visualizations contain data and information that cut across different types of software applications and building systems. Therefore, interoperability of software packages through technologies like webservices and single-sign-on authentication becomes important to fulfill the requirements through multiple software packages. Customers with requirements for a single pane of glass type interface presenting all of the metrics and visualizations may require a higher level of integration, typically at a higher cost. 7

8 From this research it is clear that concisely presenting information for operations and maintenance personnel is critical to managing building performance, and will be accomplished as much by good design of user interfaces as by presenting specific metrics and visualizations. In summary, interfaces should present information at multiple scales, across an enterprise, for specific buildings, within building zones or for specific systems and equipment, and for facility projects with clear indicators from metrics and visualizations representing overall performance and where to drill down. When drilling down, interfaces should provide sufficient information to indicate not just current conditions, but whether those conditions are within appropriate ranges, how those conditions compare to past performance, how those conditions relate to other system components, and whether those conditions represent faulty or suboptimal performance. Lastly, interfaces should provide flexibility in viewing data in many formats, with different charting types, allowing users to switch between views, and to easily overlay data or switch to related data sets. Research Tasks The research was conducted in six major tasks. These began with a scoping and review phase, in which we conducted a review of available technologies and an initial set of scoping interviews. Based on this initial research, we developed a stakeholder interview questionnaire to focus on specific metrics and graphics and conducted a second set of interviews. Then, interactive dashboard prototypes embedded in a web-based survey were created for participants to test the interfaces and communicate interface preferences. Finally, recommendations for advanced building operations and maintenance interfaces were developed based on the results of all of these tasks. Review of Metrics and Interfaces Section three of this report includes a literature review of previous research, a compilation of existing tools, and a summary of existing types of data, metrics, and graphical methods of representation used to assess building performance. Relevant research and publications are reviewed including the Performance Measurement Protocols in Commercial Buildings, the Performance Metrics Project through the U.S. Dept. of Energy s Commercial Building Initiative, ASHRAE s Building Energy Quotient, and ASHRAE Guideline 13, Specifying Building Automation Systems. These catalogue many relevant metrics, such as basic building energy use intensity (EUI), which are widely used and a foundation for assessing building performance. A database of metrics and graphics used to evaluate building performance and aid in operational and financial decision-making is available as Appendix E. The metrics database provides an overview of the types of data, metrics, and other information that is or could be made available in building automation systems, energy dashboards, and other analytics systems. The graphics database summarizes the types of graphical representations that can be used to present these metrics and information to the user from within an interface or dashboard. The graphics database includes examples of various types of visual 8

9 representation for data that are currently found in commercial tools such as calendar plots, floor plan views, rating system visualizations, and equipment graphic overlays. Participant Interviews Section four of this report describes the results of interviews with project participants, which solicited their preferences for data, metrics and visualizations for operations and maintenance. The interview questionnaires were structured into a set of 7 focused categories spanning an enterprise portfolio, building and equipment or system level, and covering topics such as consumption, cost, emissions for various utilities, and operating characteristics and diagnostics about equipment and systems. Interviewees were presented with example metrics and visualizations across these categories and asked whether they found them useful or not. Notably, interviewees had widely varying views on the most useful metrics and visualizations, and it was clear that an inflexible, fixed set of metrics and visualizations would not serve the needs of all stakeholders. Instead, widely varying needs demand flexible interfaces, which allow for different metrics to be presented in a variety of visualizations and configurations for each stakeholder. Some participant preferences were Interviewees had widely varying views on the most useful metrics and visualizations, and it was clear that an inflexible, fixed set of metrics and visualizations would not serve the needs of all stakeholders. heavily influenced by negative past experiences, including inaccurate data, unintuitive metrics, and nontransparent dashboards. Such experiences erode trust in more complex system outputs, such as fault diagnostics and avoidable costs. Many participants, especially those with engineering knowledge, preferred simple, verifiable information such as time-series graphs of key performance data and the ability to plot data from different systems on the same charts. These desires seem to be an immediate response to current pain points with existing building automation systems that have limited trending and graphing capabilities, or lack of trust in existing diagnostic information. The types of information that participants most frequently indicated were useful included metrics related to equipment fault detection, potential for LEED or other certification, system or equipment efficiency metrics, and benchmarks comparing the building performance to an ideal or simulated model. The most commonly preferred graphic visualizations included equipment and system level graphics, floor plans, and graphs showing live or historical time series data. Although participants were provided examples of the various types of graphics, it is possible that participants chose those graphics they were already most comfortable with as the most useful. Next most frequently preferred graphics included graphs showing performance data overlaid with weather data, heat maps of performance (such as zone temperature deviations) overlaid on a floor plan, energy end use icons or graphics, and performance over time overlaid on a clock or calendar. 9

10 Less useful types of representation included performance equivalents (for example, energy use represented using numbers of light bulbs), temporal maps (heat maps of performance over time), and report cards. The least popular visualizations among those who manage and operate buildings were the gauge and the scatterplot, but for different reasons. Many operational staff felt that a gauge was flashy but without substance, and many participants did not seem comfortable with the scatterplots. Two of the most popular visualization types for both portfolio and building-level management were the benchmark (visually comparing current values with historical performance or goals) and the time series. For cross-building information, participants liked color-coded portfolio or campus maps as a way to communicate high-level information only if they allowed away to drill down to detailed information. Bar charts or time series graphs of utility consumption, comparisons to past performance, and pie charts of end use breakdown over selected periods of time were predictably highly ranked. Portfolio and financial decision-makers generally had little interest in or understanding of detailed operational information, but instead preferred common financial metrics such as spending, budgets, and project or maintenance ROI. Utility consumption presented as a time-series graph, with benchmarking against goals or historical values, was a highly ranked way of viewing building performance. Facility managers generally gave high rankings to energy consumption time series, energy breakdown pie charts and time series, and energy comparison benchmarking (% different from benchmark). Understanding energy breakdowns by end use, building, tenant, or other metric was routinely ranked high by managerial stakeholders, however many were skeptical about the cost effectiveness of using metering and submetering to produce the breakdowns or other advanced metrics. Operations and engineering personnel, such as technicians, building engineers, and commissioning agents, preferred to have detailed information on equipment operation and data. Some of these technical stakeholders complained of the lack of trending and graphing capability (or flexibility) in their current systems, and they expressed a desire to see time series of operational data and simple operating state graphics condensed into one screen. Many desired to view raw data from different BAS and metering systems in one interface and to have options to view any data using multiple visualization methods. Presenting this data and related calculations on system graphics, equipment graphics, or zone graphics was well-received. Many technical stakeholders expressed a need for the ability to drill down from high level building performance metrics into system operations and diagnostics. Most participants gave high ranking to basic operating information such as current operating conditions, recent trends in operations, equipment runtimes, and setpoint compliance. Participants did express interest in diagnostic findings, which would illustrate which equipment and systems were underperforming or had faults causing performance issues, such as a leaking air handler valve causing simultaneous heating and cooling. On the other hand, many of the same participants expressed skepticism that these diagnostics could be accurate in either accurately finding faults or the projecting the energy costs of these issues. Example Data, Metrics and Visualizations for Advanced Operations 10

11 Based on feedback from the interview participants, examples of advanced operations and maintenance interfaces were created and are available to the general public at the following location: This interface includes the most commonly identified useful metrics and visualizations from the participant interviews, and some additional ones beyond interviewee preferences. Interview participants were asked to survey the mock interface and to rank each metric and visualization on a scale of one to five, from least to most useful. Participation in this follow up survey has been very limited, with only 16.5% of participants responding to this final survey, but it is still open to participants and to the general public. The results of these surveys are presented in section five of this report. Common metrics ranked highly. These included basic information such as a simple cost table of building expenditures and building energy use intensities plotted over time and relative to other buildings in the portfolio or established benchmarks. Participants regularly expressed a preference for visualizations that clearly indicated what aspect of building operations to attend to whether in time, location, or within a system. For example this might include: a campus map showing color coded buildings based on deviations from expected performance or operations; a system graphic showing the component exhibiting a fault and the nature of the fault; a table of projects or equipment prioritized by potential for savings; or calendar plots and time series indicating the points in time when issues worth investigating occurred. Participants were also asked to provide additional feedback following the ranked survey responses. Managers expressed a consistent preference for summary information about the success of energy projects. For example, one participant said The most useful section would be tracking of energy and cost savings projects. This may reflect the role of most participants, as facility managers, and their need to communicate the effectiveness of facility investments. Many participants responded that the operations and diagnostics sections are important for day-to-day operations, and often missing from available interfaces today. For example, one participant stated that the diagnostics portion of this survey would be the most useful area to identify quickly issues in the field and get them corrected. This is lacking in the industry and is now becoming the best method for continuous commissioning, while another added that it would be Even better if this [interface] is overlaid on BAS user interface. Providing clear indications of equipment operational characteristics, and importantly equipment deviating from normal or outliers, was also important. For example, one participant noted, For zone operations, would be very useful to know which zone is the worst (especially in worst-zone control schemes. 11

12 Table of Contents Acknowledgements... 2 Executive Summary... 3 Table of Contents Tables Figures Project Objectives Project Tasks and Report Structure State of the Technology Literature Review Data, Metrics, and Information for Building Performance Visualizing Building Performance Data and Information Interfaces and Dashboards for Building Operations, Monitoring, and Controls Existing Tools Existing Metrics and Graphics Metrics Database Graphics Database Participant Interviews Scoping Interviews Interview Format and Questionnaire Profile of Buildings Visited Profile of Stakeholders Interviewed Profiles of Control Systems and Dashboards Potential Value of New Information Discussion of Participant Feedback Interface Component Interviews

13 4.2.1 Interface Component Interview metrics and visualizations Interface Component Interview results Data, Metrics and Visualizations for Operations and Maintenance Example Interfaces Participant Surveys Recommendations for Advanced Operations and Maintenance Interfaces References Appendices: A. Database of Existing Tools and Graphics B. Scoping Interviews Survey and Responses C. Interface Component Interviews Survey and Responses D. Example Interface Screenshots E. Example Interface Survey and Responses 13

14 Tables Table 1 Tools in Existing Tools Database Table 2 Data visualizations in Graphics Database Figures Figure 1 Scoping interviews Building types visited Figure 2 Scoping interviews Range of building sizes visited Figure 3 Scoping interviews - Types of stakeholders interviewed Figure 4 Financial decision-making processes Figure 5 Participant sources of information about building performance Figure 6 Frequency of participant use of control systems and dashboards Figure 7 Data and information available from participant tools Figure 8 Functionalities available in participant tools Figure 9 Tasks performed by participants using control systems and dashboards Figure 10 Participant utilization of control systems and dashboards Figure 11 Participant satisfaction with existing control system and dashboards Figure 12 Rated usefulness of new metrics and information Figure 13 Rated usefulness of new graphical information Figure 14 Sample calendar plot page from Interface Component interviews Figure 15 Participant profile for Interface Component interviews Figure 16 Percent of participant approval of specific visualizations Figure 17 Energy metrics preferences for portfolio and financial managers in Questionnaire 1 Figure 18 Benchmarking options preferences for portfolio and financial managers in Questionnaire 1 Figure 19 Visualization options preferred by managerial stakeholders for specific categories in Questionnaires 1, 3, and 4 14

15 Figure 20 Visualizations preferred by operations stakeholders for all categories in Questionnaires 5, 6, and 7 Figure 21 Early prototype example interface design Figure 22 Example interface section organization Figure 23 Typical example interface page organization and navigation Figure 24 Example interface main homepage Figure 25 Example interface Costs homepage Figure 26 Example graphics from Costs page Figure 27 Example interface Utilities homepage Figure 28 Example graphics from Utilities page Figure 29 Example interface Operations homepage Figure 30 Example graphics from the Operations page Figure 31 Example interface Diagnostics homepage Figure 32 Example graphics from the Diagnostics page Figure 33 Example content from the Data page 15

16 1. Project Objectives Analyzing and interpreting building performance data to inform operations and maintenance is critical to the proliferation, retrofit and success of higher performance buildings [1] [2]9/11/2015 1:04:00 AM. Despite the growing ease in collecting building data [3], and increasing attention to performance measurement in buildings [4], there has been little research of metrics and interfaces that best serve building operations and maintenance stakeholders. There now exists a significant amount of guidance and standards on measuring the performance of buildings, primarily for bulk energy information, but with limited depth on metrics and visualizations to inform daily aspects of building operation or the unique needs of various building types [5] [6]. Recent ASHRAE research on Performance Measurement Protocols in Commercial Buildings [4] [7] has focused attention on the metrics relevant to tracking building performance. The research described in this report seeks to expand such investigations to consider graphical visualization of operational metrics and their arrangement within interfaces. This research seeks to focus attention on operations and maintenance stakeholders, including control technicians, heating ventilation and air conditioning (HVAC) technicians, service providers, commissioning agents, and facility managers. The goal of this project was to create guidance about data-driven metrics and visualization that clearly quantify and communicate building operational performance to a diverse set of building stakeholders. The objective of the first part of this research was to obtain an understanding of the current state of the technology by evaluating building automation and control systems, energy dashboards, and other analytics systems that are available in buildings today. This study included a review of relevant research, creating a compendium of known building performance metrics, and a summary of existing commercial interfaces. In addition, interviews were conducted with over 80 stakeholders with various roles responsible for managing hundreds of buildings across the U.S. During these interviews, we reviewed the types of systems and interfaces currently available to the participants, the types of data, metrics, and graphics presented in these systems, and how (or if) this information is being used. We also assessed which performance metrics and graphical representations would be most relevant to each type of participant based on their reactions to a series of example visualizations. Based on the interview responses, we determined what types of metrics and graphical methods of presentation are most useful for building operation and financial decision-making for different types of buildings and by stakeholders with different sets of needs. During the second part of this project, we used the sets of metrics and graphical visualizations selected in the first half of the project to create example interfaces. These interfaces were customized to meet the needs of several main types of stakeholders, including those with operational, energy, and financial interests. The dashboards were made available online so they could be surveyed and ranked by a group of volunteers from the original participants. This stage of the research moved beyond the static graphical examples used in the original interviews by providing participants with an interactive environment that emulated a working building performance or operations interface. 16

17 This report concludes with recommendations for the data, metrics and visualizations for interfaces that best serve the needs of advanced operations and maintenance in buildings. These recommendations are made based on the results of the state of the technology review, the initial sets of stakeholder interviews, and responses to the example interface survey. 17

18 2. Project Tasks and Report Structure This research project was conducted in six major tasks. These began with a scoping and review phase, in which we completed a review of the state of the technology and an initial set of scoping interviews. Based on our initial results, we then revised the stakeholder interview questionnaire to focus on more specific metrics and graphics and conducted a second set of interviews using the revised protocol. We then developed a series of wireframes for example interfaces based on the results of the interviews and the state of the technology review. These later evolved to become interactive dashboard prototypes, embedded in a web-based survey. We concluded the project by recruiting participants to review these interfaces and complete the survey, and by finalizing a list of recommendations based on the results of all six tasks. During Task 1 of this research project, we began gathering information about the current state of the technology, including an assessment of the type of information, data driven metrics, and dashboard interfaces currently used in building monitoring and control systems. To obtain this information, we conducted a literature review of previous research as well as a review of existing tools, including Energy Information Systems (EIS), building automation systems (BAS), energy management and control systems (EMCS), energy monitoring dashboards, and other analytics products. We also developed a database of known building performance metrics and a library of example graphics from existing tools. Once we had established the state of the technology, we used this information to develop an initial interview questionnaire and protocol for the stakeholder interviews. During Task 1, we aimed to complete roughly one half of the proposed total set of stakeholder interviews. For the first set of interviews, we met with 39 participants who worked in or managed a combined total of 23 different buildings, located primarily in the Northeast. This first round of interviews was more general in nature than later rounds and helped establish a baseline for the types of tools and metrics that participants currently had access to. The results of Task 1 are presented in Section 3: State of the Technology, and Section 4.1: Scoping Interviews. In Task 2 of the project, we developed a more detailed questionnaire and a compendium of graphics to present specific types of metrics and example visualizations to interviewees. With the project monitoring subcommittee s guidance, these were reduced to a minimal set in order to facilitate 2 hour interviews. In Task 3, interviews with a second set of 40 stakeholders were conducted using the new questionnaire to collect preferences and ideas for example interfaces. The second set of interviews took place across the U.S. and again included stakeholders in a variety of roles in building operations and decision-making. The results of Tasks 2 and 3 are presented in Section 4.2: Interface Component Interviews. During Task 4, the results of Tasks 1 through 3 were compiled and used to inform the development of interactive example interfaces with metrics and visualizations. These example interfaces were made available to participants on an online site, in which surveys were embedded to rank and collect subjective information about user preferences. The interfaces were divided into sections on costs, utilities, operations, diagnostics, and data visualization, and subdivided into portfolio, building, plant, 18

19 ventilation, and zone scale information. The goal of this project was not to determine the optimal user experience or interface design for operations and maintenance, but rather to assess which specific metrics and visualizations were useful to operations and maintenance personnel, facilities managers, and financial stakeholders. In Task 5, these interfaces were made available to participants who were requested to complete a one hour survey to provide feedback on these interfaces. The results of Tasks 4 and 5 are presented in Section 5: Data, Metrics and Visualizations for Operations and Maintenance. The final task of this project is to report on the findings of the research. This research report summarizes the work performed and resources created. It also provides recommendations from across this work on data, metrics, and visualizations considered useful specifically from an operations and maintenance perspective. 19

20 3. State of the Technology Because energy and building performance systems and dashboards are a rapidly growing and changing aspect of the building understanding, assessing the current state of available technology was critical to this research project. It is important to note, however, that this review only represents a snapshot in time for a fast-changing technology. For this study, we focused on three main areas: metrics and information for building performance, graphical representation and visualization of this information, and the use of building automation and controls systems, energy monitoring systems, and other types of dashboards for building maintenance and operations. As many of the advancements in this area are occurring directly in the marketplace, it was necessary to gather information about the tools and dashboards that are available in buildings today as well as to examine previous research. This section includes a literature review of previous research, a compilation of existing tools (initially conducted in 2012 and updated in fall 2014), and a summary of existing types of data, metrics, and graphical methods of representation used to assess building performance. 3.1 Literature Review There have been a variety of previous studies that have examined data and interfaces for building operation, particularly in the areas of metrics for measuring building performance and dashboards for visualizing performance. This section includes a summary of this research Data, Metrics, and Information for Building Performance As buildings become more complex and technology improves, building stakeholders have access to an increasing amount of data and information directly from the building itself. Information about a building s performance may be available as data, metrics, or ratings. For this study, we consider data to be numerical, Boolean, or multi-state values that are obtained directly from a meter, sensor, or control system. Examples of data include room temperature, valve position, supply air flow, chiller power consumption, and whole building electricity consumption. Data may be available from a wide variety of meters and sensors located throughout the building, and an individual building may have thousands of available data points. Data may be accessed in numerous ways, including direct readings from meters or sensors on individual pieces of equipment, through building automation and control systems, through on-site workstations and kiosks, and through web-based and remote interfaces. We also consider information about a building useful in characterizing performance for operators, such as building floor area, mechanical system types, heating degree days or other climate data, and mechanical schedule information. Performance metrics, also called performance indicators, differ from data and information in that they are generally not directly available from a sensor or meter but are instead calculated using combinations of data and other building information. Examples of metrics include Energy Use Intensity (EUI, or energy per building area), chiller kw/ton, photovoltaic cell efficiency, and occupant complaints per day. Hitchcock [8] defines performance metrics as representing the performance objectives for a building 20

21 project, using quantitative criteria, in a dynamic, structured format. Hitchcock lists a variety of objectives that may be considered using metrics, including: energy efficiency; environmental impact; life-cycle economics; occupant health, comfort and productivity; and building functionality, adaptability, durability, and sustainability. As a part of ASHRAE Special Project 115: Performance Monitoring Protocols, MacNeill et al. [9] completed a comprehensive review of literature relevant to building performance measurements. They identified the most relevant methods for quantifying building performance in several areas, including energy performance, indoor air quality, thermal comfort, acoustics and vibration, and lighting quality. They also developed an Evaluation Matrix that categorizes over 200 documents related to building performance measurements. Although a wide range of metrics exists, it is clear from MacNeill et al s research that there is currently no consensus on which metrics or sets of metrics should be used to define building performance. However, there is an ongoing effort to develop frameworks of standardized metrics, particularly for energy-related performance. The Performance Metrics Project through the U.S. Dept. of Energy s Commercial Building Initiative, the National Renewable Energy Laboratory (NREL), and Pacific Northwest National Lab (PNNL) has defined a set of performance metrics with the goal of standardizing the measurement and characterization of building energy performance [10] [11] [12]. Such metrics are highly specific and clearly defined, as the researchers involved in this study believed that reducing the possible levels of interpretation would thereby reduce the disparity among assessment results. The metrics are also organized by tier, which correspond roughly to stakeholder interest: Tier 1 includes a smaller number of more general metrics such as Net Facility Energy Use which are of interest to building owners or rating system sponsors, while Tier 2 metrics include a larger number of more specific metrics such as DHW System Efficiency, which are of interest to stakeholders involved in daily building operations. In total, the metrics were divided into six categories (energy, water, operations/ maintenance, purchasing/waste/recycling, indoor environmental quality, transportation), and 4 levels of standard performance metrics are listed with increasing granularity. For example, the metrics for energy range from monthly total building energy use and cost (and total per square foot) at level 1 to monthly individual equipment energy per square foot and per occupant at level 4. The recommended operations and maintenance metrics revolve around total annual expenditures at level 1 and move to an accounting of work orders and individual procedural costs at level 4, and the indoor environmental quality metrics similarly revolve around space temperatures, CO 2, and occupant satisfaction reports. Through this project, a set of procedures was also defined to outline how to set up the scope of a project, how to select metrics to be measured, how to identify the data and equipment required to obtain each metric, and how to analyze the metrics over time [11]. Around the time that the DOE Performance Metrics Project results were released, ASHRAE published a book on Performance Measurement Protocols, or PMP (the end result of Special Project 115 referenced above), in an effort to standardize building performance claims and measurement practices [4]. The earlier book identifies the metrics and appropriate measurement practices for building performance for six types of building information (energy, water, thermal comfort, indoor air quality, lighting, acoustics) from basic to advanced levels. At all levels, the energy metrics recommended include energy consumption and cost by source, energy use intensity (EUI), and energy normalized by weather and/or 21

22 occupancy. Intermediate and advanced performance metrics are characterized by higher frequency and more granular data, although these recommendations are accompanied by the caveat that they might be cost prohibitive for the owner. The advanced level recommendations include self-referential energy use benchmark models, such as calibrated simulations or multi-parameter regression models, and a system-level granularity of energy consumption sub-metering at hourly or daily frequencies. A second book, published in 2012, acts as a best practices implementation guide for managing and improving the performance of buildings [7]. Although the basic level recommendations can be completed without reference to the BAS, the intermediate and advanced level recommendations require a moderate to complex BAS or EIS and a certain level of utility and other sub-metering. Several other studies have considered the use of metrics for building performance assessment. Lee and Norford considered the use of energy performance metrics to benchmark a set of 49 schools in a school district in California [13]. Hitchcock s research involved the development of a model for building performance metrics that is consistent with the Industry Foundation Classes (IFC), for use across a building s life cycle [8]. O Sullivan et al. [14] used an IFC-based model of a building at University College Cork as a case study for a building energy monitoring, analyzing and controlling (BEMAC) framework for life cycle building performance assessment, and Morrissey et al. [15] proposed a Building Information Model (BIM) to support this BEMAC framework. Neumann and Jacob defined the performance metrics that would be required for different steps or levels of continuous commissioning, including benchmarking (operational rating), certification (asset rating), optimization, standard analysis, and regular inspection [16]. Building performance rating systems provide an additional way of assessing building performance. Unlike most available data and metrics, rating systems are generally used to rate or rank performance on a whole building level. Performance can be assessed as an aggregate of multiple categories of sustainability (such as with the LEED system) or it can be considered in only one category. Energy consumption or efficiency performance systems are probably the most common types of rating system. Given the many ways in which building performance is communicated, the US Department of Energy has adopted the Building Energy Data Exchange Specification (BEDES) which helps to facilitate exchange of building characteristics and consumption through a common dictionary of terms, definitions and field formats for use by software tools and or rating systems. At present, there exist several different approaches to producing a rating or score for a building. Glazer [17] evaluated a wide variety of energy rating systems and identified three broad categories of protocols: statistical (the building is rated based on where it falls in a statistical distribution of actual buildings), points (the building is rated based on how many points it gets in a long list of criteria), and prototypical (the building is rated based on comparison with good conceptual buildings, using simulations). Similarly, Olofsson et al. [18] describe three approaches for generating ratings: the simulated data approach (SDA) which compares real energy consumption to an ideal simulated version of the same building, the aggregated statistics approach (ASA) which looks at a wide population of buildings, and the expert knowledge approach (EKA) which is based on expert surveys of welldocumented buildings. A more recent examination of rating systems focused on benchmarking, rating, 22

23 and labeling as the three different types of ratings classifications, where labeling is defined as the equivalent to assigning percentile intervals to energy classes (ratings), i.e. buildings get ranked A, B, C, etc. based on where their energy performance falls [19]. One of the most popular statistical benchmarking rating systems is the ENERGY STAR Label for Buildings [20], which allows building owners and managers to compare the energy consumption in their building to that of similar buildings across the United States on a 100 point scale. To earn the Energy Star, a building must earn an Energy Star rating of 75 or higher, which indicates that it outperforms at least 75% of similar buildings. LBNL s Cal-Arch system is a similar benchmarking system that is only applicable to buildings in California [21]. The EnergyIQ tool is an updated version of Cal-Arch which provides actionoriented benchmarking, providing guidance about the potential energy impact of a set of suggested actions (for example install EMS lighting controls ) which have been generated based on the benchmarking results [22]. Although statistical benchmarking systems may be more commonly used than prototypical or simulation-based systems, the statistical databases used for such ratings may not be available for specialized buildings types such as laboratories. Labs21 is an example of a rating system that uses a simulation-based benchmarking approach to overcome this challenge [23]. Points systems are also common among rating systems, and include high-profile programs such as LEED [24] and BREEAM [25]. In the United States, LEED is possibly the most well-known rating system, although other systems include BOMA 360 [26], Green Globes [27], and CHPS [28]. Ratings systems such as LEED generally assess building performance in multiple categories to determine overall performance, and in each category, credits or points are awarded based on fulfillment of various strategies for energy efficiency or sustainability. A rating or certification is then awarded to the building based on the number of points that the building is able to achieve. For example, the LEED system has four levels of certification (Certified, Silver, Gold, and Platinum) with Platinum requiring a building to achieve at least 80% of the possible credits. For this research project, the LEED rating system was found to be important in two ways. LEED EBOM (Existing Building Operations and Management) is of particular relevance to this study, as credits are available to a building which has a building automation system, energy meters, and/or more advanced building energy management systems. Additionally, during our review of existing tools, we found that new dashboard products are being offered which track LEED points for a building attempting to achieve or maintain a LEED certification (see section 3.2). The LEED rating system may ultimately be greatly influential to the use and development of control systems and dashboards. In addition to statistical benchmarking and points-based systems, labeling systems are gaining popularity. These types of systems tend to use simple schemes to denote performance, such as report card letter grades. For example, ASHRAE s Building Energy Quotient or beq [29] is a letter-based grading system based on the actual and/or designed building EUI vs the median EUI for similar buildings. In additional to operational ratings, labeling systems may also be used to rate building assets, i.e. the energy potential of a building, such as that which is currently being developed for the state of Massachusetts [30]. 23

24 3.1.2 Visualizing Building Performance Data and Information While building data, metrics, and ratings all provide extremely valuable information about a building s operations and performance, the way in which this information is provided to a building stakeholder may be equally important. A building with a modern control system may have hundreds or thousands of data points that are updated at frequent intervals, and it would be difficult or impossible for a building operator or manager to process that much raw data in a useful or efficient way. While building performance metrics and rating systems offer ways in which raw data can be processed into more condensed non-graphical forms, display of both raw data and metrics in graphical formats such as scatter plots and daily or weekly profiles can help a building stakeholder view and analyze large amounts of building data very efficiently [31]. Graphical display of data in plots and graphs can also be helpful for diagnosing building equipment faults [32]. One important consideration for the visualization of building information is the target audience of the tool. Marini et al. [33] conducted a study in which a dashboard was installed in a federal building. Five different user categories were considered, with different granularity of information available to the different user groups. Some of the lessons learned included: information should match the user, dashboards should transform data to information, and dashboards can help knowledge lead to action. While most control system interfaces are geared towards building operators and engineers, other types of dashboards have emerged which are aimed towards different stakeholders, such as regional managers and financial stakeholders. Additionally, the term eco-visualization has been used to describe visual displays aimed at promoting sustainable behavior in building occupants. These have been proposed as public displays of information and may exist in two forms: pragmatic, which use formal elements from scientific visualization; and artistic, which may use more ambiguous imagery [34]. An example of an artistic representation is found in [35], in which visualizations of trees are used to represent carbon emissions. In existing tools today, a wide variety of plots and graphs may be used for visualizing building data and metrics (discussed further in section 3.3) Interfaces and Dashboards for Building Operations, Monitoring, and Controls Interfaces and dashboards provide interactive settings in which data, metrics, and graphical information about a building may all be displayed to a building stakeholder. Building automation and controls systems (or similarly energy management and control systems, building automation systems, energy management systems, and other names) represent one of the more common types of systems that building operators, engineers, and managers may interact with regularly in buildings today. However, a variety of other systems, such as energy monitoring dashboards, enterprise energy management systems, energy information systems (EIS), advanced analytics or fault detection and diagnostic systems, and other types of tools have emerged in recent years. The tools that are currently available will be discussed further in section 3.2. In 2014, ASHRAE released an updated version of Guideline 13, Specifying Building Automation Systems [36]. This guideline is meant to help someone construct an effective specification for a Building Automation system, and it promotes capabilities such as open protocols, system interoperability, 24

25 custom reporting, data trending and trend visualization (both time series and scatterplot), remote or portable terminals, and applications like demand limiting, energy calculations, and anti-short cycling, as well as more traditional BAS features. In Annex D, Guideline 13 also points out the management and energy saving benefits of building performance monitoring on both the building and equipment levels, either as part of the BAS, or as a separate EIS. It identifies three levels of performance monitoring, from simple data trending to sophisticated diagnostics of equipment faults, operational issues, and power quality, and calls fault detection a natural enhancement to monitoring the performance of an HVAC system. Annex D references the recent ASHRAE Performance Measurement Protocols for Commercial Buildings: Best Practices Guide [7]. In a recent cost-benefit analysis of 26 EIS case studies (23 of which were in-depth), Granderson et al. found that 21 of 23 in-depth cases attributed significant savings to the installation of an EIS [37]. Among the factors associated with greater energy savings were pre-eis site EUI (how wasteful the building was before the EIS), length of time since EIS installation, higher-granularity instrumentation, consumption benchmarking, regular load profiling, and consumption anomaly detection. Also, on the list of operational efficiency best-practices were the use of time series visualizations to study load profiles and the use of x-y scatterplots to asses load vs outdoor temperature. Much of the past research that has been done in the area of building systems and interfaces has focused on EIS, which typically include building automation and control systems in addition to tools with related functionalities such as demand response management and enterprise energy management. Granderson et al. [38] created a framework to characterize and classify EIS tools. From an overview of existing tools, they found that visualization and analytical features are distinguished by their flexibility, and that rigorous energy analyses (baselining, forecasting, anomaly detection) are not universal. They also conducted a small number of case studies in which the use of EIS tools in real buildings was evaluated. Some of the conclusions from the case studies were that data quality has significant impact on EIS usability and that while EIS may offer a wide range of features, actual use of those features may be limited. Other case studies of EIS use in real buildings include Motegi et al. [37] and Kircher et al. [39]. In addition to EISs, energy monitoring dashboards are a growing trend. Lehrer and Vasudev [40] interviewed building managers and design professionals and found that such tools are currently being used in similar ways to BAS/EMCSs. The authors found that some of the users key needs were: Highlevel overview with drill-down capabilities, integration of energy visualization features with data analysis, and compatibility with existing BASs. We will discuss the results of our stakeholder interviews, in which both BAS/EMCS and dashboard systems were evaluated, in section Existing Tools A significant aspect of this research was to identify and compile a list of existing tools for building operations, maintenance, and decision-making. These tools included general building automation and control systems, energy or resource monitoring systems, enterprise energy management systems, and 25

26 systems with more advanced analytics, such as optimization, fault detection, or demand response functionalities. The current database contains information about 70 different tools, compiled between December 2011 and November 2014 (Table 1). These tools were identified using previous studies [34] [35] [36] [37], recommendations by the PMS and others in industry, internet searches, building visits, and stakeholder interviews. For each existing tool, the database entry includes a short summary, categorization by intended audience, categorization by content or functionality, a link to a folder of example interface graphics (if available), and a website link. The database is constructed in Microsoft Excel. The excel file must reside in the same main folder as the folder of example graphics for the links to function properly. Tools were categorized based on publicly available information, some of which consisted only of marketing material, or based on feedback gathered in stakeholder interviews. All attempts were made to correctly categorize each tool; although in some cases it was not possible to fully determine what functionalities were available based on the available information. Because we were interested in the variety of tools available to different stakeholders, it was important to try to understand the audience(s) towards which each tool was targeted. The possible categories for intended audience that we considered were: financial or enterprise manager, facilities manager, field personnel, and occupants or general public. We found that most tools were relevant for facilities managers (94%), with many tools available for financial or enterprise managers (76%) and field personnel (64%). Tools for occupants and the general public were the least common (15%). In addition to intended audience, we also attempted to categorize each existing tool by content or functionality (if such information was available). The categories we considered were: educational content or public display (such as energy monitoring kiosks), enterprise or campus level views (data or information over multiple buildings available at once), energy or utilities monitoring, ENERGY STAR or LEED information, real-time equipment data (such as that typically available in a building controls system), optimization features, equipment fault detection and diagnosis (FDD), demand response (DR), and retrofit recommendations or calculated ROI. We found that the most common feature in the tools and dashboards we considered was energy or utilities monitoring (90%). While such systems are typically found only in high performance buildings today, it remains to be seen if such tools will eventually become commonplace for building operations. Other common features offered by existing tools were real-time equipment data (57%), and enterprise or campus level information (56%). The least common features were educational/public content and retrofits or ROI (both 14%), followed by FDD and DR (both 17%). 26

27 Table 1 Tools in Existing Tools Database Vendor Agilewaves (now SeriousEnergy) AirAdvice Apogee Interactive AtSite Automated Building Systems Automated Logic BCM Controls BuildingIQ C3 Energy Resource Management Carrier Chevron Energy Solutions Cimetrics CISCO Computrols CopperTree Analytics Di Mi DEXMA EcoDomus Ecova ELUTIONS EnergyICT EnergyPrint EnerNOC EnVINTA Envizi ESI esight Ezenics Facilities Dynamics FactoryIQ Field Diagnostics FirstFuel (formerly iblogix) GridPoint HARA EEM Honeywell Product Name(s) Building Optimization System and Resource Monitor BuildingAdvice, Energy Kiosk Progress Insights InSite Energy Dashboard WebCTRL BAS and Energy Dashboards BuildingIQ C3 Enterprise Energy Management Platform Building Control Systems with ivu UtilityVision Energy Kiosks and Displays, Analytika Building Network Mediator Computrols Building Automation System (CBAS) Kaizen Di Mi Speaks DEXCell Energy Manager EcoDomus Facilities Management (FM) Building Monitoring and Alerting, Continuous Building Optimization ELUTIONS Energy Management EIServer and EIDashboard EnergyPrint DemandSMART, EfficiencySMART Insight One2Five Energy, Energy Callenger, EnterprizeEM(?) Envizi Building Performance Manager (powered by SkyFoundry) esight Energy Ezenics PACRAT EnergyPoint Synergy FirstFuel Rapid Building Assessment platform GridPoint EEM Suite: Discover, Plan, Act, Innovate Energy Management Solutions, Enterprise Buildings Integrator, Attune 27

28 IBM Iconics Intelligent Energy Solutions IFCS Corp. and NRCan Integrated Building Systems Interval Data Systems Johnson Controls (EnergyConnect) Johnson Controls Johnson Controls KGSBuildings LBNL Lucid Design NorthWrite/Energy WorkSite/Onset Novar Noveda NStar Opendiem (by Building Clouds) Panoramic Power Periscope (ActiveLogix) PNNL/Honeywell/Univ. Colorado Powerit Solutions Pulse Energy (now EnerNOC) QA Graphics Quality Attributes Software (QAS) Retroficiency SAIC Selex ES Schneider Electric SCIenergy (formerly Scientific Conservation ) Serious Energy Siemens SkyFoundry Teletrol (Phillips) Trane Tridium Verisae Vizelia (Schneider France) TRIRIGA Facility Analytix, Energy Analytix Eniscope DABO Intelligent Building Interface System (IBIS) EnergyWitness GridConnect Metasys and Sustainability Manager Panoptix Clockworks EnergyIQ Building Dashboard Network & Building Dashboard Kiosk, BuildingOS Energy WorkSite Opus Energy Management System Monitors, Facilimetrix, Portfolio Operator's Portal EnergyLink Opendiem Energy Manager Energy Management Solutions Periscope Dashboard Whole Building Diagnostician (WBD) Spara EMS, Demand Control, Demand Response, and Price Response Pulse Energy Dashboard Energy Efficiency Education Dashboard IBBuilding, IBCampus, IBEnterprise Apps Retroficiency Dashboard Enterprise Energy Dashboard (E2D) DiBoss Struxureware, Resource Advisor, Energy Operations, Vista and Continuum EnergyScape Serious Energy Manager APOGEE and TALON products, Siemens Advantage Navigator SkySpark ebuilding Light Commercial System Controls, Tracer Building Management Controls Vykon Energy Suite (VESAX) vxconserve, vxmaintain Vizelia Energy Module 28

29 Wegowise Wegowise It is important to note that, at the time of writing the initial list, the industry was changing rapidly, and this list of tools grew and changed as this final version was actively updated. During 2011 to 2014 while this project was underway, several new systems were introduced into the market and a few companies merged their products. More new tools emerged as interest and demand in energy management tools with dashboards and interfaces for different stakeholders grew. This list has been updated and included in this final version of the report. Even so, this updated list of tools serves to illustrate the wide variety of products that are currently available to buildings today and the general trend towards energy and performance monitoring that has emerged over the past decade. 3.3 Existing Metrics and Graphics In addition to identifying existing tools, we developed databases of metrics and graphics used to evaluate building performance and aid in operational and financial decision-making. The metrics database attempts to provide a comprehensive overview of the types of data, metrics, and other information that is or could be made available in building automation systems, energy dashboards, and other analytics systems. The graphics database summarizes the types of graphical representations that can be used to present these metrics and information to the user from within an interface or dashboard Metrics Database The existing metrics database includes ten different categories of performance data, metrics, and information. These categories include General weather or temperature Whole facility (including utilities) Renewable energy systems Energy end use or system Cooling system components and equipment Heating system components and equipment Ventilation system components and equipment Lights and plug load components and equipment Benchmarking and standards Facilities and maintenance Within each category, different types of metrics were identified based on previous research in building performance metrics [8] [10] as well as the information available about existing tools. Each metric identified was categorized based on type (for ex., raw data, normalized, or calculated metric), relevant measurement interval(s), relevant unit(s), possible normalizations, and context (site, source, or cost). For each metric, example units were also given. For example, Energy Intensity (total building energy consumption) can exist as raw data or as a normalized metric, can be collected at intervals such as daily, weekly, monthly, or annually, can be presented in units of energy or cost, can be 29

30 normalized by building area, by time, or by number of occupants, and can be reported for site or source. Example units are kbtu/ft2-yr, kwh/ft2-yr, $/ft2-yr Graphics Database The graphics database includes examples of various types of visual data representations that are currently found in EMCS and databases. A list of graphics types and descriptions is shown in Table 2. This list of existing graphics was formed based on the literature review [31] [32] [35] [40], knowledge of existing tools, and stakeholder interviews. The graphics database includes links to example images of each graphics type in addition to the information shown in Table 3. The database is available in Microsoft Excel and must reside in the same main folder as the folder of example graphics for the links to function properly. The example images were culled from screenshots, websites, and marketing materials for existing tools and metrics. We used a small subset of these images to demonstrate different types of graphical representation during our stakeholder interviews. Table 2 Data visualizations in Graphics Database TYPE DESCRIPTION Benchmarking Calendar or Clock Checklist Educational Graphic displaying performance of a building (or system) when compared to similar buildings, ideal scenarios, or previous performance. EnergyStar is a common example of benchmarking in energy systems and dashboards. Displays performance for a week or month (raw data or calculated metrics) overlaid on a calendar or clock graphic. May be combined with color codes for "good" or "bad" performance. For LEED or other program. Displays a checklist of possible and achieved points for a given building. Educational content about the building. Enterprise/Campus Equipment Graphic Equivalents Floorplan (and heat maps) Temporal map Representation of energy consumption or other metric across multiple buildings in an enterprise or campus. May appear overlaid on a map or as a bar graph. May be used for energy competitions. Graphic displaying a piece of equipment with values of sensors or meters associated with that equipment. Common in general control systems. Representation of energy or water consumption, or of GHG emissions or waste using equivalents. For example, heating energy use may be represented as "number of houses heated in a year" or emissions may be represented as "number of cars on the road" Displays performance (for example, room temperature) overlaid on a floorplan view of each floor of a building. Often combined with stoplight color representation of performance. Displays performance (for example, building energy consumption) on a 2d image on which time of day is the Y axis and date is the X axis. Color is used to denote magnitude of performance, and 30

31 stoplight color representation is often used. Icons of consumption/waste type Odometers Report card or Grade Stoplight Colors System Graphic Trending Weather Overlay Icon or graphic representation of types of energy or water consumption, or of GHG emissions or waste. For example, energy end use graphics may display icons representative of lights, plug loads, servers, and HVAC. Displays performance (for example, chiller efficiency or whole building energy use) on an odometer or dial representation, often combined with stoplight color representations. Performance represented as a number or letter "grade", typically where 100% or A denotes excellent performance. Can appear as a single grade or a report card of multiple grades (for example, by system or by equipment). Representation of performance using colors, typically "stoplight" colors: Red = alarm or poor performance, yellow = normal or average performance, green = good performance or no alarms. Can be combined with other graphic representations (e.g. Floorplan or calendar). Graphic displaying a schematic of a system (for example, cooling plant) with all relevant equipment shown along with values or statuses of associated sensors or meters. Common in general control systems. Line, bar, or other chart displaying performance trends. May be real-time or historical data. Graphic displaying performance (for example, building energy use) with weather information overlayed, either in numerical or graphical (icon) form. 31

32 4. Participant Interviews 4.1 Scoping Interviews The first set of interviews conducted for this project were scoping interviews, intended to complement the state of the technology review by helping to define what types of tools, metrics, and graphic visualizations are currently used by various building stakeholders today. We solicited participation in the research project from a range of facility owners spanning commercial office, healthcare, education, government or national laboratories, and large multifamily buildings. Within these sectors, we targeted facilities spanning a range of climates, across the western, mid-western, southern, southeastern, and northeastern United States. Facility owners opting to participate in the project worked with us to identify key operational stakeholders within their organizations to participate in the interview process. During Task 1 of the project, we visited 23 buildings to conduct the scoping interviews. Based on recommendations from the PMS, we attempted to interview several different people in each individual building if possible. In total, we met with 39 individuals and were able to obtain 36 separate responses Interview Format and Questionnaire During each interview, we visited the participating building or campus to meet with one or more individuals. The interview consisted of two parts: first, we asked interviewees to provide general information about the building, to describe the mechanical systems if possible, and to describe the typical daily or weekly tasks that the interviewee was responsible for; next, we asked the interviewee to fill out the interview questionnaire. While the participant filled out the questionnaire, we also asked him or her to provide us with any comments or anecdotes that seemed relevant to the interview questions. We recorded these comments and anecdotes separately. The interview questionnaire was designed to determine the following information: what system(s) are available in the building, what type of information is currently available to the participant from the system(s), how the participant currently uses this information, and what types of information (in terms of raw data, metrics, and graphical display methods) would be most useful to the participant. We also tried to determine if the participant was responsible for making or informing major financial decisions and what information he or she uses to make those decisions or recommendations. During the interviews, participants were also allowed to browse a booklet of sample graphics from existing tools that served as examples of the various graphical display methods referred to in the questionnaire. The full list of interview questions and example graphics are available in Appendix A. This section presents data from the interview questionnaires. Please note that not all participants responded to every question; therefore, some sets of responses do not sum to the total number of participants interviewed. A few people chose to respond as a pair or group, therefore the total number of individual responses (36) did not match the total number of participants (39). 32

33 4.1.2 Profile of Buildings Visited Twenty three buildings were visited during Task 1. During the initial phase of the project, most of the buildings visited were located in or close to Boston, MA. As Task 1 progressed, we were also able to visit buildings in CT, NY, NJ, and NC. In total, we were able to visit 21 buildings in the Northeast and 2 buildings in the South. In Task 3, we concentrated on visiting buildings in other parts of the United States. Task 1 Interviews - Building Types Office Restaurant Retail Bank K-12 Higher Education Laboratory Health Care Facility Residential Performance/Auditorium Fitness and Recreation Other Number of Buildings Visited Figure 1 Scoping interviews Building types visited 33

34 Task 1 Interviews - Building Size < 100,000 sf 100,000 sf to 250,000 sf 250,000 sf to 500,000 sf 500,000 sf to 750,000 sf 750,000 sf to 1,000,000 sf > 1,000,000 sf Number of Buildings Visited Figure 2 Scoping interviews Range of building sizes visited Eight different building types were visited during Task 1: office, restaurant, retail, bank, laboratory, highrise residential, performance/auditorium, and fitness. The number of each type of building visited is shown in Figure 1 (Note that more than one building type was selected for mixed-use buildings, for example residential and retail). The most common building type visited during Task 1 was office (16 of the 23 buildings). During this task, the building type Other referred to data centers and daycare centers that were located within a small number of buildings. In addition to representing a variety of building types, the buildings visited in Task 1 included buildings of many different sizes, including several buildings over a million square feet in area. The distribution of square footage of the Task 1 buildings is shown in Figure 2. Several of the participants in Task 1 mentioned that sustainability and/or energy efficiency were primary goals for their buildings. Of the 23 buildings visited, five had a LEED certification (one Silver, three Gold, and one Platinum). An additional three buildings were currently working towards LEED certification. Nine buildings tracked their ENERGY STAR Target Finder score Profile of Stakeholders Interviewed There are many potential stakeholders in a building who may benefit from building automation systems, energy monitoring dashboards, or other analytics systems. The Commercial Buildings Initiative has defined different tiers of building performance information based roughly on the types of people who might use that information: Operators, energy professionals, and researchers may use metrics that are one step up from direct building data, while designers, ratings systems sponsors, energy suppliers, and owners may require more general information about building performance [41]. 34

35 In all parts of this research project, we attempted to interview a variety of stakeholders who are interested in different levels and types of information and use this information in different ways. We also attempted to interview people who were involved in more general building oversight in addition to those who were responsible for daily operations and maintenance. Where possible, multiple stakeholders were interviewed for the same building, campus, or organization. Task 1 Interviews - Stakeholder Types Building owner Facilities manager/lead engineer Facilities technician/engineer Building/property manager HVAC contractor Other Number of Stekeholders Interviewed Figure 3 Scoping interviews - Types of stakeholders interviewed During the scoping interviews, the majority of the participants identified themselves as a facilities manager or lead engineer, a facilities technician or engineer, or a building or property manager. In general, the facilities managers, lead engineers, engineers, and technicians were those who used the building automation and control system on a daily basis. The property managers generally did not have access to building automation and control systems, but instead were responsible for monitoring energy consumption and were more likely to use energy monitoring systems and dashboards. In addition to those stakeholders, we were also able to interview several people who identified themselves as building owners or representatives of building owners and one HVAC contractor. The total distribution of stakeholder job functions is shown in Figure 3 (note that some interviewees identified themselves as having two job functions). The response Other refers to a stakeholder who identified himself as a VP of Real Estate. Although several different types of stakeholders were interviewed, the majority of the participants were responsible for either recommending or directly making financial decisions. 91% of the participants responded that they recommended or made decisions about replacing equipment, and 78% of participants responded that they recommended or made decisions about capital projects for energy efficiency. Of those participants who were responsible for such decisions, all responded that experience and intuition were used to aid in making such decisions, and a large number of them also followed 35

36 recommendations from external contractors and engineers. Other sources of information used to make financial decisions included O&M manuals, control systems, dashboards, or other sources such as the online web searches and advice from colleagues (Figure 4). Figure 4 Financial decision-making processes One major goal of the scoping interviews was to determine what types of data, metrics, and information were available to different stakeholders in their existing setup. Figure 5 indicates the variety of information sources that the participants have access to in addition to control systems and dashboards, including utility bills, benchmarking tools (such as EnergyStar), reports from others, and feedback from occupants. Those who responded Other referred primarily to non-written communication with others, either informally or in meetings, or additional computerized programs, such as work-order and/or occupant complaint systems. Although this study did not focus on these additional sources of information, they are important because we found that they often provide information that is identical or analogous to the data or information that a control system or dashboard might provide (for example, a building manager may have access to utility bills instead of or in addition to an energy monitoring dashboard). 36

37 Building Information Available Utility bills and consumption information Automation and control system Energy visualization or dashboard Benchmarking tools Administrative reports from others Occupant feedback Other Number of Responses Figure 5 Participant sources of information about building performance Profiles of Control Systems and Dashboards Although all participating buildings in Task 1 had a building automation or energy management and control system, some were relatively simple while others were highly customized with many features and functionalities. Within the 23 buildings, we found 7 different control systems and 6 different dashboards. While the control systems mainly included well-known systems developed by large national or international controls companies, the dashboards we found were primarily either custommade for the building or developed by smaller regional groups such as utilities companies. The control systems found included systems by Siemens, Johnson Controls, Schneider Electric, Honeywell, Automated Logic, Trane, and Teletrol. The dashboards encountered included NSTAR EnergyLink, ies Energy Desk, Cimetrics Infometrics, GE Energy Aggregator, Duke Energy Energy Profiler Online (EPO), and Ecova Performance IQ. In general, almost all of the dashboards that we encountered fell into the category of energy or utility monitoring systems. 37

38 Number of Responses Control Systems and Dashboards - Frequency of Use Once per day or more At least once per week At least once per month Once per month or less Do not use this system Control System Dashboard Do not have this system Figure 6 Frequency of participant use of control systems and dashboards Because multiple types of stakeholders were interviewed, the participants did not all use their systems with the same frequency. Almost all of the participants who had access to building automation and control systems reported that they used these systems at least once per day (Figure 6). Many of the building engineers and operators mentioned that checking the control system was the first thing they did every day. We found that the frequency of dashboard use was more varied. 70% of the respondents who used the dashboard at least once per day were engineers or technicians who also used the control system at least once per day. However, 20% of the respondents who used the dashboard at least once per day were property managers who did not use the control system at all. Many of the dashboards we encountered were either newer systems that the participants were not comfortable with or older systems that were in the process of being phased out. One of the main goals of this study was to understand how different types of stakeholders are currently using control systems and dashboards in buildings. To determine this, we asked the participants to answer a set of questions about their control systems and dashboards, including what information is available through these systems, what functionalities are available through the systems, what tasks are performed using the systems, and how the participants use the systems to perform these tasks. The results of these questions are shown in Figures 7 to 10. Figure 7 indicates what information is available from the participants control system interfaces and dashboards. We asked the participants to try to respond as accurately as possible, even if the participant did not necessarily use all the information available. Only participants that actually used each system responded to this question. We found that, as expected, most participants responded that their control systems included equipment sensor data and equipment graphics, historical data for at least a few days, system level graphics (i.e. cooling plant), and floor plan graphics. About a third of the 38

39 respondents reported that their control systems included meter or sub-meter data, graphical metrics or visualizations (such as general line or bar graphs), or fault detection advice. We note that almost all participants referred to general alarms as fault detection in this category, not advanced fault detection and diagnosis (FDD). For dashboard systems, the most common information available was found to be historical data, utility bills, main meter data, graphs, and simplified building performance ratings (Energy Star was cited often). In general, we found that the controls systems mainly offered raw data from sensors or meters and trending for that data, while the dashboards offered a wider variety of information, including performance metrics and ratings, financial information and bills, and more advanced analytics in some cases. Equipment graphics Floor plan graphics Occupant complaints or comments Historical data Sub-meter data Meter data Benchmark or comparative information Information Available from Systems Financial information Simplified building performance ratings Control System Dashboard Number of Responses Figure 7 Data and information available from participant tools The responses to a related question, what functionalities are available in each system?, are shown in Figure 8. We found that most of the control systems and dashboards generally offered basic data collection and storage functionalities and trending options for raw data, and many of them offered remote capabilities (such as ing alerts or web-based access) as well. The dashboards tended to offer additional functionalities over the control systems, such as basic energy analysis (averages, highs/lows), energy use broken down by sub-meters where available, and financial information (such as energy costs). While none of the systems offered occupant comfort reporting, we note that many of the participants had access to additional work-order and occupant complaint systems that did offer this functionality; however, none of the work-order systems were integrated with the dashboards or control systems. Educational content was not available in any system that we encountered. 39

40 Functionalities Available In Systems Basic data collection, transmission, storage, and Display and visualization (raw data trends) Energy analysis (avgs, highs/lows, normalization, Energy end use information (sub-meter data) Advanced analysis (forecasting, FDD, statistics, Financial analysis (energy costs, savings Demand response Remote functionalities (mobile or web-based) Occupant comfort reporting Educational content for occupants or visitors Do not have/use this system Control System Dashboard Number of Responses Figure 8 Functionalities available in participant tools Tasks Enabled by Systems Alarm notification Troubleshoot alarms or equipment faults Keep track of scheduled maintenance tasks Determine or benchmark overall building Compare buildings across a portfolio or campus Feasibility studies for capital projects Demand response Optimize system settings Create reports Get occupant feedback Educate or engage occupants or visitors Control System Number of Responses Figure 9 Tasks performed by participants using control systems and dashboards In addition to determining what information and functionalities were available to the participants, we were also interested in determining what tasks the participants use their controls systems and dashboards to perform and how the systems aid them in doing so. Figure 9 indicates that the control systems are used primarily for alarm notification and to troubleshoot alarms (i.e. the participants use the system to view data related to the alarm that might help them diagnose the problem). Participants also responded that they used control systems to optimize system settings (such as scheduling and setpoints) and create reports. The dashboards were used primarily for determining building 40

41 performance and for creating reports. They were also used in some cases to compare buildings over a campus or portfolio as well as to optimize system settings (primarily to meet energy targets). The ways in which the participants used the systems are shown in Figure 10. Most of the participants reported interpreting raw data from both types of systems using their own experience and intuition to complete tasks (such as diagnosing alarms). Many participants reported using the control system to view equipment alarms directly and to trend historical data directly in the control system interface. The dashboards were used to view performance metrics, trend data, benchmark performance, and view alarms. A small number of participants also reported downloading data from the systems to manipulate manually. Figure 10 Participant utilization of control systems and dashboards Finally, we asked each participant to rate their satisfaction with their control system and/or dashboard. The responses are shown in Figure 11 and we found that for both types of systems, about a third of the respondents were very satisfied, about a third were somewhat satisfied, and the final third were ambivalent, with few dissatisfied. Based on the responses from Task 1, there does not seem to be a large difference between the levels of satisfaction regarding the two different types of systems. 41

42 Figure 11 Participant satisfaction with existing control system and dashboards Potential Value of New Information An additional major goal of this part of the study was to determine what new types of information would be valuable to the participants. This information was particularly informative to the development of the revised questionnaire used for the Task 3 interviews. To get at this information, we gave the participants lists of non-graphical data, metrics, and rankings as well as graphical visualizations, and we asked them to rate each one by how useful they believed this information would be to them in performing their expected job responsibilities. Participants were told to rate each item on the list by how useful that information currently is to them (if they already have this information available) or by how useful it might potentially be (if this information were to become available to them in a control system or dashboard). The responses are shown in Figures 12 and 13. Figure 12 indicates the participants responses regarding the usefulness of non-graphical data, metrics, and ratings. The types of information that would be most useful to participants were equipment fault detection, potential for LEED or other certification, system or equipment efficiency metrics, and benchmarks comparing the building performance to an ideal or simulated model. We note that this type of information was not found to be available in the systems that most participants currently had access to. Other types of helpful information included estimated payback on capital projects, normalized metrics, energy end use metrics, whole building efficiency, benchmarks comparing building performance to similar buildings, and occupant comfort metrics. The least helpful information included ecological footprint, carbon emissions, energy generation by on-site renewables (none of the buildings visited had these types of systems on-site), and simple building ratings or report cards. It is interesting to note that many of the participants rated suggested capital projects as less useful information, even 42

43 though estimated paybacks were considered helpful. We found that the reason that many of the participants rated this information as less useful was that they did not believe that they could trust a computer system to generate such information for them; however, they did feel that they could trust information such as automated fault detection and diagnosis. Figure 12 Rated usefulness of new metrics and information In addition to non-graphical information, participants were also asked to rate various types of graphical displays and visualizations. Figure 13 indicates the participant responses. We found that the most useful graphical information included information that the participants already had access to, such as equipment and system level graphics, floor plans, graphs showing live data, and historical trending. Although participants were provided examples of the various types of graphics, it is possible that participants chose those graphics that they were already most comfortable with as the most useful. Other potentially useful types of graphics included graphs showing performance data overlaid with weather data, heat map of performance (such as zone temperatures) overlaid on a floor plan, energy end use icons or graphics, performance over time overlaid on a clock or calendar, and checklists for LEED or other certification. Less useful types of representation included performance as equivalents (for example, energy use represented using numbers of light bulbs), temporal maps (heat maps of performance over time), and report cards. 43

44 Figure 13 Rated usefulness of new graphical information Discussion of Participant Feedback During the Task 1 interviews, we encouraged the participants to explain the reasoning behind their questionnaire responses and to add any anecdotes that they thought would be relevant to our study. We found that even among participants with the same job title, their responsibilities and experiences varied quite widely. During the portion of the interview that dealt with currently available systems and dashboards, participants had very different opinions about useful metrics and interfaces even when they had access to similar systems. This variation seemed to be due largely to job responsibilities and how much support the participant had in his or her role. For example, we encountered: A building engineer who was responsible for overseeing the operations of four very large buildings alone with a few part-time technicians. He spent most of his time responding to occupant complaints. His use of the control system was primarily to track zone temperatures and diagnose alarms. Although he had ideas regarding energy efficiency projects, he was not allowed access to meter or utility data. He also believed that his staff did not receive enough training in how to use the control system. 44

45 A team of engineers, contractors, and technicians responsible for 24/7 oversight of a large set of buildings. This team included one member whose only responsibility was to monitor the control system. In these buildings, energy use was closely monitored, and at least one building was awarded LEED certification. Alarms from the control system were text messaged to the team during all hours, and some were expected to respond to problems overnight if necessary. A building engineer whose 10 year old control system had no graphics and only a subset of sensor data integrated into the system. He often needed to visit individual pieces of equipment in order to read sensor data. He only used the control system to make sure equipment was on. A building engineer who had access to a highly customized and graphical control system. He programmed it to automatically create reports of trends of log points to use for diagnosis and feasibility studies (data only, no graphs). He would also print out tables of log points and use them directly to diagnose problems. A building manager who downloaded meter and weather data into spreadsheets to manipulate them manually to determine if building performance was adequate compared to the previous year. A building manager whose building s Energy Star rating was incorrectly calculated by another member of the staff who was not fully aware of how the utility data was collected. In this case, meter data was collected for a whole campus but incorrectly reported for a single building. In addition to these experiences, we learned a great deal from the reactions of the participants as they rated how potentially useful the various types of metrics, information, and graphics would be for their jobs. For example, we found: One engineer did not want to see any occupant complaints or comfort metrics because he believed that it would add to his workload. One building manager mentioned that he used ten different tools on a weekly basis (including the control system, dashboard, work order system, and other custom systems) and he wished that some of the information could become integrated so he could use fewer systems. One engineer had reprogrammed all of his dashboard s odometer graphics to display as trend lines, as he did not like that the odometers showed only instantaneous information and not historical information. He also believed that the equipment and system graphics available in many control systems were not helpful or necessary, as he was familiar enough with the equipment in his building that he did not require a picture of it. Many building operators and engineers expressed preference for traditional graphs (real-time and historical trends, using line or bar graphs), while some managers and financial stakeholders believed that the odometer graphics would be most useful to the operators and engineers. Many participants were most interested in information that would help them achieve LEED certification. Several participants expressed distrust of information that required the system to have some intelligence, such as equipment fault detection and diagnosis, optimization, suggested capital projects, or predicted ROI. During these scoping interviews, it was striking that we encountered so many different opinions and experiences during the building visits and stakeholder interviews. Very early in the research process it became clear that an inflexible, fixed set of metrics and visualizations would not serve the needs of all operations and maintenance stakeholders. Instead, widely varying needs demand flexible interfaces, 45

46 which allow for different metrics to be presented in a variety of visualizations and configurations for a variety of stakeholders. 4.2 Interface Component Interviews Following the scoping phase of the research, a detailed set of interview questionnaires were created to solicit specific feedback from participants. The goal of these interviews was to gather specific results identifying precisely which data, metrics and visualizations that specific types of operations, maintenance, and management stakeholders prefer. Because different stakeholders were likely to have very different goals when using an energy management dashboard, the questionnaires were structured into a set of 7 focused on data, metrics and visualizations in the following areas: 1) Portfolio energy consumption and cost 2) Portfolio budgets, expenditures, projections, and project M&V 3) Building energy consumption and cost 4) Building water consumption, cost, and carbon emissions 5) Equipment heating and cooling plant information 6) Equipment ventilation and air handler information 7) Equipment zone and occupant comfort information Each participant completed 1 to 4 of the above questionnaires, chosen based on their job description. As a companion to these questionnaires, a compendium of example visualizations was created to help participants provide feedback on graphical representations of data. To see the questionnaires themselves, see Appendix B. In addition to the questionnaires, the participants were engaged in conversation and asked a series of targeted questions meant to elicit more detailed and anecdotal reactions to their current relationship with BAS systems and energy dashboards. Participants were also asked their opinions on the most important elements of a BAS or energy dashboard and what would be required for them to trust calculations and diagnostics made by the dashboard. As in the scoping interviews, there was a diversity of answers, however some themes were surprisingly consistent across the board, and these qualitative responses also contributed to the interface designs in Task 4. For example, when asked what aspects of a potential dashboard were important to them, the two most common answers were drilldown capabilities from high level summaries to detailed exportable numerical data, and the integration of multiple control systems and dashboards into one location. Also, nearly all participants had to provide raw data to third parties or government agencies at some point, and yet system limitations on data trending was a common technical complaint. Finally, when asked what it would take to trust automated diagnostics, nearly all answers mentioned transparency and extensive commissioning. At the end of Task 3, the interviewers summarized the most common responses to these questions, which can be found in Appendix B. 46

47 4.2.1 Interface Component Interview metrics and visualizations For each focused questionnaire, the participant was asked to choose one or more specific metrics and visualizations that were useful in conveying information in the following categories: 1) Portfolio energy consumption and cost Energy use and cost over time Energy use and cost breakdowns Energy use and cost comparisons Energy and cost savings due to energy conservation measures (ECM) Energy use correlations Portfolio energy diagnostics 2) Portfolio budgets, expenditures, projections, and project M&V Projected energy and cost savings due to capital projects or targeted O&M Payback period and ROI for capital projects and O&M Measurement and Verification (M&V) O&M budgeting Expenditures 3) Building energy consumption and cost Energy use and cost over time Energy use and cost breakdowns Energy use and cost comparisons Energy use and cost forecasts Building energy and cost savings due to energy conservations measures (ECM) Energy use correlations Building energy diagnostics Specialized energy metrics for facility types 4) Building water consumption, cost, and carbon emissions Water use and cost over time Water use and cost breakdowns Water use and cost comparisons Emissions over time Emissions breakdowns Emissions comparisons Emission equivalents Water use and emissions diagnostics 5) Equipment heating and cooling plant information 47

48 Plant efficiency Compliance with setpoints, thresholds or schedules Heating and cooling loads and heat loss Equipment runtime, cycling, and on/off schedules Equipment performance correlations System and equipment diagnostics 6) Equipment ventilation and air handler information Outdoor air and Indoor Air Quality Compliance with setpoints, thresholds and schedules Equipment runtime, cycling and on/off schedules Equipment performance correlations System and equipment diagnostics 7) Equipment zone and occupant comfort information Thermal comfort conditions Outdoor air and Indoor Air Quality Lighting Acoustics Occupant feedback and complaints Equipment runtime, cycling, and on/off schedules Equipment performance correlations Zone and equipment diagnostics For example, the metrics choices in questionnaire 3 under Energy use and cost over time were: Electric energy in kwh per day Electric energy in kwh per sqft-day Heating energy (in therms, MMBTU, or lbs steam) per day Heating energy per sqft-day Cooling energy (in MMBTU or ton-hrs) per day Cooling energy per sqft-day Peak daily electric in kw Peak daily heating or cooling rate Daily electric use profile in kw Daily heating or cooling rate profile Heating energy per sqft-heating degree day Cooling energy per sqft-cooling degree day Total cost per day Cost per sqft-day Peak cost rate Other (please describe) 48

49 After the choice of metrics, the questionnaires focused on different types of tabular or graphical formats, such as raw data tables, time series graphs, or gauges. For each format, the participant was asked which general categories of information (i.e. energy use and cost over time ) would be most useful when displayed in that format. Every format page included topically appropriate and anonymized screenshots from real energy dashboards to illustrate the possibilities, and every questionnaire included sections on nearly all of the following formats: Time series (including line graphs, point graphs, and bar charts) Benchmarks (including rankings and bar chart comparisons with goals or historical data) Scatter plots/correlations (i.e. energy use vs outdoor air temperature) Pie charts (including breakdowns by building or tenant or utility) Calendar plots (including color codes or time series line graphs arranged on a calendar) Dials/Gauges (including absolute value dials and normal range dials) Diagrams/Maps (including color coded street maps, zone layout maps, and equipment schematics) Tables and text (including tabulated data, other numerical values, or text outputs) Figure 14 is an example of the page layout for the questions about data format. To view all format pages in the questionnaires, see Appendix B. 49

50 Figure 14 Sample calendar plot page from Interface Component interviews 50

51 4.2.2 Interface Component Interview results Interviews with 40 additional participants from 9 different organizations were conducted for the interface component interview phase. These interviews were performed with a wide range of stakeholders with influence on building budgets, operations, and maintenance. The breakdown of stakeholders by type, presented below, was fairly balanced. The stakeholders interviewed include technicians, lead building engineers, building or facility managers, energy or sustainability managers, commissioning agents, and financial decision makers. Participating organizations include two universities, two national labs, two offices, one hospital, one high school, and one government building (courthouse). All organizations were located in the continental United States: two in the Northeast, three in the Southeast, three in the Northwest, and one in the Midwest. In total, 87 focused questionnaires were completed, with each participant filling out between 1 and 4 questionnaires. Breakdown of Participants by Job Function Financial Decision Maker Technician Commissioning Agent Lead Engineer Energy or Sustainability Manager Building Manager Number of Participant Responses by Questionnaire 1) Portfolio Energy 2) Financial 3) Building Energy 4) Buidling Water/Emissions 5) Plant 6) Ventilation 7) Zone Figure 15 Participant profile for Interface Component interviews 51

52 Similar to the findings from the scoping interviews, feedback from participants varied greatly by stakeholder roles and responsibilities. The results emphasize the need for intuitive and flexible interfaces with data presented in a variety of visual formats. Some participant preferences were heavily influenced by negative past experiences, including inaccurate data, unintuitive metrics, and nontransparent dashboards. Such experiences erode trust in more complex system outputs, such as fault diagnostics and avoidable costs. Many participants, especially those with engineering knowledge, preferred simple, verifiable information such as time-series graphs of key performance data and the ability to plot data from different systems on the same charts. These desires seem to be an immediate response to current pain points with existing building automation systems that have limited trending and graphing capabilities. Figure 16 shows the percent of participants who were in favor of different types of visualizations. The gaps indicate visualizations that did not appear on certain questionnaires (The questionnaire numbers are defined in Section 4.2.1). The high percentages across the board indicate a desire for choice. Many of the participants checked off every visualization type, and a few explicitly stated their desire to switch between visualizations at will. There are a few notable results. The least popular visualizations among those who manage and operate buildings were the gauge and the scatterplot, but for different reasons. The attitude seemed to be that the gauge was flashy but without substance, and many participants did not seem comfortable with the scatterplots. Two of the most popular visualization types for both portfolio and building-level management were the benchmark (visually comparing current values with historical performance or goals) and the time series. The most popular equipment-level management tools were the time series and tabular raw data. Tabular values had about 80% approval across the board, and many participants commented that it would be important to them to be able to drill down to or export numerical values from any graphic on an interface. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Percent Approval of Visualizations by Questionnaire Type ALL Portfolio (Q 1,2) Building (Q 3,4) O&M (Q 5-7) Figure 16 Percent of participant approval of specific visualizations 52

53 On the multi-building level, participants liked color-coded portfolio or campus maps as a way to communicate high-level information only if they allowed away to drill down to detailed information. Bar charts or time series graphs of utility consumption, comparisons to past performance, and pie charts of end use breakdown over selected periods of time were predictably highly ranked. Surprisingly high ranked were scatterplots, whereas they were low-ranked in every other place. One explanation may be that scatterplots of energy use vs. outdoor conditions were familiar to many participants, whereas other uses of scatterplots were less well understood. Portfolio and financial decision-makers generally had little interest in or understanding of detailed operational information, but instead preferred common financial metrics such as spending, budgets, and project or maintenance ROI. They liked both current and forecasted versions of this data. Dashboards with gauges presenting basic consumption information sometimes appealed to the financial decisionmakers, not for themselves, but for the technical stakeholders (although as mentioned above, many technical stakeholders never actively used these visualizations, and a few were openly disparaging toward them). Real-time utility consumption presented as a time-series graph, with benchmarking against goals or historical values, was a highly ranked way of viewing building performance. Similarly popular was the idea of viewing the calculated energy savings due to complete energy conservation measures (see Figures 17 and 18). 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q 1 - Energy Metrics Preference Total or per Building Building Portfolio Figure 17 Energy metrics preferences for portfolio and financial managers in Questionnaire 1 53

54 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q 1 - Portfolio Benchmarking Options +/- % of goal +/- % of forecast +/- % of historical Portfolio Energy Portfolio Costs 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% +/- % of portfolio avg. kwh/sf-month +/- % of portfolio avg. thermal/sfmonth Q 1 - Building Benchmarking options +/- % of portfolio avg. $/sf-month Portfolio rankings Energy Star ratings LEED ratings Other ratings Figure 18 Benchmarking options preferences for portfolio and financial managers in Questionnaire 1 Managerial stakeholders generally gave high rankings to energy consumption time series, energy breakdown pie charts and time series, and energy comparison benchmarking (% different from benchmark) (see Figure 19). Simple tables scored surprisingly high at the building level in all categories. Understanding energy breakdowns by end use, building, tenant, or other metric was routinely ranked high by managerial stakeholders, however many were skeptical about the cost effectiveness of using metering and sub-metering to produce the breakdowns or other advanced metrics. Note that maps were only offered as an option in Questionnaire 1. 54

55 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q 1,3,4 - Breakdown Visualizations 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q 1,3,4 - Comparison Visualizations 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q 1,3,4 - Consumption Visualizations Building Energy Building Water Building Emissions Portfolio Energy/Cost Figure 19 Visualization options preferred by managerial stakeholders for specific categories in Questionnaires 1, 3, and 4 Operations and engineering personnel, such as technicians, building engineers, and commissioning agents, preferred to have detailed information on equipment operation and data. Some of these technical stakeholders complained of the lack of trending and graphing capability (or flexibility) in their current systems, and they expressed a desire to see time series of operational data and simple operating state graphics condensed into one screen. Many desired to view raw data from different BAS and metering systems in one interface and to have options to view any data using different visualization methods (see Figure 20 for the most popular visualizations). In addition, the idea of correlating historical data to events like complaints, work orders, and alarms was appealing to these stakeholders, although most did not like the idea of correlation scatterplots, preferring time series. Lastly, the idea of visually presenting this data or related calculations on system graphics, equipment graphics, or zone graphics was well-received. 55

56 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q 5,6,7 - General Visualization Preferences Plant (Q 5) Ventilation (Q6) Zones (Q7) Figure 20 Visualizations preferred by operations stakeholders for all categories in Questionnaires 5, 6, and 7 Many technical stakeholders expressed a need for the ability to drill down from high level building performance metrics into system operations and diagnostics. Most participants gave high ranking to basic operating information such as current operating conditions, recent trends in operations, equipment runtimes, and setpoint compliance. Participants did express interest in diagnostic findings, which would illustrate which equipment and systems were underperforming or had faults causing performance issues, such as a leaking air handler valve causing simultaneous heating and cooling. On the other hand, many of the same participants expressed skepticism that these diagnostics could be accurate in either the findings or the associated costs. They also expressed concerns that such a system would result in unmanageable false alarms. These stakeholders agreed that time and resource commitment from both operations and management personnel, as well as incorporation into a simple work process, would determine whether or not diagnostics were useful and effective. Detailed findings from Interface Component Interviews are presented in Appendix B. 56

57 5. Data, Metrics and Visualizations for Operations and Maintenance Having conducted a literature review, an assessment of existing interfaces, and interviews with over 80 operations and maintenance stakeholders, the project team compiled real, but anonymous data from buildings and crated a set of metrics and visualizations in an online interface to present to stakeholders for feedback. This section describes the components of the interfaces, survey feedback, and recommendations for future interface design. 5.1 Example Interfaces The exact design and user experience of the interface for advanced operations and maintenance was not evaluated as part of this research project. However, it is expected that these components will be critical to success of any interfaces for operations and maintenance. The focus of this work is to identify which metrics and visualization should be presented within these interfaces. Because it was required to create a mock interface to present these examples, a structured interface was created which presented information in two tiers. Primary sections include categories such as costs, utilities, operating characteristics, diagnostics, and raw data presentation and while secondary subsections are broken down by scale, from the portfolio level, to buildings, plant, ventilation and air handlers and zones. An early prototype for the design of the example interface is shown in Figure 21. Section organization is shown in Figure 22. Figure 21 Early prototype example interface design 57

58 Figure 22 Example interface section organization Within this framework, the project team compiled actual building automation system and utility meter data from an existing performance management platform to create an example interface. This interface is available to the general public at the following location: The example interface was constructed and presented using Google Sites. This platform was chosen as it allowed us to create an interface which simulated the flow and content of a live interactive tool. Using Google Sites, we were able to incorporate the following important features and functionality: - Ability to provide the external link to all intended participants - Embedded tables, images and multiple types of graphs/charts including those that were presented in previous sections - Interactivity with the charts and tables, for example allowing visitors to hover-over to see point values - Linkable pages to map the flow and structure found in a Live tool - Embeddable surveys with the ability to track responses During the initial research for this project, the project team identified that most building monitoring systems used by respondents could be categorized as either control systems or dashboards. The example interface created for Task 4 bridges the gap between these two categories and provides content from both as well as features that the team felt were equally informative to stake holders yet lacking from existing tools. For example, none of the work-order systems used by participants 58

59 interviewed in Task 1 were integrated with the dashboards or control systems. To provide an example of this integrated functionality, a section for project tracking was integrated into the Costs section of the interface. Most of the charts, graphs, tables, and metrics provided in this interface were taken or adapted from the Interface Component Interviews (Section 4.2). The purpose of creating this example interface was not to focus on the flow or specific metrics displayed, but rather to provide an interactive and realistic medium for the data, metrics, and visualizations that were identified as most useful by stakeholders to be reviewed. When first accessing the example interface, participants are asked to answer a brief survey regarding their background and experience with building operation and metrics tools. After completing the survey, users are then led through a tour of the site via Previous and Next page navigation arrows. The navigation arrows were provided to ensure that respondents viewed all metrics on the site and did so in organized manner. Please note that to mimic an actual tool, all navigational links on the site are live and mapped to the appropriate pages. Respondents are informed that they are able to navigate around the site freely after they complete the tour. To make the interface more user-friendly, a status bar was added to each page and instructions were provided to allow respondents to take a break and finish their review later, without having to start over. A help section was also provided in the top navigational bar that included survey instructions, Frequently Asked Questions, and an address for personal support. To record participant feedback, surveys were embedded in the mock interface to collect information about each individual graphic, table, or visualization and its corresponding metrics. An additional survey was presented after each of the primary sections to provide more detailed feedback. Below is an example of a typical page layout from the interface. The example below is a look at the main page of the Ventilation subsection under the Operations main section. It includes the following sections: (1) Help Section, (2) Main Section Navigation, (3) Subsection Navigation, (4) Typical Example Interface, (5) Typical 1-5 Rating Survey, (6) Button Submitting All Ratings, (7) Previous/Next Page Navigation Bar, (8) Tour Progress Bar. Screenshots from the home page, select graphics from each section, and descriptions of the content of each sub-section are provided in this section. Please refer to Appendix C for organized screenshots of the entire example interface. The site is still available to view here: 59

60 Figure 23 Typical example interface page organization and navigation. 1. Help section to explain survey navigation. 2. Type of Metrics and Visualizations. 3. Scale or system level. 4. Visualization. 5. Participant ranking option. 6. Form submission. 7. Page navigation. 8. Progress Bar. 60

61 Home Page After answering a survey regarding their background and experience with similar tools, respondents land on the home page to begin their review of the example interface. The home page contains graphics and metrics that provide the user with a portfolio level view of the information contained in each of the main sections. Figure 24 Example interface main homepage Costs Section The Costs section contains the following sub-sections: Portfolio: Compare energy costs of all buildings across the portfolio. Energy costs are broken out by type and normalized by square foot. Compare quarterly costs of water consumption of all buildings across the portfolio. Building: View monthly spending on electricity and gas for each building over the past year. Projects: Track the cost and savings of maintenance projects across the portfolio. 61

62 Figure 25 Example interface Costs homepage a. b. Figure 26 Example graphics from Costs page Metrics are displayed in multiple ways within the interface. For example, the figures above show the monthly net cost of utilities as well as the cost of utilities normalized by building area. 62

63 Utilities Section: The Utilities section contains the following sub-sections: Portfolio: View a comparison of the utility consumption of all buildings in your portfolio Monitor gas and electrical usage as well as CO2 emissions. Building: View the performance of the individual buildings within your portfolio. Compare the performance of each building to the rest of the portfolio. Track how each building is performing compared to previous years. Figure 27 Example interface Utilities homepage Similar to the graphics presented during the interface component interviews, metrics displayed on this example interface using multiple chart types. An example of some of these presentation methods from the Utilities Section can be seen above in in Figure

64 a. b. c. d. Figure 28 Example graphics from Utilities page Operations Section: The Operations section contains the following sub-sections: Portfolio: Monitor past and present heating and cooling modes of all buildings within the portfolio. Building: View current and weekly operating runtimes by building. Monitor key heating, cooling and ventilation performance and statistics. Plant: View current and weekly runtimes of equipment in a specific plant. Ventilation: Overview of current and weekly performances of all AHUs by building. Zones: View current and weekly performance of terminal units within each zone. 64

65 Figure 29 Example interface Operations homepage 65

66 a. b. Figure 30 Example graphics from the Operations page 66

67 Diagnostics Section The Diagnostics section contains the following sub-sections: Plant: Overview of current and weekly faults for each plant. Navigate to a specific plant for detailed information of fault types and time that faults occurred. Ventilation: Overview of current and weekly faults for each AHU. Navigate to specific building for detailed information of fault types and time that faults occurred for all AHUs within the building. Zones: Overview of current and weekly faults for each zone within a building. Navigate to a specific zone for detailed information of fault types and time that faults occurred. Figure 31 Example interface Diagnostics homepage Many of the existing dashboards and control systems investigated were able to do simple calculations and alarm generation. The Diagnostics Section incorporates similar features as well as provides advanced fault detection and diagnosis (FDD), which was only found in a very small portion (less than 20%) of the existing tools that were surveyed in Task 1. Figure 31 is one example of how an FDD finding of Simultaneous Heating and Cooling on an air handling unit was incorporated into the Diagnostics Section of the example interface. 67

68 Figure 32 Example graphics from the Diagnostics page Data Section The Data section contains the following sub-sections: Portfolio: Access raw data for the whole portfolio. Use the chart view to graph multiple data series over any data range. Building: Access raw data for each building. Use the chart view to graph multiple data series over a specified date range. Use the table view to access the data in spreadsheet format. Plant: Access raw data for each heating and cooling plant. Use the chart view to graph multiple data series over a specified date range. Use the table view to access the data in spreadsheet format. Ventilation: Access raw data for each zone. Use the cart view to graph multiple data series over a specified date range. Use the table view to access the data in spreadsheet format. The data section incorporates the popularity of producing trends of raw data (Figure 32a) as well as maintaining the ability to view raw data in a tabular format (Figure 32b). 68

69 a. b. Figure 33 Example content from the Data page 69

70 5.2 Participant Surveys Interview participants for RP1633 were asked to complete a survey of the mock interfaces and rank graphics and metrics on a 1-5 scale while reviewing each individual graphic, with 5 being a most useful rating and 1 being a least useful rating. In addition rating each individual graphic or metric, participants were also asked to provide brief feedback after each major section. Within these surveys, participants are asked how to identify which of the sections were the most useful and whether they had suggestions or comments to improve what was presented. Participation in the mock interface survey was low. Only seventeen of the original 79 interview participants completed the mock interface survey. This does not constitute a statistically significant survey of the mock interface components, and therefore their feedback has been incorporated into recommendation from this research along with the participant interview feedback, literature reviews, and existing tool reviews. The findings from these surveys are presented below. Highest Ranked Metrics and Graphics Overall The top twenty individual graphics or metrics are listed below. A complete summary of the graphics and metrics rankings are included as an appendix. The difference in ranking among the top 20 is very small, ranging from a score of 2.5 to 3.07, while the full rankings ranged from 1.44 to Summary and breakdown of building expenditures Survey score: 3.07/5 70

71 2. Weekly Calendar View of Major Faults and Severity for each Plant Survey Score: 2.9/5 3. Histogram of VAV box reheat valve operations (Percent-Hours Open) over a period of time Survey score: 2.89/5 4. Weekly Calendar View of Major Faults and Severity by Equipment Type Survey Score: 2.89/5 71

72 5. Calendar plot of energy consumption over time Survey score: 2.83/5 6. Building Summary and Energy Star Rating Survey score: 2.82/5 72

73 7. Histogram of Zone Deviations from Setpoint Over a Period of Time Survey Score: 2.78/5 8. Summary of utility Costs and Projected Spending Survey Score: 2.75/3 73

74 9. Text Summary of Fault Occurrences Survey Score: 2.75/3 74

75 10. Animated Visualization of Building Modes over Time on a Map Survey Score: 2.73/5 11. Building Annual Utility Consumption Breakdown Survey Score: 2.67/5 75

76 12. Histogram of VAV damper position (percent-hours opens) Over a Period of Time Survey Score: 2.67/5 13. Time Series of VAV box operational trends organized by supply air handler Survey Score: 2.67/5 76

77 14. Diagnostic List with Priority Rankings Survey Score: 2.67/5 15. Time Series of Fault Occurrence for a Piece of Equipment Survey Score: 2.67/5 77

78 16. Time Series of Major Project Capital Expenditures Survey Score: 2.62/5 17. Summary of major capital projects and savings to date Survey Score: 2.62/5 78

79 18. Time Series of plant equipment operating modes Survey Score: 2.55/5 19. Monthly Calendar Plot of Building Energy Performance Relative to Baseline Survey Score: 2.54/5 79

80 20. Time Series of Annual Greenhouse Gas Emissions Survey Score: 2.5/5 80

81 Most Useful Pages In addition to the individual rankings of each graphic and metric, users were asked to recall the most useful components at the end of each section. The top three views from each section, Costs, Utilities, Operations, and Diagnostics are shown below. Costs 1. Track energy and cost savings from active projects 81

82 2. Comparison of building energy costs across the portfolio, normalized by area 82

83 3. Dedicated page for energy cost of each building 83

84 Utilities 1. Building utility consumption by end use 2. Portfolio level benchmark performance (utilities cost, emissions, energy star, etc.) 84

85 3. Live gas and electricity use trends 85

86 Operations 1. Time series of when major equipment is on/off 86

87 2. Time series of building heating, cooling and occupancy 87

88 3. Heating plant diagram and operations summary 88

89 Diagnostics 1. Plant graphic highlighting current faults 2. Explanation of faults and possible causes 89

90 3a. Time series highlighting time, location and duration of all faults on a given air handler 90