White Paper by: Mario Morfin, PhD Terri Chu, MEng Stephen Chen, PhD Robby Burko, PhD Riad Hartani, PhD An Implementation of Active Data Technology October 2015 In this paper, we build the rationale for a new generation of intelligent optimization problems, better suited for the type of data in an IoT world. We introduce novel optimization techniques, recently invented by MOAI's team, to satisfy such requirements. We then focus the applicative aspects on the overall context of smart cities design, and energy systems optimization in particular. 1 MOAI From Big Data to Active Data Startup in Stealth Mode
1. Introduction The Internet of Things (IoT) is a network of interconnected objects with the ability to generate and exchange signals and sensor measurements and to respond to changes in the environment in real time. The underlying dynamics of such processes will transform how companies and individuals interact, not only by speeding up and automating their interactions, but also by creating a new environment for previously unavailable collaborative partnerships that will form the basis of new business operations. Big Data technologies are fast evolving, and they are progressively being integrated into various business processes. However, their complete potential has yet to be fully achieved. Efforts to date have been focussed mainly on building state-of-the-art data ingestion, storage, and management solutions. Leveraging such data for optimizing operations is now the immediate focus. This is precisely where MOAI, a newly formed technology startup, is developing new solutions aimed at building intelligent modelling, optimization, and analytical techniques for next generation IoT networks. In this paper, we build the rationale for a new generation of intelligent optimization and learning solutions that are better suited for the types of data and the nature of the problems that exist in an IoT world. We base this on novel optimization techniques that have been recently invented by the team, and are particularly well suited to this new class of problems. We then describe an implemented case study in the overall context of smart cities design, and energy systems optimization in particular. 2. An IoT Centric Evolution of Optimization Models Problem positioning via an illustrative industrial case study To illustrate the value of the optimization techniques we have developed, a real use case is described, in the area of pricing models based on dynamic systems analysis. Current analytical techniques in Big Data are about interactions that have already occurred in the past. They find statistical correlations within the data, which direct companies toward a better financial Efforts until now have been focussed mainly on building state-of-the-art data ingestion, storage, and management solutions. Leveraging such data for optimizing operations is now the immediate focus. 2 MOAI From Big Data to Active Data Startup in Stealth Mode
outcome in the future. In fact, the vast majority of techniques in use today, repeat the following process: 1. Data generation Users generate data by liking, sharing, relaying and a large set of possible actions 2. Data analysis Service providers collect, store, and analyze consumer data in the cloud (private, public, or hybrid). Consumers are classified by a set of preferences, which try to predict their response to a situation. 3. Decision Making From the analysis, connections are made between the actions of a provider and the reactions of a consumer. Future decisions are made in order to maximize profit, based on the validity of these connections. This in turn leads to decisions aiming at process optimality. These techniques have been successfully implemented in many industries; however, they still fail to reach the full potential of operational efficiency. When consumers communicate with each other, they become a cooperative team, which can act in the benefit of each one of its members, turning into a more resourceful and powerful economic force. This effect can turn a seller's market into a hybrid market, in which companies can capitalize on the predictability of the consumption patterns. In other words, feedback from the present and potential actions in the near future, are leveraged to make better real time optimizations. This paradigm is facing a revolution. More and more devices are able to send signals and process data. Moreover, automated response and large-scale monitoring are becoming commonplace in many industries while the possibilities for analysis and optimization widens. It will soon be possible to implement more complex models of such economic interactions as well as consumer preferences. As such, the analysis and modeling possibilities of the market can reach their natural next step, as follows Active data, the next step in the evolution of Data Technology Each step of the previous process can be enhanced through the IoT value chain: 1. Data generation + Swarm intelligence Users who share their data can cooperate. When the possibility to react to changes in the market is instantaneous and proactive, this cooperation turns clusters of users into intelligent swarms operating in the interest of all members. 2. Data analysis + Dynamic modeling 3 MOAI From Big Data to Active Data Startup in Stealth Mode
Large scale monitoring will provide a more detailed and higher level understanding of consumer preferences and tendencies in the market place. This jump takes us from a pixelated snapshot of the market place to a high-resolution movie representing a more complete and dynamic model of the market. 3. Decision Making + Preferences Using this Model as a baseline, users preferences can be as complex as the model, and the reaction to market changes is immediate. We call this step in the evolution of data technology Active Data. Big Data Active Data Active Data in the energy market. The energy market represents a relevant and immediate opportunity for leveraging these new data generation, gathering, and optimization models and for showcasing the benefits of Active Data. In general, electricity production responds directly to consumption levels, but what if consumption could respond to production? This question is particularly relevant in systems which rely heavily on clean energy sources such as wind and solar, and is beginning to be explored in part by demand response programs in municipalities around the globe. Big Data analytics, used primarily in the prediction of consumption patterns, does not sufficiently harness the capabilities of the coming technologies. The IoT creates the opportunity for bidirectional, real-time communication, which allows for the modification of consumption patterns in order to match generation. Without demand management, the absorption of renewable energy into existing electricity infrastructure will be severely limited. The use of feed forward preferences to effect coordinated, system-wide behaviours leads to enhanced efficiencies for the consumers, the grid, and the energy producers. The technologies 4 MOAI From Big Data to Active Data Startup in Stealth Mode
needed to implement real-time energy strategies are being tested in prototype Smart Cities in Seoul, Barcelona and Dubai. These environments will soon have a need for and the ability to support automated decision making through Active Data. 3. MOAI Innovations and related contributions Demand matching, load balancing, and the setting of prices are all optimization problems. The confluence of these three problems, with the restrictions imposed dynamically by the users, creates a complex search space with a dynamic set of variables and constraints. The amount of data generated in the IoT world is large, and rapidly increasing. Today s optimization techniques are not optimally designed for these complex systems present in IoT ecosystems. IoT networks face the following challenges: Geographical elasticity. In contrast to centralized infrastructures, the IoT allows for mobile and semi-mobile nodes to be connected. These nodes are part of a large-scale system of signal producers and processors. Large scale monitoring. The scale of sensors deployed by IoT networks and the monitoring systems they provide is unprecedented. The vast data lake presents new computational challenges and optimization opportunities. The greatest opportunity is to create global collaboration through localized decision making. Real time interactions. Real time interactions demand more power than batch interactions given that, in general, they require more information from the entire system. The balance between global effects and local decisions is key for achieving stability and reliability in the system as a whole. Heterogeneity. Users need private, open, and public networks. All of these factors demand better infrastructure and more powerful analytical techniques. MOAI - Innovations in intelligent optimization and contributions to IoT models In 2015, a superior algorithm for heuristic optimization, named Leaders and Followers (LaF) was invented by Prof. Stephen Chen (MOAI co-founder). It has been shown to outperform other 5 MOAI From Big Data to Active Data Startup in Stealth Mode
heuristic algorithms, especially in high-dimensional spaces (involving many variables). The breakthrough of this innovation uses the idea that in order to find an optimal solution, one should deploy a group of explorers and avoid elitism. This feature makes LaF ideal for largescale dynamic systems with high levels of interactions and dependencies. For further information about this method, please refer to: MOAI white paper IoT & Intelligent Optimization Intersect as well as descriptions of the corresponding mathematical and experimental aspects. MOAI's implementation of LaF is perfectly suited for continuously evolving systems in dynamic environments. It relies on the constant generation of inter-related partial solutions of a largescale problem. Real-time monitoring of the marketplace feeds information into the system so that these partial solutions can continue to be re-calculated. The solutions are kept in distributed systems where they are re-generated as necessary according to market conditions, such as shifts in consumer preferences or predictive variations. These partial solutions are then aggregated into a family of global solutions. LaF acts as a listener on this family of solutions monitoring for the need to make large changes while keeping track of solutions it has found which are as close to optimal as possible. In this way, the system achieves a high level of stability and is able to constantly learn and be updated from interactions in the market. It also allows for risk analysis and real time pricing of various goods traded in this environment. Predictive Data Math Model Stochastic Solution Generator Optimization Listener (LaF engine) Visualization Voltage Reader The IoT Grid A P I Real time Collection Distributed Storage Data Integration Distributed Partial Solution (fog) Statistical Analysis Data Integration A P I Controller Consumers The IoT Grid Policy engine Policy engine Wifi User settings Model setup Tasking Decision Making Controllers Risk Analysis 6 MOAI From Big Data to Active Data Startup in Stealth Mode
Implemented Case Study - Active data in the energy market application The smart grid is an excellent example for the potential of the IoT and the Active Data paradigm. Consumers produce data that include energy consumption for domestic use, e.g. heating and air conditioning, and their preferences for such activities. Moreover, through the addition of electric and thermal storage, micro-production devices such as solar panels and small generators, and an automated control system, the IoT consumer in a modern smart city becomes equipped with the tools to save energy, time shift their necessary consumption, and produce their own energy in a way as to maximize economic benefit on both a personal and global level. The data is stored and pre-processed in the fog, where partial clusters and partial load balancing can be performed. The IoT consumer can rely on MOAI's technology to find virtual teammates to balance consumption. The producers, on the other hand, can access the aggregated patterns of consumption. This information has the potential to reduce waste, lower costs, and plan for infrastructure investments. To reach this potential, an aggregated optimal schedule must be constructed. This schedule can be calculated by MOAI s LaF implementation. The well being of the grid encompasses the well being of all parties connected to it. As an intermediary, accurate pricing of energy leads to more optimal allocation of energy, and this makes improved decision making possible for the cities and institutions in charge of this essential infrastructure. Knowing accurate energy costs in real time allows for better decisions making, and the emergence of new automated patterns of consumption, which has been validated through the MOAI experiments. Specific economic benefits derive from such optimization. In fact, and based on our estimates for the province of Ontario alone, the potential savings of this technology is more than $300M. As such, the economic benefit of applying advanced modelling and optimization is highly strategic. 4. Deployment Architecture The commercial deployment of this technology is being implemented as per the following design principles: Leverages state of the art software design models The implementation of the end-to-end solution takes into account the latest developments in terms of application development. Deployment leverages software running on Infrastructure as a Service (IaaS). The implementation leverage large-scale cloud based designs, integrating micro-services software models running over virtual environments and, as needed, container based deployment models. It uses big data management engines, based on Hadoop 7 MOAI From Big Data to Active Data Startup in Stealth Mode
infrastructures, with open interfaces towards the cloud southbound, and towards the Data Sciences engines northbound. It is designed to run in a standalone environment as well, as required by the context of the industrial problem being solved, but also as a module within a larger scale industrial software solution as required. As we evolve towards the design of a new generation of smart cities, optimization problems will appear at every level of this new infrastructure. The need for a new generation of algorithms that are suitable for this new reality is essential. Runs and deploys as a Software as a Service (SaaS). MOAI's intelligence can serve the consumer to time and plan their consumption following cloud SaaS implementation and monetization methods. These services can go from the IoT end user, such as households, drivers and small businesses; to automatically monitor their use of energy, and to update their preferences. These services can include all the advantages that real time pricing brings to the consumer side of the economic chain. Integrate within Platform as a Service (PaaS) solutions Various energy companies have developed solutions as a PaaS, allowing others to deploy applications within their middleware. Examples include IBM (Bluemix), GE (Predix) and others. In this way, energy producers can access the whole platform of aggregated and partially aggregated information to improve their bottom line. As such, MOAI s computations and outputs can easily be integrated into other applications according to their specific requirements. 5. Conclusions. We have outlined the vision for and defined the key characteristics of Active Data and its implementation in the context of smart cities. Our focus has been on the design of a specific illustrative application in the energy space. The IoT and the platforms around it will deliver a rich variety of new scenarios, services, and applications at every level of the energy industry. As we evolve towards the design of a new generation of smart cities, optimization problems will appear at every level of this new infrastructure. The need for a new generation of algorithms that are suitable for this new reality is essential. These optimization techniques will share key fundamental design requirements, including flexible and adaptable architecture to tackle the massive amounts of data and the underlying computational needs, optimal orchestration and resource management of large complex systems and the ability to adapt to new innovative services and applications to be supported and required within the smart cities environment. 8 MOAI From Big Data to Active Data Startup in Stealth Mode
The quality of these techniques will benefit all the players in the ecosystem: 1. Consumers, by producing technology that can make them connected energy consumers 2. Producers, who will be able to manage and reduce costly peaks of production 3. The grid and all essential infrastructure, by constantly monitoring and digesting the vast amount of information produced by the IoT city. We believe that the optimization solutions we have brought forward at MOAI, and the way they are being applied to practical industry problems, are a fundamental step forward in this direction. 9 MOAI From Big Data to Active Data Startup in Stealth Mode