Convergence of Cloud Computing and Network Virtualization: Towards a Zero-Carbon Network

Transcription

1 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. Convergence of Cloud Computing and Network Virtualization: Towards a Zero-Carbon Network Mathieu Lemay, Kim-Khoa Nguyen, Bill St. Arnaud, Mohamed Cheriet Abstract Nowadays, reducing greenhouse gas (GHG) emissions is becoming one of the most challenging research topics in Information and Communication Technologies (ICT) because of the overwhelming utilization of electronic devices. Current solutions mainly focus on energy efficiency for saving power consumption at the micro level. Large-scale energy management strategies are still hardly taken into account. In this paper, we present a low-carbon nation-wide network built in Canada, and then expanded over the world. Based on an assumption that energy efficiency at the micro level will likely lead to an over consumption at the macro level, the proposed network is powered exclusively by renewable energy sources. Using network and server virtualization techniques, data centers services are migrated around network nodes according to renewable energy availabilities. A follow the sun, follow the wind optimization policy is deployed as a virtual infrastructure management technique. GHG reductions are evaluated as a result of our research. Introduction During the past ten years, annual mean temperature increases and sea-level rise and transition have happened as a result of global warming. Most of research agrees that the main reason of the climate changes is the greenhouse effect caused by greenhouse gases (GHG) where carbon dioxide is a key factor. It is a challenge that not only jeopardizes the sustainability of our planet; it poses significant, long-term threats to the global economy. Among the main power consumption industries, Information and Communication Technology (ICT), with its annually growing rate of 9% [1], contributes approximately two per cent to the global GHG emissions, and this amount will almost double by 2020 [2]. Fortunately, the ICT industry has the potential to reduce its current GHG emissions. The SMART 2020 report [2] showed that ICT's unique ability to monitor and maximize energy efficiency both within and outside of its own sector can lead to emission reductions five times the size of the sector s own footprint. This represents a saving of 7.8 Giga-tons of CO2 equivalent by greater than the current annual emissions of either the US or China. Using new network and distributed computing architectures to reduce or to eliminate indirect GHG emissions caused by ICT through zero-carbon data centers is one of the most promising ICT strategies to mitigate Global Warming progression. As ICT is increasingly provided by data centers as a service, including evolving forms such cloud, grid, utility computing, storage, and networking, the urgent need to reduce GHG emissions will affect both providers and customers of ICT services. Cap & trade or carbon tax will increase service provision Digital Object Indentifier /MIC /$ IEEE

2 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. costs as most of data center activities are carbon-dependent. That explains why large ICT companies, like Microsoft which consumes up to 27 megawatts of electricity at any given time [1], have built their data centers near renewable power sources. Unfortunately, many computing centers are not so close to the renewable energy sources. Thus, renewable energy distributed networks and the way they can be related to current Grid systems using Smart Grid technologies are emerging technology. A key assumption to make is that losses incurred in energy transmission over power utility infrastructures are much higher than those caused by data transmission, which makes relocating a datacenter near a renewable energy source a more efficient solution than trying to bring the energy to an existing location. Note that renewable energy and green energy may be interchangeably used. Other types of energy, like nuclear which is not renewable, are not considered green. The current approach when dealing with the ICT GHG problem is improving energy efficiency, which attempts to reduce energy consumption at the micro level. Numerous research projects have been conducted following this direction. They focused on micro-processor design, computer design, poweron-demand architectures and virtual machine consolidation techniques. However, a micro-level energy efficiency approach will likely lead to an overall increase in energy consumption due to the Khazzoom Brookes postulate (also known as Jevon s paradox) which states that energy efficiency improvements that, on the broadest considerations, are economically justified at the micro level, lead to higher levels of energy consumption at the macro level [4]. Consequently, this increases GHG emissions. Some research concluded that energy efficiency is therefore an irrelevant network design approach and the objective should be to make networks carbon neutral [2][3]. In this paper, we present one of the first nation-wide networks, so called GreenStar Network (GSN) [6], which is powered entirely by green energy. This research is inspired from the carbon neutral approach proposed in [5] and from the emerging need of a green energy distribution wide area network model for establishing a standard carbon protocol for the ICT industry. The main objective of the GSN project is to create a pilot and a testbed environment from which to derive best practices and guidelines to follow when building low carbon networks. Although the ISO standard - used to measure GHG emissions in traditionally heavily polluting industries and upon which the protocol is based - is straightforward, its specialization to ICT will require synergistic solutions relating to power and performance measurement, as well as network and system operation. Therefore, the GSN project focuses on two principal activities: i) Creation of a ISO14064 compliant protocol and enactment of a GHG reduction project based on utilization of green data centers; and ii) Development of management and technical policies that leverage virtualization s mobility to facilitate the use of renewable energy, e.g. solar and wind, within the GSN. A Zero-Carbon Network The GSN project was initiated by a Canadian consortium of industry, universities and government agencies with the common goal of reducing GHG emissions arising from ICT services. The project Digital Object Indentifier /MIC /$ IEEE

3 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. focuses on the relationship between networks and green data centers in order to provide green ICT services. The idea behind the project is that a carbon neutral network must consist of data centers built in proximity to clean power sources and user applications will be moved to be executed in data centers. Such a network must provide an ability to migrate entire virtual machines to alternate data centers locations. Regarding research recently proposed in [3], the GSN goes one step further. Whilst the former focuses on allocating physical data centers close to cheap energy sources, we are actively and dynamically migrating virtual data centers around green nodes while maintaining user services. This is supported by a highspeed optical layer having up to 1,000 Gbps bandwidth capacity. Note that optical networks have modest increase in power consumption, especially with new 100G and 1,000G waves, compared to electronic equipments such as routers and aggregators [5]. The GSN includes small and medium size data centers, which allows the network to be flexible and cost-effective regarding construction costs. Relocating small data centers can also be achieved in a timely manner, which is appropriate for online services. Indeed, advantages of small or even tiny data centers over large-scale data centers in terms of energy consumption and management have been discussed in [14][15]. Figure 1: The GreenStar Nework Digital Object Indentifier /MIC /$ IEEE

4 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. In this paper, we focus on the design and management of the GSN which provides zero-carbon ICT services. The cost of producing and maintaining network elements, such as routers and servers, is not considered, because no special hardware equipment is used in the GSN. The only difference between a regular network and the GSN and is that the latter one is able to transport ICT services to data centers powered by green energy. Such a feature is implemented at software level as described throughout this paper. As shown in Figure 1, the core GSN built in Canada includes six nodes powered by sun, wind and hydroelectricity. Solar power is used at Cybera (Calgary, AB) and CRC (Ottawa, ON). Wind power is generated at BastionHost (Truro, NS). Three nodes at Rackforce (Kelowna, BC) and ETS-UQAM (Montreal, QC) are powered by hydro energy. Since BC and QC provinces have a large capacity of hydroelectricity, there is no risk of service interruption in the network due to power outages. However, using renewable energy like wind and solar ones is considered a higher priority because hydroelectricity is unlikely considered renewable sources of energy. Therefore, applications are forced to run in solar and wind powered nodes whenever it is possible. Figure 2: Node architectures (hydro, wind and solar types) Figure 2 illustrates architectures of a hydroelectricity and two green nodes, one is powered by solar energy and the other is powered by wind. The solar panels are grouped in bundles of 9 or 10 panels, Digital Object Indentifier /MIC /$ IEEE

5 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. each panel generates a power of W. The wind turbine system is a 15kW generator. After being accumulated in a battery bank, electrical energy is treated by an inverter/charger in order to produce an appropriate output current for electrical devices. User applications are running on multiple Dell PowerEdge R710 systems, hosted by a rack mount structure in an outdoor climate-controlled enclosure. Air conditioning and heating elements are powered by green energy at solar and wind nodes; they are connected to the regular power grid at hydro nodes. The PDUs (Power Distribution Unit), provided with power monitoring features, control electrical current and voltage. A local network links servers within each node, which is then connected to a core network through GbE transceivers. Data flows are transferred among GSN nodes over 9 optical core switches of the CANARIE network spreading across Canada. The GSN node at Montreal plays a role of manager (so called the hub node) that opportunistically sets up required connectivity for Layer 1 and Layer 2 using dynamic services, then pushes virtual machines (VMs) or software virtual routers from the hub to sun and wind nodes (spoke nodes) when power is available. VMs will be pulled back to the hub node when power dwindles. In such a case, the spoke node may switch over grid power for running other services if it is required. However, GSN services are powered entirely by green energy. The VMs are used to run user applications, particularly heavycomputing services. Based on this testbed network, research experiments are performed targeting cloud management algorithms and optimization of intermittently-available renewable energy sources. The GSN is also incorporating green nodes in Ireland (HeaNET), Belgium (IBBT), Spain (i2cat), China (WiCo), Egypt (Smart-Village) and USA (ESNet). Cloud Management & Data Center Migration Converging server and network virtualization techniques, the GSN is an environment consisting of a set of clouds, each cloud represents a data center with its power and network accessories. Therefore, the key challenge is how to manage, connect, and coordinate elements within the environment in order to achieve specific tasks, while maintaining the required power level and minimizing GHG emissions. The web based management solution we propose below allows cloud users to transport their services over network to green energy sources. Cloud Management The proposed cloud management solution is based on the IaaS (Infrastructure as a Service) concept, which is considered a new technology dealing with the delivery of computing infrastructure [8]. IaaS allows clients to fully outsource services, such as servers, software, data center space or network equipments, without a need of purchasing these physical resources or dealing with some of the difficulties the equipment owners face. The key notion of IaaS is Cloud Architecture, which addresses key difficulties of large-scale data processing. Digital Object Indentifier /MIC /$ IEEE

6 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. Figure 3: Layered GSN and Cloud Computing Architectures Figure 3 compares the layered architecture of the GSN with a general architecture of a cloud comprising four layers. The GSN Data plane corresponds to the System level, including massive physical resources, such as storage servers and application servers linked by controlled lightpaths. The Platform Control plane corresponds to the Core Middleware layer, implementing the platform level services that provide running environment enabling cloud computing and networking capabilities to GSN services. The Cloud Middleware plane corresponds to the User-level Middleware, providing Platform as a Service capabilities based on IaaS Framework components [8]. The top Management plane or User level focuses on application services by making use of services provided by the lower layer services. Data center migration In the GSN project, we are interested in moving a virtual data center from one node to another. Such a migration is required for large-scale applications running on multiple servers with a high density connection local network. The migration involves four steps: i) Setting up a new environment (i.e., a new data center) for hosting the application with required configurations, ii) Configuring network connection, iii) Moving VMs and their running state information through this high speed connection to the new location, and iv) Turning off computing resources at the original node. Indeed, solutions for the migration of simple applications have been provided by many ICT operators in the market. However, large-scale data centers require arbitrarily setting their complex working environments when being moved. This results in the reconfiguration of a large number of servers and network devices. In our experiments with the online interactive application Geochronos [11], each VM migration requires 32Mbps bandwidth in order to keep the service live during the migration, thus a 10Gbps link between two data centers can transport 312 VMs in parallel. Given that each VM occupies one processor and that each server has up to 16 processors, 20 servers can be moved in parallel. If each VM Digital Object Indentifier /MIC /$ IEEE

7 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. consumes 4GB memory space, the time required for such a migration is 1000s. Figure 4: IaaS Framework Architecture Overview Currently, the GSN runs a simple energy distribution algorithm: When green energy source is available at a spoke site, perform data processing, otherwise, run applications at the hub. Each site is a location containing computing and storage resources. Each spoke site is associated with an energy broker responsible for monitoring power and triggering migration events. Other energy distribution strategies will be taken into account in our future work, such as maximizing the utilization of renewable energy sources or sharing of supported devices among different users and different data centers. The migration of data centers among GSN nodes is based on cloud management supports. The whole network is considered as a set of clouds of computing resources, which is managed using the IaaS Framework [8]. The IaaS Framework includes four main components: i) IaaS Engine used to create model and devices interactions abstractions, ii) IaaS Resource used to build web services interfaces for manageable resources, iii) IaaS Service serves as a broker which controls and assigns tasks to each VM, and iv) IaaS Tool provides various tools and utilities that can be used by the three aforementioned components (Figure 4). The Engine component is positioned at the lowest level of the architecture and maintains interfaces with physical devices. It uses services provided by protocols and transport layers in order to achieve communications. Each engine has a state machine, which parses commands and decides to perform appropriate actions. The GSN management is achieved by three types of engines: i) Computing engine is responsible for managing VMs, ii) Power engine takes care of power monitoring and control and iii) Network engine controls network devices. The engines allow GSN users to quantify the power consumption of their service. Engines notify upper layers by triggering events. The Resource Digital Object Indentifier /MIC /$ IEEE

8 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. component serves as an intermediate layer between Engine and Service. It provides Service with different capabilities. Capabilities can contribute to a resource s Business, Presentation or Data Access Tier. The Tool component provides additional services, such as persistence shared by other components. Based on the J2EE/OSGi platform, the IaaS Framework is designed in such a modular manner that each module can be used independently from others. OSGi (Open Services Gateway initiative) is a Java framework for remotely deployed service applications, which provides high reliability, collaboration, large scale distribution and wide-range of device usage. With an IaaS Framework based solution, the GSN can easily be extended to cover different layers and technologies. Through Web interfaces, users may determine GHG emission boundaries based on information providing VM power and energy sources, and then take actions to reduce GHG emissions. The project is therefore ISO compliant. Indeed, cloud management is not a new topic; however, the IaaS Framework is developed for the GSN because the project requires an open platform converging server and network virtualizations. Whilst most of cloud management solutions in the market focus particularly on computing resources, IaaS Framework components can be used to build network virtualized tools [9], allowing to flexibly set up data flows among data centers. The ability of incorporating third-party power control components is also an advantage of the IaaS Framework. Indeed, the GSN is built on top of CANARIE network, and links multiple data centers with different network architectures and paradigms, such as IP, Ethernet, MPLS (Multiprotocol Label Switching), optical. It is also required that new protocols and architectures be deployed independently without disruptions when the GSN grows. Network virtualization is therefore an appropriate solution because it allows the coexistence of different network architectures, including legacy systems. At the physical layer, we use Argia [9], a commercial version of UCLP (User Controlled Light Paths) [10]. Argia is a network virtualizer allowing end-users (people or applications) to treat network resources as software objects and provision and reconfigure optical lightpaths within a single domain or across multiple, independently managed domains. Users can join or divide lightpaths, as well as hand off control and management of their private sub-networks to other users or organizations. With a focus on optical switches, Argia enables the virtualization of a network that can be reconfigured by end-users without any interaction by the optical network manager. Layer 2 virtualization is achieved by Ether [9], which is similarly in concept to Argia, except that is designed for LAN environments. With a focus on Ethernet and MPLS networks, Ether allows users to acquire ports on Enterprise Switch and manage VLANs or MPLS configurations on their own. At the network layer, a network virtualizer created by the MANTICORE project [12] is deployed. MANTICORE is specifically designed for IP networks with an ability to define and configure physical and/or logical IP networks. It allows infrastructure owners to manage their physical as well as logical routers and to enable third parties to control them. MANTICORE also provides tools to assist infrastructure users in the creation and management of IP networks using router resources of one or Digital Object Indentifier /MIC /$ IEEE

9 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. more infrastructure owners. GHG Reduction: Estimation and Discussion Key power consuming elements in a network are data centers, core network and access network. As data centers are connected directly to the core network in the GSN, access network is not considered. In each GSN node, servers are installed in outdoor enclosures where the associated climate control is powered by green energy. So, GHG emissions from these accessories can be ignored. Current experiments are performed in data centers using an application named GeoChronos [11]. It is an infrastructure enabling the earth observation community to share data, scientific applications and to collaborate effectively. The application runs on a multi-processor server system with 48 cores in total. As a core processor consumes about 78W, the whole system consumes annually about 32.8MWh of non-renewable electricity. Prior to the GSN project, the system was powered by the power grid of Alberta fueled mostly by fossil energy, where the electricity emission factor is 930x10-6 ton/kwh [13]. Thus, the system s emission is more than 30 tons of CO2 annually. This number does not account for emissions of local switches and routers. The GHG emission of core network when migrating the data center hosting Geochronos towards a green energy source is estimated as follows. Since the GSN is an optical based network, a power consumption model for core routers proposed in [7] can be used, with a note that data centers are connected directly to the core network by optical links. Assuming the current network with 9 core nodes, the capacity of each core router is 640Gbps, the power of a router is 10.9kW, the size of each VM is 4GB (live data), the bandwidth required for each VM migration is 32Mbps, a link between two data centers is 10Gbps, and a data center is migrated in average twice a day, the power consumed by the core network for the migration of a data center is therefore 2kW/day, which is equivalent to 0.7 tons of CO2 emission annually [13]. If core nodes are all powered by fossil energy, the carbon credit saved by the GSN is = Our result remains conservative (i.e. under estimated) because an optical switch consumes less energy than a core router. In other words, we may effectively save more carbon credits. When the network grows, the amount of CO2 emitted during migrations is still very small compared to the total emission of data centers. According to our calculation model, the GSN may save up to 986 carbon credits when data centers include 1560 CPUs, assuming that the core network has 9 nodes. The GSN data centers, however, would not scale up to mega size, like Google or Microsoft ones, due to construction and energy provision costs. Moving large-scale data centers is costly in terms of network bandwidth and time. Moreover, leaving a large data center non-operational in some periods of time is not a cost-effective solution. Therefore, an appropriate solution addressing the carbon footprint of large data centers is two-folds: i) Virtualizing the data center, and then ii) Spanning it over multiple smaller physical data centers, each powered by renewable energy sources. That way, mega scale service would be powered entirely by renewable energy. Digital Object Indentifier /MIC /$ IEEE

10 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. As shown throughout the GSN project, having highspeed optical network is essential for providing neutral carbon services. As migrations could be realized frequently as a result of environmental change, bandwidth consumed by migrations would be huge. Thus, highly scalable networking infrastructure, such as CANARIE or Internet2 is required, leading to cost considerations. However, it is clear that data network is much cheaper than power transmission network. In addition, energy losses in optical network are very small compared to electrical grid. This makes neutral carbon network a realistic solution both for environment and economy, particularly when carbon tax is imposed. Conclusion In this paper, a first nation-wide network powered entirely by green energy sources in Canada is presented. Virtualization techniques are shown to be the most appropriate solution to manage the network and to migrate data centers following green energy source availabilities, such as solar and wind. With an increase in power consumption for ICT, the GSN is a promising model to deal with GHG reporting and carbon tax problems for ICT organizations, especially small and medium ones. Acknowledgement This research is funded by CANARIE Inc. The authors thank all Canadian and international partners for their contribution in the GSN project. References [1] P. Kurp, Green Computing, Are You Ready For A Personal Energy Meter?, Communication of ACM, 51(10), [2] The Climate Group, SMART2020: Enabling the low carbon economy in the information age, Report on behalf of the Global esustainability Initiative, [3] Qureshi A. et al., "Cutting the electric bill for internet-scale systems," ACM Computer Communication Review, 39(4), [4] HD. Saunders, "The Khazzoom-Brookes Postulate and Neoclassical Growth," The Energy J., 13(4), [5] S. Figuerola et al., "Converged Optical Network Infrastructures in Support of Future Internet and Grid Services Using IaaS to Reduce GHG Emissions," J. of Lightwave Technology, 27(12), [6] The GreenStar Network Project. [7] J. Baliga et al., Energy consumption in optical IP networks, J. Lightwave Technology, 27(13), [8] M. Lemay, An Introduction to IaaS Framework, 8/ Digital Object Indentifier /MIC /$ IEEE

11 This article has been accepted for publication in IEEE Internet Computing but has not yet been fully edited. Some content may change prior to final publication. [9] S. Figuerola, M. Lemay, "Infrastructure Services for Optical Networks [Invited]," J. Optical Communications and Networking, 1(2), [10] J. Wu et al., Layer 1 Virtual Private Network Management by Users, Communications Magazine, 44(12), [11] C. Kiddle, GeoChronos: A Platform for Earth Observation Scientists, OpenGridForum 28, 3/2010. [12] E Grasa, et al., The MANTICORE Project: Providing users with a Logical IP Network Service, TERENA Networking Conference, 5/2008. [13] LivClean Carbon Offset Solution, How is this Calculated?,. [14] V Valancius, et al., Greening the internet with nano data centers, CoNEXT'09, [15] K. Church, et al., On delivering embarrassingly distributed cloud services, HotNets-VII, Digital Object Indentifier /MIC /$ IEEE

12 This page intentionally left blank This page intentionally left blank This page intentionally left blank

13 Enabling Infrastructure as a Service (IaaS) on IP Networks: From Distributed to Virtualized Control Plane Kim Khoa Nguyen 1, Mathieu Lemay 2, Mohamed Cheriet 1 1 Ecole de Technologie Superieure, University of Quebec, Canada, 2 Inocybe Inc., Canada Abstract Infrastructure as a Service (IaaS) is considered a prominent model for IP based service delivery. As grid and cloud computing have become a stringent demand for today's Internet services, IaaS is required for providing services, particularly private cloud, regardless of physical infrastructure locations. However, enabling IaaS on traditional Internet Service Provider (ISP) network infrastructures is challenging because IaaS requires a high abstraction level of network architectures, protocols, and devices. Network control plane architecture plays therefore an essential role in this transition, particularly with respect to new requirements of scalability, reliability and flexibility. In this paper, we review the evolutionary trend of network element control planes from monolithic to distributed architectures according to the network growth, and then present a new virtualization oriented architecture which allows infrastructure providers and service providers to achieve service delivery independently and transparently to end users based on virtualized network control planes. As a result, current ISP infrastructures will be able to support new services, such as heavy resource consuming data center applications. We also show how to use network virtualization for providing cloud computing and data center services in a flexible manner on the nation-wide CANARIE network infrastructure. Introduction Nowadays, converged communications is considered the main evolutionary stream in the telecommunications industry, allowing underlying infrastructure to run multiple services and accessible over multiple devices. As more services are enabled on communication equipments, the line between software applications and communications applications is blurring. Traditional communications applications such as point-to-point conference or multi-cast are now requirements of business or entertainment software [1]. The aforementioned situation has an important impact on network and software solutions. Architects of network solutions will be required to understand the underlying network architectures, devices and protocols that will be used to access services. Additional frameworks and tools will be required to abstract details of the network environment. Application developers will also need to understand the protocols that will be used within their applications. These requirements will eventually lead to a

14 convergence of network control plane architectures and software solutions. In addition, the deployment of new services entailed the introduction of new protocols and link bandwidth has upgraded from megabit to gigabit rates on the Internet. New devices and protocols also increase the number of interfaces in network elements. Additional overhead resulted from more control traffic among an increasing number of peers results in difficulty of scalability, highly availability, and robustness to control plane. It appears that traditional network elements (e.g., IP routers) with centralized control plane architectures will not be able to meet new control requirements. Distributed control plane (DCP) architectures [2] are emerging solutions and have been widely used in core networks. However, the complexity and flexibility of new services, such as cloud computing and smart grids, impose new challenges on the design of network control plane architectures. In order to satisfy tremendous demands of resources from new applications, cloud computing is used to power next generation data centers and to enable service providers to lease data center capabilities for deploying applications depending on user requirements. As cloud applications have various configurations, deployment requirements, description and finding, quantifying and allocating resources are essential in order to deliver on-demand services, particularly in large scale systems. Today, ISPs are faced with an increased competition in the "bit-pipe" [1], a business model based purely on connectivity as a utility, with both lower revenue and lower margins. The bit-pipe model, rather than emphasizing content and services, is driven by operational excellence. Infrastructure consolidation, process automation, operational outsourcing are key mechanisms to reduce ISPs operating costs, driven by IP technology. New services, such as cloud computing with a huge number of resources to be managed, have placed the ISPs on the way of a new control plane evolution from current distributed architectures [3]. We witnessed that virtualization, a new paradigm being explored by the research and education community dealing with highly complex distributed environments, is an emerging technology for next generation network control plane architectures. Regarding current trends of virtualizing practically every aspect of computing (e.g., operating systems, servers, and data centers), it is necessary to have a virtualized network to interconnect all other virtualized appliances to give each of the virtual entities a complete semblance of their counterparts. Main characteristics of a network virtualization technique include: i) A warping of network elements, such as connectivity resources and traffic processing resources, ii) Dynamic establishment capability, such as flexible and efficient mechanisms to trigger and tear down service, iii) End-to-end across multiple domains, and iv) Control by the end-user, e.g., the end-user is able to operate the virtual infrastructure as if it was a dedicated physical infrastructure. This paper discusses the need of a transition from current network element s DCP architectures to network virtualization techniques in order to support new services. The rest of the paper is organized as follows. In the next section, we review network control plane architectures and focus on the currently used distributed architecture. We next investigate a paradigm of IaaS based on network virtualization.

15 Network virtualization tools are then presented with a proposed integrated control plane architecture and business and deployment models. A case study shows the deployment of network virtualization to provide cloud computing and data center services in a nation-wide network. Finally, we conclude the paper and present future work. Network Control Plane Architectures One of the key factors that enable of the extraordinary growth of the Internet is the evolution of network element architectures. As essential element of Internet, the IP router has developed from simplistic packet manipulating software implemented on a general purpose computer to a sophisticated network equipment that fully utilizes the capabilities of specialized hardware and integrates a set of functionalities, ranging from raw packet forwarding through traffic shaping, packet queuing, access control, with connection tracking all the way to distributed network protocols. Lately, it has been proposed to modularize these interspersed functions and organize them into administratively and physically distinct modules, yielding what is called the distributed router. Such distributed routers are expected to improve scalability of IP routers, open up new markets for device vendors and foster rapid innovation in the area. Control Plane Evolution The first IP networks were made with first generation routers (Figure 1-A) which contain a single central processor (CPU) and multiple interface cards interconnected through a shared bus. The CPU runs a commodity real-time operating system and implements the functional modules, including the forwarding engine, the queue manager, the traffic manager, and some parts of the network interface, especially Layer 2/Layer 3 processing logic in software. The central CPU capacity is shared among packet forwarding, running routing protocols, updating routing tables and achieving management functions.

16 Figure 1: Router generations When routers are upgraded to the second generation (Figure 1-B), more intelligence is added to the line cards, with processor, memory and forwarding caches, allowing them to perform locally some packet forwarding operations. However, control and forwarding planes still remain on the same processing unit. The third generation routers (Figure 1-C) were introduced with the concept of strict separation of control plane (software based) and data plane (hardware based), allowing the growth of service provider networks. As the shared bus is replaced by a switch fabric, which allows multiple packets to be simultaneously transferred across, data forwarding performance is significantly increased. Distributed Control Plane Architecture Due to the growing expansion of ISP networks, a network element may have to exchange control messages with hundred of peers. Such a growth in bandwidth, network traffic and network element density imposes several challenges when designing a network control plane. Particularly, the evolution

17 of traditional communication networks into multi-service networks requires control planes to be highly scalable, reliable and flexible. The monolithic architecture, where software and hardware are intertwined into a single, complex system has many limitations, which made it difficult to meet new requirements, and which has hold back the introduction of new services and applications. For example, a change in one of its subsystems may affect many other subsystems; the flexibility and performance is also limited due to its inherent complexity. Although third generation routers are still used in many of today's core networks, large ISPs are transforming their network equipments into optical based where data plane will includes optical crossconnects which provide services for IP layer through MPLS or similar technologies. This results in the development of DCP architectures, which are entirely separated from the data plane (Figure 2). Figure 2: Distributed control plane A DCP architecture [3][12] is based on the physical separation between control functions and forwarding functions. Control functions are reduced to the minimum in line cards, such as Hello protocols, neighbor discovery, and switch-over in case of failures. Control elements (CE) and forwarding elements (FE) are interconnected using an internal network, which carries control and data traffic between the elements. The internal network can be designed in various ways, using often highspeed optical network or high performance switches. Such an architecture involves three types of communications: CE-CE, CE-FE and FE-FE. ForCES [11] was introduced by IETF as a protocol for communications between elements. However, many network operators and equipment providers developed their own version of internal protocols, which seem similar to ForCES with specific features [3]. The separation of control elements from forwarding elements enables the control plane to handle complex tasks such as traffic engineering, QoS, VPN, in large scale networks. Challenges of a distributed architecture include determining function to be distributed and network element that will host that function. Solutions result in two ways of distributing control plane functionalities: functional and layered distribution [2].

18 Today, most of network operators have upgraded their control planes to distributed architectures, composed of multiple separate elements communicating through open, well-defined interfaces. The control plane and data plane are completely decoupled, running on two different devices, with a 1:N relationship, a control plane handling multiple forwarding planes. Several distributed schemes have been proposed in order to increase significantly scalability, flexibility and availability [3]. However, regarding the rapid growth of the number of network devices and VPNs needed for new cloud computing services, DCP will unlikely meet new requirements, particularly as it is related to high flexibility. Since control plane is linked to data plane by an internal network, handling change is difficult because each change to the physical infrastructure requires a corresponding modification to the control plane, such as reconfiguring the tunable parameters in the routing protocols. Infrastructure as a Service (IaaS) for Internet Provider The growing utilization of real-time services such as network telephony, video conference, has resulted in a higher need for constant connectivity, which requires more scalable management. Infrastructure as a Service (IaaS) has been introduced to meet new management requirements. Offered by Amazon, BlueLock and other companies as a renting hardware service using proprietary solutions, IaaS scales service delivery as the physical location of the infrastructure can be determined in a flexible way. This is the base of cloud architecture, where complex underlying services remain hidden inside the infrastructure provider. Resources are allocated according to users need, hence highest utilization and optimization levels can be achieved. During the duration of the service, the user owns and controls the infrastructure as if he was the owner. To ISPs perspectives, an IaaS solution allows: (i) Scaling cloud service. Since network is extensively used in cloud-based services, IaaS allows organizations to build a separate network dedicated to services they provide, given the flexibility and expected easy way of creating virtual infrastructures. IaaS opens new ways of building a backbone, particularly for private cloud customers, leading to converge routing capabilities in more centralized locations. (ii) Slicing packet-based infrastructure: If a physical device is sliced in virtual elements, it might be desirable to run different software versions on each slice. This concept already implemented in computers is now being deployable in routing systems. The virtualization allows also the upgradability of a software version to be achieved without disruption of services. (iii) Programmable network systems: a trend we observe in the industry is to integrate in the infrastructure new services up to the application level, increasing the value of the network that can be exposed to end-users. It requires more flexible ways of implementing extension of network device software, facilitating third party development and partnerships. This approach

19 (iv) of programmable infrastructure systems, such as via an operating system SDK on routers, or a standard protocol such as OpenFlow [13], will open a new dimension of innovation in communications industry. Scaling the management and service delivery: this is undoubtedly the most important concern for ISPs, in particular related to mobility in a multi-domain environment. The service delivery model used by current ISPs, which tightly couples services to the underlying transport network, fails to deliver the flexibility needed by ISPs for service innovations. ISPs need an IaaS framework that deals with service and transport independently. In addition, they want to reduce costs through service automation and streamlining of regulatory compliance. Network Virtualization The deployment of the IaaS model on current networks requires involving multiple network solutions and architectures and enables multiple networks to function as a whole. The DCP approach, which links control planes to data planes within a network element, is unable to meet this requirement. Virtualization is therefore a natural evolution from DCP architectures, since it allows the coexistence of different network architectures, including legacy systems. Network virtualization divides traditional Internet Service Providers (ISPs) into two independent entities: Infrastructure Provider, who manages the physical infrastructure, and Service Provider, who creates virtual networks by aggregating resources from multiple infrastructure providers and offer endto-end services [4]. Each service provider leases resources from one or more infrastructure providers to create virtual networks and deploys customized protocols and services, taking into account performance, topology and cost of each infrastructure. A virtual control plane (VCP) contains a network slice formed by virtual instances hosted by the physical networks. As virtualization instances are managed on a different system, the control plane scaling and resource allocation can evolve considerably and independently of the data plane. Adopting hybrid optical/packet based approach for the transport layer, the lightpath paradigm is key technology. However, we have also recognized a trend in communications industry that is to move from point-topoint deterministic pipes into the packet based transport infrastructure, such as MPLS over Ethernet. This move, from circuit oriented technologies to packet based technologies, requires also better cooperation between the packet based systems and optical cross connect systems. When the coexistence between these two trends remains, an integrated control solution based on virtualization is in need. A traditional ISP network will therefore be virtualized as in Figure 3. The Physical layer includes forwarding elements, which can be optical switches or IP routers. Each forwarding element, or a set of forwarding elements of the same kind, is managed by a VCP instance. For IP routers, the VCP contacts

20 with the routers control plane in order to setup entries in routing tables. Networking services are provided to users through the Service layer of VCPs. Figure 3: Virtualized control plane network The challenges for a virtualization solution include: i) Virtualization of network devices, such as physical equipments from different vendors, routing software, multiple configuration protocols, APIs, etc., ii) Virtualization of routing policies, in order to provide users with the ability to express potentially complex requests in a simple way, iii) Federation of user-defined autonomous systems, that allows users to create their own IP domains and choose to what other IP domains they want to peer with, and iv) Integrate lower layer resources in a converged management fashion. Virtual network solution by its nature gives full advantages for cloud-based systems. Although most of known VCP products are still developed in research projects [4][9], some commercial cloud systems, e.g., Flexiscale [14], are seen having virtual network features allowing users to rent VPNs together with virtual servers regardless of the locations of their physical servers. Nevertheless, large-scale providers, e.g., Amazon or Google, might still be concerned of performance issues when expanding their services to public utilization, resulting in no network virtualization feature has yet been present in their public cloud products. Thus, we believe that at the time-being, virtual network is more appropriate for medium-and-small enterprise customers. Incoming products from hardware providers, like Juniper Networks Control System (JSC1200) or Cisco IOS XR, fully support virtualization features. However, they focus on hardware virtualization of the point of presence (POP), while a cloud-based system needs a more flexible solution at the software level. In addition, there is a difference between public cloud and private cloud services. Public cloud services,

21 e.g., Amazon EC2, do not allow users to reconfigure their networks of virtual servers because of performance and security issues. However, private cloud services will potentially be provided with virtual network solutions. For example, Amazon Virtual Private Cloud will allow users to have complete control over their virtual networking environment [15]. Virtualized Network Control Plane Architecture A VCP architecture for network element is shown in Figure 4. It is built on top of network elements (e.g., optical cross-connect) used as data plane. The proposed architecture has been used to develop a set of VCP software, including UCLP (User-Controlled LightPaths) [5]. The Lookup Service is used to find the different instances of network elements in the network. Network Service Access Point (NSAP) advertises its service instantiation through a well-known process described by OSGi implementation such as a Web Services Description Language (WSDL) pointer or Universal Description, Discovery and Integration (UDDI) database. When a client application wants to use a NSAP service, it sends user requests to the NSAP using the Simple Object Access Protocol (SOAP) adopted for Web Services. The requests are then converted into procedure calls within the NSAP which then performs the calls on its local Service Access Point where commands are executed with the help of the other components within the system. The Traffic Engineering Service implements a set of methods to create end-to-end connections. It supports concatenating, partitioning, receiving request, using and releasing paths. There are two types of users. Normal users may invoke a connection request to create a new end-to-end connection, and the administrator may perform administrative functions, such as adding new paths, deleting paths according to changes in the physical layer and the allocating new resources (i.e., network elements). Finally, the Network Element Service encapsulates the communication protocol required to communicate with the managed network device.

22 Figure 4: Virtual control plane architecture In a traditional networking environment, routing protocols assemble routing tables used to find available resources for routing a new connection through a given network. However, no standard is available for inter-domain routing in optical networks and full knowledge of network topology as normally used for intra-domain routing is not appropriate for customer-managed networking. Therefore, when the control plane is implemented for optical network, an ad-hoc path searching mechanism can be used based on a static database, which is updated by the Lookup Service. In order to provide interfaces to upper layers, the NSAP defines management services in the context of Web Services standards, based on XML and SOAP. The XML-based SOAP protocol is used for remote method invocation. There is also a service directory where VCPs can register their list of services specified in terms of XML schemas. Client applications search this directory to find desired services and corresponding VCPs. Such a VCP can be hosted by a server separately from the network element it manages. This allows the control plane to be implemented using robust software platforms, e.g., J2EE/OSGi. Comparison of distributed and virtual control plane Regarding the complexity in management of DCP due to command line interfaces, VCP offers a clear advantage as it allows both end-user and administrator to configure network through a user-friendly GUI interface with different access levels, thus reducing the risk of errors committed by users. The

23 scalability of distributed model depends on the capacity of devices, whilst a VCP running on a dedicated server is able to manager many devices or even multiple networks. This significantly reduces the capital and operational expenditures (CAPEX/OPEX) of network providers. In addition, a DCP based on hardware components is more costly than a VCP, which is software-based. As VCP is programmable, drivers can easily be added in order to control a wide range of devices and support traffic engineering features, which is inextensible in DCP model. Similarly, security mechanisms can be implemented in a VCP for authentication, access control and user management, while security features in DCP focus mainly on protocol level. Another advantage of virtual control model is that it is by nature very flexible and easily customized. However, whilst DCPs have widely been adopted and standardized by many equipment providers, most of VCPs are still in research and finalizing phases. The aforementioned discussions are summarized in Table I. Control Distributed control plane Virtual control plane Management role Hard and error prone Administrator only Easy and user friendly with GUI End-user / Administrator Scalability Scale with device capacity Scale with network and server capacity One control plane can manage multiple devices Price Expensive, due to hardware components Cheap, as software-based component Traffic engineering/qos Limited by operators and device features. QoS is based on routing protocol extensions (e.g., RSVP-TE or NSIS) Very flexible and easily compatible with a wide range of devices. QoS is based on bandwidth broker and load balancing Security Protocol level Cover from underlying protocol level to application level Flexibility Hardware can be changed or upgraded Very flexible, customizable by end-users Standardization Standardized by many equipment providers No standard has yet been defined Table I. A comparison of distributed and virtual control planes Service delivery model The proposed IaaS service delivery model based on VCPs, as shown in Figure 5, consists of four layers. The Infrastructure layer includes physical network devices owned by infrastructure providers. These devices are usually linked within provider networks. The Virtualization layer includes servers running VCP software used to control the physical devices, such as setup and tear down on-demand paths. Bandwidth Broker is also implemented in this layer to enable traffic engineering services. The Service layer provides VCP service capabilities based on IaaSFramework components [6]. It also authenticates and handles access authorization to VCPs, enforces policy and generates usage record. The top Management plane or User level focuses on application services by making use of services provided by the lower layer services.

24 Figure 5: Layers of IaaS service delivery model In such a model, the Resource Lists Service provides the means of exchanging resources between services providers (SPs). Each SP has a resource list populated with VCPs that represent the physical network elements that the SP can access. The list of SP (A) will be sent to SP (B) when A wants to give B permission to access some of A s resources. B may then assign the network resources it receives to the network application services that B are deploying. A resource broker site, such as V-Infrastructures [6], can be used to provide SPs with resource listing, defining, browsing, and bargaining functionalities. In a typical system configuration, each SP has a set of services supported by the V- Infrastructure, including web service bundles. Although the sharing of resources among different SPs is enabled, it is important to keep administrative boundaries between SPs to avoid confusion about the ownership of assets and administrative privileges. Example: Cloud Computing Service Provisioning on CANARIE Network We now investigate an example of using virtualization techniques for providing cloud computing and

25 data center services on top of a nation-wide optical network infrastructure. Figure 6-A shows CANARIE (previously named CA*net 4), a shared network used by all the provincial Optical Regional Advanced Networks (ORANs) across Canada. It links each provincial ORAN by a set of wavelengths that can be shared among them. CANARIE provides 10Gbps optical lightpaths for research and education through multiple optical cross-connects. Based on CANARIE infrastructures, the GreenStar Network (GSN) project aims at reducing greenhouse gas emissions (GHG) arising from ICT services [7]. The GSN is made of a set of data centers linked by CANARIE, and connected to USA, Europe and Asia Pacific. The data centers are powered entirely by green energy sources, such as sun, wind, geothermal and hydroelectricity. The idea behind the GSN project is that a zero carbon network must consist of data centers built in proximity to green power sources and user applications will be moved to be executed in data centers, assume that losses incurred in energy transmission over power utility infrastructures are much higher than those caused by data transmission [8]. Such a network must be highly flexible in order to migrate an entire virtual data center (including virtual routers and servers) to alternate locations because green energy sources, like solar and wind are intermittent. Thus, the key challenge of the GSN is that the network has to move virtual servers around its nodes according to green power availability. Unfortunately, hypervisors (e.g, KVM, XEN) running virtual servers can only migrate virtual servers within a flat network. As the GSN spans multiple domains, VPNs must dynamically be reconfigured when a migration is triggered. Without VCPs, this task is very costly in terms of management as migration events are not predictable. A) GreenStar Network (Canadian portion) built on top of CANARIE

26 B) Physical connection of the GreenStar Network with VPNs established at 10AM, Feb. 22, 2011

27 C) Delay (IP traffic) between Montreal and Calgary measured during 24 hours in a regular IP routing network and on a GSN lightpath setup by a Layer 1 virtual control plane Figure 6: The Green Star Network and traffic characteristics In such a network model, CANARIE is infrastructure layer. Virtualization layer consists of VCP software. Service layer is a middleware we have implemented based on IaaSFramework, which brings services offered by VCPs to GSN users. Each VCP is considered as a resource in the middleware. The management layer handles user policies of network slices. Service provider is GSN operator, and endusers are data center service consumers. Service delivery is achieved in the GSN by VCPs for network elements at three layers. At the physical layer, we use Argia [9], a commercial version of UCLP [5]. Argia is a VCP that allows end-users (humans or applications) to treat optical cross-connects as software objects and provision and reconfigure optical lightpaths within a single domain or across multiple, independently managed domains. Users can join or divide lightpaths, as well as hand off control and management of their private sub-networks to other users or organizations. With a focus on optical switches, Argia enables the virtualization of a network element that can be reconfigured by the end-user without any interaction by the optical network manager. In order to establish Layer 2 VLAN, a network virtualization tool, named Ether [9], is used. Ether is similarly in concept to Argia, except that is designed for LAN environment. With a focus on Ethernet and MPLS networks, Ether allows users to acquire ports on an Enterprise Switch and manage VLANs

28 or MPLS configurations on their own. At the network layer, a VCP created by the MANTICORE project [10] is deployed. MANTICORE is specifically designed for IP networks with an ability to define and configure physical and/or logical IP networks. It allows infrastructure owners to manage their physical as well as logical routers and to enable third parties to control the routers. MANTICORE also provides tools to assist infrastructure users in the creation and management of IP networks using router resources of one or more infrastructure owners. As shown in Figure 6-B, data centers of the GSN is linked by several types of equipments, including optical cross-connects in the core CANARIE network, Layer 2 switch of local networks and IP routers. Therefore, reconfiguring and setting up a VPN within such a network is very challenging. In addition, the VPN needs to be flexible, i.e., its topology can be changed dynamically. VCPs allow GSN operators to enable user-controlled traffic engineering. So, networks within a single domain or across multiple independent domains can be self-provisioned and dynamically reconfigured. For example, an optical connection is setup as follows. When a path is requested between Montreal and Calgary as shown in Figure 6-B, resource objects representing ports on each switch is defined. Next, a VCP (i.e., Argia) creates a link object representing an optical connection between two switches. Then, a path is allocated. When the path is required on a SONET network (e.g., CANARIE core network), VCP deals with 10GE WAN PHY protocol. After the 10GE WAN PHY signal is converted to 10GW LAN PHY signal in Calgary, it goes to the Allied Telesis switch. The path allows virtual servers to be migrated from Calgary to Montreal as on the same LAN environment. Figure 6-C compares the delay of IP traffic between Montreal and Calgary nodes in the GSN when regular IP routing and VCP are used to establish connection. Data is collected during a period of 24 hours. In the regular IP routing service, data packets going through each intermediate switch need to be processed and sometimes converted by OEO (optical-electrical-optical) modules. This results in high delay that, in peak load periods, does not meet requirements for live migrations. VCP offers a more stable lightpath between two nodes compared to regular IP routing. As the CANARIE network is composed of multiple federated domains, each domain includes a set of network devices, VCPs are implemented on each domain to export available Resource Lists. Since the VCPs cover three underlying layers, GSN users get full control of all network elements. Network topology can therefore be reconfigured according to user requirements in a very flexible manner (e.g., changed at least two times a day) in order to move data centers following green power availabilities. Conclusion Along with the growing demand for new services on Internet, the network control plane has evolved in ISP networks through many generations. Distributed control plane architectures are being widely used. However, they are facing issues of scalability and flexibility requirements of new cloud computing

29 services. Therefore, we believe that network virtualization is a more appropriate solution, particularly in cases of very elastic networks as shown in this paper. The virtualized control plane architecture we presented have been used in a number of research and educational projects and proven to be flexible and efficient tools. Our future work will address evolving such network virtualization tools, driven by need of new complex networks, and implementing advanced optimization techniques for traffic management. References [1] A. Cuevas et al., The IMS Service Platform: A Solution for Next Generation Network Operators to Be More Than Bit Pipes, IEEE Commun. Mag., vol. 44, no. 8, 2006, pp [2] K.-K. Nguyen et al., Towards a Distributed Control Plane Architecture for Next Generation Routers, Proc. of ECUMN'2007, 2007, pp [3] K.-K. Nguyen, B. Jaumard, A. Agarwal, A Distributed and Scalable Routing Table Manager for Next Generation IP Router, IEEE Network Mag., vol. 22, no. 2, Mar. 2008, pp [4] N. M. Mosharaf Kabir Chowdhury, Raouf Boutaba, Network Virtualization: State of the Art and Research Challenges, IEEE Commun. Mag., vol. 47, no. 7, 2009, pp [5] E. Grasa et al., UCLPv2: A Network Virtualization Framework Built on Web Services, IEEE Commun. Mag., vol. 46, no. 3, 2008, pp [6] M. Lemay, An Introduction to IaaS Framework, 8/ [7] The GreenStar Network Project. [8] S. Figuerola et al., Converged Optical Network Infrastructures in Support of Future Internet and Grid Services Using IaaS to Reduce GHG Emissions, J. of Lightwave Technology, vol. 27, no. 12, 2009, pp [9] S. Figuerola, M. Lemay, "Infrastructure Services for Optical Networks [Invited]," J. of Optical Communications and Networking, vol. 1, no. 2, 2009, pp. A247-A257. [10] E Grasa et al., The MANTICORE Project: Providing users with a Logical IP Network Service, Proc. of TERENA Networking Conference, 5/2008. [11] A. Doria et al., RFC 5810: Forwarding and Control Element Separation (ForCES) Protocol Specification, IETF Draft, IETF - Network Working Group, [12] Császár et al., Converging the evolution of router architectures and IP networks, IEEE Network Mag., vol. 21, no. 4, 2007, pp [13] N. McKeown et al., OpenFlow: enabling innovation in campus networks, ACM SIGCOMM Computer Communication Review, vol. 38, iss. 2, 2008, pp

30 [14] FlexiScale Inc., Flexiscale cloud computing, [15] Amazon Inc., Amazon Virtual Private Cloud (Amazon VPC),

31 Bibliography KIM KHOA NGUYEN is a Research Fellow in the Automation Engineering Department at the École de Technologie Supérieure (University of Quebec). He is key architect of the GreenStar Network project and a member of the Synchromedia Consortium. Since 2008 he has been with the Optimization of Communication Networks Research Laboratory at Concordia University. His research includes green ICT, cloud computing, smart grid, router architectures and wireless networks. He holds a PhD in Electrical and Computer Engineering from Concordia University. MATHIEU LEMAY holds a degree in electrical engineering from the École de Technologies Supérieure (2005) and a master s in optical networks (2007). He is currently a Ph.D. candidate at the Synchromedia Consortium of ETS. He is the Founder, President, and CEO of Inocybe Technologies Inc. He is currently involved in Green IT and he is leading the IaaS Framework Open Source initiative. His main research themes are virtualization, network segmentation, service-oriented architectures and distributed systems. MOHAMED CHERIET is a Full Professor in the Automation Engineering Department at the École de Technologie Supérieure (University of Quebec). He is co-founder of the Laboratory for Imagery, Vision and Artificial Intelligence (LIVIA), and founder and director of the Synchromedia Consortium since Dr. Cheriet serves on the editorial boards of Pattern Recognition (PR), the International Journal of Document Analysis and Recognition (IJDAR), and the International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI). He is a member of IAPR, a senior member of the IEEE and the Founder and Premier Chair of the IEEE Montreal Chapter of Computational Intelligent Systems (CIS).

32 This page intentionally left blank This page intentionally left blank This page intentionally left blank

33 January 12 Estimating Emission Reductions from Low Carbon Information Technology: The GeoChronos Relocation Project Paul Steenhof tel: ex 233, fax: (613) ) Chris Weber David Aikema Randall Robinson Rob Simmonds Cameron Kiddle Abstract In this article we present an information and communications technology (ICT) project that involved moving virtualized applications between data centres in order reduce greenhouse gas emissions. The emission reductions have been accounted for according to a protocol that is conformant with ISO , an international standard for quantifying and reporting emission reduction projects. While the total amount of emission reductions attributable to the project were relatively small, the results of the article are important and illustrative as this may be one of the first examples of trying to quantify a specific project that aims to reduce emissions through the provision of low carbon ICT services. With rapid growth being experienced in the ICT services sector generally and rising energy and environmental impacts of the industry as data management related electricity consumption increases, the topic of low or zero carbon ICT services is of increasing importance from a number of perspectives. Two are subsequently discussed in this article, including the possible place of ICT within carbon 1

34 markets and also the role of such ICT projects for helping large users of data management services reduce the environmental impact of the services that they require. Key words Green ICT, virtualization and cloud computing, emission reductions Acknowledgements The project and protocol described in this article were part of a larger project called the GreenStar Network and were supported by funding received from Canada s Advanced Research and Innovation Network (CANARIE). 1 Introduction Over the last two decades the global economy has been transformed by the increasing role of data across nearly every sphere of civil society, in turn contributing to rapid growth in the data centre industry as a provider of data processing, data storage, and other data services. With this growth, there has also been a parallel increase in the power requirements associated with the millions of servers and thousands of data centres now in existence. In the United States for example, the Environmental Protection Agency estimated power requirements from data centres doubled from 2000 to 2006 to equal about 61 billion kilowatt-hours (kwh), or 1.5 percent of the country s total electricity consumption (Energy Star, 2007). It was estimated then that by 2011 this figure would likely nearly double. Global estimates of electricity usage from data centres are similar. Kommey (2008) concluded that worldwide data center power 2

35 demand in 2005 was equivalent (in capacity terms) to about seventeen 1,000 MW power plants. The rapid growth of the data centre industry and the subsequent increases in electrical requirements are also contributing to increases in the greenhouse gas emissions (GHG) associated with the electricity required by the data facilities. In 2002, the ICT sector in its entirety had an estimated global carbon footprint of 0.53 gigatonnes of carbon dioxide equivalent (Gt CO 2 e), representing about 1.25 percent of global emissions, while by 2007, this equalled about 0.83 GT CO 2 e, or about 2 percent of global emissions (Benowitz and Samuel, 2010). By 2020, it is projected that the global ICT footprint, of which datacentres contribute significantly, will equal 1.43 Gt CO 2 e, representing 2.7 percent of the global total. Put another way, this represents a 170 percent increase in emissions in the 2002 to 2020 period, or 9 percent annual growth. There are a number of major technical changes which could help lower the energy and emissions footprint of ICT services in future years, particularly in terms of moving towards location-independent cloud computing and the benefits achieved through improving the energy and carbon performance of the data centre itself. The GeoChronos Relocation Project is an example of such activity where emission reductions have been achieved by moving computing services from a data centre located in Calgary, Alberta, situated in a carbonintensive grid, to a data centre in Kelowna, British Columbia, situated in a low carbon grid. 1.1 Article purpose This article presents a case study of an ICT project that has been implemented to reduce emissions where we apply a protocol developed for quantifying and reporting emission reductions from ICT projects. The specific protocol used was developed to be conformant 3

36 with the requirements of the international standard ISO : Specification with guidance at the project level for quantification, monitoring and reporting of greenhouse gas emission reductions or removal enhancements, and is entitled The ICT GHG Reduction Protocol (referenced to as the Protocol throughout) (CSA, 2011). The use of an ISO conformant protocol for quantifying and reporting emission reductions is important since most carbon markets and players within these require the use of accepted best practices or methodologies. ISO is widely viewed as one the preeminent international standards that provides guidance for GHG quantification and reporting. We conclude the article by discussing the role of similar projects in reducing GHG emissions and the relevance of this activity in respect to both carbon trading and improving the environmental footprint of companies, institutions, or other organizations that are large users of ICT services. 1.2 Article structure The remainder of this article is structured as follows. Sections 2 through 8 largely reflect the specific requirements of the Protocol and of ISO for quantifying and reporting carbon reduction projects. Specifically, in section 2 we describe the GeoChronos Project in detail, including the applicability of the Protocol to the project, the technologies involved, and the chronological plan of the project. In sections 3 through 6 we then present an assessment of the relevant emission sources, sinks and reservoirs (SSRs) that must be quantified for both the project and in baseline, including discussion of the selection and justification of the project baseline. Sections 7 and 8 then present the methodology and quantification results, while section 9 concludes. 2 Project description The project involved moving the GeoChronos application from a data center in Calgary to new leased machines located at the RackForce Gigacenter Datacenter in Kelowna, British 4

37 Columbia. GeoChronos is a platform leveraging web 2.0, social networking, and cloud computing technologies that is aimed at enabling members of the Earth observation science community to share data and scientific applications, and to collaborate more effectively. The platform facilitates the automated collection and management of data obtained at different spatial and temporal resolutions. GeoChronos is also a major contributor to the development of community-driven online spectral libraries. 2.1 Project applicability The Protocol provides guidance for assessing the applicability of the Protocol according to the type of activity involving ICT-related emission reduction projects or initiatives. This then helps direct users to specific methodological routines that may be most relevant to the project. To do so, the Protocol presents a typology of project types and use cases that project proponents must evaluate to determine with which their project aligns. This project typology is described as follows (CSA, 2011): Project Type 1: Project activities involving improvements to ICT facilities This project type includes project activities through which GHG emission reductions are achieved by changing the environment of the ICT facility, such as improving the efficiency of the facilities or changing the energy source to renewable energy. For these projects, the ICT equipment and workload (ICT services) must remain the same between the baseline and the project. Project Type 2: Project activities involving improvements to ICT services This project type comprises project activities through which GHG emission reductions 5

38 are achieved from an improvement in the delivery of ICT services. This could involve physical improvements in information and communication technologies (so as to achieve higher workload efficiency) or the migration of machines, applications, or other services from one ICT environment to another, including virtualization and consolidation projects. It could also include improvements in other ICT infrastructure such as networks. Type 2 projects are further subdivided into the following subprojects. (1) Colocation: The proponent's physical machines are moved from the original facility to a lower carbon facility. (2) Leased Machines: The proponent s original collection of physical machines at the original facility is replaced by a leased collection of physical machines at a lower carbon facility. (3) Low or Zero Carbon Clouds and Real-Time Workload Shifting: The proponent's collection of already-virtualized machines at one or more facilities is moved to one or more physical machines at one or more lower carbon facilities. (4) Special Purpose Hardware: The proponent s original collection of physical machines at the original facility is replaced by a collection of high-efficiency physical machines, either at the original or a new facility. 6

39 (5) Virtualization & Consolidation: The proponent s original collection of physical machines at the original facility is replaced by a collection of virtual machines, either at the original or a new facility. (6) Network Improvements: The project proponent improves and installs new network capacity within or between ICT facilities which is able to transfer more or the same amount of information using less power and creating less greenhouse gas emissions while meeting all of the same customer expectations. This project was determined to be a Type 2 as described in Use Case 2 project since it involved moving and operating the GeoChronos application on new leased machines in Kelowna as compared to physical machines at the University of Calgary. 2.2 Technologies involved The project involved moving the GeoChronos application from a datacenter at the University of Calgary to new servers at a new efficient datacenter in Kelowna, British Columbia. Therefore, the project involved actually moving and operating an application remotely using a combination of virtual machines on shared hardware as well as dedicated hardware. At the University of Calgary datacenter, GeoChronos used (as of December 2010) the following hardware systems: - A set of compute nodes that run VM s performing different tasks - A file server to serve VM images - A file server to serve data via NFS. 7

40 Each of the file servers were connected to a storage area network (SAN), while network switches were employed to move data between nodes and the file-servers and between the nodes and the external clients. In addition, as part of this project, once GeoChronos had been relocated to Kelowna, all the network and data traffic associated with GeoChronos users was still initially received at the University of Calgary and then redirected from the University of Calgary over a dedicated network link to RackForce. 2.3 Chronological plan Following project setup, the GeoChronos project was operational for eight months starting in February 2011, with data from the project monitored for three months starting in July Below is the detailed chronology of the GeoChronos relocation project: Mid-June - November 2010 Setup, configure, testing of nodes and cloud infrastructure December January Migration of GeoChronos development environment and testing February September GeoChronos site run at RackForce July September Monitoring data collected at RackForce October GeoChronos relocated back to the University of Calgary 2.4 Description of datacentres involved in project There are numerous datacentres operated at or for the University of Calgary, and the datacenter involved with this project is a legacy datacenter located in the basement a large 8

41 university building. Although a new combined heat and power plant was in the midst of being built to service the campus, at the time of the project the campus and the datacenter purchased their power from the Alberta electricity grid with diesel generators serving as a source of back-up power for some circuits. The datacenter is cooled from water which is pumped from the Bow River via the Bow River Pumping Station to a chiller plant located on the campus. The Bow River loop is an open loop and runs through the chilling plant to remove heat from the closed campus cooling loop. The Bow River loop also provides cooling water to the Foothills Hospital. The campus cooling loop runs from the chilling plant to a variety of facilities on campus and is constantly recirculated back to the chillers which remove the heat from the cooling loop and transfer the heat to the Bow River water loop. Chilled water flows through building air handling units and computer room air conditioning (CRAC) units during the year to provide cooling to the building and the datacenter. GeoChronos was moved to a datacentre owned and operated by RackForce Networks. RackForce Networks is a privately held ICT service provider based in Kelowna B.C., Canada. Founded in 2001, RackForce supports thousands of customers from over 100 countries. The multimillion dollar GigaCenter facility uses a scalable and highly efficient modular design and is built to be among the greenest and most advanced data centers in the world. 3 Identification of SSRs attributable to the project A life cycle assessment was performed on the project to determine the sources, sinks, and reservoirs (SSR) of GHG emissions that are controlled, related or affected by the project. The SSRs occurring during the project were identified first, and products, materials and energy inputs/outputs were traced upstream to origins in natural resources and downstream along the life-cycle. This allowed for the identification of upstream and downstream related SSRs before, during and after project operation. It was then determined whether each of the 9

42 identified SSRs was controlled by the project proponent, related to the project, or affected by the project. The project boundary was defined by the RackForce datacenter. The datacenter shares a building with offices which support the business of RackForce, including sales, management and administration, but which are not directly related to operation of the datacenter. As such, these offices are not considered to be within the project boundaries. Although a natural gas heating system provides heat to the office portion of the building during winter months, it was not included in the description of the project as it is not part of the datacenter. One of the key assumptions is that the addition of a small amount of leased servers to the RackForce datacenter is a small insignificant increase in the amount of ICT services that RackForce will be providing and as a result there are no significant increases in any SSR s (e.g. waste disposal, coolant leakage etc) associated with the normal operation of the datacenter. The SSRs associated with the project are illustrated in Figure 1, and described in Table 1. Figure 1: Identification of all Project SSRs 10

43 Rackforce SSR s Upstream During Project P7 Power Source(s) supplying the grid to meet demand from data centre P8 Network Traffic Upstream SSR s Before Project Onsite SSR s During Projects Power Generation onsite Development of Software to Support IT facility services P1 - Software Development P9 - Back-up Power Fuel Use P10 - Power Source Maintenance (P9) Manutacture of Supporting andit equipment P2 - Raw materials used in IT equipment P3 - Manufacture of IT Equipment P11 ICT equipment maintenance P12 HVAC system P4 -Transportation of IT equipment to facility P5 Installation of IT equipment, racks/ cabinets P6 Production of Diesel fuel Downstream SSR s After Project P13 - IT Disposal and Recycling P14 - Green IT takes off Table 1: Description of the SSRs controlled, related or affected by the project 11

44 Controlled / SSR Name # Description Related or Affected Upstream SSR s before project operation Software development P1 The development of the software that will be used to run the Related application on the leased servers and any applications needed to support the GeoChronos application, including supporting management software or cloud management software. Raw materials used in P2 The acquisition and/or treatment of the raw materials used in the Related ICT equipment manufacturing of ICT equipment (e.g. silicon, lead, copper, etc.). Manufacture of ICT P3 The manufacturing of the compute, network, storage and Related equipment supplemental equipment (KVM, monitors, workstations) needed to support the GeoChronos application at RackForce. Transportation of ICT P4 Transportation of the leased servers and related ICT equipment to Related equipment to facility RackForce for installation. Installation of ICT P5 Installation of all ICT equipment, including racks/cabinets, to Related equipment, rack/cabinets support GeoChronos in RackForce. Production of diesel fuel P6 Extraction and refinement of crude oil to produce diesel used for Related backup power. Upstream SSR s during project operation Electricity from the BC P7 Electricity generation including transmission line losses needed by Related Hydro power grid the data centre for operation and provision of ICT services including any electricity required for the supporting infrastructure that is not provided by onsite electricity. Network Traffic P8 The network traffic that users of GeoChronos generate in using the application at the RackForce datacenter Related and Controlled 1 Onsite Project SSR s Back-up Diesel P9 Operation of the diesel back up generator as needed to provide Controlled Generator backup power to the RackForce datacenter. Power source P10 Maintenance of the back-up power source. Controlled maintenance (P9) 1 Part of the network between the University of Calgary datacenter and the Rackforce datacenter in Kelowna is controlled by the project proponent and their partners and part of it is not controlled or a related SSR. 12

45 Controlled / SSR Name # Description Related or Affected ICT equipment P11 Additional maintenance of the servers and related ICT equipment Controlled maintenance and supporting infrastructure as a result of the suporting GeoChronos at RackForce. HVAC system P12 Additional use and leakage of refrigerant associated with Controlled operation of the HVAC at RackForce as a result of the GeoChronos relocations. Also includes any additional maintenance of the HVAC system, particularly to meet the cooling requirements of the data centre. Downstream SSR s after project termination ICT disposal and P13 Disposal and demanufacturing of servers and related ICT Related recycling equipment being used to support GeoChronos for recycling purposes, shredding of the hard drives, and transport of this equipment to recycling or waste facilities. Green ICT takes off P14 The project s success spurs a large growth of similar low carbon Affected ICT services. (market impact). 4 Selection and justification of the baseline scenario As stipulated in ISO , the baseline represents what would have occurred in the absence of the project, inclusive of energy sources and the technologies used. Here, the baseline technology must be able to provide the same product or service as the technology in the GHG project so that they may be directly compared. As stated, based on the project typology outlined in the ICT Greenhouse Gas Reduction Protocol, this project involved a Type 2 use case 2 project activity. For these project types, the process for determining and quantifying emissions reductions is outlined as follows; 13

46 Type 2: Project activities involving improvements to ICT services This project Type includes project activities where emission reductions are achieved from an improvement in the delivery of ICT services This could involve physical improvements in information and communication technologies so as to achieve higher workload efficiency or the migration of machines, applications, or other services from one ICT environment to another, including virtualization and consolidation projects. (2) Leased Machines: The proponent s original collection of physical machines at the original facility is replaced by a leased collection of physical machines at a lower carbon facility. The software packages are equivalent in the original and new implementations; however the assignment of software packages to physical machines may be different, and the physical machines comprising the new collection may be different in number and type from the original collection. The baseline for this type of project activity has been specified in the Protocol as the following; The standard baseline comprises substitution of the original physical machines with up-to-date commonly available physical machines at the baseline facility, using the project s assignment of software packages if possible. In the case of this project, following the requirements of the protocol it therefore has been assumed that the project proponent would have, in the absence of moving the application to 14

47 RackForce, installed up-to date commonly available physical machines at the current datacenter. These machines would have replaced the older original physical machines while remaining at their current facility at the University of Calgary. The baseline facility was assumed to be the current IT facility at the University of Calgary where GeoChronos is currently being hosted as operated and supported by the Grid Research Centre. 5 Identification of SSRs attributable to the baseline As was done in the project case, a life cycle assessment was performed on the baseline to determine and identify the SSRs of GHG emissions that are controlled, related or affected. The SSRs occurring during the baseline scenario were identified and a mass and energy balance was done on the various components to determine the upstream and downstream related SSRs before, during and after the baseline. These are illustrated in Figure 2, and described in Table 2. Figure 2: Identification of all Baseline SSRs 15

48 GRC Baseline SSR s Upstream During Project B7 Electricity from electricty grid in Calgary (EPCor) B8 Water Cooling of Data Center B9 Network Traffic Upstream SSR s Before Project Onsite SSR s During Projects Power Generation onsite Development of Software to Support IT facility services B1 - Software Development B10 - Back-up Diesel Generator B11 - Power Source Maintenance (B9) Manutacture of Supporting andit equipment B2 - Raw materials used in IT equipment B3 - Manufacture of IT Equipment B12 ICT equipment maintenance B13 HVAC system B4 -Transportation of IT equipment to facility B5 Installation of IT equipment, racks/ cabinets B6 Production of Diesel fuel Downstream SSR s After Project B14 - IT Disposal and Recycling Table 2: Description of the SSRs controlled, related or affected in the baseline Controlled / SSR Name # Description Related or Affected Upstream SSR s before project operation Software development B1 The development of the software that will be used to run the Related application at the Grid Research Center and any applications needed to support the GeoChronos application, including supporting management software or cloud management software. Raw materials used in B2 The acquisition and/or treatment of the raw materials used in the Related ICT equipment manufacturing of ICT equipment (e.g. silicon, lead, copper, etc.). Manufacture of ICT B3 The manufacturing of the compute, network, storage and Related equipment supplemental equipment (KVM, monitors, workstations) needed 16

49 Controlled / SSR Name # Description Related or Affected to support the GeoChronos application at the Grid Research Center. Transportation of ICT to B4 Transportation of the servers and related ICT equipment to GRC Related facility for installation. Installation of ICT B5 Installation of all ICT equipment, including racks/cabinets, to Related equipment, rack/cabinets support GeoChronos at GRC. Production of diesel fuel B6 Extraction and refinement of crude oil to produce diesel used for Related backup power. Upstream SSR s during project operation Electricity from B7 Electricity generation including transmission line losses needed by Related electricity grid in Calgary the data centre for operation and provision of ICT services (EPCOR) including any electricity required for the supporting infrastructure that is not provided by onsite electricity. Water Cooling of Data B8 The datacenter at the University of Calgary which supports Related Center GeoChronos is cooled with water taken from the Bow River. There emissions result from the energy (electricity) used to cool the water, power the air conditioners, withdraw the water and pump it through the loop to the university and the datacenter and back to the Bow River. Network Traffic B9 The network traffic that users of GeoChronos generate in using the Related application at GRC. Onsite Project SSR s Back-up Diesel B10 Operation of a diesel back up generator as needed to provide Controlled Generators backup power to the University of Calgary Datacenter. Power source B11 Maintenance of the back-up power sources. Controlled maintenance (B10) ICT equipment B12 Additional maintenance of the servers and related ICT equipment Controlled maintenance and supporting infrastructure as a result of the supporting GeoChronos at GRC. HVAC system B13 Additonal use and leakage of refrigerant associated with operation Controlled 17

50 Controlled / SSR Name # Description Related or Affected of the HVAC at GRC needed to support GeoChronos. Also includes any additional maintenance of the HVAC system, particularly to meet the cooling requirements of the data centre. Downstream SSR s after project termination ICT disposal and B14 Disposal and demanufacturing of servers and related ICT Related recycling equipment being used to support GeoChronos for recycling purposes, shredding of the hard drives, and transport of this equipment to recycling or waste facilities. 6 Selection of relevant SSRs for quantification or estimation of GHG emission reductions The protocol provides guidance in order to determine the relevant SSRs for quantification and reporting. The protocol s guidance was developed based on the following criteria that were applied according to the ISO requirements and principles: 1- If the SSR did not result in GHG emissions it was considered not relevant except if the activity level for this SSR was necessary to calculate the emissions of another SSR. 2- If the SSR was similar in both the project and the baseline scenario qualitatively and quantitatively then the SSR was considered not relevant since it would not have any impact on the GHG emission quantification 3- If the SSR was similar qualitatively in the baseline and the project scenario, and the GHG emissions were greater in the baseline than in the project scenario then it was possible to consider the SSR not relevant since it is conservative to do so. 18

51 Additional criteria for removing GHG sources from estimation and measurement include if they are deemed to be minor and too difficult to measure. Using the information and criteria listed above the following SSRs were considered either relevant or not relevant for this quantification (see table 3). Table 3. Comparison of SSR s and Selection of Relevant Ones SSR # Controlled SSR Name (Project / Related Relevancy and justification if not relevant Baseline) Affected Software development P1 / B1 Related Not Relevant. This SSR does not have significant amount of greenhouse gas emissions associated with the move from the University of Calgary to RackForce. Raw materials used in ICT equipment P2 / B2 Related Not Relevant. Some very small differences in the amount and type of raw materials that go into the manufacture of one type of server versus another. Same SSR. Manufacture of ICT equipment P3 / B3 Related Not Relevant. The same amount of energy with very small differences go into the manufacture servers. There is no difference or a very small difference between the manufacture of a leased server versus an up-to date commonly available server. Same SSR or insignificant 19

52 SSR # Controlled SSR Name (Project / Related Relevancy and justification if not relevant Baseline) Affected small differences Transportation of ICT equipment to facility P4 / B4 Related Not Relevant. The differences in transportation are insignificant for a server going to Calgary vs. Kelowna. Same SSR or insignificant small differences Installation of ICT equipment, rack/cabinets Production of diesel P5 / B5 Related Not Relevant. The same process of installation is required for a leased server as compared to an up-to date commonly available one installed in Calgary. Same SSR or insignificant small differences P6 / B6 Related Not Relevant. Same SSR or insignificant small differences fuel (for backup) Power Source(s) supplying the grid to meet demand from the data centre P7 / B7 Related Relevant. The grid providing electricity to the two datacenters are significantly different, with one being primarily powered by coal-fired power plants and the other one being powered by hydropower. In addition, the amount of power being required by the IT facility and the ICT equipment to support the application could be different in RackForce as compared to the University of Calgary datacenter and could also be a source of emission reductions. This would be a Scope II emission or an electricity indirect emission reduction. Water Cooling of Data B8 Related Relevant. SSR is only a source at the University of Calgary. Center Network Traffic P8 / B9 Related Partially Relevant. All network traffic for GeoChronos is routed through the datacentre at the University of Calgary regardless if GeoChronos is actually at RackForce or at the University of Calgary. During the project the additional network traffic and associated energy use because of routing of the network traffic from the University of Calgary to RackForce in Kelowna via Vancouver using the CANARIE/BC Net network will need to be quantified. However, only the project SSR (P8) is actually relevant as the number of requests and the amount of traffic arriving at GeoChronos will be the same in the baseline and the project, but the project will have additional emissions associated with the network traffic to RackForce and back again. 20

53 SSR # Controlled SSR Name (Project / Related Relevancy and justification if not relevant Baseline) Affected However, only part of the network is controlled by the proponent and it s partners. Back-up Diesel Generator Power Fuel Use P9 / B10 Controlled Relevant. Same SSR, but there could be significant differences if the generator is used more in one location (different activity level). At the baseline facility, the emissions from the backup generators used in the baseline facility are minimal and can thus be ignored. Most of the equipment that would have been used at the baseline facility is not connected to generator-backed circuits. Only the storage array is connected in such a manner, and the GeoChronos project is only responsible for a small fraction of its use (1 TB of 20 TB total in the baseline scenario). Total carbon emissions during the time that the generator is operational appear unlikely to be significantly higher than at other times as the total power consumption decreases due to the remainder of the resources used by GeoChronos not consuming power at such times. The generator is also used to power much additional equipment - the critical services of the university- meaning that the storage array also accounts for a small percentage of generator output. The emissions from this SSR will only be calculated at the project scenario resulting in a conservative estimate of emissions for this SSR. Power Source Maintenance P10 / B11 Controlled Not Relevant. Same SSR with insignificant small differences. Assume that the same power source maintenance (i.e. test runs) is carried out in the two locations and if there are any differences they would result in small differences in emissions. ICT equipment maintenance P11 / B12 Controlled Not Relevant. Same level of ICT equipment maintenance will be required for the GeoChronos servers whether leased or at the University of Calgary. Same SSR or insignificant small differences. HVAC system P12 / B13 Controlled Not relevant. The amount of additional impact on the HVAC system is minimal and comparable in both datacenters. Same SSR or insignificant small differences. 21

54 SSR # Controlled SSR Name (Project / Related Relevancy and justification if not relevant Baseline) Affected ICT disposal and P13 / B14 Related Not Relevant. Same SSR or insignificant small differences. recycling Green ICT takes off (market impact) P14 Affected Not Relevant. Not an SSR in the baseline, may be a source of emissions in the project due to increased demand for Green ICT. This is actually negative leakage. Likely that the emission reductions are as a result of moving from another less green data centre to the new one. Will not be estimating the emission reduction because of difficulty in doing so and because it is negative leakage. 7 Method to quantify/estimate GHG emissions and/or removals in the baseline and project Using the methodology as outlined in the Protocol, emissions in both baseline and in the project were determined based on the SSR s identified for both. The basic methodology for estimating emissions from the baseline scenario is formulized as follows; t p (1) EM ( Ba) ( PICT (Pr) PUE( Ba) EF ( Ba)) Where: EM(Ba) = Emission associated with the baseline (in kg CO 2 e) P ICT (Pr) = Power measured from the ICT equipment in the project, in GJ PUE (Ba) = Power Usage Effectiveness (PUE) of the baseline ICT facility EF (Ba) = Emission factor for source energy in the baseline, in kg CO 2 e / GJ. This emission factor reflects all of the source energy used to provide the baseline facility s energy requirements 22

55 As stipulated in the Protocol, for project type 1 and the first three use cases for project type 2, ICT power is made equivalent between the baseline and the project reflecting that emission reductions are related to the data centre environment and not the ICT equipment itself. Emissions from the project scenario from the SSRs in the datacenter were similarly calculated as was done for the baseline. However, in addition to the SSRs identified for the baseline, for the project this also included the project SSRs identified with the network traffic from the University of Calgary datacenter to RackForce and back again over the CANARIE network. As a result of these additional SSRs, emissions from the project were determined as follows; (2) EM (Pr) ( P (Pr) PUE(Pr) EF(Pr) ( Energy EF )) ICT Network Network Where: Energy Network = The electricity required to operate and maintain the network for GeoChronos traffic between the University of Calgary and RackForce in Kelowna, EF Network = The emission factor of the source power providing electricity to the network. Each SSR as identified in table 15 in the previous section was matched with the values in the formula as shown below and each of these factors in the equations were measured, monitored or estimated to determine the emission reductions associated with the GeoChronos relocation project. 23

56 7.1 Determining emissions in the baseline scenario: Recall that emissions from the baseline scenario for all SSR s in the datacenter are calculated according to equation 1, as follows; (1) EM ( Ba) ( PICT (Pr) PUE( Ba) EF ( Ba) The PUE captures the share of the total energy that contributes to directly powering the ICT equipment as well as the portion that contributes to the cooling and other supporting infrastructure of the datacenter. For the baseline, the PUE is therefore defined according to the energy sources used by the datacentre; (3) Energy PUE Ba 1 CRAC Energy Energy CLOOP ICTEquipment Energy Other Where: Energy CRAC = The amount of power used by the CRAC units in the datacentre (kwh or GJ) Energy CLOOP = The amount of power used by the cooling system (kwh or GJ) Energy Other = The amount of power used by other supporting services such as lighting of the datacenter (kwh or GJ) Determining the PUE of the baseline facility As indicated by equations 1 and 2, the PUE for both the baseline scenario and project was required in order to estimate emissions and emission reductions. While for the project facility this could be measured and monitored, there were a number of challenges for determining the 24

57 PUE for the datacenter at the University of Calgary. One of the primary reasons was that the primary electricity feed to the datacenter was not dedicated as it also provided power to neighboring university infrastructure (i.e. a parking lot where block heaters are source of power consumption in the winter). As a result, it was not possible to determine the total amount of energy supplied to the datacenter from the various energy sources (i.e. the electricity grid, the cooling loop and any back-up generator) in a cost-effective manner. In addition, it was not possible to determine the amount of energy used for powering the ICT equipment at the datacenter or for determining what amount of energy was devoted to cooling and providing other support services for the data center. It is also important to note that the baseline datacentre from which Geochronos was moved had been retrofitted a number of times with new overhead cooling units as well as more efficient CRAC units. Therefore, based on a survey of literature investigating the PUE of datacentres as well as consideration of the technical characteristics of the University of Calgary facility, a number of assumptions were made to arrive at a conservative estimate for the data centre. In surveying relevant literature, a 2007 Report to Congress assumed an average PUE of 2.0 for US data centers for the period from 2000 to 2006 in order to estimate the total energy use attributable to datacenters (Energy Star, 2007). The average PUE of 2.0 was based on an energy use benchmarking study of 22 datacenters performed by the Lawrence Berkeley National Lab. In the same report, the EPA states that they expected the average PUE to decline to 1.9 by Other data from authoritative organizations which make conclusions based on the EPA study are similarly aligned, including from the Green Grid who state that PUE values of 2.0 are achievable with proper design (Green Grid, 2008). Digital Realty Trust, meanwhile, 25

58 conducted a survey of about 300 data centers which was reported in 2011 concluded that the average reported PUE energy efficiency rating for respondents' data centers was 2.9 and that one in six respondents report PUE ratings of less than 2.0 for their facilities (Digital Realty Trust, 2011). With this and the above analysis in consideration, we assume that the PUE at the University of Calgary is 1.7 and that for the reasons stated above this can be considered to be a conservative assumption of PUE (i.e. it is conservative to have a lower estimate of the PUE of the baseline facility as this will mean a lower estimate of baseline emissions) Determining the weighted emission factor of source energy The datacenter at the University of Calgary receives energy from three sources; the electricity grid, the backup generation and the cooling loop. These must all be considered when deriving the weighted emission factor of the facility. The sources of energy for the University of Calgary datacentre are shown schematically in figure 3; Figure 3: Sources of energy to University of Calgary datacentre Sources of Energy Electricity Grid Backup Generator Cooling Loop Energy Consumption ICT equipment Cooling provided by CRAC units Other Services (lighting) etc. 26

59 For the baseline, we have assumed that the backup generator does not provide significant power to the datacenter. As a result, the sources of energy to the datacentre are in actuality the electricity grid and the cooling loop. To determine the emission factor of the cooling loop, we required the following information: The total amount of power provided to the Bow River Pumping Station to draw the water from the Bow River The amount of water from the Bow River water loop that is provided to the campus relative to the total amount that is drawn from the Bow River The power needed by the pumps along the Bow River loop to get the water to the chillers The amount of power needed by the chillers to chill the water in the campus loop by transferring the heat from the campus loop to the Bow River loop (or by transferring the cold from the Bow River loop to the campus loop) The power needed by the pumps along the campus loop to circulate the water around the campus The total amount of cooling provided to the campus by the chilled water loop Since the electricity used within the pumping and cooling process is from the Alberta electricity grid, the grid electricity factor was used to derive emissions from purchased electricity. In order to estimate emissions associated with the cooling loop, 12 months of data was collected on each of the parameters listed above. The resulting power usage and emission calculations are summarized in the table below (see table 4). 27

60 Table 4: Calculation of Emission Factor for the Campus Cooling Loop Month Bow Percent Chillers Total Emissions Total Emission River sent to Campus factor Pumping campus Chilled Station water KWh % KWh KWh Kg CO 2 e KWh g CO 2 e per KWh 10-Jun 174, ,053,344 1,186,528 1,044,145 4,945, Jul 239, ,396,703 1,601,669 1,409,469 6,375, Aug 220, ,230,080 1,410,895 1,241,587 5,591, Sep 139, , , ,948 3,279, Oct 149, , , ,852 3,577, Nov 159, , , ,598 1,671, Dec 139, ,986 85,347 1,461, Jan 143, ,969 81,812 1,473, Feb 129, ,340 72,459 1,309, Mar 152, ,011 87,130 1,565, Apr 176, , , ,962 1,559, May 152, , , ,004 2,955, Total 1,975, ,904,546 7,352,629 6,470,313 35,766, As shown, the average annual emission factor of the energy supplied via the cooling loop equalled grams/kwh. This is roughly ¼ of the emission intensity of the power grid in 28

61 Alberta, reflecting in part that during the coldest months of the year additional cooling of water from the Bow River is not required. In arriving at a combined emissions factor, sources familiar with the university s datacentre suggested that 70 percent of the datacentres total energy should be attributed to electricity purchased from electricity grid and 30 percent attributed to energy from the cooling loop. Based on the grid average emission factor, this would suggest a combined emission factor for source energy of g CO 2 e per kwh. 7.2 Determining emissions for the Project: Recall that emissions at the project will be determined by: (2) EM (Pr) ( P (Pr) PUE(Pr) EF(Pr) ( Energy EF )) ICT Network Network Where: EM(Pr) = Emission associated with the project (in kg CO 2 e) P ICT (Pr) = Power measured from the ICT equipment in the project, in GJ PUE (Pr) = Power Usage Effectiveness of the project ICT facility EF (Pr) = Emission factor for source energy in the project, in kg CO 2 e/gj Energy network = The electricity required to operate and maintain the network for GeoChronos traffic between the University of Calgary and RackForce in Kelowna, EF network = the emission factor of the source power providing electricity to the network. This is expected to be based on the emission factor of the grid in Alberta and B.C. 29

62 Where the PUE is determined by: (5) Energy PUE 1 Energy Total ICTEquipment Therefore five variables need to be determined in order to estimate the emissions associated with the project project ICT power usage, the PUE of the project facility, the emission factor of source energy for the ICT facility, the energy used by the network involved with transferring data, and the emission intensity of the electricity used for the network Determining the project ICT power usage The GeoChronos application was hosted on the following pieces of hardware at RackForce; A storage area network on a rack, A dedicated server on a rack with other servers, A virtual machine running in the cloud environment at RackForce on shared hardware and, Two blade servers on a chassis with a total of 8 blades. The power to each of these pieces of equipment was monitored as follows; Storage Area Network: For the SAN, the circuits providing power to the rack was monitored at the Power Distribution Unit (PDU) using specific power management software called Power Xpert. The 30

63 amount of power used by GeoChronos at the SAN was apportioned based on the amount of storage used (or allocated) to GeoChronos. The same method was used to determine the amount of power used by GeoChronos at GRC attributable to the SAN. Dedicated server: A Raritan managed powerbar was used to measure the power used by this dedicated server. This data will be taken weekly by logging into the IP based interface on the powerbar. The power reading was then multiplied by the number of hours in the week to provide a number in kwh. The reading was taken when the workload on GeoChronos was not at a low level or possibly at a high level in order to be conservative. Virtual Machine: GeoChronos also made use of a virtual machine (VM) in a shared cloud environment at the RackForce Gigacenter datacenter. In reviewing the shared resources in the cloud ( i.e. RAM and CPU), it was seen that CPU assigned to any particular VM or group of VMs was typically under utilized or oversubscribed by as much as 300% whereas RAM was generally fully utilized (greater than 80%) by these same VMs and was not oversubscribed. Therefore RAM has been determined as a reasonable proxy to determine the fractional power usage for the GeoChronos VM on the cloud. The total RAM assigned to the cloud segment where the GeoChronos VM resides was divided by the RAM assigned to the GeoChronos VM. This ratio was multiplied by the amount of power used by the cloud. Power readings were taken from the internal management systems that are built into the blade center computers that are used to provide compute resources to the cloud. These readings were taken weekly and the watts multiplied by the hours in a week in order to provide a number in kwh. 31

64 Blade Servers: The power used by the blade servers was measured in the following manner; - Internal management systems built into blade servers provide power consumption in real time - The chassis power consumption was also measured using the internal power measurement and management software of the chassis. - The chassis power consumption was subtracted from the power consumption of all of the blades in that chassis and the difference was the amount of overhead power to be allocated to each blade (split equally across each blade). The results showed that each blade used 10 watts per hour while the chassis used 88 watts per hour. After allocating the overhead power to the 2 blades used by GeoChronos, this meant that the application was responsible for 22 watts per hour or 3.7 kwh per week Determining the PUE of the project facility The total energy provided to the project datacenter is from two sources; electricity from the power grid as well as energy from the back-up diesel generator. Each of these were monitored and/or estimated at the project scenario site during the project in terms of both total energy use and the energy used specifically for ICT equipment. In particular, the amount of electricity provided by the electricity grid was determined by monthly utility bills from Fortis BC. Meanwhile, the load time of the generator was used to determine the amount of electricity generated from the diesel generator in order to support the datacenter as well as information about the total fuel used. Monitoring these showed that RackForce had a PUE of 2.0. Here, it should be emphasized that this is higher than the baseline facility largely because 32

65 at the time of analysis was relatively under utilized relative to its full potential. Thus, there was a much large overhead of support services (cooling, etc), compared to when the facility will be more fully utilized. At that point, it is expected that the PUE of the facility will fall well below that of the average data centre. Nonetheless, for the purposes of this analysis a PUE of 2.0 has been used Network traffic emissions source As indicated in equation 2, an additional relevant SSR which contributes emissions in the project scenario is the network between the University of Calgary datacenter and the datacenter in Kelowna. In the baseline, any user requests for GeoChronos go to the University of Calgary datacenter and are processed by the GeoChronos application. In the project, all of the requests associated with GeoChronos continue to be directed to the University of Calgary datacenter. They are then redirected to the RackForce datacenter in Kelowna. This additional network traffic is the source of addition power consumption and emissions. Consequently this source of additional emissions needs to be quantified in the project. The network between the University of Calgary and RackForce is provided by two service providers, CANARIE and Shaw with equipment being owned by three organizations. CANARIE has a dedicated 10 Gbps wavelength (pipe) which is active and up 24 hours a day. CANARIE dedicated 10 % of this (1 Gbps) for the GeoChronos relocation. The CANARIE network handoffs any GeoChronos traffic to a transfer switch owned and operated by RackForce in the Vancouver station. The RackForce switch then hands-off the GeoChronos traffic to another service provider (Shaw) for the transfer of data from Vancouver to the RackForce Kelowna datacenter. 33

66 To determine the power consumption and emission for the CANARIE network between the University of Calgary and Vancouver, since the CANARIE wavelength pipe is always operating and the GeoChronos lightpath is similarly always open, power usage can be approximated by a constant. We have used the maximum power specifications for all the equipment and stations on the CANARIE wavelength to estimate the power consumption attributable to this SSR. In addition, based on the location of the station, we have multiplied by the emission factor associated with the power grid in that province. We applied similar logic for the RackForce owned and operated network component. 8 Quantification of GHG emission reductions and removals Emission calculations are shown for the baseline, the project, and the emission reductions associated with the project. 8.1 Quantifying emissions for the baseline Based on the monitoring done at the baseline and the project, and based on the methodology laid out in section 7, the data and emissions for the baseline are shown below: Table 5: Estimate of baseline emissions Emission PUE Power Cooling Emission Weighted P ICT Emissions Factor (Grid) Grid Loop Factor Emission (kwh) (Baseline) (g CO 2 e / Share Share (Cooling Loop) Factor kg kwh) (g CO 2 e / kwh) ,738 Over the life of the monitored portion of the project (13 weeks), the baseline emissions would have been 2,738 kg of CO 2 e. 34

67 8.2 Quantifying emissions for the project The project operated from June 28, 2011 to September 25, 2011 for a total duration of 13 weeks. Emissions associated with the project occurred in two locations, the datacenter and the network. The datacenter emissions are shown in the table below; Table 6: Estimate of project datacentre emissions Emission PUE P ICT Emissions Factor (Grid) (g (kwh) (Project) kg CO2e / kwh) Emissions from the network are shown as follows; Table 7: Estimate of project network emissions Province Power consumed Emission Emissions per Emissions Emissions per week (kwh) Factor week (kg Network (project) (g CO 2 e/kwh) CO 2 e) (13 weeks) kg Alberta , BC Total emissions from the network over the length of the project attributable to GeoChronos totalled kg of CO 2 e. When factoring in emissions from the data centre, total emissions from the project over the 13 weeks totalled kg of CO 2 e. 35

68 8.3 Estimated emission reductions Emission reductions are estimated as the difference between the project and the baseline. These are estimated to equal 2,412 kg of CO 2 e over the 13 week project. Therefore, if the GeoChronos Relocation Project was run for a full year, the project would result in total emission reductions of 11,405 kg of CO 2 e. 9 Conclusions and discussion of relevance for carbon trading and corporate sustainability In this article we have presented a quantification of emissions and emission reductions resulting from an ICT-related project. This is a complex topic involving emerging technologies and processes. However, the topic is of one of growing importance seeing the increasing role of ICT across the economy, the importance of data management for specific industries as well as the rising energy requirements for ICT and associated increases in GHG emissions,. This leads to a final discussion of the relevance of ICT and ICT projects, such as was presented in this article, for the broader topics of carbon trading and improving corporate environmental sustainability In regards to carbon trading, this might include any activities tied to regulatory frameworks such as the Kyoto Protocol and regional trading systems tied to government climate change programs. However, increasingly, carbon markets are being impacted by voluntary actions to improve carbon performance, and it is here likely where ICT related projects will fit in. On the demand side, the voluntary carbon market is largely made up of entities choosing to voluntarily purchase offsets to lower the carbon footprint associated with their activities, while on the supply side, this involves providers of verified emission reductions selling into 36

69 this market. The market is therefore intertwined and reinforced by rising public awareness of environmental issues such as climate change as well as by organizations and businesses undertaking efforts to lower their carbon footprint or claim carbon neutrality through the purchase of carbon offsets. In 2009, suppliers of emission offsets in the voluntary market transacted 93.7 million metric tonnes of carbon dioxide equivalent (MtCO 2 e), worth about $387 million in value, with expectations that the transactions could be as large as 1,200 MtCO 2 e by 2020 (Hamilton et al., 2010). Rising demand will likely be driven by the processes described above, namely the increasing public attention placed on the reduction of carbon in regards to the products they consume and the services they use. This will continue to drive corporations and other entities to lower the carbon footprint of their operations and products, in turn underpinning demand for offsets originating from sources such the delivery of lower carbon ICT. Further to the use of emission reductions for carbon trading, sourcing low or zero carbon ICT services may come more and more on the radar of large corporations, institutions and users of data services. This is particularly relevant when considering the widespread inclusion of carbon in corporate sustainability reporting as well as the thousands of organizations who measure and disclose their greenhouse gas emissions and climate change strategies through efforts such as the Carbon Disclosure Project (PricewaterhouseCoopers, 2010). It is also relevant when considering the potential for the large-scale adoption of cloud computing and the rapid increases in the outsourcing of ICT services that has occurred during the last number of years and decades. 37

70 One approach to understanding the potential scale of a large scale ICT project or initiative is to consider the power use requirements attributable to the operation of datacentres directly owned and operated by a large corporation or organization. While in the past such information was often difficult to obtain and often not disaggregated from the power and energy use footprint of the corporation, increasingly, corporate environmental reporting has been including more details on the corporation s environmental impact. In 2010, for example, the Royal Bank of Canada (RBC) reported that the datacentres owned and operated by the company required nearly 92,250 MWh of electricity (RBC, 2010). If theoretically all of these were located in grids with an emissions factor equalling 0.50 kg CO 2 e/kwh (representative of a grid that is a mix of fossil fuels, renewables, etc), then this would mean that the datacentres owned and operated by this large bank would result in emissions equalling over 46,000 tonnes per year. This scale of emissions and potential emission reductions therefore helps indicate that focusing on lowering the carbon impact of ICT could in fact result in large emission reductions for a company or organization, thereby improving their environmental performance. With burgeoning requirements for data management, the ICT services industry has been experiencing rapid growth in recent years. While cloud computing and virtualization are key technological processes that will likely continue and even accelerate this growth, these also link into actions that could help reduce the carbon intensity of data management. This article has focused attention on one example of how to lower or eliminate the carbon and environmental impacts of ICT services. However, there are many more, as stipulated in the protocol used and sourced in this article. Thus, in the future heightened attention could be given to the sector in terms of contribution to climate change mitigation. 38

71 References Benowitz, M. and S. Samuel, 2010, Green Information and Communications Technology (ICT) for Eco-Sustainability Overview. Bell Labs Technical Journal 15 (2), 1-5. Canadian Standards Association (CSA) (2011). ICT Greenhouse Gas Reduction Project Protocol: Quantification and Reporting (version 1). Canadian Standards Association, Mississauga, Ontario. Canada Digital Realty Trust, 2011, Accessed September 2011 from Energy Star, 2007, Report to Congress on Server and Data Center Energy Efficiency Public Law , Accessed September 2011 from available from _Report_Congress_Final1.pdf. Green Grid, 2008, Green Grid Data Centgre Power Efficiency Metrics: PUE and DCIE, Accessed October 2011 from _PUE_and_DCiE_Eff_Metrics_30_December_2008.ashx?lang=en. Hamilton, K., M. Peters-Stanley and T. Marcello (2010). Building Bridges: State of the Voluntary Carbon Markets 2010, Ecosystem Marketplace, Bloomberg. Koomey, J. G., 2008, Worldwide electricity used in data centers. Environmental Research Letters 3 (3), Pricewaterhouse Coopers (2010). Carbon Disclosure Project 2010: Global 500 and S&P 500 Report Highlights. London, PricewaterhouseCoopers:

72 Royal Bank of Canada (RBC) (2010). RBC 2010 Corporate Responsibility Report and Public Accountability Statement - Full Report, Royal Bank of Canada. 40

73 This page intentionally left blank This page intentionally left blank This page intentionally left blank

74 Telecommun Syst DOI /s Lightpath scheduling and routing for green data centres Jing Wu James Yiming Zhang Michel Savoie Her Majesty the Queen in Right of Canada 2011 Abstract Optical networking technologies enable data centres to be located near sources of green energy (i.e., renewable energy). Since some green energy sources are intermittent and are not always available, we need to dynamically connect distribution networks to the green energy powered data centres. On the other hand, the availability of green energy is reasonably predictable, and we are thus able to schedule connectivity to data centres in advance. We propose a WDM network planning model, which allows lightpaths to slide within their desired timing windows with no penalty on the optimization objective, and to slide beyond their desired timing windows with a deteriorating greenlevel. Our simulation results show the tradeoffs between the consumption of brown energy (i.e., energy generated by carbon-intensive means), the capability of providing required connectivity to data centres, network resource utilization, and overall operation objective. Keywords WDM networks Lightpath scheduling Data centre interconnection Green energy Network resource utilization 1 Introduction Optical networks are widely used in connections between data centres, which are facilities that primarily host elec- J. Wu ( ) J.Y. Zhang M. Savoie Communications Research Centre (CRC) Canada, 3701 Carling Avenue, Ottawa, Ontario, Canada, K2H 8S2 jing.wu@crc.gc.ca J.Y. Zhang yizhang@site.uottawa.ca M. Savoie michel.savoie@crc.gc.ca tronic equipment for data processing, data storage, and communications. In connecting data centres to each other and also to their respective distribution networks, optical networks have attractive advantages, such as high speed, large capacity and energy efficiency [1, 2]. With the advanced optical networking technologies, data centres can be located near sources of green energy (i.e., renewable energy). In this way, instead of the transmission of electricity from sources of green energy to data centres, data can be transmitted to and among data centres over optical networks, which results in less loss in electricity transmission and more efficient use of green energy. We aim at reducing the overall use of brown energy by data centres and on the other hand, maximizing their use of green energy. Green data centres are the data centres that mainly consume the electricity generated by renewable energy, such as wind, solar, tide and hydro energies [3]. Green data centres can avoid the use of brown energy that is generated by carbon-intensive means, such as coal or gas burning power plants. At the same time, green data centres may use brown energy as secondary or backup options. However, using brown energy costs more than using its green counterpart, and more so in social and environmental senses. The cost varies for using different power sources at different times. Because some sources of green energy are intermittent and are not always available, we need to dynamically change the connectivity of data centres. We use the terminology connectivity of data centres to refer to two types of connections: connections between data centres, and connections from data centres to their respective distribution networks for end users [4]. The availability of green energy highly depends on weather and other environmental factors, making it intermittent and fluctuating [5 7]. However, measurement and predication of green energy availability, together with

75 J. Wu et al. the use of rechargeable batteries, make green energy reasonably predictable [8 10]. The operation of green data centres must be planned to take into account the predicted green energy supply at different times and sites. Accordingly, we can schedule the change of connectivity of data centres in advance to make better use of greener time slots. There are daily or weekly patterns of workloads and connectivity requirements of data centres. It has been observed that workloads of transaction-oriented servers vary significantly depending on time of day, day of week, or other external factors [11 14]. Network traffic at the optical layer periodically fluctuates. For core optical networks such as Wavelength Division Multiplexing (WDM) networks, network traffic is observed following daily patterns [15], or weekly patterns [16]. Such traffic patterns can be predicted from historical statistics, which repeat every day (or week) with minor variations in timing and volume. It is expected that a large portion of the traffic that is carried by optical networks are related to data centres. The knowledge about workloads and traffic patterns provides an opportunity to schedule the connectivity of data centres. Using scheduled lightpaths to connect green data centres has different timing requirements from other lightpath scheduling problems. Existing methods for the lightpath scheduling problems assume that a lightpath should be set up either at a given time, or within a given time window, which makes the lightpath scheduling inflexible for connecting green data centres. For example, in [17 21], network planning was conducted for a set of lightpath requests, each having a pre-specified starting and ending time. In [22 25], static WDM network planning was conducted for fixed holding-time lightpath requests, each one being allowed to slide within its given time window. However, for connecting green data centres, timing flexibility in lightpath scheduling is very important, not only because of the variability of the green power, but also because of the network operations. Network operators are concerned about not only timing violations (related to the green-level of data centres), but also the resource utilization, as well as lightpath rejections (related to the connectivity of data centres). Network operators need tools to make wise tradeoffs between these goals, e.g., network operators would rather adjust lightpaths scheduling timing, than reject lightpaths that cannot be accommodated due to their strict timing requirement or impractical timing windows. To connect green data centres, scheduled lightpaths can be allowed to slightly slide in time, without deteriorating the performance. Sliding-timing scheduling potentially provides better network resource utilization than fixed-timing scheduling. Since lightpaths are scheduled based on the statistical availability of green energy at different data centres and traffic characteristics, minor timing slides should not impact much on the performance of the traffic adaptation, while dramatic timing slides, on the other hand, should be avoided. For example, due to the availability of sunlight at a given solar powered data centre, the network operator needs to provision a lightpath between the data centre to a distribution network during the day time. It usually does not make much difference if the timing slide is far below the variance of green energy availability, e.g., starting the lightpath from 8:00 AM or 10:00 AM. However, setting up the lightpath at 5:00 AM cannot efficiently transfer data from the data centre, due to the lack of sufficient solar power at the data centre. In the above-mentioned applications, the extent of the timing satisfaction or violation needs to be quantitatively measured. Moreover, a timing window should not be used in a binary way, i.e., it can either be satisfied (thus the corresponding lightpath is accepted), or not (thus the corresponding lightpath is rejected). Network operators normally would prefer scheduled lightpaths being centered on their desired timing, with a decreasing tolerance level, which can take into consideration both the green level and the timing punctuality, as scheduled lightpaths move away from their desired timing windows. This requires proper modelling of the extent of timing satisfaction or violation, which has not been done by the existing methods for the static lightpath scheduling, and therefore motivates this study. Our study aims at planning scheduled lightpaths to adapt to relatively stable traffic patterns and predictable availability of green energy. Static scheduled lightpath demands are generated based on forecast traffic patterns and green energy availability at data centres, and are input to our network planning problem. Our problem is different from dynamic lightpath scheduling problems, which generally do not assume any a priori information of how traffic patterns change [26 28]. In our approach, once a lightpath is pre-planned, it becomes available to carry traffic at its scheduled time. In contrast, dynamic lightpath scheduling cannot guarantee the availability of a lightpath. Only when a request arrives, the network operator makes real-time decisions depending on the network resource availability at the moment of the request. Our approach achieves a better coordination of lightpaths than dynamic lightpath scheduling by taking advantage of known traffic patterns, as well as green energy availability for data centres [28]. This paper is organized as follow. In Sect. 2, we outline the energy efficiency problems, challenges, possible solutions and emerging opportunities of data centres. In Sect. 3, we summarize the networking requirements of green data centres, as well as assumptions used in our model. In Sect. 4, we present our model, followed by numeric results in Sect. 5. We conclude this paper in Sect. 6.

76 Lightpath scheduling and routing for green data centres Fig. 1 Typical electrical systems in a data centre [29] 2 Green data centres The energy efficiency problems, challenges, possible solutions and emerging opportunities of servers and data centres are highlighted in a comprehensive report developed by the United States Environmental Protection Agency (EPA). EPA issued a report to the US congress on energy efficiency of servers and data centres in August 2007 [29]. Some of its key findings are: In 2006, servers and data centres in the US consumed 61 billion kwh (kilowatt hours), and their total electricity cost was US $4.5 billion; Their energy consumption grew rapidly and the trend continues. From 2000 to 2006, their total electricity consumption doubled; On average, their site infrastructure consumed half of the electricity for cooling systems, power delivery, and so on; Among the information processing equipment, volume servers consumed 68% of the electricity in 2006; while network equipment steadily consumed approximately 10% of the electricity over Typical electrical systems in a data centre include main and backup power supplies, electrical systems for the facility, and power regulatory systems for data processing equipment. Connections of electrical systems in a data centre are illustrated in Fig. 1. Due to the stringent requirements of data processing equipment, power conditioning systems are normally used, such as power distribution units, rechargeable batteries/uninterruptible power supplies. Data centres suffer from low utilization and low energy efficiency. Normally, the capacity of data centres is designed based on the peak demand. However, the real demand varies over time. Servers operate most of the time at 10% 50% of their maximum utilization levels [12, 13, 30 34]. Unfortunately, when the utilization of servers is low, their power consumption remains high. Currently, servers still consume over 50% of their peak power usage, even when the servers are idle, resulting in low energy efficiency of servers [12, 30, 34, 35]. Since on average data centres consume half of the electricity for cooling systems, power delivery, and so on [29], low energy efficiency of servers results in even lower energy efficiency of data centres. Servers at low utilization still generate significant heat, which requires cooling systems [36]. Even idle servers need power to be delivered, which causes power delivery loss. The capability of on-demand switching off servers increases the energy efficiency of data centres. In another word, an easy step to increase energy efficiency of data centres is turning off the power of idle servers, and turning them on when necessary [11, 34]. Virtual machine migration allows the relocation of active tasks from one physical host server to another, without any major performance degradation on the active tasks running on servers [37 43]. Active tasks running on low utilized servers may be dynamically consolidated into fewer servers, creating opportunity to scale back the power consumption of idle servers (including powering-off idle servers) [12, 13, 32, 44 49]. When additional service capacity is needed, the servers that are powered off may be powered on and provide service in minutes [50]. Software applications may be intelligently mapped to underlying servers based on the workload, capability and power consumption profiles of these servers [51 58]. As proposed in [59], intelligent distribution of computational workload was explored across geographically distributed data centres to optimize energy consumption and cost. It took advantages of the cost-saving factors, including different and variable electricity prices, peak-demand prices versus off-peak-demand prices due to time zone differences, and green energy versus brown energy. Making data centres use green energy has different requirements from minimizing the total energy consumption or reducing the peak power consumption of data centres. For example, the total energy consumption may be slightly increased, when we aim at maximizing the usage of green energy. However, our goal is viable due to its long-term social, environmental, and economic benefits. Previous studies addressed the problem of distributing tasks in web servers to minimize their total energy consumption and at the same time to satisfy performance requirements [13, 47, 60 64]. There are also other efforts in reducing peak power requirements at chips, servers, racks and data centres [65].

77 J. Wu et al. Fig. 2 A data centre interconnected by WDM networks 3 Networking requirements of green data centres 3.1 Networking requirements of green data centres High-bandwidth networks are used to connect data centres, as well as key systems within a data centre. The data processing equipment within a data centre consists of three key systems (shown in Fig. 2): front and back end gateways, and back end application servers. The front end gateway is connected to distribution networks, e.g., IP service provider networks. The front end gateway supports client-to-server communications and provides the high-speed server-to-server networking with the back end gateway and other front end gateways. The front end gateway is composed of firewalls, and servers for content caching, load balancing, and intrusion detection. The back end gateway is connected to the back end gateways at other data centres, which could be located beyond the same metropolitan domain. In this example, we use WDM networks due to their advantages of protocol transparency, high-bandwidth offered by multiple optical channels, and most importantly their energy efficiency. However, other high-bandwidth networks such as metropolitan area Ethernet networks, and SONET/SDH networks are also viable solutions for data centre interconnection [44]. Optical networks are ideally suited for bridging the gap between the desired locations of data centres from the facility s perspective and from the information processing s perspective. There are benefits to locate data centre facilities in rural and remote areas that provide data security from natural disasters, green energy generation, lower land cost, lower external surrounding temperature, and running water for cooling. However, the information processing should ideally be located close to users to reduce delay and network congestion. Optical networks provide huge bandwidth and point-to-point IP links, while consuming less energy than any other communication networks. By using optical networks, we effectively remove the need of the long-haul transmission of electricity from power generation stations to data centres. We pre-plan schedules for setting up lightpaths to connect data centres to the requesting distribution networks. We assume that high-volume digital contents are hosted by large data centres. Each piece of content is duplicated to 2 or 3 mirror data centres. Lightpaths originate from the back end gateway of the data centre that store the digital content, and terminate at the front end gateway of the data centre that is located close to the requesting distribution network. Network power cost is considered constant, regardless of carried traffic. We assume that all wavelength channels in all WDM fibres are available for the lightpath scheduling and routing. In reality, optical virtual private networks may be used, where a set of wavelength channels are allocated by the lightpath scheduling and routing algorithm. Lightpath demands are created based on the predicted green energy availability for data centres and the requests for contents that are hosted by the data centres. If the preferred schedule for a lightpath cannot be satisfied, a lightpath may be shifted in its schedule with a higher cost, due to its increased use of brown energy and the reduced timing satisfactory level. Our problem is to provide the best scheduling and Routing and Wavelength Assignment (RWA) schemes of lightpaths, which are generated off-line by our optimization algorithm, to provide the required connectivity of data centres and at the same time to minimize the use of brown energy. 3.2 WDM network operations, modeling and assumptions We consider wavelength-routed WDM networks with mesh topologies. We model a general topology WDM mesh network of N nodes interconnected by E links. Each node represents a data centre, including its front and back end gateways. Each link consists of a pair of fibres, each fibre for one direction and having W non-interfering wavelength channels. Two nodes can be connected through a lightpath defined as a concatenated sequence of wavelength channels [66]. We assume no wavelength conversion is used due to its high cost and little benefit for the static WDM planning problem [67]. So a lightpath must use the same wavelength all the way from its source node to destination node. Lightpaths are scheduled to be set up at the beginning of their starting time slots and be torn down at the end of their finishing time slots. Network-wide synchronous time slots are used for resource allocations and lightpath scheduling. All time slots have the same fixed duration, which should be one hour or larger. We assume the time to set up or tear down a lightpath (i.e., signalling time) is negligible compared to the duration of a time slot. The holding time of a lightpath is fixed and known in advance, measured by the number of time slots. Without losing generality, we number the time

78 Lightpath scheduling and routing for green data centres slots in our planning time horizon sequentially from 0 to Z 1 (0 t<z). The complexity of our scheduling problem increases, as the number of time slots in the planning time horizon increases. A detailed analysis of computation complexity can be found in [68]. We aim at scheduling and allocating network resources to lightpaths, i.e., planning network operations for Scheduled Sliding Lightpath Demands (SSLDs). The network resources primarily include wavelength channels. We provide an accepted SSLD with a schedule (i.e., starting time slot) and an RWA scheme (i.e., a list of allocated wavelength channels). The same RWA scheme is used for the entire holding time of an SSLD, i.e., once an SSLD is accepted, it stays connected from its starting time slot to its finishing time slot. If there is insufficient resource for an SSLD during its holding time, the SSLD is rejected. 4 Problem formulation 4.1 Notations For the remainder of this paper, the following notations and variables are used: Network model related parameters: V the set of all nodes in the network; e ij the fibre between node i(i V)and node j(j V); E the set of all fibres in the network, i.e., {e ij },(i V, j V); W the number of wavelengths used in the network; N the number of nodes in the network. SSLD related parameters: l h the hth SSLD. If it is accepted, we use the same notation to refer to the lightpath that is provided to the SSLD; L the total number of SSLDs, 0 h<l. Scheduling related parameters: Z the total time slots of our scheduling problem; t h the holding time of SSLD l h. Based on our assumption, it is the same as the lifespan of lightpath l h,if SSLD l h is accepted; [b h,b h ] the desired window of starting time for SSLD l h, 0 b h b <Z; y h the weight for earliness penalty of lightpath l h ; r h the weight for tardiness penalty of lightpath l h ; E h the overall timing violation penalty of lightpath l h. Cost and revenue related parameters: h ij t the cost of using a wavelength channel on link e ij (e ij E) for time slot t; C h the routing cost of lightpath l h, i.e., the cost of wavelength channels used by lightpath l h ; D h the dual routing cost of lightpath l h ; P the penalty for rejecting an SSLD, i.e., the loss of revenue if an SSLD is rejected. Decision variables: α h the binary integer variable indicating the admission status of SSLD l h. It is one, if SSLD l h is accepted. Otherwise, it is zero; β h the starting time slot of lightpath l h, 0 β h <Z; δij h ct the binary integer variable representing the use of the cth wavelength channel on fibre e ij (e ij E,0 c< W) at time slot t by lightpath l h. It is one, if lightpath l h uses such wavelength channel at such time slot. Otherwise, it is zero; A the admission status of all SSLDs, i.e., (α h ), 0 h< L; B the starting time slots of all lightpaths, i.e., (β h ), 0 h<l; h the RWA scheme of lightpath l h, i.e., (δij h ct ) h; the RWA schemes of all lightpaths, i.e., ( h ), 0 h<l. 4.2 Objective function Our goal is to provide as much required connectivity for data centres as possible for their desired peak operation time slots, so that their usage of green energy is maximized. We model our goal by an optimization objective as minimizing the rejection of requests, the resource usage and the timing violations of lightpaths. We want to accept as many profitable requests as possible, and for the accepted requests, we want to find lightpath schedules that respect their timing preference as much as possible, while at the same time provide them with RWA schemes that use as few resources as possible. Our objective function is to minimize the function J, i.e., [ min {J }, where J (1 αh )P + α h (C h + E h ) ]. A,B, 0 h<l The overall penalty consists of the rejection penalty (i.e., P ), the resource usage cost (i.e., C h ) and the timing violation penalty (i.e., E h ). The timing violation penalty could be either an earliness penalty or a tardiness penalty. The shortage of effective service time caused by schedule earliness and tardiness is shown in Fig. 3. When SSLD l h is scheduled sooner than its desired starting time, the lightpath will be removed sooner than the desired ending time, causing a shortage of the effective service time after the lightpath is removed. This is because we assume the lifespan of lightpath l h is exactly the same as the holding time of SSLD l h. On the other hand, when lightpath l h is scheduled later than its desired starting time of SSLD l h, there will be a shortage of effective service time before the lightpath starts.

79 J. Wu et al. either fully or partially powered by green energy. When the data centre operates outside of the green energy availability time window, it has to use brown energy. To maximize its use of green energy, its connectivity needs to be scheduled to match its green energy availability time window as much as possible. We impose higher operational penalty, if its connectivity does not perfectly match its green energy availability. In this paper, we adopt earliness and tardiness penalties defined in [69]: Fig. 3 Shortage of effective service time caused by schedule earliness and tardiness y h (b h β h ) 2 if β h <b h (i.e., earliness penalty) 0 if b h β h b h (i.e., no penalty, since the E h = lightpath starts within its desired starting time window) r h (β h b h )2 if β h >b h (i.e., tardiness penalty) 0 h<l (1) where y h and r h are the weights for earliness and tardiness penalties of lightpath l h. Our earliness and tardiness penalties reflect a high penalty as a data centre is forced to use brown energy, when its connectivity moves away from its desired timing window (shown in Fig. 5). When y h and r h are set to infinitely large positive values and the resource costs are set to zero, this formulation then becomes the same as the fixed time-window scheduling problem, which does not allow any timing violations. The cost of routing lightpath l h is denoted as C h and defined as the total cost of using wavelength channels. We use this definition to illustrate that the cost of a routing lightpath is a weighted summation of certain parameters related to links and nodes. Such a definition may be extended to incorporate other parameters without changing the framework proposed in this paper. Fig. 4 Green energy availability and brown energy consumption of a data centre for a period of one day Some sources of green energy are intermittent and are not always available. As an example, we illustrate green energy availability and brown energy consumption of a data centre for a period of one day in Fig. 4. If the data centre operates at its peak capacity, there are two cases of its required energy: C h = β h t<(β h +t h ) ( h ij t e ij E 0 c<w δ h ij ct ), 0 h<l (2) Our design variables are the admission status of all SSLDs (A), the starting time slots of all lightpaths (B), and the RWA schemes for all lightpaths ( ). Our design variables are not completely independent. They represent three inter-related sub-problems, i.e., lightpath request admissions, lightpath scheduling, and RWAs.

80 Lightpath scheduling and routing for green data centres Fig. 5 Example of earliness and tardiness penalties reflecting a decreasing tolerance level as a scheduled lightpath moves away from its desired timing window 4.3 Constraints Lightpath continuity constraints If SSLD l h is admitted, its RWA must be continuous along its path and be terminated at its two end nodes. δij h ct δjict h j V 0 c<w j V 0 c<w α h if i is the source node of s h = α h if i is the destination node of s h 0 otherwise, 0 h<l,i V,0 t<z (3) If l h is accepted (i.e. α h = 1), at its source node, there is one lightpath going out; at its destination node, there is one lightpath coming in; at any intermediate node, this lightpath does not contribute to the number of lightpaths that terminate at this node. For any node that is not related to this lightpath, or when l h is rejected (i.e., α h = 0), this lightpath does not contribute to the number of lightpaths that terminate at the node. These constraints confine that if and only if α h = 1, during the lifespan of lightpath l h (i.e., β h t<(β h +t h )), there must be a lightpath from its source node to its destination node Exclusive wavelength channel usage constraints δij h ct 1, e ij E,0 c<w,0 t<z (4) 0 h<l Every wavelength channel at any time slot t cannot be used by more than one lightpath Lightpath persistency constraints δ h ij cx = δh ij cy, 0 h<l,e ij E,0 x<z,0 y<z (5) During the lifespan of lightpath l h, its RWA scheme must remain the same for all time slots. Fig. 6 Example network for performance evaluation (NSFNET) 5 Numeric results We study the design tradeoffs in an example network operating under randomly generated connectivity requirements of data centres, which represent their best utilization of green energy. Our example network is a mesh topology network (i.e., NSFNET) with 14 nodes and 21 links. Each node represents a data centre, whose internal structure is shown in Fig. 2. The network topology is shown in Fig. 6, which marks the sequence number of nodes and links. In this section, we present results for one particular connectivity requirement of data centres, while the same trends are observed under several other connectivity requirements. The number of SSLDs for all node pairs is listed in Table 1, where the number on the ith row and the jth column represents the total number of SSLDs demands from node i to j. We randomly assign their values between 0 and 3. The total number of SSLDs is 286. Their timing requirements are also randomly generated. We applied a Lagrangian Relaxation and Subgradient Method (LRSM) to the formulated optimization problem. We aim at obtaining near-optimal solutions to our problem, while providing a tight performance bound that can be used to evaluate the optimality of our solution. Details on LRSM can be found in [70]. In our first example, we study the tradeoffs between multiple design criteria: Consumption of brown energy; Capability of providing required connectivity to data centres;

81 J. Wu et al. Table 1 Number of SSLDs for all node pairs Network resource utilization; and Overall operation objective. We first set the tardiness penalty to a fixed value and study the impact of the earliness penalty. We vary the weight of the earliness penalty for SSLDs, so that when an SSLD s starting time is earlier than its desired starting time, different earliness penalties are imposed on the operation objective. The indices of the consumption of brown energy are measured by our timing violation function: Brown Energy Consumption Earliness Index = y h (b h β h ) 2 (6) 0 h<l Brown Energy Consumption Tardiness Index = r h (β h b h )2 (7) 0 h<l To better understand the tradeoffs between network parameters, we introduce two measurements of timing violation: the Sum of Earliness Violations (SEV), and the Sum of Tardiness Violations (STV), defined as: Sum of Earliness Violations (SEV) = min{0,(b h β h )} (8) 0 h<l Sum of Tardiness Violations (STV) = min{0,(β h b h )} (9) 0 h<l As the weight for earliness penalty y h increases, fewer earliness violations are observed (shown in Fig. 7). The reason is that with a given total cost budget for each lightpath setting to a fixed value (i.e., P = 100), a high weight for earliness penalty quickly uses up the total cost budget. Fig. 7 Sum of Earliness Violations as y h varies Fig. 8 Total number of rejected SSLDs vs. the Brown Energy Consumption Earliness Index as y h varies For the same reason, as the weight for earliness penalty y h increases, the total number of rejected SSLDs increases, because SSLDs with insufficient cost budget are rejected (shown in Fig. 8). When the weight for earliness penalty y h reaches a certain value, SSLDs are either scheduled strictly respecting their preferred timing requirements, or rejected due to their insufficient cost budget to cover the high penalty for earliness (shown in Fig. 8). The Brown Energy Consumption Earliness Index sharply increases, as the weight for earliness penalty y h increases (shown in Fig. 8). At a certain point, the Brown Energy Consumption Earliness Index gradually decreases (shown in Fig. 8), since the total number of earliness violations decreases at a faster pace (shown in Fig. 7). When all accepted SSLDs are scheduled strictly respecting their timing require-

82 Lightpath scheduling and routing for green data centres Fig. 9 Achieved optimization objective and its bound as y h varies Fig. 11 Sum of tardiness violations as r h varies Fig. 10 Total number of rejected SSLDs vs. the Brown Energy Consumption Tardiness Index as r h varies Fig. 12 Number of rejected SSLDs as W varies ment, the Brown Energy Consumption Earliness Index stays atzero(showninfig.8). No early operation at data centres is scheduled before their green energy approximately reaches peak. However, completely disabling earliness in data centre operation schedule is not optimal for the overall objective, which is evidenced by a higher achieved objective value (shown in Fig. 9). In Fig. 9, we also demonstrate that our results are highly optimal (all within 3% from the bound). We now set the earliness penalty to a fixed value and study the impact of the tardiness penalty, i.e., we vary the weight of the tardiness penalty for SSLDs. The tradeoff between the consumption of brown energy and the capability of providing required connectivity to data centres is observed in Figs. 10 and 11. In our second example, we study the impact of available network resources on the optimization objective. We vary the number of wavelength channels on each fibre (denoted by W). The impact on the number of rejected SSLDs is showninfig.12. The achieved optimization objective and its bound are shown in Fig. 13. In this example, we set parameters h ij t = 4, P = 100, y h = 20 and r h = 20. In this way, a lightpath may be scheduled up to two time slots ahead or behind of its desired timing, which causes a penalty of = 80 within its 100 total budget. In Fig. 12, we can see that as W reduces, the number of rejected SSLD reduces. We do not observe any obvious trend in changing the timing violations as W changes. In Fig. 13, we can see again that our algorithm consistently produces a near-optimal solution that is very close to the lower bound.

83 J. Wu et al. Fig. 13 Achieved optimization objective and its bound as W varies Fig. 14 Impact of h ij t on the average hop count In our third example, we study the impact of the cost of a hop of a lightpath, which includes the cost of one adjacent node and a wavelength channel. We fix the other parameters and vary the cost of wavelength channels (denoted by h ij t ). The results are shown in Figs. 14 and 15. We grouped the SSLDs into three groups based on their holding times. We can see in Fig. 14 that as h ij t increases from 0 to 40, the average hop counts of each group drop at different rates. In Fig. 15, we demonstrate that the performance of our scheduling results is mostly optimal for this study. The network operator can thus easily control the hop number of the routings by adjusting the h ij t value. Please note that although for the simplicity of our numerical experiments, we only set the penalty for rejecting an SSLD (i.e., P )tothe same value for all SSLDs, our model allows to set different values for individual SSLDs. The penalty for rejecting a given SSLD is an artificial value that operators of data centres are willing to pay for setting up the SSLD. 6 Conclusions In this paper, we study the benefits and tradeoffs of using scheduled lightpaths to connect data centres for optimizing the use of intermittent renewable energy sources. The prior knowledge of server workload and traffic patterns, as well as

84 Lightpath scheduling and routing for green data centres Fig. 15 Achieved optimization objective and its bound as h ij t varies the green energy sources availability provides an opportunity to schedule lightpaths to serve our goal of maximizing the use of green energy at the network planning stage. We proposed a WDM network planning model, which allows the operator of networks and data centres to set the scheduling and cost related parameters, making tradeoffs between competing operation goals. We reveal the tradeoffs between the consumption of brown energy and the capability of providing required connectivity to data centres. Our simulation results show how timing flexibility improves network resource utilization and reduces rejections. We also study the impact of available network resources and the cost of network resources on the optimization objective. Our future work includes modeling lightpaths with variable holding times. References 1. Zhang, Y., Chowdhury, P., Tornatore, M., & Mukherjee, B. (2010). Energy efficiency in telecom optical networks. IEEE Communications Surveys and Tutorials, 12(4), Bolla, R., Bruschi, R., Davoli, F., & Cucchietti, F. (2011). Energy efficiency in the future internet: a survey of existing approaches and trends in energy-aware fixed network infrastructures. IEEE Communications Surveys & Tutorials, 13(2), Rahman, S. (2003). Green power: what is it and where can we find it? IEEE Power & Energy Magazine, 1(1), Systems, Cisco (2006). Transport Options for Data Center Optical Interconnection. optical/ps5724/ps2006/prod_white_paper0900aecd803e884f.pdf. 5. Wen, J., Zheng, Y., & Donghan, F. (2009). A review on reliability assessment for wind power. Renewable & Sustainable Energy Reviews, 13(9), Skea, J., Anderson, D., Green, T., Gross, R., Heptonstall, P., & Leach, M. (2008). Intermittent renewable generation and maintaining power system reliability. IET Generation, Transmission & Distribution, 2(1), Karki, R., Hu, P., & Billinton, R. (2006). A simplified wind power generation model for reliability evaluation. IEEE Transactions on Energy Conversion, 21(2), El-Tamaly, H. H., & Mohammed, A. A. E. (2006). Impact of interconnection photovoltaic/wind system with utility on their reliability using a fuzzy scheme. Renewable Energy, 31(15), Georgilakis, P. S., & Katsigiannis, Y. A. (2009). Reliability and economic evaluation of small autonomous power systems containing only renewable energy sources. Renewable Energy, 34(1), Watson, S. J., & Ter-Gazarian, A. G. (1996). The optimisation of renewable energy sources in an electrical power system by use of simulation and deterministic planning models. International Transactions in Operational Research, 3(3 4), Lefurgy,C.,Rajamani,K.,Rawson,F.,Felter,W.,Kistler,M.,& Keller, T. W. (2003). Energy management for commercial servers. Computer, 36(12), Chen, G., He, W., Liu, J., Nath, S., Rigas, L., Xiao, L., & Zhao, F. (2008). Energy-aware server provisioning and load dispatching for connection-intensive internet services. In 5th USENIX symposium on networked systems design and implementation (NSDI 2008) (pp ). San Francisco, California, USA, April 16 18, Chase, J. S., Anderson, D. C., Thakar, P. N., Vahdat, A. M., & Doyle, R. P. (2001). Managing energy and server resources in hosting centers. Operating Systems Review, 35(5), Special issue Proceedings of 18th ACM Symposium on Operating Systems Principles (SOSP 2001), October 21 24, 2001, Banff, Alberta, Canada. 14. Zhu, Q., Chen, Z., Tan, L., Zhou, Y., Keeton, K., & Wilkes, J. (2005). Hibernator: helping disk arrays sleep through the winter. Operating Systems Review, 39(5), Special issue Proceedings of 20th ACM Symposium on Operating Systems Principles (SOSP 2005), October 23 26, 2005, Brighton, UK. 15. Gencata, A., & Mukherjee, B. (2003). Virtual-topology adaptation for WDM mesh networks under dynamic traffic. IEEE/ACM Transactions on Networking, 11(2), Thompson, K., Miller, G. J., & Wilder, R. (1997). Wide-area Internet traffic patterns characteristics. IEEE Network, 11(6), Kuri, J., Puech, N., Gagnaire, M., Dotaro, E., & Douville, R. (2003). Routing and wavelength assignment of scheduled light-

85 J. Wu et al. path demands. IEEE Journal on Selected Areas in Communications, 21(8), Gagnaire, M., Koubaa, M., & Puech, N. (2007). Network dimensioning under scheduled and random lightpath demands in alloptical WDM networks. IEEE Journal on Selected Areas in Communications, 25(9), Wang, B., & Li, T. (2010). Survivable scheduled service provisioning in WDM optical networks with iterative routing. Optical Switching and Networking, 7(1), Skorin-Kapov, N. (2006). Heuristic algorithms for the routing and wavelength assignment of scheduled lightpath demands in optical networks. IEEE Journal on Selected Areas in Communications, 24(8), Lee, S. S. W., Chen, A., & Yuang, M. C. (2010). A Lagrangian relaxation based near-optimal algorithm for advance lightpath reservation in WDM networks. Photonic Network Communications, 19(1), Jaekel, A., & Chen, Y. (2009). Resource provisioning for survivable WDM networks under a sliding scheduled traffic model. Optical Switching and Networking, 6(1), Wang, B., Li, T., Luo, X., Fan, Y., & Xin, C. (2005). On service provisioning under a scheduled traffic model in reconfigurable WDM optical networks. In 2nd international conference on broadband networks (BROADNETS 2005) (Vol. 1, pp ). Boston, Massachusetts, USA, October 3 7, Su, W., Sasaki, G., Su, C. F., & Balasubramanian, A. (2006). Scheduling of periodic connections with flexibility. Optical Switching and Networking, 3(3 4), Nguyen, H. H., Gurusamy, M., & Zhou, L. (2009). Scheduling network and computing resources for sliding demands in optical grids. IEEE/OSA Journal of Lightwave Technology, 27(12), Shen, L., Yang, X., Todimala, A., & Ramamurthy, B. (2007). A two-phase approach for dynamic lightpath scheduling in WDM optical networks. In IEEE international conference on communications (ICC 2007) (pp ). Glasgow, UK, June 24 28, Tornatore, M., Ou, C., Zhang, J., Pattavina, A., & Mukherjee, B. (2005). PHOTO: an efficient shared-path-protection strategy based on connection-holding-time awareness. IEEE/OSA Journal of Lightwave Technology, 23(10), Lucerna, D., Tornatore, M., Mukherjee, B., & Pattavina, A. (2009). Dynamic routing of connections with known duration in WDM networks. In IEEE global telecommunications conference (GLOBECOM 2009), Honolulu, Hawaii, USA, November 30 December 4, Environmental Protection Agency (2007). EPA report to Congress on server and data center energy efficiency (August). downloads/epa_datacenter_report_congress_final1.pdf. 30. Barroso, L. A., & Holzle, U. (2007). The case for energyproportional computing. Computer, 40(12), Tsai, C. H., Shin, K. G., Reumann, J., & Singhal, S. (2007). Online web cluster capacity estimation and its application to energy conservation. IEEE Transactions on Parallel and Distributed Systems, 18(7), Pinheiro, E., Bianchini, R., Carrera, E. V., & Heath, T. (2001). Dynamic cluster reconfiguration for power and performance. In 2nd workshop on compilers and operating systems for low power (COLP 2001) (pp ). Barcelona, Spain, September 9, Bohrer, P., Elnozahy, E. N., Keller, T., Kistler, M., Lefurgy, C., McDowell, C., & Rajamony, R. (2002). The case for power management in web servers. In R. Graybill & R. Melhem (Eds.), Power aware computing (pp ). Norwell: Kluwer Academic. 34. Meisner, D., Gold, B. T., & Wenisch, T. F. (2009). PowerNap: eliminating server idle power. ACM SIGPLAN Notices, 44(3), Special issue Proceeding of the 14th international conference on Architectural support for programming languages and operating systems (ASPLOS 2009), March 7 11, 2009, Washington, DC, USA. 35. Fan, X., Weber, W. D., & Barroso, L. A. (2007). Power provisioning for a warehouse-sized computer. Computer Architecture News, 35(2), Nathuji, R., Somani, A., Schwan, K., & Joshi, Y. (2008). CoolIT: coordinating facility and it management for efficient datacenters. In 2008 conference on power aware computing and systems (Hot- Power 2008) (p. 15). San Diego, California, USA, December 7, Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., & Warfield, A. (2003). Xen and the art of virtualization. Operating Systems Review, 37(5), Special issue Proceedings of 19th ACM symposium on operating systems principles (SOSP 2003), October 19 22, 2003, Bolton Landing (Lake George), New York, USA. 38. Clark, C., Fraser, K., Hand, S., Hansen, J. G., Jul, E., Limpach, C., Pratt, I., & Warfield, A. (2005). Live migration of virtual machines. In 2nd conference on symposium on networked systems design & implementation (NSDI 2005) (Vol. 2, pp ). Boston, Massachusetts, USA, May 2 4, Sugerman, J., Venkitachalam, G., & Lim, B. H. (2001). Virtualizing I/O devices on VMware workstation s hosted virtual machine monitor. In USENIX annual technical conference (ATC 2001) (pp. 1 14). Boston, Massachusetts, USA, June 25 30, Nelson, M., Lim, B. H., & Hutchins, G. (2005). Fast transparent migration for virtual machines. In USENIX annual technical conference (ATC 2005) (p. 25). Anaheim, California, USA, April 10 15, Figueiredo, R. J., Dinda, P. A., & Fortes, J. A. B. (2003). A case for grid computing on virtual machines. In 23rd IEEE international conference on distributed computing systems (ICDCS 2003) (pp ). Providence, Rhode Island, USA, May 19 22, Milojicic, D. S., Douglis, F., Paindaveine, Y., Wheeler, R., & Zhou, S. (2000). Process migration. ACM Computing Surveys, 32(3), Bobroff, N., Kochut, A., & Beaty, K. (2007). Dynamic placement of virtual machines for managing SLA violations. In 10th IFIP/IEEE international symposium on integrated network management (IM 2007) (pp ). Munich, Germany, May 21 25, Nathuji, R., & Schwan, K. (2007). Virtual power coordinated power management in virtualized enterprise systems. Operating Systems Review, 41(6), Special issue Proceedings of 21st ACM Symposium on Operating Systems Principles (SOSP 2007), October 14 17, 2007, Stevenson, Washington, USA. 45. Mahadevan, P., Sharma, P., Banerjee, S., & Ranganathan, P. (2009). Energy aware network operations. In 28th IEEE international conference on computer communications (INFOCOM 2009) workshops (pp ). Rio de Janeiro, Brazil, April 19 25, Bianchini, R., & Rajamony, R. (2004). Power and energy management for server systems. Computer, 37(11), Heath, T., Diniz, B., Carrera, E. V., Meira, W. Jr., & Bianchini, R. (2005). Energy conservation in heterogeneous server clusters. In 10th ACM SIGPLAN symposium on principles and practice of parallel programming (PPoPP 2005) (pp ). Chicago, Illinois, USA, June 15 17, Weddle, C., Oldham, M., Qian, J., Wang, A. I. A., Reiher, P., & Kuenning, G. (2007). PARAID: a gear-shifting power-aware RAID. ACM Transactions on Storage, 3(3), Qureshi, A. (2008). Plugging into energy market diversity. In 7th ACM workshop on hot topics in networks (HotNets 2008),Calgary, Canada, October 2008.

86 Lightpath scheduling and routing for green data centres 50. Lu, L., & Varman, P. (2008). Workload decomposition for power efficient storage systems. In 2008 conference on power aware computing and systems (HotPower 2008) (p. 13). San Diego, California, USA, December 7, Nathuji, R., Isci, C., Gorbatov, E., & Schwan, K. (2008). Providing platform heterogeneity-awareness for data center power management.cluster Computing, 11(3), Special Issue on Autonomic Computing. 52. Moore, J., Chase, J., Ranganathan, P., & Sharma, R. (2005). Making scheduling cool : temperature-aware workload placement in data centers. In USENIX 2005 annual technical conference (ATC 2005) (pp ). Anaheim, California, USA, April 10 15, Verma, A., Ahuja, P., & Neogi, A. (2008). pmapper: power and migration cost aware application placement in virtualized systems. In V. Issarny & R. E. Schantz (Eds.), Lecture notes in computer science (LNCS) Subseries: programming and software engineering: Vol th ACM/IFIP/USENIX international conference on middleware (Middleware 2008), Leuven, Belgium, December 1 5, 2008 (pp ). New York: Springer. 54. Bash, C., & Forman, G. (2007). Cool job allocation: measuring the power savings of placing jobs at cooling-efficient locations in the data center. In 2007 USENIX annual technical conference (ATC 2007) (pp ). Santa Clara, California, USA, June 17 22, Burge, J., Ranganathan, P., & Wiener, J. L. (2007). Cost-aware scheduling for heterogeneous enterprise machines (CASH EM). In 2007 IEEE international conference on cluster computing (pp ). Austin, Texas, USA, September 17 20, Beloglazov, A., & Buyya, R. (2010). Energy efficient resource management in virtualized cloud data centers. In 10th IEEE/ACM international conference on cluster, cloud and grid computing (CCGRID 2010) (pp ). Melbourne, Australia, May 17 20, Berral, J. L., Goiri, I., Nou, R., Julia, F., Guitart, J., Gavalda, R., & Torres, J. (2010). Towards energy-aware scheduling in data centers using machine learning. In 1st international conference on energy-efficient computing and networking (e-energy 2010) (pp ). Passau, Germany, April 13 15, Kusic, D., Kephart, J. O., Hanson, J. E., Kandasamy, N., & Jiang, G. (2009). Power and performance management of virtualized computing environments via lookahead control. Cluster Computing, 12(1), Special Issue on Autonomic Computing. 59. Le, K., Bianchini, R., Martonosi, M., & Nguyen, T. D. (2009). Cost- and energy-aware load distribution across data centers. In Workshop on power aware computing and systems (HotPower 2009). Big Sky, Montana, USA, October 10, Bertini, L., Leite, J. C. B., & Moss, D. (2010). Power optimization for dynamic configuration in heterogeneous web server clusters. The Journal of Systems and Software, 83(4), Rajamani, K., & Lefurgy, C. (2003). On evaluating requestdistribution schemes for saving energy in server clusters. In 2003 IEEE international symposium on performance analysis of systems and software (ISPASS 2003) (pp ). Austin, Texas, USA, March 6 8, Rusu, C., Ferreira, A., Scordino, C., & Watson, A. (2006). Energyefficient real-time heterogeneous server clusters. In 12th IEEE real-time and embedded technology and applications symposium (RTAS 2006) (pp ). San Jose, California, USA, April 4 7, Pakbaznia, E., Ghasemazar, M., & Pedram, M. (2010). Temperature-aware dynamic resource provisioning in a poweroptimized datacenter. In Conference on design, automation and test in Europe (DATE 2010) (pp ). Dresden, Germany, March 8 12, Pakbaznia, E., & Pedram, M. (2009). Minimizing data center cooling and server power costs. In 14th ACM/IEEE international symposium on low power electronics and design (ISLPED 2009) (pp ). San Francisco, California, USA, August 19 21, Nathuji, R., Schwan, K., Somani, A., & Joshi, Y. (2009). VPM tokens: virtual machine-aware power budgeting in datacenters. Cluster Computing, 12(2), Chlamtac, I., & Ganz, A. (1993). Lightnets: topologies for highspeed optical networks. IEEE/OSA Journal of Lightwave Technology, 11(5/6), Zhang, J. Y., Wu, J., Bochmann, G., & Savoie, M. (2009). A proof of wavelength conversion not improving the Lagrangian bound of the static RWA problem. IEEE Communications Letters, 13(5), Zhang, J. Y., Mouftah, H., Wu, J., & Savoie, M. (2010). Lightpath scheduling and routing for traffic adaptation in WDM networks. IEEE/OSA Journal of Optical Communications and Networking, 2(10), Jin, B., Luh, P. B., & Thakur, L. S. (1999). An effective optimization-based algorithm for job shop scheduling with fixedsize transfer lots. Journal of Manufacturing Systems, 18(4), Zhang, Y., Yang, O., & Liu, H. (2004). A Lagrangian relaxation and subgradient framework for the routing and wavelength assignment problem in WDM networks. IEEE Journal on Selected Areas in Communications, 22(9), Jing Wu obtained a B.Sc. degree in information science and technology in 1992, and a Ph.D. degree in systems engineering in 1997, both from Xi an Jiao Tong University, China. He is now a Research Scientist at the Communications Research Centre Canada (Ottawa, Canada), an Agency of Industry Canada. In the past, he worked at Beijing University of Posts and Telecommunications (Beijing, China) as a faculty member, Queen s University (Kingston, Canada) as a postdoctoral fellow, and Nortel Networks Corporate (Ottawa, Canada) as a system design engineer. Currently, he is also appointed as an Adjunct Professor at the University of Ottawa, School of Information Technology and Engineering. He has contributed over 70 conference and journal papers. He holds three patents on Internet congestion control, and two patents on control plane failure recovery. His research interests include control and management of optical networks, protocols and algorithms in networking, optical network performance evaluation and optimization. Dr. Wu is a cochair of the sub-committee on network architecture, management and applications of the Asia Communications and Photonics Exhibit and Conference (ACP), Dr. Wu is a Senior Member of IEEE.

87 J. Wu et al. James Yiming Zhang obtained his Ph.D. degree in electrical engineering from University of Ottawa, Ontario, Canada. He received both his B.S. and M.S. degrees in electronic science from Zhejiang University, China. He has done extensive research work in various fields including optimizations for the communication networks and the manufacturing processes, object detection in images and pattern recognition, machine learning, high-speed digital hardware design and biometrics. He is currently working on natural language processing in Idilia Inc., Montreal, Canada. Michel Savoie is the research program manager for the Broadband Applications and Optical Networks group of the Broadband Network Technologies Research Branch at the Communications Research Centre Canada (CRC), an Agency of Industry Canada. He maintains expertise in broadband systems and related technologies such as: Infrastructure as a Service (IaaS), Green ICT, Application Oriented Networking (AON), advanced IP and WDM based Optical Networks in order to provide advice on important national initiatives and to demonstrate the application of CRC technologies in a real operational environment. He has managed two projects funded under CANARIE s directed research program on UCLP and was involved with EUCALYPTUS: A Service-oriented Participatory Design Studio project funded under the CANARIE Intelligent Infrastructure Program (CIIP), PHOSPHORUS: A Lambda User Controlled Infrastructure for European Research integrated project funded by the European Commission under the IST 6th Framework and the Health Services Virtual Organization (HSVO) project under CA- NARIE s Network-Enabled Plaftorms program. He is involved with the High Performance Digital Media Network (HPDMnet) research initiative and the GreenStar Network project under CANARIE s Green IT Pilot program. CRC and its partners are currently engaged in the further development of the IaaS Framework software. Mr. Savoie holds a B.Sc. and M.Sc. in Electrical Engineering from the Univ. of New Brunswick.

88 This page intentionally left blank This page intentionally left blank This page intentionally left blank

89 ENERGY EFFICIENCY IN COMMUNICATIONS Leveraging Green Communications for Carbon Emission Reductions: Techniques, Testbeds, and Emerging Carbon Footprint Standards Charles Despins, Prompt Inc. and École de Technologie Supérieure Fabrice Labeau and Tho Le Ngoc, McGill University Richard Labelle, The Aylmer Group Mohamed Chériet, Claude Thibeault, and François Gagnon, École de Technologie Supérieure Alberto Leon-Garcia, University of Toronto Omar Cherkaoui, Université du Québec à Montréal Bill St. Arnaud, St. Arnaud, Walker & Associates Jacques McNeill, Prompt Inc. Yves Lemieux, Ericsson Canada Mathieu Lemay, Inocybe Technologies ABSTRACT Green communication systems and, in broader terms, green information and communications technologies have the potential to significantly reduce greenhouse gas emissions worldwide. This article provides an overview of two issues related to achieving the full carbon abatement potential of ICT. First, green communications research challenges are discussed, notably as they pertain to networking issues. Various initiatives regarding green ICT testbeds are presented in the same realm in order to validate the green performance and functionality of such greener cyber-infrastructure. Second, this article offers a description of ongoing international efforts to standardize methodologies that accurately quantify the carbon abatement potential of ICTs, an essential tool to ensure the economic viability of green ICT in the low carbon economy and carbon credit marketplace of the 21st century. INTRODUCTION As climate change has emerged as a global challenge over the last decade, the international community is now increasingly unified in a call to action. It is a challenge that not only jeopardizes the sustainability of our planet; it poses significant long-term threats to the global economy. According to former U.K. Government and World Bank Chief Economist Lord Stern [1], if no action is taken, the overall costs and risks of climate change will be equivalent to losing at least 5 percent of global gross domestic product (GDP) each year. Not acting now will incur a wider range of impacts and the damage estimates could rise to 20 percent of global GDP or more. The information and communications technologies (ICT) industry is estimated to contribute 2 to 3 percent [1] of global greenhouse gas (GHG) emissions, a share that is quickly rising. More significantly, however, the ICT industry also has the potential to reduce current global GHG emissions by percent [1], notably as a result of ICT s pervasiveness in various industry and societal sectors. In 2008 alone, green ICT, green communications, and related applications helped to eliminate 376 million metric tons of carbon [1]. By 2013, this volume could exceed 1.2 billion metric tons. This creates a huge potential for the ICT industry, academia, and governments to demonstrate leadership on the climate change issue while at the same time exploiting an economic opportunity estimated [1] at US$800 billion annually in worldwide energy cost savings by Moreover, with the mitigated outcomes of the 2009 United Nations Climate Change Summit, green ICT presents an unexploited and immediate opportunity for the ICT industry. Green ICT has the potential to reduce carbon emissions without some of the difficult compromises that typically stifle progress in such international summits. IEEE Communications Magazine August /11/$ IEEE 101

90 Not only must the aspect of load be considered, but also the data center cooling requirements generated by a service have to be measured and accounted for before migrating operations. This kind of management is realized in a manner that is transparent to end users. This article provides an overview of two of the main challenges that face the ICT industry in its contribution to reducing GHG emissions. First, we present a description of some key research issues and potential solution avenues that will help reduce the carbon footprint of communication systems, particularly from the point of view of network architectures; these sections also describe various initiatives regarding green ICT testbeds that can be used to validate the performance of such greener ICTs. Second, we study ways in which the green impact of such technologies can be quantified; the economic viability of these new technologies in the low carbon economy and carbon credit marketplace of the 21st century will rely on such measures of impact, particularly when applied [1] in non-ict sectors where ICT has the greatest GHG emission reduction potential. As such, we describe ongoing efforts to define standardized methodologies to accurately quantify the carbon abatement potential of ICTs. GREEN NETWORKING RESEARCH: REDUCING ICT S FOOTPRINT This section describes research challenges and potential solutions to be considered when trying to reduce the carbon footprint of communications. The creation of a green cyber-infrastructure must rely on advances in many areas, from energy-efficient hardware components to better software. Life-cycle management issues, while critical in greening the ICT industry, are not addressed in this article as it targets energy consumption and carbon emission abatement. Advances in the hardware arena have been numerous: energy consumption is being reduced in processors, memory modules, and ICT peripheral and network devices. Various policies promote the concept of power on demand, as opposed to power always on, to reduce energy consumption. New technologies, notably at the physical layer level, are more energy efficient as well as more technically proficient. In this section, however, we focus on an area that is less commonly thought of as an enabler of a green cyber-infrastructure: network architecture. High-speed optical networks allow ICT infrastructure, data centers, to be located in remote sites where renewable energy sources are found. Virtualization, infrastructure as aservice (IaaS) middleware, and optical networking are the key elements that enable green networks. Virtualization allows the flexible allocation of resources to different communities of users according to their specific requirements. Network virtualization relies on technologies that combine or divide computing resources, presenting to network providers one or many operating environments using methodologies like hardware and software partitioning or aggregation. Network virtualization has consisted until now in deploying network services (VLAN, VPN, etc.) or virtual routers. Today, it is evolving into the deployment of multiple distinct networks over the same physical infrastructure. IaaS uses service-oriented principles to identify, manage, and reconfigure resources in a shared computing and communications infrastructure. Optical networks provide the agility that enables the effective sharing of the resources in this infrastructure. As key ICT GHG emission sources include data centers and communication networks, we now describe how virtualization can help the ICT industry to deal with GHG emission reductions. GHG EMISSION REDUCTION IN DATA CENTERS Virtual machines (VMs) that encapsulate virtualized services can be moved, copied, created and deleted according to management policies. Although energy efficiency can be achieved by hardware consolidation, server hibernating or hardware design optimization, the degree of GHG reduction based on an energy efficient strategy in data centers is limited. Virtualization allows services not to be managed within a data centre site, and to be moved to other sites where they can operate in the most environmental friendly way. Not only must the aspect of load be considered, but also the data center cooling requirements generated by a service have to be measured and accounted for before migrating operations. This kind of management is realized in a manner that is transparent to end users. GHG EMISSION REDUCTION IN NETWORKS AND PROTOCOLS Reducing GHG emissions for communication networks involves dealing with the trade-offs between performance, energy savings, and quality of service (QoS). Migrating to optical networks is the current solution to reduce GHG emissions, as in general, optical devices consume less energy than electronic ones. Network devices can also be designed with features that create an opportunity for energy-saving operation such as turning off network interfaces and throttling of processors. Network protocols could also be optimized, or even be redeveloped in a way that enhances the energy-efficient operation of the network elements. However, we believe that an essential part of the solution is to transfer services from energy-inefficient to greener devices. This can be achieved by delegating protocol processing operations (i.e., routing) to servers located at green powered data centers. Network devices will simply perform signaling operations that need low bandwidth, thus reducing the total amount of GHG emissions of the communication network. In such a network model (e.g., Fig. 1), customer premise flows feed into edge routers, which in turn are aggregated in core switches that are interconnected to other core switches, including those attached to large data centers. A green network architecture requires the relocation of physical infrastructure to sites with renewable energy. We believe that this is possible through new traffic grooming and virtualization techniques. The optimum configuration of green networks can be formulated as an optimization of the placement and sizing of the resources shown in Fig. 2 to meet traffic demand forecasts. In [2], this model was used to investigate the design of IP over wavelength-division multiplexing (WDM) networks to minimize overall energy consumption. The study found that routers currently 102 IEEE Communications Magazine August 2011

91 account for approximately 90 percent of the overall power consumed in a range of network scenarios. Consequently, the optimum designs essentially minimize the number of router ports and maximize the use of optical bypass to minimize the number of router hops. In [3], future packet and circuit switch technologies were compared, and it was found that the relative power consumption performance between electronic routers and optical circuit switches will persist. Clearly, placement of router functionality at renewable energy sites will be an important element in the design of green networks. Consider now an architecture where routing functionality is concentrated at sites with renewable energy as shown in Fig. 2. As before, edge router flows feed into core routers; however core router ports are now virtualized and the edge router flows are aggregated and transported using optical paths, possibly including sub-wavelength multiplexing for efficient use of wavelengths, to remote sites where the physical resources that provide router functionality are located. Existing large-scale router designs may initially provide router functionality, so the aggregation will lead to better port fill levels. However, entirely new physical system designs to provide virtualized router functionality also become possible. We speculate that the economy of scale benefits of cloud computing may also accrue to these green networks. The lack of sustained availability of wind and solar power require the development of monitoring, prediction, and reallocation mechanisms to ensure uninterrupted operation of green networks. Virtualization enables a separation between the IP and underlying substrate and facilitates the replacement of the underlying physical resources with the availability of renewable energy. New network topology design and fault tolerance methodologies will be thus required to ensure that green networks provide uninterrupted service. Edge Edge Core Edge GREEN CYBERINFRASTRUCTURE TESTBEDS A major obstacle to the development of new network architectures that can reduce GHG emissions is the ability to experiment with proposed network architectures in large-scale environments that involve large numbers of end users. To address this problem, several initiatives, including GENI in the United States and FEDERICA in Europe, are designing and building large-scale testbeds for networking research. In North America, the Canada-California Strategic Innovation Partnership (CCSIP) also regroups researchers from various disciplines to collaborate on the development of a zero-carbon Internet. In general, the development of green ICT cyber-infrastructure requires testbeds that combine new ICT infrastructures and renewable sources of energy. In the following, we report on various Canadian efforts to develop such testbeds. A GREEN NETWORK PILOT: THE GREENSTAR PROJECT Edge Figure 1. Typical IP over WDM configuration. The GreenStar Network (GSN) [4] is a green cyber-infrastructure pilot testbed to share infrastructure and maximize lower-cost power with follow the wind, follow the sun networks. Several zero-carbon energy sites have been selected for the location of network and computing resources in a testbed that can be managed using IaaS. This project includes international collaboration between Canada, the United States, Spain, Ireland, Australia, and China. GSN, an alliance of Canada s leading ICT companies, universities, and international partners, is led by the École de Technologie Supérieure (ÉTS) in Montreal. GSN will develop the world s first Internet network whose nodes will be powered entirely by hydroelectricity, wind, and solar energy and yet will provide the same reliability to users as the current Internet network does. The GreenStar Network will be applied to two green ICT service provision scenarios: Edge Core Edge Edge Edge Edge Edge Edge Edge Virtual core Figure 2. IP over WDM configuration with core routing functions virtualized and instantiated in a routing center. IEEE Communications Magazine August

92 Extended GSN nodes RackForce Cybera CRC ETS UQAM SigmaCo BastionHost S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 L2 switch L2 switch L2 switch L2 switch L2 switch L2 switch L2 switch 1... n Canarie, Canada OME 6500 Vancouver OME 6500 Calgary OME 6500 Winnipeg OME 6500 ThunderBay OME 6500 Toronto OME 6500 Ottawa3 OME 6500 Ottawa2 OME 6500 Montreal OME 6500 OME 6500 Fredericton Halifax PNWG Star light MANLAN Nether light L2 switch L2 switch L2 switch L2 switch L2 switch L2 switch S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 S1 S2 S3 UCSD MCNC Esnet HEAnet, Ireland i2cat, Spain IBBT, Belgium Figure 3. GSN connectivity diagram, upper: core nodes (Canada partner nodes); middle: CANARIE optical network; lower: international partner nodes (UCSD, MCNC, and ESnet from the United States; HEAnet, I2CAT, and IBBT from the European Union). All the nodes are connected to the CANARIE network using lightpaths at layer 2 (Canada) and layer 3 (international). The creation and enactment of a green IT protocol based on the utilization of a green data center The development of management and technical policies that leverage virtualization mobility to facilitate use of renewable energy within the GreenStar Network The GSN Carbon Measurement Protocol will also be the world s first. Although the International Organization for Standardization (ISO) standard upon which the protocol is based is straightforward, its specialization to ICT will require synergistic solutions relating to power and performance measurement as well as network and system operation. The protocol s success will be measured by its uptake within industry, academia, and government. The Extended GreenStar Network diagram is shown in Fig. 3 and illustrates additional sites that are expected to connect to the Canadian GSN core. The idea behind the GSN project is that a carbon neutral network must consist of data centers built in proximity to clean power sources, and user applications will be moved to be executed in such data centers. Such a network must provide an ability to migrate entire virtual machines (routers and servers) to alternate data center locations with an underlying communication network supported by a high-speed optical layer having up to, say, 1000 Gb/s bandwidth capacity. Leveraging clean power sources means a greater dependency on power from renewable energy sources which however can be unreliable and unpredictable. While storage systems and large distributed electrical grids are part of the solution, ICT can play a critical role in developing smart solutions where products and services can adapt to this variable and unpredictable power availability. The GSN is the first example of such thinking, where applications are shuttled around a high-speed network to sites that have available power. Figure 4 illustrates the layered architecture of the GSN. The data plane layer includes massive physical resources, such as storage servers and application servers linked by controlled lightpaths. The platform control plane layer implements the platform-level services that provide cloud computing and networking capabilities to GSN services. The cloud middleware plane layer provides platform as service capabilities based on IaaS framework components. The top management plane layer focuses on applications by making use of services provided by the lower layers. In each GSN node, servers are installed in outdoor enclosures where the climate control is powered by green energy. Therefore, only data centers and core network equipment are considered to evaluate the GSN GHG emission reduction potential. As an example, experiments are 104 IEEE Communications Magazine August 2011

93 Although energy Management plane Cloud middleware plane Platform control plane Resource Engine Resource Engine User policies, management techniques Resource laasframework component Network virtualization tool efficiency is an important target in terms of cost savings and can in some cases directly contribute to reducing GHG emissions, it can also, in many other situations, have little or no impact on reducing these GHG emissions. VM Data plane Data center Lightpath Figure 4. Layered GSN architecture. performed in GSN data centers using a University of Calgary application, GeoChronos, which enables the Earth observation community to share data and scientific applications. The application runs on a 48-core multiprocessor server system. According to our estimation using [5], this system emits 30 tons of GHG annually, assuming it is powered by the Alberta fossilbased electrical grid. This number does not account for emissions of local switches and routers providing communications to GeoChronos. If the system is moved to a green powered server, 30 tons of GHG emissions may be saved yearly. However, such a migration consumes about kw/day, which is equivalent to 1.5 tons of GHG emission annually, assuming that a nine-node core network is composed of IP routers powered entirely by fossil-based energy and that migrations are done twice a day. Consequently in this example, the GSN yields annual GHG emission savings of 28.5 tons. GHG emissions of a core network and data centers are also compared in Fig. 5, assuming both are powered by fossil energy. For example, for a service with 520 Gbytes data volume, 1357 tons of GHG (the difference between the two curves) may thus be saved if the service is provided by the GSN. The preceding example also underscores the importance of a focus on GHG emission reductions for the ICT industry. Currently, green ICT R&D mostly focuses on energy efficiency, a theme where ICT researchers generally excel. Although energy efficiency is an important target in terms of cost savings and can in some cases directly contribute to reducing GHG emissions, it can also, in many other situations, have little or no impact on reducing these GHG emissions. To understand the relationship between energy efficiency and GHG emissions, it is important to note that GHG emissions from ICT are indirect in that they do not directly produce GHG gases through their operation but rather through the generation of electricity needed to power them. In GHG parlance, they are referred to as scope 2 emissions. Primary emissions from a power plant are classified as scope 1. In most of the world, over 50 percent of this electricity comes from coal-powered electrical generating stations, the primary source of ICT GHG emissions. As most ICT infrastructures operate 24 hours a day, they thus have very little impact on a utility s peak power consumption. Although there are a number of energy efficiency projects underway to reduce power consumption of PCs and consumer devices when they are not in use, this will have little impact on their base load power usage. Base load power is primarily produced by coal, as the cost of power from these plants is very low and they are most efficient at full power operation. More expensive gas and hydro-electricity are generally used to provide peak power only. Therefore, although many power utilities may claim a power mix that is relatively balanced between coal, gas, nuclear and hydro, the ICT sector consumes a disproportionate percentage of coal power generated IEEE Communications Magazine August

94 Tons of C CO2 - Network CO2 - Servers Application size (GB) Figure 5. GHG emission comparison between 48-core data servers and a ninenode core network. base load electricity. Most power utilities will also not commit to reducing the power from their coal plants commensurate with reductions in energy consumption. Instead, they will resell the power to other users as it is very costly not to have a coal plant operating at 100 percent capacity. Alternatively, they may reduce operation of the more expensive gas plants and/or reduce imported hydroelectric power or the use of solar and wind-based energy. As such, the link between energy efficiency and GHG emission reductions is far from direct. While improving the energy efficiency of ICT products and services remains an important target, the GSN concept therefore emphasizes the importance of an ultimate focus on GHG emission reductions. As the GSN leverages virtualization to facilitate the use of renewable energy, two other virtualization testbeds are described below. THE VANI TESTBED The Virtualized Application Networking Infrastructure (VANI) [6] is a University of Toronto testbed allowing university researchers and application providers to utilize their internal resources to rapidly create and deploy networked systems, and to experiment with new layer three protocols as well as new applications. The VANI node is designed using virtualization and service-oriented principles and is ideal for investigating the green networking concepts presented in this article. A VANI node offers virtualized resources such as processing, storage, networking, and programmable hardware that can be interconnected by a 10GE switch to provide programmable virtual network/computing nodes on demand. A service-oriented control and management plane allows VANI nodes to be interconnected using layer 2 and optical networks into virtual networks to support applications operating in the applications plane. VANI is designed to provide an isolated and secure environment for researchers to carry on their experiments and develop their networked applications. VANI is also developing wrappers to allow interconnection to GENI testbeds. THE NETVIRT PROJECT The Netvirt project, launched in 2009 at the Université du Québec à Montréal and École Polytechnique, is investigating how to build virtualized nodes. In this project, virtualization is pushed over each node of the network. The virtualized node approach reduces the size of network infrastructure and hence energy consumption. The Netvirt project investigates the main challenges of node virtualization, notably in the following areas: the discovery and advertisement of network resources; the creation and management of a sliced node across diverse resources; the extension of virtualization to wireless and optical links; the implementation of virtualization across diverse resources and across layers of a protocol stack; the management of slices; the service offering from the network infrastructure providers to the network providers; and the set of capabilities enabled by infrastructure virtualization. The project proposes a new concept named Network Aware Hypervisor (NAH) in order to share the resource between multiple instances of virtual node. The NAH is composed by four components: the resource manager, the partition manager, the admission control and the domain manager. Underlying these research challenges, the Netvirt project is implementing a virtualized node, in order to validate and experiment with the new infrastructure. STANDARDIZATION AND METRICS As described earlier, the economic viability of new Green ICTs in the low carbon economy of the 21st century will rely on quantifying the green impact of these technologies, particularly when applied to non-ict sectors where ICT has the greatest GHG emission reduction potential (e.g. intelligent transport systems, smart buildings, optimization of manufacturing processes, etc.) [1]. As such, one of the key issues related to the role that ICTs play in abating climate change is the development of a methodology to accurately measure the impact of ICTs on climate change specifically and on the environment in general. In order to measure the contribution of ICTs to GHG emission abatement, standard methods of measurement and analysis must be developed. The international community has yet to agree to such standards. One of the key organizations involved in tackling this question of how to measure energy consumption and GHG emissions of ICTs with a standardized approach is the International Telecommunication Union (ITU). These standards will define how to measure energy consumption and GHG emissions, along with the appropriate metrics. Once the standards-based measures are universally accepted and applied to collect the data, developing a response can then be addressed using a combination of policy and technical tools. Standards are one such policy instrument. While the immediate impacts of direct ICT use on material utilization and energy consumption are relatively straightforward to determine, it remains a challenge to measure the environmental benefits that accrue from using ICTs in 106 IEEE Communications Magazine August 2011

95 other sectors (e.g. through more efficient energy and material use as a result of smart [1] grids, buildings, logistics, and transportation). This has led some researchers to conclude that traditional environmental assessment approaches are insufficient to accommodate the digital technology revolution and cannot accommodate the challenge of measuring the impacts of ICT on environmental sustainability [7]. Policies and standards have led to ICT-based reductions in energy consumption and GHG emissions. These include such standard-based programs or initiatives that have been driven either by governments and/or by the ICT industry itself or by consumers. These include the EnergyStar program, the 80 PLUS, and the European Union directives 2002/95/EC on the reduction of hazardous substances and 2002/96/EC on waste, electrical, and electronic equipment (WEEE) that required the substitution of heavy metals and flame retardants in all electronic equipment put on the market starting on July 1, The ITU Radiocommunication Sector (ITU- R) reports that while there are no special standards on green radio technologies, the implementation of some Regional Agreements with international treaty status and the application of some ITU-R Recommendations have led to significant reductions of GHG emissions [8]. The most impressive example is the Regional Agreement related to the planning of digital broadcasting service which involved 120 countries in the development of a new digital broadcasting Plan GE06 that envisages significant reduction (almost 10 times) of transmitter power due to the use of digital modulation [9]. Moreover, the number of transmitters (there are tens of thousands of transmitters around the world with power of up to kw each) may be reduced due to the possibility of transmitting several TV and radio programs in one channel (instead of 1 TV program per channel). ITU standards (Recommendations in ITU terminology) such as Recommendations ITU-R BT and ITU-T H.262 are used as the technical basis for switching from analog to digital broadcasting. Several standards have been or are being developed that address the environmental impact of ICT. Various research bodies (Intergovernmental Panel on Climate Change, the GHG Protocol Initiative), UN agencies (the Secretariat of the United Nations Framework Convention on Climate Change, the Climate Change Secretariat or UNFCCC), and standards organizations 1 have looked at these issues and in some cases, standards have been established [10]. Most of these standards deal with measuring GHG emissions and the contribution of ICTs to GHG emissions. None deal with the mitigation of GHG emissions possible through the use of ICTs in other sectors of the economy or beyond. The ITU has documented these and undertaken a gap analysis to identify areas that could benefit from further research and standards development. To address these concerns, the ITU has appointed a committee to develop standards to study methodologies for calculating the amount of GHG emissions from ICTs, and the amount of reduction in the GHG emissions in other sectors as a result of using ICTs. The main objective is to create a standard methodology for calculating the carbon footprint. ITU established the Focus Group on ICTs and Climate Change in July 2008 [11]. One of the deliverables produced by the focus group in early 2009 was a list of policies and ICT-based technologies that could be used to directly reduce GHG emissions. These include checklists for developing and promoting eco-friendly standards for ICT architecture, devices, providers and systems, as well as for the safe disposal and recycling of ICT systems. The focus group also reported the potential GHG savings that could be achieved by ICT related interventions that could reduce GHG emissions, such as video conferencing systems (travel replacement), intelligent transportation systems, etc. The work of this focus group ended in 2009 and was continued under a new study group, Study Group 5 (SG-5) described below. The study group s recommendations are not close to becoming standards, but point to further areas of research for standardizing the contribution of ICTs to mitigate GHG emissions in non- ICT sectors of human endeavor. The focus group did produce some interesting results evaluating GHG emissions savings from telemedicine applications and tackled issues such as the environmental impact assessment of ICTs [12]. In April 2009, the Telecommunication Standardization Advisory Group (TSAG) of the ITU-T designated Study Group 5 (SG-5) as the lead study group on environment and climate change, and established the Joint Coordination Activity on ICTs and Climate Change (JCA- ICT&CC) to coordinate work among the ITU-T Study Groups, and liaise with the ITU-R and ITU-D sectors. The work of SG 5 is now beginning its work under the following five work areas [13]: The coordination and planning of ICT and climate change related standardization (Question 17 of ITU-T SG 5) Methodology of environmental impact assessment of ICT (Q18/5) Power feeding systems (Q19/5) Data collection on energy efficiency for ICTs over the life cycle (Q20/5) Environmental protection and recycling of ICT equipment/facilities (Q21/5) One key output of this work will be the development of a Universal power adapter and charger solution for mobile terminals and other ICT devices [13]. The charger specifies general requirements and covers chargers for mobile phones. A future version will cover other ICT devices. ITU is also approving a standard on e- waste. A classification of outside plant waste has been updated which will help promote recycling. The SG 5 focus group is also working on a standard for GHG measurement and evaluation methods and specifically on developing a method for environmental impact assessment of ICT related goods and/or services at the organization and the country levels [13]. Other outputs [13] will be standards, based on life cycle analysis, for promoting recycling and e-waste reduction. Work will also be undertaken on efficient power feeding systems within ITU-R reports that while there are no special standards on green radio technologies, the implementation of some Regional Agreements with international treaty status and the application of some ITU-R Recommendations have led to significant reductions of GHG emissions. 1 ITU, ISO, ETSI, IEC, ECMA, and Alliance for Telecommunications Industry Solutions (ATIS). IEEE Communications Magazine August

96 The SG-5 focus group is also working on a standard for GHG measurement and evaluation methods and specifically on developing a method for environmental impact assessment of ICT related goods and/or services at the organization and the country levels. data centers and will deal specifically with power feeding systems, evaluating their performance, environmental impact, and safety-related issues. Dr. Hamadoun Toure, ITU Secretary General, hopes to see the ICT environmental impact standard published shortly [14]. And SG-5 Chair, Mr. Keith Dickerson, head of standards at British Telecom (BT) Design, said [14] BT would use these standards to drive its suppliers to reduce their power consumption and hence CO2 emissions as a way of meeting its own targets. The first suppliers likely to be affected are those supplying equipment for BT s 21st century network. ACKNOWLEDGMENTS The authors thank the following organizations for supporting the work described herein: the International Telecommunications Union, the Natural Sciences and Engineering Research Council of Canada, the Canada Foundation for Innovation, the CANARIE research network, the Prompt research consortium, Ericsson Canada, the Fonds Québécois de la Recherche sur la Nature et les Technologies and the Québec Ministry of Economic Development, Innovation and Export Trade. REFERENCES [1] Smart 2020, Enabling the Low-Carbon Economy in the Information Age, The Climate Group, London, U.K., 2008, [2] G. Shen and R. Tucker, Energy-Minimized Design for IP over WDM Networks, IEEE J. Opt. Commun. Networks, June 2009, pp [3] S. Aleksic, Analysis of Power Consumption in Future High-Capacity Network Nodes, IEEE J. Opt. Commun. Networks, Aug. 2009, pp [4] [5] J. Baliga et al., Energy Consumption in Optical IP Networks, J. Lightwave Tech., vol. 27, no. 13, 2009, pp [6] H. Bannazadeh et al., Virtualized Application Networking Infrastructure, Int l. Conf. Testbeds and Research Infrastructures for the Development of Networks and Communities (TridentCom), May [7] L. Ly and H. R. Thomas, A Review of Research on the Environmental Impact of e-business and ICT, Environ Int l. 2007, Aug. 33, vol. 6, pp [8] A. Vassiliev, Study Group Counsellor, ITU Radiocommunication Sector ITU-R, Private Communication, [9] ITU, 2006, Final Acts of the Regional Radiocommunication Conference for Planning of the Digital Terrestrial Broadcasting Service in Parts of Regions 1 and 3, in the Frequency Bands MHz and MHz (RRC-06), [10] ITU, 2010, Record of Standards Activity on ICT and Climate Change, updated ; [11] ITU, 2009, Focus Group on ICTs and Climate Change (FG ICT&CC), Report to TSAG/Deliverables; [12] ITU, 2009, Deliverable 3: Methodologies; [13] R. Scholl, ITU and Climate Change Standardization Landscape, PowerPoint presentation, ETSI Green Agenda Seminar 26 Nov. 2009, ITU, Cannes, France. [14] I. Grant, ITU to Develop Standard ICT Climate Impact Measure, ComputerWeekly.Com, Sept. 10, BIOGRAPHIES CHARLES DESPINS [SM] (cdespins@promptinc.org) has contributed as both an industry executive and an academic researcher to the ICT industry. He has held various posts in the private sector at CAE Electronics, Microcell Telecommunications, and Bell Nordiq Group (a network operator in rural and northern Canada), where he was vice-president and chief technology officer. He has also worked as a consultant for wireless network deployments in India and China. Since 2003 he has been president and CEO of Prompt Inc., an ICT university-industry R&D consortium. In addition to being a guest lecturer in the executive M.B.A. program at McGill University in Montreal, he is also a faculty member (on leave) at École de Technologie Supérieure (Université du Québec) with research interests in wireless communications. He holds a Bachelor s degree in electrical engineering from McGill University as well as M.Sc. and Ph.D. degrees from Carleton University, Ottawa, Canada. He was Co-Chair of the 2006-Fall IEEE Vehicular Technology Conference. He is a past recipient (1993) of the Best Paper of the Year award in IEEE Transactions on Vehicular Technology as well as a Fellow (2005) of the Engineering Institute of Canada and a recipient (2006) of the Outstanding Engineer award from IEEE Canada. FABRICE LABEAU [SM](fabrice.labeau@mcgill.ca) is an associate professor in the Electrical and Computer Engineering Department at McGill University, Montréal, Québec, Canada. He received his Electrical Engineer degree in 1995 rom Université Catholique de Louvain (UCL), Belgium, and his Diplo^me d études spécialisées en Sciences Appliquées, orientation Télécommunications, also from UCL, in From 1996 to 2000 he was with the Communications and Remote Sensing Laboratory of UCL. From January to March 1999 he was a visiting scientist at the Signal and Image Department (TSI) of ENST Paris, France. He received a Ph.D. degree in September 2000 from UCL. His research interests include multirate processing, joint source channel coding, data compression, and error control coding. He was part of the organizing committee of ICASSP 2004 in Montréal. He was a Technical Program Committee co-chair for the 2006-Fall IEEE Vehicular Technology Conference in Montréal, and is TPC co-chair for the 2012-Fall VTC and ICIP RICHARD LABELLE (rlab@sympatico.ca) is an independent consultant based in Gatineau, Québec, Canada. He has over 29 years experience on issues related to the use of ICTs in support of human, economic, social, and sustainable development in developing countries. He has traveled to over 58 developing countries to consult with development actors for this purpose. Recently his work has also focused on using ICTs to abate climate change with a focus on the developing world. He has completed a scoping study on ICTs andclimate change entitled ICTs for e-environment and a follow-up study entitled e-environment Toolkit and Readiness Index for the International Telecommunication Union, and is completing a study on the use of wireless technologies for abating climate change. He has developed an in-depth training program on ICTs for abating climate change and has been promoting Green Growth for the Korea-based United Nations Asian and Pacific Training Centre for Information and Communication Technology for Development (APCICT), a part of the Economic and Social Commission for Asia and the Pacific (ESCAP). MOHAMED CHÉRIET [SM] (mohamed.cheriet@etsmtl.ca) received M.Sc. and Ph.D. degrees in computer science from the University of Pierre et Marie Curie (Paris VI) in 1985 and 1988, respectively. Since 1992 he has been a professor in the Automation Engineering Department at the École de Technologie Supérieure (University of Québec), Montréal, and was appointed full professor there in He is the founder and director of Synchromedia, which targets multimedia communication in telepresence applications. Computational intelligence is one of the many areas of his expertise that benefits the Consortium, providing Synchromedia s expertise in advanced open overlay self-scaling network architecture based on distributed and virtualized resources and the grid computing paradigm. He is the lead of the GreenStar Network (GSN) project, the world s first initiative for reducing GHG emissions arising from ICT services. He is the founder and former Chair of the IEEE Montrál Chapter of Computational Intelligent Systems. ALBERTO LEON-GARCIA [F] (alberto.leongarcia!utoronto.ca) is a professor of electrical and computer engineering at the University of Toronto. He is a Fellow of the Engineering Institute of Canada. He received the 2006 Thomas Eadie Medal from the Royal Society of Canada and the 2010 IEEE Canada A. G. L. McNaughton Gold Medal for his contributions to the area of communications. He holds a Canada Research Chair in Autonomic Service Architecture. His current research interests are focused on application-oriented 108 IEEE Communications Magazine August 2011

97 networking and autonomic resource management with a focus on enabling pervasive smart infrastructure. His research team is currently developing a network and applications testbed that supports at-scale experimentation in new network protocols and distributed applications. OMAR CHERKAOUI (cherkaoui.omar@uqam.ca) is a professor of computer science at the University of Québec in Montréal. He has (co-) authored more than three books, 50 peer-reviewed technical publications, multiple invited, keynote, and tutorial presentations, technical reports, and many patent disclosures. He created the research laboratory in computer networks (Lab Téléinformatique), where he supervised multiple projects in teh fields of high-speed network management, grid computing, virtualization, and cloud computing. THO LE-NGOC [F] (tho.le-ngoc@mcgill.ca) obtained his B.Eng. degree (with distinction) in electrical engineering in 1976, his M.Eng. in 1978 from McGill University, Montréal, and his Ph.D. in digital communications in 1983 from the University of Ottawa. During he was with Spar Aerospace Limited and involved in the development and design of satellite communications systems. During he was an engineering manager of the Radio Group in the Department of Development Engineering of SRTelecom Inc., where he developed the new point-to-multipoint DA-TDMA/TDM Subscriber Radio System SR500. During he was a professor in the Department of Electrical and Computer Engineering at Concordia University. Since 2000 he has been with the Department of Electrical and Computer Engineering at McGill University. His research interest is in the area of broadband digital communications. He is a senior member of the Ordre des Ingéneurs du Québec and a fellow of the Engineering Institute of Canada, the Canadian Academy of Engineering, and the Royal Society of Canada. He is the recipient of the 2004 Canadian Award in Telecommunications Research and the IEEE Canada Fessenden Award He holds a Canada Research Chair (Tier I) on Broadband Access Communications and a Bell Canada/NSERC Industrial Research Chair on Performance and Resource Management in Broadband xdsl Access Networks. BILL ST. ARNAUD (bill.st.arnaud@gmail.com) is a green IT and Internet consultant providing a variety of university, industry, and government clients strategic advice on the Internet, broadband, and cloud computing, and particularly their potential to mitigate against the effects of climate change. He works with a number of national and international clients to identify new Internet business opportunities in a futre low carbon economy. These projects are intented to reduce global warming by reducing CO 2 emissions at universities and businesses, and in society in general through the use of Internet applications and clouds, In 2002 he was featured by TIME Magazine Canada as the engineer who is wiring together advanced Canadian science.in 2005 In June 2010 he received an honorary Doctorate of Science from the University of Athabasca in recognition of his contribution to the advancement of research and Internet networking in Canada, and his leadership in promoting green ICT for a low carbon economy. He is also the recipient of the ORION Leadership Award for He is a frequent guest speaker at numerous conferences on the Internet and resarch and education networking. He is a graduate of Carleton University School of Engineering. JACQUES MCNEILL (jmcneill@promptinc.org) is the coordinator of Prompt s Green ICT Initiative. He is president of Technoprise, Inc., an ICT business consulting firm that participates in the early stages of new innovation development activities. His career spans the industry, academic, and government realms of the ICT sector. As a new venture entrepreneur and project developer, he is involved in leading edge partnership opportunities in Canada and abroad. He provides hands-on executive management experience and addresses ICT market opportunities through strategic alliances, often with early stage technology ventures and leading industry players. He is a former board member and board chair of Prompt. In 2008 he coordinated the launch of the Green ICT Initiative. He graduated from McGill University with a B.Sc. in biochemistry and an M.B.A. in marketing and international business, which led to a career as a high-tech entrepreneur in the ICT industry. His present green ICT focus not only represents the industry s convergence toward ecological sustainability but also that of his education, professional experience, and personal convictions. YVES LEMIEUX (yves.lemieux@ericsson.com) joined Ericsson in 1994 and currently holds the position of research engineer. He has a number of patents and publications to his credit in the fields of cellular system synchronization selection, network resiliency, LTE core network congestion control, and so on. His main interests are now vested in 3GPP-based end-to-end QoS and also vitualization for cloud computing. Prior to working at Ericsson, he was a systems design engineer at AT&T Canada and a radio/fiber manager at Rogers Wireless. He received his Bachelor s degree in electrical engineering from the University of Sherbrooke in December 1981 and a Master s degree in computer engineering from Polytechnique in Canada in June MATHIEU LEMAY (mlemay@inocybe.ca) is CEO and president of Inocybe Technologies Inc., a company specializing in infrastructure as a service (IaaS) products and services. He started his career at the Communications Research Centre, a Canadian government laboratory, where he started working on user empowered networking under CANARIE s UCLP and UCLPv2 directed research programs. In 2005 he created Inocybe Technologies Inc. to promote the business models of infrastructure providers and services. This led to the creation of the IaaS Framework on which different projects and middleware solutions have been developed by research projects and future of the Internet initatives. He is promoting the impact that utility ICT, virtualization, and their respective business models will have on GHG emission reductions by optimizing the use of physical equipment by different organizations. Inocybe Technologies Inc. and its partners have been participating in many international projects under FP7 and NSF programs creating different cyberinfrastructure environments for the research and education communities. He holds a B.S. in electrical engineering and an M.S. in optical networks, and is currently finishing his Ph.D. on infrastructure slice orchestration for virtual infrastructures. He is now a recognized expert on service-oriented architectures for federated network and computing management. CLAUDE THIBEAULT [SM] (claude.thibeault@etsmtl.ca) received his Ph.D. from École Polytechnique de Montréal, Canada. He is now with the Electrical Engineering Department of École de Technologie Supérieure, where he serves as a full professor. His research interests include current-based test and diagnosis, delay testing, as well as design and verification methodologies targeting ASICs and FPGAs. He holds 11 U.S. patents and has published more than 120 journal and conference papers, which have been cited more than 540 times. He received the best paper award at DVCON 05, verification category. He has been a member of different conference program committees, including the VLSI Test Symposium, for which he was program chair in FRANÇOIS GAGNON (francois.gagnon@etsmtl.ca) received B.Eng. and Ph.D. degrees in electrical engineering from École Polytechnique de Montréal. Since 1991 he has been a professor with the Department of Electrical Engineering, Écike de Technologie Supérieure. He chaired the department from 1999 to 2001, and now holds the NSERC Ultra Electronics Chair, Wireless Emergency and Tactical Communication, at the same university. His research interest covers wireless high-speed communications, modulation, coding, high-speed DSP implementations, and miltary point-to-point communications. He has been very involved in the creation of the new generation of high-capacity line-of-sight military radios offered by the Canadian Marconi Corporation, which is now Ultra Electronics Tactical Communications Systems. The company has received, for its product, a Coin of Excellence from the U.S. Army for performance and reliability. He is a recognized leader in research management with an annual budget of $1,600,000; he supervises 18 graduate students and leads a brilliant team of seven research professionals; and he maintains activities with more than 10 companies, including Ultra, ISR Technology, Sita, Ericsson, Lipso, Nortel, Bell, Octasic Semiconductors, Sierra Wireless, Boomerang, and IREQ. He was awarded the 2008 NSERC Synergy Award (Small and Medium-Sized Comapnies category) for the fruitful and long lasting collaboration with Ultra Electronics TCS. IEEE Communications Magazine August

98 This page intentionally left blank This page intentionally left blank This page intentionally left blank

99 Future Generation Computer Systems 28 (2012) Contents lists available at SciVerse ScienceDirect Future Generation Computer Systems journal homepage: Distributed computing for carbon footprint reduction by exploiting low-footprint energy availability Ward Van Heddeghem, Willem Vereecken, Didier Colle, Mario Pickavet, Piet Demeester Department of Information Technology (INTEC) of Ghent University - IBBT, Gaston Crommelaan 8, B-9050 Gent, Belgium a r t i c l e i n f o a b s t r a c t Article history: Received 29 October 2010 Received in revised form 11 April 2011 Accepted 7 May 2011 Available online 15 May 2011 Keywords: Green ICT Energy-efficiency Power consumption Distributed computing Grid computing Carbon footprint Low carbon footprint energy sources such as solar and wind power typically suffer from unpredictable or limited availability. By globally distributing a number of these renewable sources, these effects can largely be compensated for. We look at the feasibility of this approach for powering already distributed data centers in order to operate at a reduced total carbon footprint. From our study we show that carbon footprint reductions are possible, but that these are highly dependent on the approach and parameters involved. Especially the manufacturing footprint and the geographical region are critical parameters to consider. Deploying additional data centers can help in reducing the total carbon footprint, but substantial reductions can be achieved when data centers with nominal capacity well below maximum capacity redistribute processing to sites based on renewable energy availability Elsevier B.V. All rights reserved. 1. Introduction Data center power consumption is significant, and growing The last decade has seen a steady rise in data center capacity and associated power consumption. In 2008, the yearly average worldwide data center power consumption was estimated to be around 29 GW [1]. This is comparable to the total electricity consumption of Spain in the same year [2], a country that ranks in the top 15 of the list of electricity consumption per country. In [3], it was estimated that the aggregate electricity use for servers worldwide doubled over the period With the predicted growth of Internet-based services for social networks and video, and with the growing usage of mobile thin clients such as smart phones that require a server back-end [4], it seems unlikely that this increase will halt soon. Using renewable energy, in addition to energy-efficiency, is key to mitigate climate change While the growing energy consumption in data centers presents some issues both economically and technically, there has been a growing concern from an environmental point of view as well, with electricity consumption contributing to greenhouse gas (GHG) emission. Two high-level approaches can help in reducing GHG emissions: (a) an improvement in energyefficiency to reduce the amount of electrical energy used, and (b) use of energy that contributes little to GHG emissions. What Corresponding author. Tel.: ; fax: address: ward.vanheddeghem@intec.ugent.be (W. Van Heddeghem). concerns the latter, this electrical energy will typically come from renewable energy sources such as solar and wind power. Adding renewable energy to the current energy mix still poses some issues While renewable energy is indeed already promoted and used to mitigate climate change both in ICT and non-ict sectors, significantly increasing the amount of renewable energy as part of the regular energy mix raises a number of issues [5]. First, because most good sites for renewable energy sources may be located in distant areas with limited transmission capacity, and it might take many years for the required transmission infrastructure to become available [6]. Second, the distributed power generation poses many challenges for the existing distribution infrastructure, especially with respect to protection and control strategies due to new flow patterns [6,7]. Third, with renewable energy sources likely to be located in distant areas, the transmission losses will increase; current transmission losses are already estimated to be around 6.5% of the total electricity disposition 1 for the USA in 2007 [8]. Fourth, with hydro-power usually reserved for peak power handling [9], other renewable energy sources such as wind and solar power are usually characterized by intermittent power delivery, resulting in periods of peak power being available and no power being available at all. 1 To be correct, the loss percentage is calculated as a fraction of the total electricity disposition excluding direct use. Direct use electricity is electricity that is generated at facilities that is not put onto the electricity transmission and distribution grid, and therefore does not contribute to transmission and distribution losses [8] X/$ see front matter 2011 Elsevier B.V. All rights reserved. doi: /j.future

100 406 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) we provide a detailed and realistic quantification of the parameters in our mathematical formulation (Section 4), we show that the manufacturing carbon footprint is a nonnegligible factor in footprint reduction evaluations, and that under certain conditions minor footprint savings are possible when deploying additional sites where jobs are distributed according to the FTSFTW approach (Section 5), we show that larger relative footprint savings are possible when applying the FTSFTW scenario to distributed data centers where the nominal load is well below the maximum capacity (Section 6). It should be noted that the theoretical model we present in Section 3 can be applied, with or without slight modifications, using other metrics than carbon footprint. 2. Related work Fig. 1. Distributed data center. Data centers are uniquely positioned to provide an alternative solution Data centers have become more and more globally distributed for a number of reasons as summarized by [10]: the need for high availability and disaster tolerance, the sheer size of their computational infrastructure and/or the desire to provide uniform access times to the infrastructure from widely distributed client sites. This geographical distribution of data centers, combined with the availability of low-power and highspeed optical links, allows them to be located near renewable energy sites. With technology currently available to migrate live virtual machines while minimizing or avoiding downtime altogether [11 13], jobs can be dynamically moved from a data center site where renewable power dwindles to a different site with readily available renewable power. This approach has previously been referred to as Follow The Sun/Follow The Wind (FTSFTW) [5]. Fig. 1 illustrates this concept with solar powered data center sites. As the sun sets in the top-right data center (and the capacity of potential backup-batteries fall below a critical value) the site s data and jobs are moved to a different site (top left) where solar power has become available. In this paper we will evaluate the carbon footprint and potential footprint savings of such a FTSFTW-based distributed data center. We will generalize on the notion of renewable energy, and instead consider low-footprint (LF) energy and high-footprint (HF) energy. As a metric for the carbon footprint we will use grams of CO 2 - eq, unless otherwise indicated. CO 2 -eq indicates CO 2 -equivalent emissions, which is the amount of CO 2 that would have the same global warming potential when measured over a given time horizon (generally 100 years), as an emitted amount of a long-lived GHG or a mixture of GHGs. The contributions of this paper are the following: we provide a mathematical model for calculating the carbon footprint and savings of such a distributed data center infrastructure which is powered by a fixed mix of LF and HF energy (Section 3), Next to the work already pointed out in the previous section, below are some earlier references and publication related specifically to the FTSFTW approach. One of the first papers to suggest locating data centers near renewable energy sources is [14]. The primary reason given is that it is cheaper to transmit data over large distances than to transmit power. The paper does not discuss or explore this issue in any more detail. The first paper to our knowledge to discuss and mathematically evaluate load distribution across data centers taking into account their energy consumption, energy cost (based on hourly electricity prices) and so-called low-footprint green energy and highfootprint brown energy is [10]. It presents and evaluates a framework for optimization-based request distribution, which is solved using heuristic techniques such as simulated annealing. The paper shows that it is possible to exploit green energy to achieve significant reductions in brown energy consumption for small increases in cost. It does not consider the manufacturing carbon footprint. Similarly, in [15] load distribution across data centers is discussed, but only to optimize energy costs by exploiting energy price differences across regions. In [5] the FTSFTW scenario is discussed in more detail and an Infrastructure as a Service (IaaS) approach is suggest to turn this in a viable business model. It outlines the main arguments for employing such a scenario. The key idea put forward is that the FTSFTW scenario provides a zero-carbon infrastructure for ICT, thereby somewhat optimistically ignoring the potential contribution of the manufacturing carbon footprint. The GreenStar Network project [16] is a proof-of-concept testbed for the FTSFTW strategy. The project started in 2010 and is deployed across the Canadian-based CANARIE research network and international partners distributed across the world. It consists of a number of small-scale nodes powered by renewable energy (especially hydro, solar and wind power) which provide energy for the routers, switches and servers located at the node. Applications are running inside virtual machines, with multiple virtual machines per server, and are migrated live from node to node. The expected outcome of the project is a number of tools, protocols and techniques for deploying green ICT services. A framework for discovering carbon-minimizing resources in networks similar to those deployed by the GreenStar Network project, is described in [17], but again the manufacturing carbon footprint is not considered.

101 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) The total carbon footprint F of the above described distributed data center infrastructure, averaged over a long-enough period, will be the sum of the manufacturing footprint F m, the usage footprint F u and the communication footprint F c : F = F m + F u + F c. (1) The manufacturing footprint will be the carbon emitted during the manufacturing of the sites and the equipment (servers, network equipment etc.) inside. The usage footprint will be the result of the electrical energy used during the use phase. The communication footprint will be the carbon emitted by migrating data and jobs from site to site. All three footprints will be expressed in g CO 2 -eq. Before we elaborate on each of these footprints, it is useful to point out the following assumptions we will make for our theoretical model: Fig. 2. Distributed data center infrastructure overview, consisting of m = 5 sites with n = 3 sites active. The independent LF energy availability per site is p = Theoretical model In this section we will outline the details of the scenario that we consider and develop a theoretical model for estimating its total carbon footprint. The quantification of the various parameters in our formulation will be done in Section 4. To introduce our theoretical model, we consider the distributed simplified data center infrastructure that is shown in Fig. 2. It consists of m equally-sized sites. Of these m sites, on average n sites are active. When a specific site becomes non-active, data and processing is moved to another active site, keeping the number of active data centers equal to n at all times. At this point it is important to point out that, although we use the term data center, our model will be independent of the size of the data center. A data center site could be an energy-optimized building housing thousands of servers, or it could be as small as a single server. In the context of this paper, it might be helpful to think of a data center site as a computing node of any possible size. Each site is powered by either LF or HF energy. The average availability of LF energy versus HF energy is considered equal, but uncorrelated, for each site. This availability ratio p might be the result of an average temporal availability of a specific renewable energy source (for example, solar or wind power), or specific service level agreements between the data center operator and the utility provider. To reduce the total footprint, the usage of LF energy will be maximized by migrating operation of a data center powered by HF energy to a data center where LF energy is available. When no LF energy is available, HF energy will be used to guarantee service delivery. We assume each site in the distributed data center to be of uniform size. We assume instant site migration. That is, we assume that a migration takes no time and produces no extra overhead not accounted for in the communication footprint. If the migration frequency is relatively low (say, limited to a few times a day), this assumption will hold. We do not consider a surplus of LF energy. That is, if for example 4 out of 5 sites have LF renewable energy available, but we only require 3 sites for daily operation, the electricity generated in the 4th site is wasted. There is potential for using this energy for other less-critical purposes, or for selling or trading it for carbon credits. However, for simplicity and generality, our model does not take using surplus available power into account. We assume that a non-active data center site consumes no energy. While this is an optimistic assumption for large data centers, this is certainly feasible for micro-scale data centers consisting of a few servers (remember that, although we use the term data center, our model is independent of the data center size). The energy for a non-active site could be reduced to (nearly) zero by for example suspending all servers Usage footprint Let us call p the chance that a site is powered by LF energy. Let us call k the total number of data center sites that are powered by LF energy and P k the chance of this number being k. This chance is given by the probability mass function of the binomial distribution: m P k = p k (1 p) m k. (2) k Eq. (2) can be understood intuitively as follows. The chance for exactly k sites powered by LF energy is p k. The chance for the m k remaining sites to be not powered by LF energy is (1 p) m k. The number of ways to choose k sites out of a total of m sites m is given by the binomial coefficient and can be calculated as m k = m! k!(m k)!. Given L the carbon footprint of the total usage phase of a single site when powered exclusively by LF energy and H the carbon footprint when powered exclusively by HF energy. The total usage footprint F u for all sites is then: If k n (that is, if LF energy is available in enough or more sites than required): F u = nl. (3) Else: F u = (n k)h + kl. (4) k

102 408 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) Thus, using the chances of k being a certain value, the total usage footprint F u becomes: F u = m n 1 [P k nl] + [P k ((n k)h + kl)]. (5) k=n k=0 The first term describes the weighted footprint if enough sites are powered by LF energy, the second term when this is not the case. When substituting Eq. (2) in (5) we get for the total usage footprint F u of the distributed data center infrastructure: m m F u = nl p k (1 p) m k k k=n n 1 m + p k (1 p) m k ((n k)h + kl). (6) k k=0 The usage footprint results exclusively from electrical energy. The emission intensity of electricity describes the GHG emissions in gram CO 2 -eq/kwh. We use I L and I H to denote the emission intensity for LF and HF electricity respectively. With E u the energy used by a single site during the entire use phase, L and H can thus be expressed as: L = IL E u (7) H = I H E u Manufacturing footprint The total manufacturing footprint F m is a function of the carbon footprint cost M for manufacturing one data center site, and the number of data centers sites m: F m = mm. (8) As we will see in Section 3.4, it is convenient to consider the manufacturing fraction f, which is the ratio of the manufacturing carbon footprint M of a single site over the usage carbon footprint H of a single site: f = M H. (9) Equipment where the manufacturing emits less GHG than the typical GHG emitted during its use phase will have a manufacturing fraction f < 1. Given Eq. (9), we can rewrite Eq. (8) as: F m = mfh = mfi H E u. (10) Note that we considered the equipment to be manufactured with HF energy, by expressing M as a function of H instead of L Communication footprint Migrating jobs or data across data centers incurs an extra amount of carbon emissions. This will mainly be due to the energy consumed for (a) the transportation over an optical network, (b) the preparation and duration of the migration and (c) switching the data center to the non-active state or vice versa. In this section we show that the overhead of the above three factors is negligible with respect to the carbon emitted in the manufacturing and use phase, and can thus be ignored for now. Data centers are typically connected by optical networks. Power consumption in the optical core network is dominated by the IP router power consumption, with high-end IP routers consuming in the order of 10 W/Gbps [18]. Accounting for redundancy, cooling and power supply overhead, and client and network interface, we have approximately 100 W/Gbps, or an energy of kwh needed to transport one Gbit. Further, we assume two migrations per site once a day, i.e. one inbound migration and one outbound migration. We consider each server in a data center site to be capable of running four virtual machines, with each virtual machine to be about 10 GB in size. For each server s data to be migrated, this totals to 640 Gb/day. Considering a server use phase of 4 years, this sums up to Gb per use phase. Using our estimation from above, this requires approximately 26 kwh of energy. With a world-average emission intensity of 500 g CO 2 -eq/kwh, this results in about 13 kg CO 2 -eq emitted due to migration (for one server, during its entire use phase). This equals to less than 3% of the current manufacturing footprint of a server (about 500 kg CO 2 -eq, see Table 1), or about 0.5% of the current total carbon emissions. With respect to the energy overhead induced by migration preparation and duration, transmitting our exemplary 640 Gb/day would take less than 15 min/day over a 1 Gbps link. This accounts for only about 1% of the time. Likewise, as the daily migration frequency is low, the time and energy overhead to switch a data center from the active to nonactive state (or vice versa) should be relatively low as well. Also, the active/non-active switchover time will probably depend on the kind of jobs and data that the data center is running. Although the above estimate is based on the current situation of the average absolute carbon footprint of servers and current virtualization technology, we feel that it is a fair assumption for current and short-term future to neglect the contribution of the communication footprint F c to the total footprint Total footprint Combining Eqs. (6) and (10), the total footprint is given by: m m F = mfi H E u + nl p k (1 p) m k k k=0 k=n n 1 m + p k (1 p) m k ((n k)h + kl). (11) k The above equation depends on the value of E u, the single site usage energy. This value will vary depending on the data center size and type, and on the jobs and data processed. We can eliminate this parameter, if we normalize the total footprint over the single site usage energy E u. By doing so, we can conveniently express this total normalized footprint F norm as a function of the LF energy emission intensity I L, the HF energy emission intensity I H and the fraction f : F norm = F E u m m = mfi H + ni L p k (1 p) m k k k=0 k=n n 1 m + p k (1 p) m k ((n k)i H + ki L ). (12) k We now have a metric for the carbon footprint which is independent from the data center size and type, and with unit (g CO 2 -eq/kwh). 4. Parameter quantification Our model constructed in the section above consists of a number of parameters. In this section we discuss realistic values for each of these parameters.

103 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) Table 1 Manufacturing fraction values according to different studies. Reference Description Manufacturing phase Use phase (4 years) f PE International [19] Simple office server a 500 kg CO 2 -eq/unit 1030 kg CO 2 -eq 0.49 Malmodin ITU [20] PC a 400 kg CO 2 -eq/unit 640 kg CO 2 -eq 0.63 Malmodin ITU [20] Server 500 kg CO 2 -eq/unit 5200 kg CO 2 -eq 0.10 Malmodin, Moberg [21] Data centers b 10 Mton CO 2 -eq in Mton CO 2 -eq in a Overhead power in use phase not included (PUE = 1). See text for more information. b This includes data center equipment and buildings. Data based on 10 million new servers and 35 million servers in use; this translates roughly to a use phase of 4 years. Use phase emission intensity in [21] = 0.6 kg CO 2 -eq/kwh. Table 2 Average CO 2 emissions per kwh from electricity and heat generation for a number of countries and regions, data for 2008 [24]. Region World 502 United States 535 Canada 181 European Union 351 China 745 India Manufacturing fraction (f ) Intensity (g CO 2 /kwh) The manufacturing fraction represents the ratio between the manufacturing carbon footprint and the usage carbon footprint. Detailed life cycle analysis (LCA) studies that report on the carbon emissions of data centers during the manufacturing phase and the use phase are scarce. Moreover, the resulting manufacturing fraction is influenced by the use phase lifetime of the equipment and the emission intensity of the energy used during the use phase. In addition, it is important to know if reported use phase values include power consumed for overheads such as cooling. This overhead is typically expressed by the power usage effectiveness (PUE). For example, a PUE of 2 (a typical accepted value for data centers 2 ) indicates that for each Watt consumed by useful equipment such as servers and switches an additional Watt is consumed through overheads. Table 1 lists emission values and the derived manufacturing fraction f according to a number of studies. All data, except for the Simple office server and the PC, includes overhead power consumption. For the Simple office server probably no overhead is included ([19] is not completely clear on this); correcting for this with a PUE of 2, the use phase power consumption doubles and thus the manufacturing fraction value halves, bringing the values roughly in line with the other data. Based on the data in Table 1 we will use, unless otherwise specified, a value of f = High-footprint energy emission intensity (I H ) The parameter I H indicates the emission intensity of regular (HF) electrical energy. As already stated, the emission intensity indicates the amount of GHGs emitted for each kwh of electrical energy, and is typically expressed in grams of CO 2 -eq/kwh. The value for I H differs from country to country, and for larger countries even from region to region, depending on the primary energy sources (such as coal or gas) and technologies (such as open cycle gas turbines or combined cycle gas turbines) used for generating electricity, see for example Table Recently deployed high-capacity data centers with a focus on energy efficiency show much lower PUE values, such as Google claiming to reach a yearly average of 1.16 at the end of 2010 [22]. However, as the LCA data is based on 2007 estimates, the for that year typically accepted PUE value of 2 is used [23]. 3 The table reports the CO2 emissions instead of the CO 2 -eq emission (which takes a number of other GHGs into account). However differences are minor and irrelevant for our study. For this paper, we will consider the world average value of 500 g CO 2 -eq/kwh Low-footprint energy emission intensity (I L ) The emission intensity I L for low-footprint electricity is obviously lower than the regular HF energy emission intensity I H. Indicative, Fig. 3 lists the estimated emission intensity for a number of low-footprint sources (typically renewable energy such as hydro, wind or solar power), as reported by [9]. Roughly similar numbers are given in the slightly older study of [25]. In this paper, we assume a state-of-the-art LF energy emission intensity of 10 g CO 2 -eq/kwh Low-footprint energy availability (p) The parameter p represents the chance of each site being powered by LF energy. For example, with p = 0.6, each site has an independent chance of 60% to be powered by LF energy at any point in time. Or otherwise put, 60% of the time, each site will be powered by LF energy. While it might seem tempting to try to relate the value for p to the availability of a specific LF energy source (say, wind energy), this is not necessary for our model. After all, the availability of LF energy sufficient for powering a data center site will largely be a matter of monetary cost. This cost will be reflected either in the negotiated service level agreement (SLA) with the utility provider, or in the cost to install the required capacity of LF energy sources to deliver the required nominal power even during periods of low availability of e.g. sun or wind. Thus, a higher value for p will usually require higher investments. Note that it is key for the validity of our footprint model to know what kind of power (LF or HF) is used at what point in time, so as to be able to migrate the data to a different site if needed (and if possible). We assume p = 0.6, as we will see later that this results in maximum savings. 5. Case study I: the Added Distributed Data centers (ADD) scenario Can we reduce the footprint of a regular data center, by distributing additional sites across the globe as to benefit from uncorrelated and potentially complementary availability of renewable energy sources which offer a lower usage footprint? This is the question we will examine in this section. We refer to this scenario as the Added Distributed Data centers (ADD) scenario. Consider a data center that requires n = 3 sites for daily operation. Each site has an LF energy availability of p = 0.6, and we consider the current estimation for the manufacturing fraction f = Since we want to reduce the footprint of the complete data center, we would like to be able to run our applications on three data centers that have LF energy available. The chance of success increases with an increased number of data centers to choose from, that is, if we increase the total number of sites m to a value higher than 3.

104 410 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) Fig. 3. Lower and upper emission intensity estimates for various low-footprint sources [9] (CSP: Concentrated Solar Power). Fig. 4. The total normalized footprint F norm and corresponding relative emission savings as a function of the total number of data centers m. Savings are calculated with respect to the baseline scenario. (Parameter values: n = 3, f = 0.25, p = 0.6, I L = 10 g CO 2 -eq/kwh, I H = 500 g CO 2 -eq/kwh and E u = 1.) Fig. 4 shows the use phase, manufacturing phase and total footprint as we increase the total number of data centers m beyond 3. With each additional data center, the use phase footprint decreases as a result of the increased chance of finding a data center that runs on LF energy. Initially, this decrease is large enough to make up for the linearly increasing manufacturing footprint, resulting in a decreasing total footprint. However, when the number of data centers is approximately double the number of data centers required, the total footprint increases and eventually overtakes the first scenario footprint. Taking the first scenario (where m = n = 3) as a baseline, we see initial footprint savings until too many data centers are deployed, resulting in a net loss. Taking the first scenario as the baseline makes sense, since this corresponds to the current practice of operating a number of sites with a mix according to p of LF and HF energy, without migrating data or processing capacity based on LF energy availability Influence of manufacturing fraction (f ) As we have seen in the above case, the usage footprint reduction was initially able to make up for the linearly increasing manufacturing footprint. What if the manufacturing fraction f is higher, say f = 0.5? Fig. 5 shows the normalized footprint (upper figure) and relative savings (lower figure) for different values of f. Clearly, footprint reduction becomes smaller and even impossible for higher values of f. Even more so, our current rough estimate of f = 0.25 seems critical: with a slightly higher value for f = 0.3 savings are almost negligible (a mere optimistic 5%) and might be completely annihilated if we take more subtle factors (such as the migration footprint and management overhead) into account. In the inverse case, for lower values of f the savings increase. At the Utopian case of having manufacturing for free (f = M = 0), Fig. 5. The total normalized footprint F norm and relative emission savings for n = 3 as a function of m for different manufacturing fractions f. savings are obviously maximal and converge to the usage footprint cost nl Influence of low-footprint energy availability (p) Perhaps counterintuitive, an increase of LF energy availability of p towards 100% does not unconditionally result in additional savings. While the footprint indeed decreases monotonically with an increase of p (because the usage footprint becomes smaller), the baseline scenario footprint (where m = n) will also decrease. Fig. 6 shows that for the scenario n = 3, m = 6 (i.e., twice as much data centers as required for daily operation) the savings are maximum around p = For p = 0 there is a net loss due to the increased manufacturing footprint not yet being offset by a greener usage footprint. For p = 1 the baseline scenario runs entirely on LF energy whereas the FTSFTW approach has an increased manufacturing footprint due to the extra sites deployed. As we have already argued that p will be cost driven, a casebased cost study will have to find the optimal value for p. In retrospect, this also explains our decision for taking p = Influence of n and m values Because of the binomial coefficient, we cannot simply generalize the footprint savings obtained for e.g. n = 3 and m = 6 to apply to any other combination of n and m with the same ratio, e.g. n = 1 and m = 2, or n = 10 and m = 20. For higher values of n, footprint savings already occur for higher (i.e., worse) manufacturing fractions. For example, when we consider n = 10 (see Fig. 7), already for f = 0.5 minor savings are available (2% maximum), whereas for the previous case where n = 3 this was not the case (see Fig. 5). Because of the higher number of sites, the chance for finding enough sites where LF energy is available has increased. It should be noted that the total footprint will have increased as well.

105 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) Fig. 6. The total normalized footprint and the relative savings (with respect to the baseline scenario where m = n = 3) as a function of the LF energy availability p (n = 3, m = 6 and f = 0.25). Fig. 8. Relative emission savings as a function of the HF energy emission intensity I H (for f = 0.25, p = 0.6 and I L = 10 g CO 2 -eq/kwh). Fig. 7. Relative emission savings for n = 10 (all other parameters are equal as before). This finding suggests to favor a large number of small, distributed data center sites, over a few large ones. However, in that case, care should be taken that the combined manufacturing footprint of the small sites is not larger than the manufacturing footprint of the few larger sites. Taking the idea to extremes, largescale distributed computing projects such as Folding@Home [26] where small consumer entertainment devices are involved [27] might be a perfect fit, if both the manufacturing footprint and usage footprint (standby power consumption issues etc.) from these devices is small enough Influence of emission intensity difference The HF emission intensity (500 g CO 2 -eq/kwh) and LF emission intensity (10 g CO 2 -eq/kwh) that we consider in this paper following our findings in Sections 4.2 and 4.3 are relatively large in difference; I L is only 2% of the I H. In some countries or regions, the regular emission intensity is substantially lower (or higher) than the world average value, as can be seen in Table 2. Will the FTSFTW approach still be sustainable under those conditions? Fig. 8 shows the relative savings with changing values of I H. It is immediately clear from this figure that for values below the world average, the savings quickly become negligible. For emission intensities below the average European value, savings become negative, i.e. more carbon dioxide will be emitted. On the contrary, for geographical regions where the regular electricity has high emission intensities (such as China and India), the savings offered by FTSFTW are much higher. Note that we consider the manufacturing carbon footprint cost M (see Eq. (8)) to be fixed, even with changing I H value. This means that in this case we have fixed the instance of I H Fig. 9. Relative emission savings as a function of the LF energy emission intensity I L (for f = 0.25, p = 0.6 and I H = 500 g CO 2 -eq/kwh). in Eq. (10) to the world-average emission intensity. Fixing the manufacturing footprint makes sense, as it represents the case where the equipment remains manufactured as before, but is used in a region with a different HF energy emission intensity. Similarly, we can also consider different values for the LF energy emission intensity I L. The value of I L = 10 g CO 2 -eq/kwh we assumed in Section 4.3, is based on state-of-the art renewable energy, typically from wind turbines. For other energy sources with higher emission intensities, the savings will obviously be smaller. Fig. 9 shows the savings for increasing values of I L, with the HF energy emission intensity fixed at 500 g CO 2 -eq/kwh. As can be seen, the savings rapidly dwindle, to the point where they become marginal. As such, using less emission-saving renewable energy sources such as solar PV installation should be evaluated carefully if the main goal is saving on total carbon emission by employing the ADD scenario. To summarize, with current estimates for the manufacturing and usage footprint, carbon emission savings up to around 14% are possible by deploying additional data center sites. Actual savings depend mainly on the manufacturing fraction (lower is better), the LF energy availability (optimum around 50% 70%) and the number of sites deployed (optimum around times as much data centers as required for daily operation). For geographical regions with higher HF emission intensities, the possible savings by employing the ADD scenario are much higher than 14%; likewise, for intensities below the world average savings quickly turn negative. 6. Case study II: the Low Load Redistribution (LLR) scenario The main conclusion from the above scenario is that the manufacturing carbon footprint is a non-negligible factor, and should

106 412 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) (a) Regular scenario: 3/5 load distribution under the regular approach. (b) LLR scenario: 3/5 load distribution under the FTSFTW approach. Fig. 10. Nominal load distribution in a distributed data center. (a) Shows the regular scenario where a nominal load of 60% is distributed equally over all data center sites. (b) Shows the LLR scenario, where the same nominal load is distributed according to the FTSFTW approach, resulting in an optimal usage of sites with LF energy availability. be taken into account when evaluating potential carbon footprint savings. However, there are cases where the manufacturing footprint is already expended. Data centers are not constantly running at peak capacity, but instead operate at a nominal load well below the peak capacity, typically servers operate most of the time between 10% and 50% of their maximum utilization levels [28]. We could redistribute the load using the FTSFTW approach, resulting in what we will refer to as the Low Load Redistribution (LLR) scenario. Regular approach Fig. 10(a) shows the regular approach (without applying LLR). The load is equally distributed among the different sites. To calculate the total carbon footprint, we consider a data center with peak capacity m to run at nominal load n. We assume unused servers to be powered down. The total footprint of a data center running at this nominal load is then: F nominal = F u + F m = n (pl + (1 p) H) + mm. (13) LLR approach What would happen if we apply the FTSFTW approach to optimally distribute processing to sites where LF energy is available (Fig. 10(b))? We can use Eq. (11) or (12) to calculate the footprint in that case as well, with m representing the peak capacity, and n representing the (varying) nominal load. Fig. 11 plots the footprint for both scenarios for a distributed data center consisting of 5 sites (m = 5), for an increasing load (i.e., n increasing from 0 to m). The LF energy availability p per site has been taken equal to 0.6. As can be seen, for the nominal load being half of the peak capacity, savings around 20% are possible by employing FTSFTW. These are savings over the total footprint, that is, the sum of the use phase and manufacturing phase footprint. If we only consider the savings over the usage phase, which would be an equally valid approach since the manufacturing phase has no savings, the savings are as high as 90% when running at 20% of the capacity and still reach more than 60% when running at half the peak capacity. The savings itself vary for different values of p. This is shown in Fig. 12. It is important to remark that from the above results we should not conclude to design distributed data centers to run well below their maximum capacity. This results in a large total manufacturing carbon footprint. First, and foremost, data center capacity should be scaled to their nominal loads as much as possible, taking into account such factors as redundancy and peak loads. Once this is done, carbon emissions can be reduced using the LLR scenario outlined above. Fig. 11. Relative footprint (with respect to the maximum load) of a distributed data center running at various loads both under a regular scenario and an LLR scenario (m = 5, p = 0.6). The Savings, total are the relative savings over the total footprint (both manufacturing F m and usage F u ). The Savings, usage are the relative savings over the usage footprint only. 7. Conclusions Fig. 12. Savings for various values of p. The carbon footprint from data centers is significant, and growing. Besides improvements in energy-efficiency, the use of low footprint energy (typically from renewable energy sources such as wind or solar power) is key to reducing data center carbon emissions. Data centers are in a unique position to overcome some of the issues currently associated with renewable energy sources.

107 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) They can be located near renewable energy sites, and jobs and data can be migrated from site to site as renewable energy intermittent by nature comes and goes. This approach has been referred to as follow the sun/follow the wind. In this paper, we researched if carbon emissions can be reduced by applying this technique to take advantage of the resulting increased availability of low footprint renewable energy. To this purpose, we have build a mathematical model to calculate the carbon footprint of such a distributed data center infrastructure that is powered by a mix of low-footprint (LF) and high-footprint (HF) energy. We have shown that for footprint reduction the manufacturing carbon footprint of data centers is a critical parameter to consider. Based on the available LCA data for data centers, footprint savings in the order of 14% over the total footprint are possible by deploying additional data center sites to take advantage of the resulting increased available of LF energy. Reductions of the manufacturing footprint relative to the usage footprint will lead to improved savings. However, a number of factors heavily influence the actual savings, which could easily turn into an increased carbon footprint if not evaluated carefully. As the savings are strongly influenced by the HF electrical emission intensity, it is of no use to deploy the follow the sun/follow the wind approach in regions with emission intensities below the current world average value. And, consequently, it makes more sense to use the approach in regions with high emission intensities. Carbon footprint savings also depend on the LF energy availability per site: optimal availability varies for different configurations, but is in the order of 50% 70%. Optimum savings can be gained at architectures that deploy around times as much data centers as required for daily operation. Bigger savings up to 60% 90% are possible by applying the follow the sun/follow the wind strategy to data centers where the nominal load is well below the peak capacity. Finally, it should be noted that our model is not restricted to carbon footprint metrics. It can easily be used or modified to evaluate other metrics. For example, the low and high emission intensities can be replaced by low and high energy prices (requiring an appropriate quantification of the manufacturing fraction in that case) to evaluate the cost benefits in the light of fluctuating energy prices. However, this is outside the scope of this paper. Acknowledgments The authors would like to thank Koen Casier, Kevin Mets and Tom Verschueren for their helpful advice and discussions. We also thank the Greenstar Network partners for their valuable comments. The work described in this paper was carried out with the support of the BONE project (Building the Future Optical Network in Europe) and the TREND project (Towards Real Energyefficient Network Design), both a Network of Excellence funded by the European Community s Seventh Framework; the IBBT-project GreenICT and the STRONGEST project funded by the European Community s Seventh Framework Programme FP7/ under grant agreement n References [1] M. Pickavet, W. Vereecken, S. Demeyer, P. Audenaert, B. Vermeulen, C. Develder, D. Colle, B. Dhoedt, P. Demeester, Worldwide energy needs for ICT: the rise of power-aware networking, in: nd International Symposium on Advanced Networks and Telecommunication Systems, p. 3. [2] EIA, International energy statistics, total electricity net consumption, aid=2. [3] J. Koomey, Estimating total power consumption by servers in the US and the world, [4] L. Deboosere, P. Simoens, J. De Wachter, B. Vankeirsbilck, F. De Turck, B. Dhoedt, P. Demeester, Grid design for mobile thin client computing, Future Generation Computer Systems (2010). [5] S. Figuerola, M. Lemay, V. Reijs, M. Savoie, B. St. Arnaud, Converged optical network infrastructures in support of future Internet and grid services using IaaS to reduce GHG emissions, Journal of Lightwave Technology 27 (2009) [6] A. Ipakchi, F. Albuyeh, Grid of the future, IEEE Power & Energy Magazine 7 (2009) [7] T. Ackermann, G. Andersson, L. Söder, Distributed generation: a definition, Electric Power Systems Research 57 (2001) [8] EIA, Total annual losses estimates related to electrical transmission and distribution, [9] M. Jacobson, Review of solutions to global warming, air pollution, and energy security, Energy & Environmental Science 2 (2009) [10] K. Le, R. Bianchini, M. Martonosi, T. Nguyen, Cost- and energy-aware load distribution across data centers, Proceedings of HotPower (2009). [11] C. Clark, K. Fraser, S. Hand, J.G. Hansen, E. Jul, C. Limpach, I. Pratt, A. Warfield, Live migration of virtual machines, in: NSDI 05: Proceedings of the 2nd Conference on Symposium on Networked Systems Design & Implementation, USENIX Association, Berkeley, CA, USA, 2005, pp [12] F. Travostino, P. Daspit, L. Gommans, C. Jog, C. De Laat, J. Mambretti, I. Monga, B. Van Oudenaarde, S. Raghunath, Seamless live migration of virtual machines over the MAN/WAN, Future Generation Computer Systems 22 (2006) [13] G. Kecskemeti, G. Terstyanszky, P. Kacsuk, Z. Neméth, An approach for virtual appliance distribution for service deployment, Future Generation Computer Systems 27 (2011) [14] A. Hopper, A. Rice, Computing for the future of the planet, Philosophical Transactions of the Royal Society. Series A (2008). [15] A. Qureshi, R. Weber, H. Balakrishnan, J. Guttag, B. Maggs, Cutting the electric bill for internet-scale systems, in: Proceedings of the ACM SIGCOMM 2009 Conference on Data Communication, ACM, 2009, pp [16] The greenstar network project, Project website: [17] A. Daouadji, K.-K. Nguyen, M. Lemay, M. Cheriet, Ontology-based resource description and discovery framework for low carbon grid networks, in: SmartGridComm [18] W. Van Heddeghem, M. De Groote, W. Vereecken, D. Colle, M. Pickavet, P. Demeester, Energy-efficiency in telecommunications networks: link-bylink versus end-to-end grooming, in: Conference on Optical Network Design and Modeling, 14th, Proceedings, IEEE, 2010, p. 6. [19] C. Hermann, Environmental footprint of ICT equipment in manufacture, use and end of life, in: ECOC 2008, Brussels, p. 24. [20] J. Malmodin, Global carbon footprint of ICT: method and results, in: ITU FG ICT & CC, Geneva, [21] J. Malmodin, A. Moberg, D. Lundén, G. Finnveden, N. Lövehagen, Greenhouse gas emissions and operational electricity use in the ICT and entertainment & media sectors, Journal of Industrial Ecology (2010). [22] Google, Data center efficiency measurements, html. [23] US Environmental Protection Agency, Report to congress on server and data center energy efficiency, development/downloads/epa_datacenter_report_congress_final1.pdf. [24] IEA, CO 2 emissions from fuel combustion highlights, [25] M. Pehnt, Dynamic life cycle assessment (LCA) of renewable energy technologies, Renewable Energy 31 (2006) [26] Stanford University, Folding@home, Project page: [27] E. Luttmann, D. Ensign, V. Vaidyanathan, M. Houston, N. Rimon, J. Øland, G. Jayachandran, M. Friedrichs, V. Pande, Accelerating molecular dynamic simulation on the cell processor and playstation 3, Journal of Computational Chemistry 30 (2009) [28] L. Barroso, U. Holzle, The case for energy-proportional computing, Computer 40 (2007) Ward Van Heddeghem received his Masters degree in Applied Engineering Electromechanics from Hogeschool Gent, Belgium (1999) and his M.Sc. degree in Computer Science Engineering from Vrije Universiteit Brussel, Belgium, (2009). He is a Ph.D. student in the IBCN research group at INTEC UGent/IBBT. His research interests are in the field of environmental impact of ICT and energyefficient network architectures.

108 414 W. Van Heddeghem et al. / Future Generation Computer Systems 28 (2012) Willem Vereecken received his M.Sc. degree in electrotechnical engineering from Ghent University, Belgium, in July He is a Ph.D. student in the IBCN (IN- TEC Broadband Communication Networks) research group at the Department of Information Technology at Ghent University/Interdisciplinary institute for BroadBand Technology (INTEC UGent/IBBT). His research is funded by a Dehousse Ph.D. grant. His main research interests are the environmental impact of ICT systems and the design of energy efficient network architectures. Mario Pickavet (M 99) is associate professor at Ghent University. His current research interests are related to broadband communication networks (WDM, IP, (G-) MPLS, Ethernet, OPS, OBS) and include design, longterm planning, techno-economical analysis and energy efficiency of core and access networks. He is currently involved in several European and national projects, such BONE, DICONET, ECODE, ALPHA and OASE. He is co-author of the book Network Recovery: Protection and Restoration of Optical, SONET-SDH, IP, and MPLS. Didier Colle (M 01) received his M.Sc. degree in Electrotechnical Engineering (option: Communications) from Ghent University in Since then, he has been working as a researcher in the IBCN research group at IN- TEC UGent/IBBT. His research led to a Ph.D. degree in February His work is focused on optical transport networks, to support the next-generation Internet. He was involved in several European projects like NOBEL, LASAGNE and TBONES and COST-action 266 and 291. Piet Demeester received his Masters degree in Electrotechnical engineering and his Ph.D. degree from Ghent University, in 1984 and 1988, respectively. In 1992, he started a new research activity on broadband communication networks resulting in the IBCN (INTEC Broadband Communications Networks) research group. In 1993 he became a professor at Ghent University, where he is responsible for research and education on communication networks. The research activities cover various communication networks, including network planning, network and service management, telecom software, internetworking, network protocols for QoS support, etc. He is a member of the editorial board of several international journals and has been a member of several technical program committees (ECOC, OFC, DRCN, ICCCN, IZS).

109 This page intentionally left blank This page intentionally left blank This page intentionally left blank

110 P1: Gopal December 2, :55 K13681 K13681 C006 Chapter 6 Ontology-Based Resource Description and Discovery Framework for Low-Carbon Grid Networks K.-K. Nguyen, A. Daouadji, M. Lemay, and M. Cheriet Contents 6.1 Introduction Low-Carbon Network and Resource Management Virtualized Management ICT Energy Consumption Ontology Proposed System Architecture Carbon-Aware Resource Discovery Experimental Results Conclusion Acknowledgments References

111 P1: Gopal December 2, :55 K13681 K13681 C Communication and Networking in Smart Grids Using smart grids to build low-carbon networks is one of the most challenging topics in the ICT industry. The GreenStar Network is the first worldwide initiative completely powered by renewable energy sources across Canada. Smart grid techniques are deployed to migrate data centers across network nodes according to energy source availabilities, thus reducing CO 2 emissions to minimal. Such flexibility requires a scalable resource management support, achieved by virtualization enabling the sharing, aggregation, and dynamic configuration of a large variety of resources. Such a virtualized management is based on an efficient resource description and discovery framework, dealing with a large number of heterogeneous elements and the diversity of architectures and protocols. In this chapter, we present an ontology-based resource description framework, developed particularly for ICT energy management, where the focus is on energy-related semantics of resources and their properties. We propose then a scalable resource discovery method in large and dynamic collections of ICT resources, based on semantics similarity inside a federated index using a Bayesian belief network. The proposed framework allows users to determine the cleanest resources to fulfill their requirements, regarding the energy source availabilities. Experimental results are shown to compare the proposed method with a traditional one in terms of GHG emission reductions. 6.1 Introduction Nowadays, reducing greenhouse gas (GHG) emissions is becoming one of the most challenging research topics in information and communication technologies (ICT) because of the alarming growth of indirect GHG emissions resulting from the overwhelming utilization of ICT electrical devices. The current approach when dealing with the ICT GHG problem is improving energy efficiency, which aims to reduce energy consumption at the micro level. Research projects following this direction have focused on microprocessor design, computer design, power-on-demand architectures, and virtual machine consolidation techniques. However, a microlevel energy efficiency approach will likely lead to an overall increase in energy consumption due to the Khazzoom-Brookes postulate (also known as Jevon s paradox), which states that energy efficiency improvements that, on the broadest considerations, are economically justified at the micro level, lead to higher levels of energy consumption at the macro level [1]. Therefore, we believe that reducing GHG emissions at the macro level is a more appropriate solution. Large ICT companies, like Microsoft, which consumes up to 27 megawatts of energy at any given time [15], have built their data centers near green power sources. Unfortunately, many computing centers are not so close to green energy sources. Thus, green energy distributed networks using smart grid technologies are an emerging technology. An

112 P1: Gopal December 2, :55 K13681 K13681 C Ontology-Based Resource Description and Discovery Framework important assumption to make is that losses incurred in energy transmission over power utility infrastructures are much higher than those caused by data transmission, which makes relocating a data center near a renewable energy source a more efficient solution than trying to bring the energy to an existing location. Such a new green ICT network-based approach has to face a number of issues, such as the complexity of networks, the large number of elements, the diversity in architectures and organizations, and highly flexible requirements. The GreenStar Network [2] is the first nationwide network in the world that is powered only by green energy. Virtualization, a new paradigm being explored by the research and education community dealing with highly complex distributed environments, is deployed to address resource management issues in such large-scale smart grid networks. Virtualized management will likely become one of the key solutions interconnecting appliances in order to give each of the virtual entities a complete semblance of their counterparts. Main characteristics of a virtualization technique include: (1) a warping of network elements, such as connectivity resources and traffic processing resources, (2) dynamic establishment capability, such as flexible and efficient mechanisms to trigger and tear down service, (3) end-to-end across multiple domains, and (4) control by the end user, e.g., the end user should be able to operate the virtual infrastructure as if it were a dedicated physical infrastructure. Key techniques supporting a scalable network management include resource description and discovery. An inefficient resource description solution could be a barrier for developing powerful resource discovery mechanisms, which in turn makes the resource management inefficient. For example, resources may have different descriptive information, using different formats. They may also be indexed using different keywords referring to various characteristics. This imposes several difficulties for searching and discovery. In the context of virtualized resource management, this chapter presents a framework for resource description and discovery, which targets lowcarbon network management based on smart grid technologies. The resource description is based on an energy-oriented ontology proposed for ICT resources. Resource information is represented using Resource Description Framework (RDF) graph models [8] and web semantics, which enables resource classification, indexing, keyword searching, and semantic analyses. The resource discovery is based on a Bayesian semantic approach to enable various resource description methods. Thesaurus-based searching is also implemented addressing the keyword uncertainty problem, where we define a priori probability on each keyword, in order to improve the performance of the resource discovery engine. The main contribution of this chapter includes an energy-oriented ontology for ICT equipment, a description of grid resources based on proposed

113 P1: Gopal December 2, :55 K13681 K13681 C Communication and Networking in Smart Grids ontology, and a resource discovery method that takes into account the energy consumption and GHG emissions of ICT resources. The remainder of the chapter is organized as follows. In the next section, we present the key elements to build a low-carbon network with a resource management framework covering a large set of resources. A virtualization management approach is then described, with a proposed ontology for ICT resource focusing on energy consumption. A resource discovery mechanism is next provided where queries are built in a flexible and extensible manner with the help of a Bayesian network model and a knowledge base. We also provide an example of using the proposed resource description and discovery framework in the GreenStar Network in order to reduce GHG emissions. Finally, we conclude the chapter and present future work. 6.2 Low-Carbon Network and Resource Management The research presented in this chapter is positioned in a Canadainternational project, aimed at building the first nationwide network, the GreenStar Network, which is powered only by green energy sources. The project is inspired from the carbon-neutral approach proposed in [3] and from the emerging need of a green energy distribution wide area network model for establishing a standard carbon protocol for the ICT industry. The key objective of the GreenStar Network project [2] is to create a pilot and a testbed environment from which to derive best practices and guidelines to follow when building low-carbon networks. The idea behind the Green- Star Network project is that a neutral carbon network must consist of data centers built in proximity to clean power sources, and user applications will be moved to be executed in such data centers (Figure 6.1). Such a network must provide an ability to migrate entire virtual machines (routers and servers) to alternate data center locations. An underlying communication network is supported by a high-speed optical layer having up to 1,000 Gbps bandwidth capacity. Note that optical networks have a modest increase in power consumption, especially with new 100 G and 1,000 G waves, in comparison to electronic equipment such as routers and aggregators [3]. The key technology of the GreenStar Network is virtualization. The migration of a data center over network nodes is indeed a result of a combination of server and network virtualizations as virtual infrastructure management. Figure 6.1 shows the connection plan of the GreenStar Network. The Canadian section of the network has the largest deployment of six nodes powered by sun, wind, and hydroelectricity; each node represents a data center. The core GreenStar Network is connected to the European green nodes in Ireland (HEAnet), Spain (i2cat), and the USA (CalIT2, through

114 P1: Gopal December 2, :55 K13681 K13681 C Ontology-Based Resource Description and Discovery Framework Egypt (Smart Village) China (WICO) CANADA Green Star Core CANARIE GEANT USA (ESNet & CalIT2) Iceland (NORDUner) Netherlands (SURFnet) Belgium (IBBT) Ireland HEAnet Spain (i2cat) Figure 6.1 The GreenStar Network. ESNet). Other nodes in Belgium (IBBT), the Netherlands (SURFnet), Iceland (NORDUnet), China (WiCo), and Egypt (Smart Village) are also planned to connect to the network. In such a network, data centers are smart grids that are capable of turning on/off and migrating application data to remote locations in order to utilize available green energy sources. This requirement is achieved by a flexible and robust resource description and discovery framework where energy control a is key characteristic. Such a framework is not yet available in current grid resource management frameworks. Indeed, most of current grid resource description and discovery research focuses on XML structured data, such as the Globus Toolkit s Monitoring and Discovery System (MDS4) [4,6,7], in order to enable Web service interfaces. Nevertheless, a semantic relationship among resources is not fully taken into account, particularly as it is related to energy consumption interpretation, which results in big challenges for building low-carbon networks. Additionally, the current method based on collecting and publishing aggregated information can

115 P1: Gopal December 2, :55 K13681 K13681 C Communication and Networking in Smart Grids decrease search performance. Some research has suggested handling the semantic aspect based on the Web Services Resource Framework (WSRF) specification [9]. However, WSRF will unlikely be appropriate for managing heterogeneous environments. In addition, current methods usually maintain a large number of sophisticated components, which impose several challenges for implementation and maintenance. Until very recently, many grid resource management systems still did not support semantic resource description and discovery. For example, Gridbus broker [10], Gridway [11], and the Monitoring and Discovery System (MDS) [12] simply use Globus middleware services to gather grid resource information; thus, it is impossible to understand the semantic relationship between the available resource information and the requested information. Their resource discovery mechanism is based on conventional keyword matchmaking. Some other research has proposed various support levels for semantic relationships among resources [13], for example, using ontology to describe and select web-based services used in specific domains, like life sciences and aeronautical engineering [14]. However, no virtualized management approach is proposed. One of the key challenges for a resource description and discovery system is automation. Such a system must be able to integrate knowledge from resource providers, service providers, and users in order to enrich system databases and to empower searching performance. In [9], authors proposed a completed framework for ICT resource management in grid-based networks using a predefined ontology. Their research, however, focuses on resource utilization and searching performance. Energy-related aspects and semantics expressing relationships among resources in terms of energy consumption interdependence are not taken into account. 6.3 Virtualized Management The GreenStar Network is built with multiple layers, resulting in a large number of resources to be managed. In order to scale management activities along with the growth of grid resources, virtualized management has been proposed for service delivery regardless of the physical location of the infrastructure, which is determined by resource providers. This keeps complex underlying services hidden inside the infrastructure provider. Resources are allocated according to user s need, and hence highest utilization and optimization levels can be achieved. During the service, the user owns and controls resources as if he were the owner. Therefore, users will be able to run their application in a virtual infrastructure powered by green energy sources. Such a service provisioning model has three layers: infrastructure provider, service provider, and end users (Figure 6.2). End users send jobs

116 P1: Gopal December 2, :55 K13681 K13681 C Ontology-Based Resource Description and Discovery Framework End Users App App App Service Providers Resource Resource Resource Resource Infrastructure Providers Figure 6.2 Virtualized service provisioning. to infrastructure providers through service providers and get back results. The service provider layer consists of several functionalities, such as resource registry, reservation, work scheduling, resource aggregation, data routing, and so on. The virtualized management approach for grid resources requires a flexible and extensible resource description and discovery support. The interdependence between a service provider and an infrastructure provider, as well as resource sharing and aggregation capabilities, must be represented semantically. One of the appropriate solutions for such a description is using a particular ontology. 6.4 ICT Energy Consumption Ontology Ontologies (often also referred to as domain model) are generally defined as a representation of a shared conceptualization of particular domain. As a traditional textual keyword-based approach is not efficient in resource description and discovery regarding energy consumption characteristics, an ontology should be provided to represent relationships of resources in this domain.

117 P1: Gopal December 2, :55 K13681 K13681 C Communication and Networking in Smart Grids Solar Wind Hydro Gas Oil Coal is-a is-a is-a is-a is-a is-a Green Dirty is-a is-a Energy is-a Used-by ICT Resource is-a Mobility Aggregate is-a Networking Computing has-a Bandwidth has-a has-a CPU has-a Memory has-a Storage has-a Hard disk Figure 6.3 ICT energy consumption ontology. The proposed ontology includes knowledge models for ICT elements, hosting application, and energy. These models are mapped into classes and properties assigned to these classes. The ontology contains a hierarchy of semantically linked elements and describes their attributes and capabilities. As shown in Figure 6.3, the proposed ontology is a combination of two large domains: ICT and energy. It includes the following classes: The ICT resources class is a generic type of ICT resources. It has some common characteristics, like name, ID, location, and power consumption. ICT resources has three subclasses: (1) computing class represents all computing resources, such as servers, PCs, or handheld devices; (2) networking class represents all resources used for interconnecting ICT devices, such as routers, switches, hubs, optical cross-connects, and multiplexerdemultiplexer, and (3) storage class represents all storage resources, such as hard disk, CDs, and multirack storage devices. There is a relationship between ICT resources and energy that defines the kind of energy consumed by ICT devices, e.g., hydroelectricity, solar, wind, or fuel. A relationship between ICT resource and energy also defines the amount of energy consumed by an ICT device during a time unit (i.e., an hour). ICT resources is associated with a concept, named mobility, to

118 P1: Gopal December 2, :55 K13681 K13681 C Ontology-Based Resource Description and Discovery Framework define whether the resource can be portable or not. A resource may also have an aggregation capability, which defines whether the resource can be aggregated in order to achieve a given task. An energy source can be green energy or dirty energy. Green energy can be solar, wind, or hydroelectricity. Dirty energy includes natural gas, heating oil, or coal. Each energy class has an associated cost. Dirty energy is more costly than green energy. In the context of the GreenStar Network project, we define energy cost as CO 2 emissions of an energy unit. These classes are implemented using RDF [8], which is considered the best representation for ontology. Unlike traditional databases where tables are fixed and cannot handle semantic aspects, RDF supports the semantic representation and information is mapped directly to models, making the method flexible and extensible. Reasoning is also enabled in RDF, using a triple object-attribute value: an object O has an attribute A with value V [8]. 6.5 Proposed System Architecture Figure 6.4 shows the global architecture of the proposed resource description and discovery framework used for the GreenStar Network. There are two stacks in the figure. The resource provider stack aims at determining ontology concepts and description for exposed resources. On the other hand, the end user stack is dedicated for searching appropriate resources according to each user request. The system maintains a local index to register all resources. Resource providers define description and capabilities for their resources. One or several keywords can be used for each resource to facilitate indexing and searching. The semantic analyzer performs information processing based on resource descriptions in order to determine the type of resources and associates resources to appropriate ontology concepts. The semantic analyzer may also extract keywords from a description. Newly found keywords or concepts will be added to the semantic analyzer knowledge base. Finally, resource registry service is invoked to register resources in the RDF data model base. The end user stack begins with a request for resources. The query is analyzed and keywords are extracted and processed by the semantic analyzer. Based on its knowledge base, the semantic analyzer determines the required resources and their locations. If a resource is available, it will be triggered. The proposed system focuses on the energy aspects. Therefore, the set of resources will be returned, taking into account the minimal GHG emissions possible. In order to provide a powerful search method based on the proposed ontology, we use a Bayesian semantic graph, which combines a semantic

119 P1: Gopal December 2, :55 K13681 K13681 C Communication and Networking in Smart Grids Resource Provider Resources Query End User Key words Description Key words Extrator Bayesian Network Ontology Knowledge Base Semantic Analyzer RDF Model + Resource Registration Query Analyzer + Resource Launcher Resource Database Cluster 1 Cluster 2 Cluster n Figure 6.4 Global architecture. inference and a probabilistic one. The key idea is to integrate semantic links and reasoning rules, by processing keywords and taking into account the context, which determine the cluster the resource belongs to. As proposed in our ontology, a resource will be placed in one of the three categories: computing, storage, and network. However, a user query like RAM = 64 Mb, IP address = , may lead to a confusion, because the result can be a server or a router. If further information, such as bandwidth = 1G, is added, a more accurate network resource could be found. The Bayesian semantic analyzer is used to deal with such confusion. It processes all the words in each resource description or user query and calculates the probability that resource belongs to each cluster. Such a mechanism improves significantly the search operation. Generally, providers and end users use different manners to describe resources and express their requests. The knowledge base is used to find

120 P1: Gopal December 2, :55 K13681 K13681 C Ontology-Based Resource Description and Discovery Framework Table 6.1 Example of Clusters Word CPU RAM Bandwidth Concept Compute Storage Network the similarity between these different keywords. It is built based on a thesaurus dictionary. When the input keyword is not found, a synonym-based searching will be triggered. Thus the searching operation is more efficient, since many of the bad formed requests or poor descriptions can be addressed. In order to define the concepts and clusters for a Bayesian network, a probabilistic table is built, as proposed in [5], which assigns joint probability values. The assignments are based on expert judgment and may be improved over time. The probability table is a matrix, as shown in Table 6.1, which has as rows words and columns concepts. A value in each matrix cell represents the probability of the word to be involved in the concept. When a resource has been analyzed and its keywords have been determined, it is represented by an RDF graph. This operation is achieved using an RDF request language, such as SPARQL [18] in our implementation. In the user stack, user queries are also analyzed in order to extract keywords. Similarly, probability is calculated for each keyword, and then target concept and cluster are determined. A substituted synonym can be used if no cluster is found. Finally, an appropriate resource will be obtained from the resource database. In the worst case, where no resource is found, users will be suggested to use alternate keywords. 6.6 Carbon-Aware Resource Discovery The proposed resource discovery mechanism aims at minimizing the CO 2 emissions produced by ICT resources for each service request. For example, a user request for 25 CPU, 2 GHz speed each, can be achieved by many available servers distributed across the GreenStar Network. However, at a given period of time, only some of them are powered by green energy sources (i.e., solar energy is not available during the night). Thus, the proposed resource discovery mechanism tries to find a list of optimal resources in terms of CO 2 emissions. The problem can be formulated as follows.

121 P1: Gopal December 2, :55 K13681 K13681 C Communication and Networking in Smart Grids Given N resources in the network, each resource R i has a set of capabilities {C ij } and a power P f. Resource R i is powered by an energy source E m, which is associated with a CO 2 emission factor E mi. A user request for a set of L capabilities: Q ={Q 1 }, where Q 1 = C 1 is a sum of capabilities C l. The resource discovery engine has to decide which resource in the network should be assigned for the request Q. The following matrix of binary variables represents the resource selection result: X i = { 1 if Ri is selected 0 otherwise (6.1) The mathematical optimization problem is to optimize: min N x i P i E mi (6.2) i subject to: N x i C il Q l (6.3) i,lψ L This formulation is referred to as the original mathematical optimization problem. Since the objective function is nonlinear and there are nonlinear constraints, the optimization model is a nonlinear programming problem with binary variables. Generally, a solution for this problem cannot be obtained by mathematical programming solvers. In this chapter, we use a simple greedy algorithm to determine solutions for each user request, assuming a relatively small number of resources in the network (i.e., order of 10,000). The algorithm first tries to find a list of resources that can meet user requirements. The resources in the list will then be sorted according to energy sources. They will be picked from the top of the list until the required number is filled. 6.7 Experimental Results In the current GreenStar Network, there are 6 different data centers powered by different energy sources. According to the Canadian GHG Registries [17], electricity emission factor (i.e., tons of CO 2 per Kwh) varies from province to province. For example, Québec has the lowest emission factor and Alberta has the highest one, due to the fact that AB uses fossil power while QC uses hydroelectricity.

122 P1: Gopal December 2, :55 K13681 K13681 C Ontology-Based Resource Description and Discovery Framework Number of Resources QC BC ON NS AB Figure 6.5 Distribution of resources over network nodes according to a user request (CPU = 2 GHz, RAM >= 1,024 MB, OS = Ubuntu). In our experiments, a user query such as CPU = 2 GHz, RAM >= 1, 024 MB, OS = Ubuntu may return a bunch of servers. A user application service, like GeoChronos [16], developed by the University of Calgary, requires 48 servers with that configuration. Figure 6.5 shows the distribution of resources found for such a user request. In a traditional resource discovery, when the GHG emission aspect is not taken into account, the first 48 servers in the finding results will be returned regardless of their locations or energy sources. It would lead to a nonoptimal resource allocation scheme from the point of view of ecologists. The proposed resource discovery engine will try to allocate resources powered by greenest energy sources; thus the GHG emission is minimal. In reality, resources located in Québec are the most efficient in terms of GHG emissions because they are all powered by green energy (i.e., hydroelectricity). Unfortunately, their number and availability make it is impossible to always allocate resources in Québec for all user requests. Therefore, a best-effort discovery must be provided in order to find an optimized possible resource list. Figure 6.6 compares GHG emissions resulting from the traditional and proposed resource discoveries. The amount of CO 2 emissions is calculated based on [17]. As shown in the figure, when the number of required resources is 900, we may save up to 128 credits of CO 2. The carbon credit savings is significant, particularly when the number of resources allocated increases, which is very useful for large IT companies when a carbon tax is imposed (i.e., which is currently the case in British Columbia).

123 P1: Gopal December 2, :55 K13681 K13681 C Communication and Networking in Smart Grids Tons CO 2 /Year Traditional resource discovery method Proposed resource discovery method Number of Resources Figure 6.6 CO 2 emission in the traditional and proposed resource discovery schemes. 6.8 Conclusion In this chapter, we have presented a framework for resource description and discovery in large-scale smart grid networks, leveraged by the aim of reducing GHG emissions of ICT resources. Results of this research will be used for GreenStar Network, the first nationwide network powered entirely by green energy. An energy-oriented ontology has been proposed to support semantic descriptions of ICT resources in terms of energy consumption. Taking into account the inefficiency of current resource description and discovery frameworks, a web semantic approach has been deployed for resource description and persistence. The resource discovery engine we developed integrates a knowledge base and a Bayesian network in order to improve searching performance. GHG reductions resulting from experimental evaluation suggest potential applications of the research in large-scale grid networks. Our future work includes resource control and migration, driven by the change of the green energy sources, such as solar and wind. The resource discovery algorithm will also be improved, regarding the increasing number of resources in the network. Optimization mechanisms will be investigated, with respect to combined objective functions, such as QoS and energy together. Acknowledgments We thank CANARIE for funding the GreenStar Network project under its G-IT pilot program. The authors acknowledge all GreenStar Network partners for their contribution in this project, from network implementation to carbon assessment and quantification.

124 P1: Gopal December 2, :55 K13681 K13681 C006 Ontology-Based Resource Description and Discovery Framework 151 References 1. H.D. Saunders, The Khazzoom-Brookes Postulate and Neoclassical Growth, Energy J., 13(4), , The GreenStar Network Project, 3. S. Figuerola, M. Lemay, V. Reijs, M. Savoie, B. St. Arnaud, Converged Optical Network Infrastructures in Support of Future Internet and Grid Services Using IaaS to Reduce GHG Emissions, J. Lightwave Technol., 27(12), , S.M. Pahlevi, I. Kojima, Towards Automatic Service Discovery and Monitoring in WS-Resource Framework, inproceedings of Semantics, Knowledge and Grid SKG 05, 2005, p K. Kyoung-Min, H. Jin-Hyuk Hong, C. Sung-Bae, Intelligent Web Interface Using Flexible Conversational Agent with Semantic Bayesian Networks, in Proceedings of Next Generation Web Services Practices NWeSP 05, 2005, pp A. Chervenak, J.M. Schopf, L. Pearlman, S. Mei-Hui, S. Bharathi, L. Cinquini, M. D Arcy, N. Miller, D. Bernholdt, Monitoring the Earth System Grid with MDS4, inproceedings of e-science and Grid Computing e-science 06, 2006, p J.M. Schopf, I. Raicu, L. Pearlman, N. Miller, C. Kesselman, I. Foster, M. D Arcy, Monitoring and Discovery in a Web Services Framework: Functionality and Performance of Globus Toolkit MDS4, J. Physics, 46, , S. Decker, S. Melnik, V. Van-Harmelen, D. Fensel, M. Klein, J. Broekstra, M. Erdmann, I. Horrocks, The Semantic Web: The Roles of XML and RDF, IEEE Internet Comput., 4(5), 63 73, B.R. Amarnath, T.S. Somasundaram, M. Ellappan, R. Buyya, Ontology-Based Grid Resource Management, Software Practice and Experience, 39(17), , S. Venugopal, R. Buyya, L. Winton, A Grid Service Broker for Scheduling E- Science Applications on Global Data Grids, Concurrency and Computation: Practice and Experience, 18(6), , E. Huedo, R.S. Montero, I.M. Llorente, The Gridway Framework for Adaptive Scheduling and Execution on Grids, Scientific International Journal for Parallel and Distributed Computing, 6(3), 1 8, K. Czajkowski, S. Fitzgerald, I. Foster, C. Kesselman, Grid Information Services for Distributed Resource Sharing,Proceedings of the Tenth IEEE International Symposium on High-Performance Distributed Computing (HPDC-10), O. Corcho, P. Alper, L. Kotsiopoulos, P. Missier, S. Bechhofer, C. Goble, An Overview of S-OGSA: A Reference Semantic Grid Architecture, Journal of Web Semantics, 4(2), , M.J. Murphy, M. Dick, T. Fischer, Towards the Semantic Grid: A State of the Art Survey of Semantic Web Services and Their Applicability to Collaborative Design, Engineering, and Procurement, Communications of the IIMA, 8(3), 11 24, 2008.

125 P1: Gopal December 2, :55 K13681 K13681 C Communication and Networking in Smart Grids 15. W. Binder, N. Suri, Green Computing: Energy Consumption Optimized Service Hosting, in SOFSEM 2009: Theory and Practice of Computer Science, vol. 54, C. Kiddle, GeoChronos: A Platform for Earth Observation Scientists, Open- GridForum 28, LivClean Carbon Offset Solution, How Is This Calculated?, livclean.ca. 18. E. Prud hommeaux, A. Seaborne, SPARQL Query Language for RDF, W3C Recommendation,

126 This page intentionally left blank This page intentionally left blank This page intentionally left blank

127 Green Networking and Resilience Ward Van Heddeghem, Willem Vereecken, Jens Buysse, Chris Develder, Mario Pickavet, Piet Demeester IBCN Ugent - IBBT DRCN, Krakow (Poland) October 10-12, 2011

128 Power Consumption 2007 (worldwide) 16,000 GW Electricity = 30% of primary energy consumption 4,800 GW Primary Energy 1,900 GW Electrical Energy M. Pickavet, et al., Worldwide energy needs for ICT: the rise of poweraware networking, in: ANTS

129 Future estimations (BusinessAsUsual scenario) Source: Ghent University - IBBT Conclusions: 1/7 th of electricity goes to ICT use phase in

130 Network Power Consumption Typical Operator 100% Total: TWh Typical incumbent operator Only network side, not home Source: T-Systems 80% 60% 40% 20% 0% Mobile Fixed Line User Premises Operator Source: Nokia-Siemens 4

131 Transmission Energy Efficiency Power (W / (Gb/s) ) Access Backbone, Single Wavelength Multi Wavelength Source: T-Systems Energy efficiency : aggregation level and physical medium: Optically transparent WDM technology most efficient Backbone equipment more efficient than access equipment 5

132 Access and Core - Comparison & Evolution Source: R. Tucker & al

133 Routers 100 Power consumption (kw) 10 1 Juniper: J-series E-series M-series T-series Cisco: CRS-1 CRS Source: Juniper & Cisco datasheets Throughput (Gbps)

134 Trends core networks Source: Willem Vereecken et.al., Power Consumption in Telecommunication Networks: Overview and Reduction Strategies, Communication Magazine 2011,

135 Trends access networks Source: Willem Vereecken et.al., Power Consumption in Telecommunication Networks: Overview and Reduction Strategies, Communication Magazine 2011

136 Data centers computational efficiency Koomey s law Computational energy-efficiency doubles every 1.6 years (=CAGR 55%) Aligns well with our observations (CAGR 43%) Koomey, et.al., Implications of Historical Trends in The Electrical Efficiency of Computing. IEEE Annals of the History of Computing, Vol 33-3, March 2011

137 Data centers total power consumption Total power consumption growth slowing down: : +100% : +56% Worldwide: #servers and Total Power CAGR: 15% total power CAGR: 9.5% total power projection Financial crisis Virtualization More focus on energy-efficiency # servers CAGR: 15% # servers CAGR: 6% Based on: Koomey, Jonathan. August Growth in Data center electricity use 2005 to Oakland, CA: Analytics Press. <

138 Green ICT Requires two high-level approaches Reduce energy Use clean energy

139 Content Introduction Approach 1: Improve energy-efficiency Example: Thin client in cloud Resilience in the cloud: server relocation Approach 2: Use clean energy

140 Thin Clients Instead of on individual computers, the tasks are executed in a server farm (cloud). Network is used to transmit I/O signals Advantages: Low power usage at user premises More power management in server farm Improved life cycle Disadvantages: Overhead of thin client protocol More elements that consume power 14

141 Thin Clients

142 Thin Client Power Consumption Power consumption in thin client scenario is clearly advantageous. P(W) Power management of servers is key to handle idle users and variable computation load. 0 Desktop PC Thin Client (ADSL2) Thin Client (VDSL2) Thin Client (PON) PC/terminal Base Power Server Base Power Network (local) CPU load (local) CPU load (server) Network (other) Source: Willem Vereecken et.al., Power Efficiency of Thin Clients, European trans on telecommunications

143 Thin Client Passive Users Power Saving 100% 80% 60% 40% 20% 0% -20% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% -40% -60% -80% -100% Passive User Fraction No shutdown Idle servers shutdown Idle servers & ONU shutdown

144 Resilience in the cloud anycast routing JOB? C. Develder, et al., "Survivable optical grid dimensioning [...]", ICC 2011, Kyoto, Japan

145 Resilience in the cloud Exploiting relocation Dimension network so that it is resilient against failures Exploit anycast principle: allow rerouting jobs to other destinations primary jobs secondary C. Develder, et al., "Survivable optical grid dimensioning [...]", ICC 2011, Kyoto, Japan

146 Results: Network dimensions 1,600 Number of wavelengths 1,400 1,200 1, Number of connections Total, Reloc., 1LS Total, No Reloc., 1L Total, Reloc., 1L Exploiting relocation: Saves 17% of wavelengths (avg) Adding server site resilience: Adds 26% (avg) C. Develder, et al., "Survivable optical grid dimensioning [...]", ICC 2011, Kyoto, Japan

147 Results: Server dimensions 600 Number of servers Total, Reloc., 1LS Total, No Reloc., 1L Total, Reloc., 1L Number of connections Exploiting relocation: adds 11% of servers (avg) Adding server site resilience: adds 55% (avg) C. Develder, et al., "Survivable optical grid dimensioning [...]", ICC 2011, Kyoto, Japan

148 Content Introduction Approach 1: Improve energy-efficiency Approach 2: Use clean energy Why needed? (increasing absolute growth) Availability of renewable energy (wind, in this case) Google chiller-less data centers LCA study: Follow the wind/follow the sun

149 Energy-efficiency, right metric? For COST For ENVIRONMENT (CO2 emissions)? Worldwide electricity always increasing despite more efficient laptops, fridges, lightning, aircos,

150 So wee need renewable energy

151 MW Renewable energy - availability Ireland: 1 year of wind output (all wind turbines) 5 days no wind D. McKay, Sustainable Energy without the hot air, UIT Cambridge, 2008,

152 Renewable energy - availability Solve intermittency with pumped storage? Energy Power 720 GWh (1 day) 33 GW (Britain s baseload) 30 GWh 2.8 GW (Britain s pumped Storage capacity) North Wales pumped storage reservoir On in 12 seconds! D. McKay, Sustainable Energy without the hot air, UIT Cambridge, 2008,

153 Google handling failures $$

154 Google handling failures 2003 Google Distributed Filesystem (GFS) cheap server: fails easily Ghemawat, et al, Google File System, 19th ACM Symposium on Operating Systems Principles, 2003.

155 Google Google has begun operating a data center in Belgium that has no chillers [ ] Rather than using chillers part-time, the company has eliminated them entirely So what happens if the weather gets hot? On those days, Google says it will turn off equipment as needed in Belgium and shift computing load to other data centers. Google s chiller-less data center,

156 Data centers using green energy Distributed data centers offer alternative way to cope with intermittency

157 Data centers are already distributed Google data centers (2008) Source map:

158 Follow the wind / follow the sun (FTWFTS) job Core network

159 Follow the wind / follow the sun (2) FTWFTS addresses a number of issues: Deal with intermitent availability of renewable energy Locate data centers near renewable energy sites Reduce carbon emissions Greenstar Network project (Canadian) testbed for FTWFTS

160 Model Total carbon footprint F = F manufacturing + F usage Four main parameters: Emission intensities (= carbon per kwh) of regular electricity I H green electricity I L Manufacturing fraction f = CO 2 manuf. / CO 2 usage Green energy availability p p = 50% 1-p = 50% Van Heddeghem, W. et al., Distributed computing for carbon footprint reduction by exploiting low-footprint energy availability, FGCS Vol 28, issue 2, Febr 2012, / j.future

161 Emission intensities (carbon / kwh) source: M. Jacobson, Review of solutions to global warming, air pollution, and energy security, Energy & Environmental Science 2 (2009) IEA, CO2 Emissions from Fuel Combustion - highlights, I L = 10 g CO2/kWh I H = 500 g CO2/kWh

162 Manufacturing fraction f = CO 2 for manufacturing CO 2 in use phase Source Description Manufacturing phase PE International Simple office server 500 kg CO 2 -eq/ unit Malmodin ITU PC 400 kg CO 2 -eq/ unit Malmodin ITU Server 500 kg CO 2 -eq/ unit Malmodin, Moberg Data Centers 10 Mton CO 2 -eq in 2007 Use phase 1030 kg CO 2 -eq kg CO 2 -eq kg CO 2 -eq Mton CO 2 -eq in 2007 f 0.09 We assume f = 1/8 = 0.125

163 Underutilized data centers Nominal load in data centers typically lower than maximum capacity (reasons: redundancy, provision for peak load, future expansion, ) what if we would redistribute to sites where green energy is available? Source: L. A. Barroso and U. Hölzle, The Case for Energy-Proportional Computing, IEEE Computer, vol. 40, no. December, pp , 2007.

164 Underutilized data center p = 60% Savings over usage phase only 1-p = 40% 3/5 load: regular 3/5 load: FTWFTS f =1/8 Van Heddeghem, W. et al., Distributed computing for carbon footprint reduction by exploiting low-footprint energy availability, FGCS Vol 28, issue 2, Febr 2012, doi: / j.future

165 Free lunch? Just add sites? p = 50% 1-p = 50% m = 5 n = 3 m total number of datacenters n datacenters required for daily operation

166 Model - Idea 3 data centers 4 data centers Electrical Energy CO 2 add 1 data center Electrical Energy CO 2 Use phase Manufacturing Total CO 2 during lifetime CO 2 savings

167 Results different manuf. fraction f = 0 f = 1/8 Likely in between f = 1/4 f = 1/2 f = 1/3 Van Heddeghem, W. et al., Distributed computing for carbon footprint reduction by exploiting low-footprint energy availability, FGCS Vol 28, issue 2, Febr 2012, / j.future

168 Conclusion Resilience Green ICT Resilience and green ICT can go hand in hand

169 This page intentionally left blank This page intentionally left blank This page intentionally left blank

170 CANARIE GREENSTAR PROJECT Green Sustainable Cloud & IT Service Network Mohamed Cheriet, GSN Project Lead École de technologie supérieure, Montréal, Québec, Canada Laboratory for Multimedia Communication in Telepresence

171 ICT Energy & Environment Climate Change is not reversible Urgent need to develop low carbon solutions ICT is a major consumer of power (8% in the US) and CO 2 production which is growing at 6% per year The problem requires not only Energy Efficiency BUT also GHG Reduction using Renewable resources The Goal is to provide ICT Services with renewable energies ICT sector can be a leader in GHG reduction 2

172 ICT Energy & Environment The current Internet 3

173 ICT Energy & Environment Example of Data Centers Bejing Olympic Data Center, fs, 80 MVA Lakeside Technology Center, Chicago fs, 100 MVA Next Generation DCT Newport, GB fs, 90 MVA 4

174 ICT Energy & Environment Challenges - Solutions What is the number one challenge your data center faces today? * What do Canada & Quebec offer? Required temperature 40 for data centers Quebec City, 30 Quebec Sacramento California Washington, DC Paris, France Beijing, China Mumbai, India Sydney, Australia Required temperature for data centers: 20 o C to 22 o C, or even lower (15 o C) *Source: IDC Inc. 5 5

175 The GreenStar Network Solution How to keep IT service green? Energy efficiency techniques Data centers powered by renewable energy sources (e.g. hydroelectricity, solar, wind) Separate infrastructure ownership and maintenance from usage All-optical Core network IT service is migrated around nodes when power dwindles Virtualization and Cloud Computing must be implemented Scalable & Flexible management for intermittent energy sources 6

176 The GreenStar Network Solution World s First Zero Carbon Internet & Cloud GSN Green IT Pilot Goals: (i) Distributed Cloud Software (ii) Follow the Sun/Wind Energy (iii) Carbon protocol, GeoChronos relocation Compute Resource 1 M Compute Resource n M Network Resource 1 M Compute Cluster Infrastructure Resources Network Equipment Network Resource n M Sensor Resource 1 M Infrastructure Metrics Aggregation Sensor Resource n M Decision Engine Instruments, Sensors Metrics (M) Data Centre n GHG Measurement Engine (ISO 14064) GreenStar Network Central Hub Hydro-powered node Spokes Sun or Wind powered nodes 7

177 The GreenStar Network Map

178 The GreenStar Network Transcoding and Video Streaming service provision based on Follow the Sun and the Wind Video Streaming VM VM DLNA Server Transcoding Server 9

179 GV- CANARIE AGM Live demonstration Simulation on global scale Simulation.mp4 10

180 Carbon Protocol How Real Reductions will be Achieved 11

181 Protocol-related Accomplishments Finished ISO based Protocol Draft in Preparation for TAG meeting (held in June 2011) Continuation with GHG Project report for GeoChronos Relocation Development of proposal for online documentation tool

182 Online Documentation Tool The online documentation and calculator will be able to be used for the following purposes: For documenting low or zero carbon ICT projects and creating project reports that project proponents could try to third party verify For the quantification and reporting of low or zero carbon ICT initiatives within corporate sustainability reports The tool will be based on the ICT GHG Reduction Project Protocol

183 Academia The GreenStar Network Partners Not-for-Profit Corporations Industry Government ideal inc. SIGMACO International Partners USA Belgium Ireland Spain China 14

184 Thank you! Contact : Prof. Mohamed Cheriet, Eng., Ph.D., SMIEEE Director / Synchromedia Consortium Website: Synchromedia