Efficient Router Architecture, Design and Performance Exploration for Many-core Hybrid Photonic Network-on-Chip (2D-PHENIC)

Transcription

1 Efficient Router Architecture, Design and Performance Exploration for Many-core Hybrid Photonic Network-on-Chip (2D-PHENIC) Achraf Ben Ahmed, Michael Meyer, Yuichi Okuyama, Abderazek Ben Abdallah Graduate School of Computer Science and Engineering University of Aizu, Aizu-Wakamatsu , Japan [d , d , okuyama, Abstract Nowadays, increasing emerging application complexity and improvement in process technology have enabled the design of many-core processors with tens to hundreds of cores on a single chip. Photonic Network-on-Chips (PNoCs) have recently been proposed as an alternative approach with high performance-per-watt characteristics for intra-chip communication. While providing large bandwidth through WDM (Wavelength Division Multiplexing), the main design challenge of conventional hybrid PNoC lies in the control layer, which is generally used for path set-up and also for short message communication. In this paper, we propose architecture and design of an efficient router for control and communication in heterogeneous Many-core Hybrid Photonic Network-on-Chip (2D-PHENIC) 1. In addition, we present detailed complexity and performance evaluation of the proposed architecture. Keywords-Hybrid, Optical Network-on-Chip; Many-core Systems-on-Chip; Architecture; Path Setting. I. Introduction Photonic Network-on-Chip (PNoC) [1], [2] is a novel concept enabling high bandwidth especially when combined with Wavelength Division Multiplexing (WDM) to concurrently transfer multiple parallel optical streams of data through a single waveguide. This contrasts with the Electronic Network-on-Chip (ENoC) [3] that requires a unique metal wire per bit stream. The key power saving comes from the fact that once a photonic path is established, the optical data is transmitted in an end-to-end fashion without the need for buffering, repeating, or regenerating. In fully optical architectures, a significant portion of the used spectrum is dedicated to perform the necessary arbitration and control functions instead of using it for the data transfer itself. In this fashion, the benefits of photonic properties are not fully exploited for fast data transfer. This has huge impact on lowering the bandwidth of PNoC systems and the need to optimize this bandwidth has become imperative. In order to solve the above issue, the concept of using an electronic layer to set the required path and an optical layer to establish a high throughput end-to-end communication was proposed. 1 PHENIC project is supported by University of Aizu Competitive Research Funding, Ref. P Despite the huge bandwidth that we can get from the hybrid architecture, the Electronic Control Network (ECN) is considered as the main source of latency and power consumption, as discussed in the next section. This overhead might be caused by the use of an inappropriate message size, a non optimized physical channel width, or especially the used path setup protocol which is a source of both power and latency overhead. We previously proposed PHENIC [1] system, which is a hybrid PNoC architecture. We showed preliminary evaluation results without taking into consideration the previously mentioned problems. In PHENIC, the electronic router was based on our previous 2D-OASIS router [3]. In this work, we present an efficient router architecture and design to be used in 2D-PHENIC, which aims to reduce the area and power overhead caused in the ECN. We also propose a complete architecture of 2D-PHENIC system, for both Electronic Control Network (ECN) and Photonic Communication Network (PCN). To find out the optimal configuration of the proposed router, we first conduct a performance exploration based on the message size variation to see its impact on the overall performance. Second, we define the optimal packet size that should be carried in the ECN and we define a selection policy to choose if the message/packet is transmitted electronically or optically. Finally, we show the design and synthesis results of the electronic router using Cadence and Synopsis CAD tools. II. 2D-PHENIC System Architecture The system consists of two types of networks. The upper one is the Photonic Communication Network (PCN) and is based on silicon broadband photonic switches interconnected by waveguides. The bottom layer is called Electronic Control Network (ECN) and is used for path reservation and configuration of the optical switches at the PCN layer by mainly powering ON/OFF the microrings resonators (MRs). Each IP core is connected to a local electrical router and also connected to the corresponding gateway (modulator/detector) in PCN. Complete details of the involved components in 2D- PHENIC system architecture are given in [1].

2 Figure 1: 2D-PHENIC s Router Architecture. A. Electronic Control Network (ECN) The ECN is based on Mesh topology. The packets are forwarded among the network using Wormhole-like switching policy and then routed according to Dimension-Ordered- Routing (DOR-XY). As a flow control, ECN adopts Stall-Go mechanism and Matrix-Arbiter as a scheduling technique. The ECN s packet format. does not contain a payload since all data will be transmitted in optical manner. Thus, all fields are for control purposes and all packets generated in the ECN are Path Setup Control Packet (PSCP). We set the maximum size of the optical message to 16Kb according to the work in [8]. The router is considered as the back-bone element in the whole ECN design. The ECN s router architecture is based upon OASIS NoC router (ONoC Router) [3], [9] with the addition of the optical switches arbitration module. Figure 1 illustrates ECN s router architecture. The routing process at each router can be defined by three main pipeline stages: Buffer writing (BW), Routing Calculation and Switch Allocation (RC/SA), and finally the Crossbar Traversal stage (CT). B. Photonic Communication Network (PCN) The PCN is based on a broadband photonic switches connected in mesh topology. 2D-PHENIC s optical switch is based on multiple 1 2 and 2 2 optical switching elements. These switching elements are based on micro-ring resonators. The functionality of the whole switch depends on the state of these micro-ring resonators, which are set by electrical signals coming from the ECN layer according to the routing requirements of each packet. In fact, when the next-port is granted and the PSCP can be forwarded to the next node, the arbiter module sets the required micro-ring resonators in parallel according to a state table where all micro-ring resonators states are stored (0 for OFF and 1 for ON). Thus, the corresponding photonic switch resources are also reserved (input and output port). When the PSCP reaches the destination, an ACK signal is issued from the destination to the source informing it to start the data transmission. After the data transmission is done, a tear-down signal is generated, where at each hop the arbiter will release the previously reserved resources in both electronic router and photonic switch. For the time being, we are using a 5x5 photonic bloking switch [10], based on a 4x4 non-blocking one presented by [11]. Figure 2 shows the block diagram of the used 5x5 photonic switch. Up to now, the 2D- PHENIC s ENC layer supports only PSCP packet and all data are transmitted optically regardless of the message size. In the next section, we present a performance exploration of 2D-PHENIC and we focus on the message size impact to define the optimal size to be used in the ECN and PCN. Figure 2: 5x5 Photonic switch block diagram. III. Performance Exploration of 2D-PHENIC In a typical computer system communication, we can find a wide range of message sizes. This range

3 starts from the small messages, which are usually control messages for memory, cache coherence protocols or barriers synchronization. From few bytes, for a control message (such as a memory read requests or cache coherency snoop messages) to a few kilobytes for a data packet such as the exchange of memory pages or long cache lines. In conventional ENoC, all messages are transported electronically by dividing the message into small flits according to the channel width. However, in 2D-PHENIC we have to choose whether to use the electronic layer or the photonic layer to transmit the data. Before we discuss the selection policy that we will adopt (if the message is transmitted optically or electronically), let us first evaluate the performance of a pure Electronic NoC (ENoC) [3] and 2D-PHENIC for different message sizes. Figures 3 and 4, respectively, show the average latency and the average bandwidth before the network saturation for 256 cores under random traffic for different message sizes (from 32 bytes to 16 kilobytes). As can be seen in Fig. 3 (a) up to a message size of 512 bytes ENoC outperforms the hybrid one. Starting from 1kb the latency for the ENoC keeps increasing linearly with the message size reaching 13 µs for 16Kb message size, while for 2D-PHENIC the latency stagnates between 1 to 1.5 µs for all message sizes. Figure 4: Achieved bandwidth before the saturation under random traffic for 256 cores and different message sizes. Figure 3 (b) shows the latency breakdown before the network saturation for 256 cores under random traffic for different message sizes starting from one kilobyte to 16 kilobytes. As can be seen, the setup latency dominates the overall latency. This is explained by the fact that once the path is set in the electronic layer the optical transmission is independent from the distance and the data stream transported. Thus, there is a need to optimize the latency in the Electronic Control Network (ECN) and to minimize the overhead of the path setup process as much as possible to increase the overall performance. Figure 4 shows that for less than 1Kb message, the bandwidth is basically the same for both ENoC and 2D- PHENIC with a slight advantage for ENoC. From 1Kb to 16Kb message size, the bandwidth of ENoC increases only by 17%, while it increases by 500% for 2D-PHENIC for the same range. The last part of the evaluation in this section will be dedicated to the power consumption in 2D-PHENIC. Figure 5 (a) shows the power consumption evaluation for 2D-PHENIC and ENoC for different network sizes and different synthetic benchmarks. The three benchmarks are Random, Bitreverse and Neighbor. Random is a communication pattern where the destinations are randomly picked uniformly each time a new communication occurs. Bitreverse is a communication pattern where each node takes the ones complement of its ID, resulting in very long communications. Neighbor traffic pattern is where communication is limited to only one hop. In addition to be more power-hungry than the 2D-PHENIC, the ENoC is sensitive to the communication distance. In fact, as the network size increases, the difference between the power consumption resulting from the three different communication patterns increases. We can see that for 16x16 network size, the highest power consumption is for the Bitreverse. On the other hand, 2D-PHENIC is insensitive to the traffic pattern, where for the three network sizes the performances of the three traffic patterns are the same. For example, for 16x16 network size the Bitreverse traffic pattern can generate very long communications but it has the same power consumption as Neighbor traffic pattern which generates only one hop distance. Despite this good feature that comes with 2D- PHENIC, the power consumption is still increasing with the network size. This increase can be explained in Fig. 5 (b) which shows the power consumption breakdown before the saturation under random traffic for 256 cores and different message sizes. As can be seen, most of the consumed power is taken place in the ECN and the portion of the power generated by the PCN is only 7% for 1Kb message size and around 22% for 16Kb message size. Unlike optical transmission, the power consumption generated by an electronic transmission depends on the bit stream and the distance generated by the traffic pattern. Although the ECN just carries a small path setup packet, it is still one of the main drawbacks of the 2D-PHENIC system and it needs to be optimized further to reduce the latency and the power consumption. In the next section, we present the design details and evaluation results of our proposed electronic router for 2D- PHENIC and the selection policy that we will adopt.

4 (a) Figure 3: Average latency before the saturation under random traffic for 256 cores and different message sizes, (a) total average latency, (b) latency breakdown. (b) (a) Figure 5: Power consumption before the saturation for : (a) different network sizes and different benchmarks (b) random traffic for 256 cores and different message sizes. (b) A. Selection policy IV. Electronic Router for 2D-PHENIC According to the results shown in the previous section and the behavior of real CMP systems under realistic applications, we add a second type of small packets to be carried in the ECN in addition to the PSCP. We named this packet, Small Control Packet (SCP) with 74 bits length. We set the payload to 64 bits in addition to the source and destination addresses, and the tail. We choose 64 bits as payload according to the payload of the control packet in MOESI protocol. For the optical transmission, it is set from 1 Kb to 16 Kb. We set the maximum message size to 16Kb according to the work in [8]. B. Evaluation results We propose the design and evaluation results of an electronic control router to be used for 2D-PHENIC. Since all packets carried on the ECN are still relatively small, we set the channel width (i.e, flit size) equal to the largest packet, which is the SPC. We aim to not divide any of the packets carried in the ECN. We evaluate the 2D-PHENIC s router power consumption and area before and after the packet size and channel width optimizations. Table I illustrates the hardware complexity results of 2D- PHENIC s router in terms of area and total power when compared to the baseline router. Decreasing the channel width from 128 bits to 74 bits will lead to a decrease by 48% in the power consumption and 56% in the area. This can be explained by the fact that decreasing the channel width also reduces buffer utilization at each input-port as they are the most area consuming components in an ENoC system. The area overhead has a direct impact on the power consumption explaining the power reduction obtained when reducing the channel width.

5 Table I: 2D-PHENIC s router hardware complexity evaluation results before and after optimization. System / Parameter Area Total Power µm µw Baseline (Before optimization) Proposed (After optimization) Figure 6 shows the layout of 2D-PHENIC s electronic router obtained by Cadence SoC-Encounter at 45nm CMOS process technology [12]. Tokyo Japan, in collaboration with Synopsys, Inc. and Cadence Design Systems, Inc.. References [1] A. Ben Ahmed, A. Ben Abdallah, PHENIC: Towards Photonic 3D-Network-on-Chip Architecture for Highthroughput Many-core Systems-on-Chip, IEEE Proceedings of the 14th International conference on Sciences and Techniques of Automatic control and computer engineering, pp. 1-9, Dec [2] Y. Ye et al., 3-D Mesh-Based Optical Network-on-Chip for Multiprocessor System-on-Chip, Computer-Aided Design of Integrated Circuits and Systems, IEEE Transactions on, vol.32, no.4, pp , April [3] K. Mori, A. Esch, A. Ben Abdallah, K. Kuroda, Advanced Design Issues for OASIS Network-on-Chip Architecture, IEEE Proc. of the 5th BWCCA-2010, Nov. 2010, pp [4] Thomas Moscibroda, Onur Mutlu, A case for bufferless routing in on-chip networks, Proceedings of the 36th annual international symposium on Computer architecture, June 20-24, 2009, Austin, TX, USA Figure 6: PHENIC s electronic router layout. V. Conclusion and Future Work In this work, we proposed an efficient router architecture and design targeted for our 2D-PHENIC Hybrid Photonic Network-on-Chip. We provided information about the complete 2D-PHENIC system architecture including both Electronic Control Network (ECN) and Photonic Communication Network (PCN). We also conducted a performance exploration by studying the impact of the message size on the performance of the proposed router to find out its optimal configuration. The study shows that for less than 1Kb message size the pure electronic NoC outperforms 2D- PHENIC system. In addition, we observed that most of the consumed power (between 78% and 95%) is taken place in the electronic network. According to these results, we presented the synthesis results of the proposed electronic router which supports both path-setup and small control packets. The results show that the proposed router can reduce the power by 48% and the area by 56%. In our future work, we intend to focus on the path setup protocol to minimize its latency overhead which is the main source of latency in hybrid PNoC architectures. Acknowledgment This project is partially supported by UoA Competitive Research Funding, It is also supported by VLSI Design and Education Center (VDEC), the University of [5] A. Kahng, B. Li, L.-S. Peh, and K. Samadi, ORION 2.0: A power-area simulator for interconnection networks, IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 20, no. 1, pp , jan [6] M. Hayenga, N. E. Jerger, and M. Lipasti, SCARAB: a single cycle adaptive routing and bufferless network, in Proceedings of the 42nd International Symposium on Microarchitecture, [7] C. Fallin, X. Yu, G. Nazario, and O. Mutlu, A highperformance hierarchical ring on-chip interconnect with low-cost routers, Computer Architecture Lab (CALCM), Carnegie Mellon University, Tech. Rep., [8] A.Shacham, K.Bergman, L.Carloni, Photonic Networkson-Chip for Future Generations of Chip Multiprocessors, IEEE Transactions on Computers, vo1.57, n.9, pp , Sep [9] A. Ben Ahmed and A. Ben Abdallah, Architecture and Design of High-throughput, Low-latency, and Fault-Tolerant Routing Algorithm for 3D-Network-on-Chip (3D-NoC), The Journal of Supercomputing, Vol. 66-3, pp , Dec [10] J. Chan, G.Hendry, A. Biberman, K. Bergman, L. Carloni, PhoenixSim: A Simulator for Physical-Layer Analysis of Chip-Scale Photonic Interconnection Networks, DATE 2010,paper [11] H. Wang, B. G. Lee, A. Shacham, K. Bergman, On the Design of a 4x4 Nonblocking Nanophotonic Switch for Photonic Networks on Chip, Frontiers in Nanophotonics and Plasmonics, Guaruja, SP Brazil (Nov 2007). [12] Nangate 45nm open cell library,