XtratuM integration on bespoke TTNoCbased. Assessment of the HW/SW.

Transcription

1 XtratuM integration on bespoke TTNoCbased HW. Assessment of the HW/SW. Deliverable Project acronym : MultiPARTES Project Number: Version: v1.1 Due date of deliverable: September 2014 Submission date: 19/11/2014 Dissemination level: Public Author: FENT, TUW, IKERLAN, UPV Part of the Seventh Framework Programme Funded by the EC - DG INFSO

2 Table of Contents 1 DOCUMENT HISTORY LIST OF ACRONYMS EXECUTIVE SUMMARY INTRODUCTION PURPOSE OF THE DOCUMENT OUTPUT OF THE WORK STRUCTURE OF THE DOCUMENT ASSESSMENT OF TEMPORAL AND SPATIAL ISOLATION ASSESSMENT OF THE TTNOC-BASED HARDWARE Spatial Isolation Temporal Isolation Fault/error containment Security Heterogeneity ASSESSMENT OF XTRATUM Spatial Isolation Temporal Isolation Task synchronization ASSESSMENT OUTLINE BESPOKE TTNOC-BASED HW EVALUATION INTRODUCTION SYSTEM ARCHITECTURE FUNCTIONAL EVALUATION PERFORMANCE EVALUATION TEMPORAL INTERFERENCE ANALYSIS EVALUATION RESULTS INTRODUCTION FUNCTIONAL TEST RESULTS PERFORMANCE EVALUATION Partition Context Switch (PCS) Effective slot time of partition execution Partition overhead Overhead due to number of partitions Overhead due to the number of slots in the plan Stability of the plan Inter-partition communications Service costs /09/2013 IST

3 7.3.9 TTNoC services costs DSMC services costs Inter-partition-core communication message TEMPORAL INTERFERENCE ANALYSIS COMPARISON WITH RESPECT TO COTS SOLUTION TEMPORAL INTERFERENCE Computation time CONCLUSION REFERENCES /09/2013 IST

4 1 Document History Version Status Date V0.1 Initial draft 13/09/2014 V0.2 Update with TUWIEN contribution 29/09/2014 V0.3 Update contents 20/10/2014 V0.4 Major Internal review 02/11/2014 V1.0 First release 13/11/2014 V1.1 Ready to be delivered 19/11/2014 Approval Name Date Prepared UPV 13 September 2014 Updated TUWIEN 29 September 2014 Updated fentiss 18 October 2014 Updated IKERLAN 20 October 2014 Reviewed All Project Partners 18 November 2014 Authorised Salvador Trujillo 19 November 2014 Circulation Recipient Date of submission Project partners 20/10/2014 European Commission 19/11/ /09/2013 IST

5 2 List of Acronyms AMP COTS DSM DSMC FPGA IPC MMS MPSoC MPT NoC PCS SMP TISS TTNoC Asymmetric Multiprocessing Commercial off-the-shelf Distributed Shared Memory Deterministic Shared Memory Controller Field-Programmable Gate Array Inter-Process Communication Multiple Module Schedule Multiprocessor System-on-Chip MultiPARTES Project Network-on-Chip Partition Context Switch Symmetric multiprocessing Trusted Interface Subsystem Time Triggered Network on Chip 01/09/2013 IST

6 3 Executive Summary In addition to the initial dual-core hardware platform for LEON3 evaluated in the WP4 (referenced in this document as COTS platform), WP6 has consolidated two platforms that the goal was to experiment with hardware solutions in order to avoid the temporal interference of cores when partitions run in parallel. This document provides the assessment for the functional and performance evaluation report of the multicore platform virtualization layer developed in the frame of work package 6 based on TTNOC. Deliverables D6.5.1 ( XtratuM integration on bespoke TTNoC-based HW. Hardware design and implementation ) detail the hardware modifications performed in order to avoid the temporal interference and D6.5.2 ( XtratuM integration on bespoke TTNoC-based HW. Virtualization layer design and implementation. ) explain the modifications at virtualization layer required to adapt it for this hardware architecture. The main goal of this document is to analyse, evaluate and assess the hardware and software with respect to the functionalities achieved in the initial hardware platform and provide specific test evaluation of the expected improvements with respect to the temporal interference. The evaluation is based in the test suites used and detailed in D4.5 adapted to new execution platform. Additionally, a set of new tests has been defined to evaluate specific aspects of the platform. 01/09/2013 IST

7 4 Introduction 4.1 Purpose of the document This document is one of the output of task 6.4 activities, Xtratum integration on bespoke TTNoC-based HW. The goal of this document is to present the analysis, performance evaluation and assessment of the results of MultiPARTES multicore platform virtualization layer. The general dependence of the task in the new reconfiguration of WP6 and the deliverables is shown on Figure 1. Figure 1 MultiPARTES work package and task dependences 4.2 Output of the work The main result of this document is the assessment with respect to the system properties, in special with respect the temporal interference and the analysis of the system performance of the integration of the MultiPARTES virtualization layer on bespoke TTNoC-based HW. One important aspect to be pointed out is the focus on the SPARC-V8 platform that has been considered in the activities carried out in WP6. x86 platform is out of scope of this analysis. 01/09/2013 IST

8 The output of this work will be used in order to investigate future upgrades of the MultiPARTES platform and virtualization layer with respect to system performance Structure of the document This document is organised as follows: Chapter 4 offers the summary and structure of this document. Chapter 5 provides the assessment of the system properties considering as system the hardware and the virtualization layer Chapter 6 defines the metric to evaluate the platform and the set of tests adapted or designed to evaluate it. Chapter 7 provides the evaluation results. Chapter 8 performs the comparison of the results obtained with respect to the COTS SPARC. Chapter 9 concludes the results of this deliverable. 01/09/2013 IST

9 5 Assessment of Temporal and Spatial Isolation The following assessment provides an overview of the system properties required for mixedcriticality integration. We evaluate the platform according to its capabilities to provide temporal isolation, spatial isolation, and implications on fault containment and security. In this chapter we analyse the platform on an analytic basis and evaluate an influence of these critical points on system properties. In further chapters an experimental evaluation will be performed, that verifies the theoretical assumptions regarding system properties, as well as performance aspects of the system. In the analytical assessment of the platform we will evaluate both hardware and software individually and their integration at the platform level. This platform introduces several radical changes in hardware. The assessment of the COTS based platform showed inadequacies for safety-critical applications with hard real-time requirements. As a solution for this problem two platforms were proposed which provide a higher level of temporal isolation and determinism. This document provides a description of the platform based on a TTNoC and asymmetric processor (AMP) configuration. In this section we assess the extent of advantages introduced by this platform in regard to mixed-criticality systems. In this project we defined key properties which should be satisfied such that mixed-criticality applications can be hosted: Spatial isolation: The separation of functional units in hardware or software, such that no unit is allowed to access private resources of any other unit. Temporal isolation: The execution timing of a functional unit must be uninfluenced by execution of any other. Fault tolerance / error containment: The ability of a system to tolerate faults and deny error propagation from one subsystem to the rest. Security: The platform design supports security properties and provides a secure environment for mixed-criticality integration. The analysis of the platform will concentrate on these key points and try to identify critical points which will then be investigated further using experimental evaluation. The resources considered are: memory, communication channels, and IO devices. 5.1 Assessment of the TTNoC-based Hardware The analysis of the platform based on COTS hardware components showed that it lacked full guarantees w.r.t. temporal isolation, which are required for hard real-time systems. On the other side the COTS-hardware platform (D3.4 Implementation of the generic virtualization layer) is able to guarantee complete spatial isolation in software using static resource management. The sharing of resources is avoided either by dividing them (i.e. memory, comm. channels) or strict reservation to a single partition (IO devices). The platform used a symmetric multiprocessor which allowed two processor cores to work in parallel. And while the spatial 01/09/2013 IST

10 interference access could be avoided the temporal concurrency couldn t. If the schedule of two tasks on two processors overlapping and they both require a specific resource during the overlap period one of them would be forced to wait. This could disrupt the scheduling plan from the point of view of effective execution time of partition, i.e. the use effective of CPU for partitions could be different to than expected in the assignation of resources. In order to avoid this problem, an evaluation of the execution predictability and worst case execution analysis must be performed. This information allows us more precise scheduling policies that are able to guarantee temporal isolation. The main obstacle for this task is complexity, which increases additionally with each software iteration. On top of this even if this task was plausible there could still be schedulability boundaries. The central part of the proposed TTNoC platform is a Time-Triggered Network on Chip (TTNoC). This on-chip communication concept has proven itself as a reliable and deterministic communication mechanism for interconnected on-chip systems. The concept has been used in the European project ACROSS with great success. The main properties of the TTNoC approach are: encapsulated communication channels, a trusted interface sub-system (TISS), a global time base, deterministic communication. These components allow deterministic and reliable on-chip communication, core decupling via generic interfaces and synchronisation. These properties directly support temporal and spatial isolation, and provide a strong platform for an implementation of mixed-criticality systems. The proposed hardware platform is based on an AMP LEON3 processor with two cores. Each core is equipped with a local working memory, which stores all transient data of the core. In the further text we will refer to the combination of the CPU and local memory as a host. The XtratuM hypervisor, designated to a specific core, has an exclusive control over this local memory. The segregation of partitions in local memory is controlled by the hypervisor. The temporal isolation of partitions is controlled by hypervisor s scheduler. Each core is controlled by a virtualization layer copy or instance following the Asymmetric Multiprocessing (AMP) approach. Both virtualization layer instances require to be synchronized and services for partition management and communications between partitions allocated in different cores. The proposed platform also provides a shared memory. It is an off-chip memory coupled with a special memory controller. The memory controller is a single core LEON3 processor with dedicated software for memory control. It is attached to the rest of the system via TTNoC. It allows a complete deterministic access to the shared memory to the cores. The communication with IO devices is performed using a special gateway component such as an IO-Server partition. The cores use TTNoC to access IO gateway. 01/09/2013 IST

11 5.1.1 Spatial Isolation Mixed-criticality integration requires a clear segregation among sub-systems. Each sub-system is allowed to access only resources which are designated to that specific sub-system. In the earlier work (see D3.5) we established that software based methods for the implementations of spatial isolation are quite sufficient. Using these methods an operating system can efficiently control access to resources such that complete spatial isolation is satisfied. The proposed architecture has two hosts. The local memory of each host is completely decupled from the rest of the system and only a local core is able to access this memory. This means the working memory of each core on MPSoC level is also isolated in hardware. The local memory is partitioned using software methods and regulated by XtratuM. In earlier evaluations XtratuM showed that it is capable of implementing complete spatial isolation among partitions (see D3.5). The TTNoC provides encapsulated communicated channels. Each channel occupies specific ports designated to specific hosts. If the architecture uses a static configuration method the channel configuration cannot be changed during runtime Temporal Isolation The deterministic time behaviour is crucial for systems with high-real time requirements. Systems based COTS hardware are almost exclusively designed to accommodate performance requirements. In the analysis of COTS hardware we identified several sources of temporal interferences. The proposed TTNoC architecture abandons the SMP approach, used in the original COTS based platform, and introduces a time-triggered communication between the cores. This eliminates an influence of the bus on the temporal behaviour of the system. The second significant change is related to the working memory. Each core has a local working memory now. By removing these two shared resources (memory and bus) the temporal interferences caused by waiting on access to the given resource are eliminated. The temporal separation is performed on the TTNoC level. There is no implicit synchronization among hosts, but one could be established if needed by using synchronisation signals that propagate over the TTNoC. These signals could be used as a task triggers and thus provide synchronized start of tasks. This is only possible if the overlaying software architecture supports these features. In Section 5.2 we will provide a further analysis of the temporal alignment between TTNoC and the XtratuM hypervisor. 01/09/2013 IST

12 5.1.3 Fault/error containment A system ability to contain and tolerate faults is a very important aspect of mixed-criticality integration. The fault is considered to be an anomaly in the correct and anticipated behaviour of the system. We distinguish two types of faults: design faults (also known as systematic faults) and random faults (physical faults). Design faults are anomalies introduced during the development process. They could be part of software or hardware (e.g. programming bugs, engineering miscalculation etc.). Random faults are caused either by an external event, or damage to the system not caused by design. Under random faults we understand single event upsets (SEU), manufacturing faults, or damage due to physical stress. Hardware is generally prone to both types of faults and we can observe containment regions according to type of the fault. The platform is implemented as a combination of several independent computational units: processor cores with local memory, TTNoC, and shared memory unit. The spatial and temporal isolation are ensured in hardware and by design they provide fault containment regions for systematic faults. Each unit is able to operate in complete isolation even if all other components fail. We assume that a random fault would affect all components regardless of the relation between them. The MPSoC in general is observed as a fault containment unit with regard to random faults Security In the earlier reports we evaluated native security of the hypervisor and underlying architecture. The security is evaluated by the capability of the system to establish and preserve security related properties (e.g., confidentiality, integrity, authenticity). Hardware architectures are rarely built with security tools implemented in hardware. Security is in most cases a responsibility of higher level developers. The hardware architecture must ensure that there are no design faults which could cause a violation of security properties. Separation of entities using different policies is a cornerstone for enforcing security. Spatial isolation is therefore a natural pre-requisite for any security architecture. The spatial isolation allows confidentiality of local data among hosts, and confidentiality of data transferred through encapsulated TTNoC channels. The security of the data communicated with the outside world must be enforced using additional methods, e.g. encryption. 01/09/2013 IST

13 A set of security properties depends highly from the application requirements. The platform supports security properties but they final implementation requires additional security measures. These must be applied either on the application level or using middleware security services Heterogeneity The proposed architecture enables an integration of heterogeneous computational units without breach of, above mentioned, basic system properties. The TTNoC is connected to processor cores via TISSes and they are fully decoupled from the rest of the system. Each TISS has a private memory area, where the messages and configuration data are stored. The configuration of the TISSes is static and is performed a priori. This principle allows an implementation of a deterministic gateway towards other types of processing cores without compatibility problems. The global time base allows both types of systems to stay phase aligned even on the application layer. (Salloum C., Elshuber, Hoftberger, Isakovic, & Wasicek, 2012) The heterogeneity in this case is not bounded on a specific system integration layer, and it can be applied on device level, as it is in this case. It can be also applied on chip level where two different cores are connected via TTNoC on a single chip. 5.2 Assessment of XtratuM In this section we evaluate the impact of the platform shift on basic properties of the virtualisation layer. Again, our focus will be on the properties needed for safe and reliable mixed-criticality integration, spatial and temporal isolation. In D3.5 we established that on COTS hardware the majority of tasks involved in ensuring spatial and temporal isolation, are responsibility of software. The OS layer is responsible for resource separation and execution of the tasks in correct temporal order. The COTS platform used in the original evaluation provided very low amount of support towards mixed-criticality integration. Despite this fact the hypervisor was able to provide full spatial isolation among different tasks and partial temporal isolation. The encountered boundaries were a direct consequence of the hardware architecture Spatial Isolation A separation of different applications is critical for mixed-criticality systems. It is also a foundation for other system properties e.g. fault tolerance and security. On the proposed 01/09/2013 IST

14 platform the CPU implementation changed from SMP to AMP. This means that basic computer resources are not shared among CPUs anymore. The task of the hypervisor in an SMP version in regard to spatial isolation was to establish a safe and secure way for tasks, partitions, and CPUs to manage resources. With an AMP approach the responsibility of the hypervisor on this matter reduces. The memory and other resources (e.g. bus, cache etc.) are not shared among CPUs anymore. Therefore the hypervisor must only manage resource separation among partitions. Previous functional tests (D3.5 Assemsment of XtratuM, 2013) (D4.5 Platform validation report, 2013) showed that XtratuM fully implements spatial isolation and we can conclude that the same properties hold for the adapted version. According to (D6.5.2 XtratuM integration on bespoke TTNoC-based HW. Virtualization layer design and implementation, 2014) no changes have been done such that these properties could be violated Temporal Isolation The second property crucial for mixed-criticality applications is the ability to isolate different tasks in time. An execution of each task cannot be violated by any external event within its execution window. In the SMP version of the MultiPARTES platform a time isolation was affected by due to interferences created by shared memory, shared cache, and system management interrupts (e.g., Intel x86). In the AMP version the temporal violations are possible only in case of under dimensioning of task execution windows (i.e., partitions). A task must be evaluated using worst case execution time (WCET) analysis, to establish execution time bounds and to be able to correctly schedule tasks in the system. These conflicts can be resolved a-prior without any risks. The task execution can also be pre-empted by the hypervisor itself. These interrupts can be measured and calculated in schedule. The effects of under dimensioned partitions have been explored in and earlier assessment (D3.5 Assemsment of XtratuM, 2013). We can apply the same conclusions here, eventual error in configuration and integration could cause temporal interferences Task synchronization The proposed platform provides deterministic means of communication among different cores, as well as a notion of global time between them. This allows tasks to coordinate their actions based on time. The proposed concept allows coordination of tasks over TTNoC using specialized synchronization signals. The signals are implemented either through TTNoC or via dedicated lines. The synchronization is performed on two levels of abstraction: hypervisor and user. 01/09/2013 IST

15 The hypervisor coordinates the start of the schedule with the rest of the system using dedicated signal. It allows schedules on different cores to be phase aligned. This functionality allows systems to reintegrate on reset or switch schedules if needed. The coordination of tasks on heterogeneous systems with independent clocks requires additional synchronization or phase alignment methods, and a deterministic communication system between two parties (Salloum C. E., Elshuber, Höftberger, Isakovic, & Wasicek, 2013). In principle the architecture allows phase alignment of tasks on heterogeneous processors. 5.3 Assessment Outline In the sections above we analysed the proposed platform with regard to temporal and spatial isolation, as basic properties for mixed-criticality systems. We also identified additional properties that are also closely related, and emerging effects on these properties. The hardware part of the platform allows execution of complete spatial and segregation of overlaying virtualisation layers. It provides means for the virtualization layers to coordinate their actions, and it is included in basic services of the platform. As an optional service there is a capability of a temporal coordination between user tasks. The platform provides a minimal local memory for each core, this way spatial isolation between cores is guaranteed in hardware. The shared memory is still accessible through the deterministic shared memory controller. This increases temporal and spatial isolation, and overall reliability and dependability. On the other side, it reduces the performance factor of memory accesses to shared memory. The matter of temporal isolation is also solved by dividing hardware cores in separate independent computational units. Of course there is still a risk of false scheduling and dimensioning of execution windows. This can be avoided by applying a sort of WCET analysis during the configuration of the system. The concept of the proposed architecture doesn t allow temporal interferences due to non-deterministic hardware. If the system possesses the above mentioned characteristics, it improves other properties. It increases fault and error containment of the system, and it provides certain security properties by design. The heterogeneous system can be built in a deterministic fashion using COTS hardware, and without major trade-offs. This improves overall compensability of a system. The weak spot of the platform is a performance decrease. This is a typical trade-off in mixedcriticality systems. A more deterministic hardware design with high-performance capabilities oriented for mixed-criticality is a high-rated future challenge. 01/09/2013 IST

16 6 Bespoke TTNoC-based HW Evaluation Deliverable v Introduction This section provides an overview of the metrics used, tests and results obtained for the evaluation of the XtratuM on top of the bespoke TTNoC-based HW developed in work package WP6. The goal of the evaluation is to analyse that the hardware platform fulfils the same functionalities and solves the problems raised in the COTS platform. The main problem raised was the temporal interference due the parallel execution of application code on different cores. In the following sections the performance metrics, benchmark scenarios and respective results are presented. 6.2 System architecture The hardware and software architecture has been described in D6.5.1 and D We summarise here the main aspects to be consider for the evaluation purposes. Figure 2 shows the hardware and software architecture. Each LEON3 core is handled by one instance of the virtualization layer (AMP software architecture) which supports the execution of a set of partitions. Each LEON3 core has a local memory and can access to the shared memory through the TTNoC bus and the Deterministic Shared Memory Controller (DSMC). Figure 2: HW and SW architecture For the evaluation purposes, the aspects to be considered are: - The processing mode of the virtualization layer is Asymmetric multiprocessing (AMP) which implies that each core is controlled by one instance of the virtualization layer. This implies: 01/09/2013 IST

17 o Partitions have to be monocore o Virtualization layer communication and synchronization is managed through the TTNoC o The synchronization between Virtualization layer instances is required for: system management services partition management involving partition allocated to different cores plan management synchronization - Shared memory is accessed through a DSMC (distributed shared memory controller) - System build, load and boot. 6.3 Functional evaluation In order to evaluate the functional behaviour of the hardware solution and virtualization layer adaptation, we have selected as reference test suite for the MPT Abstraction Layer: Conformance test suite defined in section 7 of D4.4. It defines the set of tests for System, Partition, Inter-Partition Communication (IPC), Time, Multiple Module Schedule (MMS), and Health Monitor. As the tracing is local and no changes to the tracing subsystem have been made, this set of tests have not been included in this new evaluation. However, the modifications of the hardware have required the modification of the virtualization layer and the addition of new hypercalls. These new services are considered in the functional evaluation. In order to fulfil the functionalities, a set of new requirements and tests have been included. New functionalities have been detailed in Deliverable D due to the H- AMP software architecture. System status and actions between hypervisor instances Partition status and actions between partitions allocated in different hypervisor instances. We will refer as different core. Plan synchronization between cyclic scheduling running executed by different hypervisor instances. Communication services between partitions allocated in different cores. Specific services for shared memory management New tests for these services have been designed and implemented. Table 1 summarizes the MPT Abstraction Layer test. Columns define: Test status: details which is the test status. o New test: the test has been defined due to the hardware modifications. o Adapted: the test has been adapted from a previous test due to architecture constraints o : test is executed without modification o Not Applicable NA : test can not be executed due to architecture constraints Test No.: defines a unique test identifier. Test Purpose: defines the goal of the test Comments: provides the constraints of the test. 01/09/2013 IST

18 o Same core: test applicability restricted to partitions allocated in the same hypervisor instance o Different core: test applicability restricted to partitions allocated in different hypervisor instance o New hardware: test has been defined due to the new hardware Table 1: MPT Abstraction Layer test definition Test Status Test No Test Purpose Comments System Management T-API-SYS-0110_0010 Successful system status check Adapted T-API-SYS-0110_0011 Successful system status check Different node T-API-SYS-0110_0020 T-API-SYS-0110_0030 Successful management of service being called by a non-system partition. Successful management of the service being called with an invalid SYSTEM_STATUS reference address. NA T-API-SYS-0120_0010 Successful system mode setting New T-API-SYS-0120_0011 Successful system mode setting T-API-SYS-0120_0020 Partition Management Successful management of service being called by a non-system partition. T-API-PM-0210_0010 Successful system partition status check Adapted T-API-PM-0210_0011 Successful system partition status check Different node Adapted Adapted Adapted T-API-PM-0210_0020 T-API-PM-0210_0030 T-API-PM-0220_0010 T-API-PM-0220_0011 T-API-PM-0220_0020 T-API-PM-0220_0021 T-API-PM-0220_0030 T-API-PM-0220_0031 T-API-PM-0220_0040 Successful handling of invalid input parameters error cases. Successful handling of the case when the calling partition is not a system partition. Successful setting of a partition mode to WARM_START from states NORMAL and WARM_START. Successful setting of a partition mode to WARM_START from states NORMAL and WARM_START. Successful setting of a partition mode from NORMAL to IDLE and back to NORMAL. Successful setting of a partition mode from NORMAL to IDLE and back to NORMAL. Successful setting of a partition mode from NORMAL to COLD_START and back to NORMAL. Successful setting of a partition mode from NORMAL to COLD_START and back to NORMAL. Successful handling of NORMAL to NORMAL mode transition request. Reset the system is supported by a programmable external device that generates a signal. Different node Different node Different node 01/09/2013 IST

19 T-API-PM-0220_0050 T-API-PM-0220_0060 T-API-PM-0230_0010 Inter-Partition Communication Successful handling of invalid values passed as service parameters. Successful handling of the error case when the calling partition is not a system partition. Successful idling of a partition until its next scheduling slot or interrupt reception. T-API-IPC-0310_0010 Successful creation of a queuing port. T-API-PM-0310_0020 Successful management of the service being called with wrong or invalid parameters values. T-API-PM-0310_0030 Successful management of the service being called when the calling partition is in NORMAL state. T-API-IPC-0320_0010 Sampling port creation. T-API-IPC-0320_0020 T-API-IPC-0320_0030 Sampling port creation (when called with invalid parameters). Sampling port creation (when called by partition in NORMAL state). T-API-IPC-0330_0010 Successful queuing port ID check. T-API-IPC-0330_0020 Successful management of the service being called when the passed port name does not exist. T-API-IPC-0340_0010 Successful queuing port status check. T-API-IPC-0340_0020 Queuing port status check failure because of invalid port id passed. T-API-IPC-0350_0010 Successful sampling port current status check. T-API-IPC-0350_0020 Successful management of service call with an invalid port Id. T-API-IPC-0360_0010 Successful sampling port ID check. T-API-IPC-0360_0020 Successful management of service call with an invalid port Name. T-API-IPC-0370_0010 Successful sampling port status check. T-API-IPC-0370_0020 Successful handling of service call with an invalid port Id T-API-IPC-0380_0010 Successful sampling message readout. T-API-IPC-0380_0020 Successful sampling message readout (message age>refreshperiod) T-API-IPC-0380_0030 Successful sampling message readout. T-API-IPC-0390_0010 Successful conditional sampling message readout. T-API-IPC-03100_0010 Successful updated sampling message readout. T-API-IPC-03110_0010 Successful queuing message readout. T-API-IPC-03110_0020 T-API-IPC-03110_0030 Successful Management of service being called when the queuing port is empty. Successful management of service being called when the port has overflowed since the last time it was read. 01/09/2013 IST

20 T-API-IPC-03110_0040 T-API-IPC-03110_0050 T-API-IPC-03110_0060 Successful management of service being called when the given queuing port is not configured to operate as a destination port. Successful management of service being called when the given timeout value is not valid. Successful management of service being called when the given queuing port does not exist. T-API-IPC-03120_0010 Successful queuing message sending. T-API-IPC-03120_0020 T-API-IPC-03120_0030 T-API-IPC-03120_0040 T-API-IPC-03120_0050 T-API-IPC-03120_0060 Successful management of service being called when there is not sufficient space left in the queuing port for the new message. Successful management of service being called when the message is larger than the maximum allowed message size. Successful management of service being called when the port passed is not configured to operate as a source port. Successful management of service being called when the passed time out value for the port is invalid. Successful management of service being called when the passed port id does not identify an existing queuing port. T-API-IPC-03130_0010 Successful sampling message writing. T-API-IPC-03130_0020 T-API-IPC-03130_0030 T-API-IPC-03130_0040 Successful management of the service being called when the input message length is greater than the maximum allowed length. Successful management of the service being called when the passed port is not configured to operate as a source. Successful management of the service being called when the passed port id does not identify an existing port. TTNoC Management New T-API-TTN-03200_0010 Successful creation of a ttnoc port. New hardware New T-API-TTN-03200_0020 Unsuccessful ttnoc port creation (when called with invalid parameters). New hardware New T-API-TTN-03200_0030 Successful ttnoc port ID check. New hardware New New T-API-TTN-03200_0040 T-API-TTN-03200_0050 Successful management of the service being called when the passed ttnoc port name does not exist. Successful management of service call with an invalid ttnoc port Id. New hardware New hardware New T-API-TTN-03200_0060 Successful ttnoc message readout. New hardware New T-API-TTN-03200_0070 Successful ttnoc message writing. New hardware DSMC Management 01/09/2013 IST

21 New T-API-TTN-03300_0010 Successful management of the service being called when invalid parameters in the call New hardware New T-API-TTN-03300_0020 Successful dsmc message readout. New hardware New T-API-TTN-03300_0030 Successful dsmc message writing. New hardware Time Management T-API-TM-0410_0010 Health Monitor Management T-API-HM-0510_0010 T-API-HM-0520_0010 T-API-HM-0520_0020 T-API-HM-0520_0030 Multiple Module Schedule T-API-SCH-0610_0010 T-API-SCH-0610_0020 T-API-SCH-0620_0010 T-API-SCH-0630_0010 T-API-SCH-0630_0020 T-API-SCH-0630_0030 Successful get access service Successful error status confirmation. Successful application error raising. Successful management service being called with an invalid length of error message. Successful management service being called with an invalid provided error code. Successful retrieval of the current schedule Id. Successful management of the service being called with a non-existent schedule name. Successful retrieval of scheduling status information. Successful changing of module schedule. Successful management of the service being called when the passed schedule Id does not exist. Successful management of the service being called by a non-system partition. Adapted T-API-SCH-0640_0010 Successful start of major frame New hardware supporting the external MAF synchronization Section 7.2 ( Functional test results ) details the results, considerations and limitations of this test suite execution on the target of the evaluation. 6.4 Performance evaluation In order to evaluate the bespoke TTNoC-based HW MultiPARTES platform, we have used the same performance metrics as defined in deliverable 4.6 Platform validation report. The metric consist on the following elements: 1. Maximum and average Partition Context switch time: maximum and average time for switching between two partitions in the scheduling plan. 2. Effective slot time of partition execution: time available to the partition code in a slot. 3. Partition overhead due to the virtualization layer. Performance loss of the partition due to its execution on top the hypervisor. 01/09/2013 IST

22 4. Stability of the plan with respect to the mid and long term: time variability of the Major Frame (MAF) occurrences. 5. Inter-partition communication message transfer: time needed to send a message to resp. receive a message from another partition 6. Service cost: evaluation of the maximum and average execution time of the hypervisor services. The detailed specification of the test cases for the collection of the above metrics was presented in D4.6. Additionally, some new elements are going to be considered: 1. Inter-partition communication message transfer between partitions allocated in different nodes. Evaluation of the time used to send and receive messages. 2. Time to access to shared memory blocks. Evaluation of the time needed to read and write memory blocks. These new tests are described below. Table 2 Inter core communication Test Case Id Test purpose Test Configuration T-PERF To measure the time to send/receive a message between two partitions allocated in different cores The scenario is built with 2 partitions. Each partition is allocated to one of the instances of the Virtualization Layer (VL). P1 is allocated to VL1 and P2 to VL2. The scheduling plan consists of one slot with a duration that is large enough to execute the partition code (50 ms). Test Description In every MAF, P1 sends a message of maximum size 26 bytes to P2 measuring the cost of the hypercall and waits for the reception of a message from P2 with the time the message was read. P1 stores the total time and prints a summary after 50 MAFS. P2 waits for a message and, when received, reads the clock value and sends back a message with the ACK and the clock value. Test constraints Test measurement The TTNoC limits the messages to 26 bytes for the message Local time stored during the execution. 01/09/2013 IST

23 Table 3 Memory block read/write Test Case Id Test purpose Test Configuration Test Description Test constraints Test measurement T-PERF To measure the time to read/write a memory block The scenario is built with 2 partitions. Each partition is allocated to one of the instances of the Virtualization Layer (VL). P1 is allocated to VL1 and P2 to VL2. The scheduling plan consists in one slot with a duration that is large enough to execute the partition code (500 ms). Every MAF, P1 writes a buffer (maximum size 26 bytes) in shared memory, measures the cost of the hypercall and send a message to P2 to inform about the write completion. P2 waits for the P1 message and reads the memory block and measures the cost of the hypercall. P2 verify the contents of the message (predefined). The TTNoC limits the messages to 26 bytes for the message. Several read/write hypercalls are needed to read/write buffers with larger size. Execution time of the hypercalls. Message content is the same. 6.5 Temporal interference analysis The design of this hardware platform was aimed to avoid the effects of the temporal interference of the parallel execution of two partitions on two different cores. In order to evaluate the temporal interference, the same experiment as described in section of D3.5 has been adapted to and performed on the current platform. Next, a summary of this experiment is presented. The goal of the experiment is to analyse the effect of the parallel execution of two partitions on different cores with different time intervals of partition overlaps. Table 4 Test case description Test Case Id Test purpose Test Configuration T-MTP-INTERF XXX To analyse the effect of the parallel execution of two partitions on different cores with different overlapping time interval The scenario is built with 2 partitions. Each partition is allocated to one of the instances of the Virtualization Layer (VL) running on different cores. P1 is allocated to VL1 and P2 to VL2. The scheduling plan consists of one slot with a duration that is large enough to execute the payloads of both partitions (20ms). The test is instantiated with a value for the overlap. This figure shows 01/09/2013 IST

24 the example when no overlap is produced. Test Description Partition P1 and P2 execute a known predefined payload that has been measured in isolation. Test constraints To force the maximum impact of the memory interference, the memory areas of both partitions P1 and P2 are configured as non-cacheable. Test measurement Time that P1 and P2 execute their payloads in the different test cases, i.e., for different partition overlaps (see Table 5). The following tests have been considered. Table 5: Test cases Test Case Id T-MTP-INTERF T-MTP-INTERF T-MTP-INTERF T-MTP-INTERF T-MTP-INTERF Test Configuration P1 and P2 overlap in 0% of their duration P1 and P2 overlap in 25% of their duration P1 and P2 overlap in 50% of their duration P1 and P2 overlap in 75% of their duration P1 and P2 overlap in 100% of their duration 01/09/2013 IST

25 7 Evaluation results 7.1 Introduction This section provides the results obtained with the different evaluation items described in the previous section. 7.2 Functional test results This section provides the results obtained with the different test scenarios for the functional evaluation (see Section 6.3 Functional evaluation ). The term OK/KO is used to show the test result. The term PASSED/NO PASSED is used to show the fulfilment of a global functionality. PASSED is used when all test in the category are OK. Table 6: MPT Abstraction Layer test results Status Test No Result Status Test No Result System Management PASSED Inter-Partition Communication PASSED T-API-SYS-0110_0010 OK T-API-IPC-0310_0010 OK Adapted T-API-SYS-0110_0011 OK T-API-PM-0310_0020 OK T-API-SYS-0110_0020 OK T-API-PM-0310_0030 OK T-API-SYS-0110_0030 OK T-API-IPC-0320_0010 OK NA T-API-SYS-0120_ T-API-IPC-0320_0020 OK Adapted T-API-SYS-0120_0011 OK T-API-IPC-0320_0030 OK T-API-SYS-0120_0020 OK T-API-IPC-0330_0010 OK Partition Management PASSED T-API-IPC-0330_0020 OK T-API-PM-0210_0010 OK T-API-IPC-0340_0010 OK Adapted T-API-PM-0210_0011 OK T-API-IPC-0340_0020 OK T-API-PM-0210_0020 OK T-API-IPC-0350_0010 OK T-API-PM-0210_0030 OK T-API-IPC-0350_0020 OK T-API-PM-0220_0010 OK T-API-IPC-0360_0010 OK Adapted T-API-PM-0220_0011 OK T-API-IPC-0360_0020 OK T-API-PM-0220_0020 OK T-API-IPC-0370_0010 OK Adapted T-API-PM-0220_0021 OK T-API-IPC-0370_0020 OK T-API-PM-0220_0030 OK T-API-IPC-0380_0010 OK Adapted T-API-PM-0220_0031 OK T-API-IPC-0380_0020 OK T-API-PM-0220_0040 OK T-API-IPC-0380_0030 OK T-API-PM-0220_0050 OK T-API-IPC-0390_0010 OK T-API-PM-0220_0060 OK T-API-IPC-03100_0010 OK T-API-PM-0230_0010 OK T-API-IPC-03110_0010 OK TTNoC Management PASSED T-API-IPC-03110_0020 OK 01/09/2013 IST

26 New T-API-TTN-03200_0010 OK T-API-IPC-03110_0030 OK New T-API-TTN-03200_0020 OK T-API-IPC-03110_0040 OK New T-API-TTN-03200_0030 OK T-API-IPC-03110_0050 OK New T-API-TTN-03200_0040 OK T-API-IPC-03110_0060 OK New T-API-TTN-03200_0050 OK T-API-IPC-03120_0010 OK New T-API-TTN-03200_0060 OK T-API-IPC-03120_0020 OK New T-API-TTN-03200_0070 OK T-API-IPC-03120_0030 OK DSMC Management PASSED T-API-IPC-03120_0040 OK New T-API-TTN-03300_0010 OK T-API-IPC-03120_0050 OK New T-API-TTN-03300_0020 OK T-API-IPC-03120_0060 OK New T-API-TTN-03300_0030 OK T-API-IPC-03130_0010 OK Time Management PASSED T-API-IPC-03130_0020 OK T-API-TM-0410_0010 OK T-API-IPC-03130_0030 OK Health Monitor Management OK T-API-IPC-03130_0040 OK T-API-HM-0510_0010 OK T-API-HM-0520_0010 OK T-API-HM-0520_0020 OK T-API-HM-0520_0030 OK Multiple Module Schedule PASSED T-API-SCH-0610_0010 OK T-API-SCH-0610_0020 OK T-API-SCH-0620_0010 OK T-API-SCH-0630_0010 OK T-API-SCH-0630_0020 OK T-API-SCH-0630_0030 OK Adapted T-API-SCH-0640_0010 OK These results show that all tests have been successfully executed and passed. Comparing with the results obtained on the COTS platform (D4.5), the set of tests that have run unchanged provide the same functionalities than the COTS platform. Adapted tests, as result of the hardware modification, provide also the same functionalities New tests have satisfactory validated the new functionalities. 7.3 Performance Evaluation This section presents the performance-test cases and different configurations that were used in order to retrieve the performance metrics presented in Section 6.4 of this deliverable that is based in the test description detailed in D4.6. Two configurations have been used: 1. XtratuM and partitions in each core have been allocated in static memory (SRAM) 2. XtratuM and partitions in each core have been allocated in dynamic memory (DDR) 01/09/2013 IST

27 7.3.1 Partition Context Switch (PCS) Deliverable v1.0 The next table summarizes the results from measurements that assessed the maximum, minimum and average time values required for the partition context-switch (PCS). Same test T- MTP-PERF (defined in D4.6) has been used. Last column shows the results obtained in the COTS platform. Table 7: Overall Partition Context Switch test results Processor LEON3 SRAM LEON3 DDR COTS LEON3 DDR Average (µsec) Maximum (µsec) Minimum (µsec) Standard deviation 0,721 1,432 1,663 These results show that the maximum PCS is 61 µsecs and 205 µsecs when the software runs on static (SRAM) or dynamic (DDR) memory Effective slot time of partition execution The effective slot time used by a partition is the time when the partition executes its own code. Next figure shows the scheme. Black rectangle refers to the partition context switch executed by the hypervisor. Figure 3: Effective partition execution slot time scheme The tests defined in D4.6 (T-MTP-PERF XXX) have been executed for the different configurations of slot duration specified. In these tests, the columns Avg, Max and Min show the average, maximum and minimum counter increments in a fixed period of 1 second. Diff column computes the differences with respect to a reference value (average counter with slot of 1000 milliseconds). Loss obtains in percentage the difference with respect to the reference value. PCS1 estimates the difference in counts with respect to the number of PCS that a partition has required. For instance, in the case of slot duration of 100ms, the partition has required to execute 10 times the slot in 1 second and, as consequence, 9 PCS have been needed. PCS2 estimates the value of the PCS in µsec taking into account PCS1 and the number of counts that have been measured in the reference value. Next table summarizes the results for the SRAM and DDR configurations. 01/09/2013 IST

28 Table 8: Effective slot time for SRAM configuration SRAM Configuration Slot (ms) Avg (counts) Max (counts) Min (counts) Diff (counts) Loss (%) PCS1 (counts) PCS2 (µsec) Table 9: Effective slot time for DDR configuration DDR Configuration Avg (counts) Max (counts) Min (counts) Diff (counts) Loss (%) PCS1 (counts) PCS2 (usec) Avg (counts) This scenario estimates the partition context switch indirectly through the performance loss of a partition when it is split in several slots. The performance loss corresponds to the external computation required to execute the partitions that are directly associated to the hypervisor scheduling. The reported results confirm the previous measurements of the PCS for the SRAM and DDR configurations which were measured by instrumenting the hypervisor code Partition overhead Overhead due to the virtualization layer Previous scenarios permit to estimate the performance lost due to the virtualization layer. It strongly depends on the effective time of the partition with respect to the allocated time. From previous table, we summarize the virtualization-layer overhead (last line of the following tables). Table 10: Virtualisation layer overhead for SRAM configuration Test ID Slot(ms) Avg (counts) /09/2013 IST