Experience with the integration of distribution middleware into partitioned systems Héctor Pérez Tijero (perezh@unican.es) J. Javier Gutiérrez García (gutierjj@unican.es) Computers and Real-Time Group, University of Cantabria Berlin (Germany), June 2013 GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 1
Outline 1. Introduction and objectives 2. The XtratuM hypervisor 3. Integration of distribution middleware into XtratuM 4. Case study 5. Conclusions and future work GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 2
Introduction Partitioned systems Development of safety-critical software - certification requirements Memory and time isolation properties - the execution of partitions is restricted to predefined intervals PARTITION #1 PARTITION #2... PARTITION #N PARTITIONING KERNEL DEVICE #1 DEVICE #2 DEVICE #N GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 3
Introduction (cont d) Distributed partitioned systems Special purpose networks Common distribution middleware increases complexity Research on middleware for safety-critical systems Custom distribution facilitites - automatic generation of source code from system models - e.g., PolyORB-HI Distribution facilities based on standards - Safety-critical DDS - Ada Ravenscar or Safety-critical Java profiles GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 4
Introduction (cont d) Partitioning based on a hypervisor Thin layer with low overhead Independent execution environments (multiple OS) - mixed-criticality partitions on top of different OS Eases the integration of distribution middleware Benefits of using distribution middleware in partitioned systems Enables transparent communications between subsystems - network services, connection management - regardless of partitions in different/same core module - interoperability (standards) Allows to schedule a multicore as if it were distributed - partitions statically allocated to CPUs, fixed scheduling parameters GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 5
Objective CORE MODULE PARTITION #A PARTITION #M PARTITION.#N.. PARTITION #T PARTITION #Z MW MIW MW MW COMMUNICATION SERVICES HYPERVISOR DEVICE #GP DEVICE #RT Two approaches Specific profile of distribution standards (high-criticality) Standard distribution middleware (low-criticality) GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 6
XtratuM hypervisor CORE MODULE Partition #1 Linux OS... Partition #N Other OS XtratuM API XtratuM Services Device #1 Device #2 Device #3 Device #N Design based on the ARINC-653 standard - time and space isolation features Partition as an application executed on top of the OS GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 7
XtratuM Communication Services XtratuM I/O virtualization engine (XMIO) Device and transport virtualization Only Linux-based partitions ARINC-like communication ports Sampling ports - storage for a single message Queuing ports - storage for a fixed number of messages (FIFO) Channels to interconnect them CORE MODULE XTRATUM XMIO SERVICES HARDWARE CORE MODULE XTRATUM API XTRATUM ARINC-LIKE SERVICES HARDWARE PARTITION #A V-NIC... V-NETWORK PARTITION #N V-NIC PARTITION #A... PARTITION #N SENDING PORT RECEIVING PORT GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 8
XtratuM Communication Services: ARINC-like communication ports CORE MODULE Partition #1 Partition #N... XtratuM API XtratuM ARING-Like Services Sending port Receiving port Hardware Sampling and queuing ports - offline configuration - allow one-way operation mode - provide non-blocking communications - single source of messages for a receiving port GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 9
XtratuM Communication Services: ARINC-like communication ports CORE MODULE I/O Partition Partition #N... XtratuM API XtratuM ARING-Like Services Sending port Receiving port Hardware NIC Management of devices left to partitions Exclusive access to device - drivers must be implemented by partitions - offline configuration GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 10
System architectures Integration of middleware within partitioned systems Communications between core modules Communications between partitions CORE MODULE PARTITION I/O... PARTITION #X CORE MODULE PARTITION I/O... PARTITION #N CORE MODULE PARTITION I/O... PARTITION #N XTRATUM API XTRATUM HARDWARE NIC HARDWARE NIC ARINC-LIKE SERVICES NETWORK HARDWARE NIC (A) (B) GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 11
System architectures (cont d) Communications between core modules 1. I/O partition with distribution middleware - message processing, open systems 2. "Bare" I/O partition to forward messages - opaque messages, static connections Platform design issues Fixed and predefined communication channels - static routing table for the I/O partition - may be automatically generated from configuration GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 12
System architectures (cont d) Communications between partitions Use of distribution middleware Platform design issues Asynchronous communications - synchronous remote calls are not allowed Non-blocking communications - middleware cannot be blocked awaiting for incoming messages Single source of messages for a receiving port - common strategy in middleware (single listening port) - multiple communication channels between partitions GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 13
Proposed system architecture (cont d) CORE MODULE #1 PARTITION #X MIDLEWARE I/O PARTITION CORE MODULE #2 I/O PARTITION PARTITION #Y... MIDLEWARE XTRATUM API XTRATUM API XTRATUM ARINC-LIKE SERVICES XTRATUM ARINC-LIKE SERVICES HARDWARE NIC HARDWARE NIC NETWORK SENDING PORT RECEIVING PORT CHANNEL N (#X => #Z) CHANNEL M (#Z => #X) CHANNEL T (#X => #Y) I/O PARTITION PARTITION #Z... NIC MIDLEWARE CORE MODULE #3 XTRATUM API XTRATUM ARINC-LIKE SERVICES HARDWARE GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 14
Case Study: Video-surveillance SENDING PORT CORE MODULE #1 VIDEO_RECORDER MIDDLEWARE RTOS IO_SERVER RTOS NETWORK RECEIVING PORT XTRATUM API CHANNEL 1 XTRATUM ARINC-LIKE CHANNEL 2 SERVICES HARDWARE NIC THIRD-PARTY APP NETWORK CORE MODULE #2 CORE MODULE #N - Multiple display monitors may request video captures from recording app. - Key feature: reliability of recording application GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 15
Case Study: Objectives Performance of the proposed architecture overhead introduced by hypervisor Interoperability between heterogeneous subsystems partitioned and non-partitioned systems Development of a prototype to validate the approach retrieval of a previous distributed real-time platform - PolyORB as distribution middleware - application personalities, microkernel and protocol personalities - RT-EP as the real-time network - token management in partitioned systems - MaRTE OS as the real-time operating system integration is not straightforward GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 16
Case Study: Prototype APPLICATION ADA CODE ADA CODE ADA BINDINGS (XTRATUM API) ADA CODE MIDDLEWARE POLYORB-CORBA POLYORB-CORBA POLYORB-ICMC POLYORB-ETHERNET COMMUNICATION SERVICES ARINC-LIKE ARINC-LIKE & ETHERNET ETHERNET OPERATING SYSTEM HYPERVISOR MARTE OS XTRATUM MARTE OS XTRATUM MARTE OS VIDEO_RECORDER IO_SERVER CLIENT GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 17
Case Study: Configuration 1. Assignment of partitions to core modules 2. Number and type of ports in each partition - decoupled model 3. Definition of communication channels 4. Cyclic scheduling plan - IO_Server partition should fulfils the I/O requirements of remaining partitions - execution of I/O operations "in one go" to minimize idle times MAF CPU-0 <VIDEO_RECORDER> <IO_SERVER> 0 400 500 SCHEDULING CYCLIC PLAN (TIMES IN μs) GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 18
Case Study: Preliminary results Platform: 800 Mhz embedded nodes, 100 Mbps Ethernet Time required to request and obtain a video capture Three experiments - Non-partitioned - Single partition - Partitioned OVERHEAD PERFOMANCE LOSS COMMENTS HYPERVISOR < 1 % ARINC-LIKE PORTS ARCHITECTURE 29 % 25 % PARTITION CONFIGURATION 4 % OVERHEADS GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 19
Conclusions Integration of distribution middleware into partitioned systems Inter-core modules communications - I/O partition required, middleware may be required Inter-partitions communications - asynchonous and non-blocking communications - multiple reception ports per partition Benefits of the proposed integration - avoid complexity in communications between partitions - interoperability among subsystems GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 20
Future work New features of XtratuM v3 multicore support - dedicated core to the I/O partition asynchronous management of communication ports Exploring an asynchronous and decoupled distribution model source and destination are unknown at partition-level incoming safe-critical profile for DDS Other communication approaches to ease portability XMIO communication service GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 21
Questions? GRUPO DE COMPUTADORES Y TIEMPO REAL Héctor Pérez Tijero 22