Persistent Publish/Subscribe Messaging in Medical Devices Justin Moon (Product Manager, Medical) Ben VandenBelt, B.Eng (Senior Developer, Medical Team) jmoon@qnx.com,bvandenbelt@qnx.com Abstract Many medical devices must bring together a disparate array of hardware and software components, as well as support a sophisticated, multi-layered Human- Machine Interface (HMI). Persistent Publish/Subscribe (PPS) messaging offers versatile, easy-to-use and reliable messaging that simplifies system design and facilitates HMI implementation. Introduction The design, development and preparation for market of an electronic medical device can entail significantly more time, effort and expense than are required for, say, consumer-grade devices of similar technical complexity. In addition to the generic development and validation requirements, medical devices must meet stringent functional safety and certification requirements. These requirements imply a strictly defined and managed design, development and validation environment, and extensive and sophisticated device validation against functional safety requirements. And, of course, a device must receive the appropriate certifications by the competent authorities in every jurisdiction where it will sold and used: FDA 510(k) pre-market notification, MDD and other supranational and national bodies, and so on. This paper describes PPS messaging in a medical data aggregator and publisher (QNX Medical Demo, or MD) that brings together in a portable demonstration application a blood pressure monitor, spirometer, pulse oximeter, ECG, and insulin pump. These components connect to a QNX-Continua interoperability manager and use QNX PPS messaging to communicate with a Qt HMI. PPS also provides messaging to a remote manager, for secure internet communication through the cloud to a records database and a portable tablet. Thanks to how easily a system with PPS messaging can integrate disparate components, we have concluded that this messaging paradigm is well-suited for a system like the QNX MD application. Figure 1: The QNX MD (medical device) demo.
PPS Messaging in Medical Devices Asynchronous Messaging Asynchronous messaging is well known and widely implemented, and does not require detailed elaboration here. It is the solution of choice for many systems, but has some characteristics that make it less than ideal for systems that must integrate a wide range of devices and software components. Figure 2. With asynchronous messaging, a process does not wait for a reply from the intended receiving process. In the context of messaging models for complex medical devices, it is worth noting, for example, that asynchronous messaging is a low-level solution that pushes the burden of error handling, end-to-end semantics and buffer management up to the application level. That is, architects designing a system that uses asynchronous messaging must develop a protocol or protocols to ensure the correct behavior of messaging across all applications, as they must ensure that these applications have sufficient memory allocated for message buffers under high-load conditions. While in simple systems these tasks may require no great effort, they can pose daunting challenges when designing or upgrading complex systems. Further, the complex burdens they place on the application level and the application development process can adversely affect, not just design and development schedules, but also device validation, and, hence, device certification. Binary or Human-readable objects? A PPS service can be designed to use either binary or human-readable objects. We chose to use human-readable objects and attributes for QNX PPS, considering that the benefits to development and debugging outweigh the cost of the larger objects. Human-readable objects allow debugging of from the command-line using simple filesystem utilities, such as cat for subscribe and echo for publish. For example: or: cat /pps/media/playcurrent cat /pps/media/.all?wait echo "attr::value">>/pps/objectfilename Similarly, debugging information, including PPS object and attributes, can be retrieved by a simple program that subscribes to an object and prints. With messaging thus closely coupled, a change to one software component may require changes to other software components, slowing or hindering system development and increasing system fragility. Solutions designed and implemented based on initial connectivity requirements are very brittle when new requirements are introduced. They typically rely on direct, point-to-point connections between communicating applications/components 1. Send/Receive/Reply Send/receive/reply (or synchronous) messaging is less common than asynchronous messaging. It is of particular value for real-time environments, where many processes require responses to their messages before they can proceed. Unlike asynchronous messaging, with send/receive/reply messaging, the system framework takes on the burden of handling messaging errors and message buffers. Every server communicates directly with its clients, and must know how to respond to all client messages. Figure 3. With synchronous messaging, a process blocks until it receives a reply from the intended receiving process. In other words, with send/receive/ reply messaging, as a system increases in scale and as diverse components are added to it, that system rapidly grows 1 Jerry Krasner. Making the case for commercial communication integration middleware. Embedded.com (Dec. 29, 2009). 2
PPS Messaging in Medical Devices in complexity and becomes increasingly brittle difficult to upgrade and scale while ensuring performance and, critical for medical devices, reliability. Persistent Publish/Subscribe Send/receive/reply messaging is well-suited and is even mandatory for a real-time OS such as the QNX Neutrino RTOS, which by definition must meet stringent reliability and availability requirements. Like asynchronous messaging, however, send/receive/reply messaging may not be the optimal choice for the application level in complex systems, especially if these systems must be designed to easily integrate disparate applications and functionality. Send/receive/reply messaging closely couples sender and receiver. Publish/Subscribe has been around in some form or other for at least 20 years. A similar messaging model, virtual synchrony, was described by K. P. Birman and T. A. Joseph in 1987 as a fault-tolerant asynchronous bulletin board mechanism 2. Twenty years ago, Nortel Networks implemented an analogous model for fault monitoring on telephone switches, such as the DMS-100, and used the technique in network monitoring and reporting systems. A quick search on the internet will provide many examples of publish/subscribe implementations, as will a search on, for instance, the ACM portal bring up hundreds of papers dealing with some aspect of publish/subscribe or other observer pattern models in computing. Our discussion focuses on how a publish/subscribe model that also offers persistence across reboots, or Persistent Publish/Subscribe (PPS), can aid in the design and deployment of embedded applications that must not only support a wide range of devices and software components, but also communicate with a sophisticated HMI. We have used a Qt-based HMI for the QNX MD, but we suggest that the advantages offered by the PPS messaging model could also apply to HMIs built with other technologies. We have, in fact, used PPS for other systems, including the QNX CAR Application Platform, and a Smart Energy reference application, which both have Adobe Flash-based HMIs. 2 Kenneth P. Birman and Thomas A. Joseph. Exploiting Virtual Synchrony in Distributed Systems. Cornell University. February, 1987. See also Kenneth P. Birman, Building Secure and Reliable Network Applications. Greenwich: Manning, 1996. An Object-based System PPS is an object-based service with publishers and subscribers in a loosely-coupled messaging architecture. Any PPS client can be a publisher only, a subscriber only, or both a publisher and a subscriber, as required by the implementation. Publishing is asynchronous. PPS objects are integrated into the PPS filesystem pathname space. Publishers modify objects and their attributes, and write them to the filesystem. When any publisher changes an object, the PPS service informs all clients subscribed to that object of the change. PPS clients can subscribe to multiple objects, and PPS objects can have multiple publishers as well as multiple subscribers. Thus, publishers with access to data that applies to different object attributes can use the same object to communicate their information to all subscribers to that object. PPS clients must of course know which PPS objects are of interest. If they publish, they must know what to publish and when; if they are subscribers, they must know to which objects they must subscribe, and which object attributes interest them. However, PPS clients do not have to manage errors, or buffers beyond what they need for open(), read() and write() POSIX API calls, confirming that they can make sense of what they read, and determining if they want their reads to be blocking or non-blocking. The PPS service Push or Pull? In its default implementation, the QNX PPS service acts as a push publishing system; that is, publishers push data to objects, and subscribers read data upon notification or at their leisure. However, some data, such as packet counts on an interface, changes far too quickly to be efficiently published through PPS using its default push publishing. QNX PPS therefore offers an option that allows a subscriber to change PPS into a pull publishing system. When a subscriber that opened an object with this option issues a read() call, all publishers to that object receive a notification to write current data to the object. The subscriber s read blocks until the object s data is updated, then returns with the new data. With this pull mechanism, the PPS subscriber retrieves data from the publisher at whatever rate it requires in effect, on-demand publishing.
PPS Messaging in Medical Devices handles the rest. Clients need only know that they have read a message, and that they are able to parse what they have read. Similarly, because PPS subscribers use read() calls to retrieve objects, they do not need to manage buffers for these objects. Language independence QNX PPS leverages the services of standard POSIX file systems, and can thus work with any programming language or application environment, including C, C++, Java, Adobe Flash, ksh scripting language, and so on. A component written in one language can communicate with components written in any other language. No special knowledge of the other components is required. delay decisions on module connection points and data flow until runtime. Because such decisions are neither hardcoded nor directly linked, they can be adapted as situations or requirements evolve; they can even change dynamically as the system runs. The loosely-coupled PPS messaging model also simplifies the integration of new software components. Since publisher and subscriber do not have to know each other, developers adding components need only to determine what these new components should publish, and what data they need other PPS clients to publish. No fine-tuning of APIs is required, and system complexity does not increase as components are added. Persistence A Persistent Publish/Subscribe service maintains data across reboots. It maintains its objects in memory while it is running, but will save its objects to persistent storage, either on demand while it is running, or at shutdown. It restores its objects on startup, either immediately, or on first access (deferred loading). Of course, the underlying persistent storage depends on a reliable filesystem and storage media, such as a hard disk, NAND or NOR flash. As well as ensuring data persistence across reboots, the PPS messaging model can simplify startups. For example, in a system that uses another messaging model, if a client comes up after the server, this client must request up-to-date data from the server, in case something changed during the period between the time the server started up and the time the client started up. This requirement is also true if, for whatever reason, a client loses contact with a server, and it is true for each and every client on the system, to which, of course, the server must respond. With PPS, however, the service restores its objects on startup and maintains them as they change. Any client no matter when it starts or reconnects needs only to read these objects to acquire current data. System Scalability With PPS, publisher and subscriber do not know each other; their only connection is an object that has a meaning and purpose for both of them. This messaging model gives developers great flexibility when designing a system: they can, if necessary, Figure 4: A screenshot of the QNX MD home screen. QNX MD (Medical Demo) As part of our Medical Device program at QNX Software Systems, we designed and built the QNX MD data aggregation and publishing application on systems running the QNX Neutrino RTOS specifically for the limited computing resources available to a portable medical device. It brings together a typical set of devices using a Continua-based interoperability manager, QNX PPS, and a sophisticated smart screen HMI built with Qt. Qt and CESL We chose Qt for the user interface and the Continua Enabling Software Library (CESL) for our interoperability manager because both these technologies are widely known in the medical device industry. Qt offers a well-defined set of UI widgets in a C++ development environment, and has a long successful history of successful implementations on devices that received FDA and other required certifications. 4
PPS Messaging in Medical Devices Figure 5: The QNX MD data aggregation and publishing application with PPS messaging. Note that PPS provides all the messaging between the HMI and the interoperability manager, as well as with the remote manager. With Qt, we had at our disposal all the elements needed to build clear, effective smart screens supporting the most exacting design requirements, including layout, layering, and multimedia support. Similarly, the communication protocols in the Continua library offered, not only a simple method for communicating with disparate medical devices, but also standardized protocols with a history of successful deployments in medical devices. In short, the Qt and Continua technologies met all our requirements, and are familiar and trusted in the medical device industry. Simplified Architecture One of the key benefits of using PPS messaging for our QNX MD demo application is that this looselycoupled messaging model enables a flexible architecture. If, for whatever reason, such a change were required, very little work would be required to substitute another library in the place of the Continua library, or to replace Qt for the HMI. Changing the HMI technology, for instance, would imply no changes to the interoperability manager or the remote manager, just as changing these managers would require no changes to the HMI. In addition, the PPS messaging model facilitates the addition of new devices, which can be connected to the system using standard Continua protocols over USB, Bluetooth, or even TCP transport. For instance, we could add an EEG to our demo simply by using Continua protocols to connect it to the interoperability manager, adding appropriate PPS objects to communicate data to the database, the local HMI and the remote HMI on the tablet, and, of course, adding the relevant displays and controls to the HMI. In a system using another messaging model, the components would be tightly coupled to each other and to the HMI. Each component would know about every other component with which it needs to exchange data an architecture which of course makes changing or adding anything difficult, time-consuming and fraught with risks. A further benefit of the PPS messaging architecture is that it simplifies testing and functional safety validation, because adding new components does not require revisiting the messaging between all other components. Finally, PPS simplifies re-branding, localization and interface updates. Because the HMI communicates with the rest of the system through PPS objects, user interface designers only need to ensure that a new or changed HMI subscribes and publishes to the same PPS objects as the previous HMI. They do not have to change a line of code below the HMI. This separation means that product line of devices can be built with exactly the same underlying system, but with different interfaces, different features enabled, or simply in different regions with different HMI designs (for example, to accommodate different writing systems or different color preferences and meanings.
PPS Messaging in Medical Devices reduced risk of inadvertent compromises to the system. References Figure 6: A high-level view of the QNX MD application connecting to an external database. Conclusion The QNX MD application shows how PPS messaging can be used in a loosely-coupled medical device architecture. This design enables versatile and robust communication betweenthe HMI and an interoperability manager supporting standard Continua protocols. This interoperability manager communicates with external component devices. Since components simply publish and subscribe to PPS objects as required by the needs of the implementation, they do not require knowledge of each other, and thus little work is required to scale or modify the system. Changes, upgrades and expansions become relatively benign tasks with a Birman, Kenneth P. and Thomas A. Joseph. Exploiting Virtual Synchrony in Distributed Systems. Cornell University. February, 1987. See also Kenneth P. Birman, Building Secure and Reliable Network Applications. Greenwich: Manning, 1996. <http://www.cs.cornell.edu/home/rvr/sys/p123- birman.pdf> Cieplak, Kris. Rapid Development and Reusable Design for the Connected Car. QNX Software Systems, 2010. 21321> Hobbs, Chris. Building Functional Safety into Complex Software Systems, Part I. QNX Software Systems, 2011. 21862> Hobbs, Chris. Building Functional Safety into Complex Software Systems, Part II. QNX Software Systems, 2011. 21978> Krasner, Jerry. Making the case for commercial communication integration middleware. Embedded.com (Dec. 29, 2009). <http://v2.embedded.com/design/222002429> Leroux, Paul. Exactly When Do You Need an RTOS?, 2009. 8090> 6 About is the leading global provider of innovative embedded technologies, including middleware, development tools, and operating systems. The component-based architectures of the QNX Neutrino RTOS, QNX Momentics Tool Suite, and QNX Aviage middleware family together provide the industry s most reliable and scalable framework for building high-performance embedded systems. Global leaders such as Cisco, Daimler, General Electric, Lockheed Martin, and Siemens depend on QNX technology for vehicle telematics and infotainment systems, industrial robotics, network routers, medical instruments, security and defense systems, and other mission- or life-critical applications. The company is headquartered in Ottawa, Canada, and distributes products in over 100 countries worldwide. www.qnx.com 2011 GmbH & Co. KG, a subsidiary of Research In Motion Ltd. All rights reserved. QNX, Momentics, Neutrino, Aviage, Photon and Photon microgui are trademarks of GmbH & Co. KG, which are registered trademarks and/or used in certain jurisdictions, and are used under license by Co. All other trademarks belong to their respective owners. 302205 MC411.92