The Data Distribution Service [Universal Standard for Real-Time Data Sharing]

The Data Distribution Service [Universal Standard for Real-Time Data Sharing] Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech angelo.corsaro@prismtech.com

Speaker Short Bio Angelo Corsaro, Ph.D. is Chief Technology Officer (CTO) at PrismTech. As CTO, Angelo directs the technology strategy, planning, evolution, and evangelism. Angelo leads the strategic standardization at the Object Management Group (OMG), where he co-chairs the Data Distribution Service (DDS) Special Interest Group. Prior to joining PrismTech Angelo worked as a Technology Scientist within the FINMECCANICA CTO Directorate looking after group-wide middleware strategy. Angelo is a widely known and cited expert in the field of real-time and distributed systems, middleware, and software patterns, has authored several international standards and enjoys over 10 years of experience in technology management and design of high performance mission- and business-critical distributed systems. Angelo received a Ph.D. and a M.S. in Computer Science from the Washington University in St. Louis, and a Laurea Magna cum Laude in Computer Engineering from the University of Catania, Italy. Contacts: email: angelo@icorsaro.net Mobile: +33 6 42 30 75 65

Background

Data Distribution Service Introduced in 2004 to address the Data Distribution challenges faced by a wide class of Defense and Aerospace Applications Key requirement for the standard were to deliver very high and predictable performance while scaling from embedded to ultralarge-scale deployments Scale Real-Timeliness Parallelism Systemic Signal Processing Data Processing Parallel Systems Determinism Real-Time Information Processing Throughput, Availability Near Real-Time FaultTolerant Information Processing Distributed Systems Scalability, Persistence, Security Complex Information Management Copyright 2011, PrismTech All Rights Reserved. For Real-Time Systems

Data Distribution Service Recommended by key administration worldwide, e.g. DoD, MoD, EUROCAE, etc. Widely adopted across several different domains, e.g., Automated Trading, Simulations, SCADA, Telemetry, etc. Scale Real-Timeliness Parallelism Systemic Signal Processing Data Processing Parallel Systems Determinism Real-Time Information Processing Throughput, Availability Near Real-Time FaultTolerant Information Processing Distributed Systems Scalability, Persistence, Security Complex Information Management Copyright 2011, PrismTech All Rights Reserved. For Real-Time Systems

Defense and Aerospace Integrated Modular Vetronics Training & Simulation Systems Naval Combat Systems Air Traffic Control & Management Unmanned Air Vehicles Aerospace Applications

Agricultural Vehicle Systems Large Scale SCADA Systems Smart Cities Train Control Systems Complex Medical Devices High Frequency Auto-Trading Copyright 2011, PrismTech All Rights Reserved. Commercial Applications

Smart Cities

City of Nice -- France

City Service - Architecture Urbio&ca APPLICATIONS User, City agent Urbio2ca MESH network City Message BUS Opensplice DDS Data warehouse Esper Park Control Exis2ng systems

About Opensplice DDS Publica2on Service Filtering and reading a ContentFilteredTopic an hundred 2me is faster than reques2ng once on DB Started with OpenSplice DDS for sensor data, now using OpenSplice everywhere, even as cloud messaging on Amazon Cloud!

Smart-Grids

U.S. Army Corps of Engineers 13 Grand Coulee Dam The Grand Coulee Dam is the largest hydro-electric power plant in the United States The dam network connects a 40,000-point SCADA system controlling 30 generators and the transmission switchyard In September 2011 USACE successfully deployed using

Grand Coulee Dam is used as the communication mechanism for the Generic Data Acquisition and Control System (GDACS) The dam network connects a 40,000+ point SCADA system controlling 30 generators and the transmission switchyard was selected because of its unique scalability, determinism, and robustness in presence of overload situations

Cool Vehicles

16 Agricultural Vehicle Systems GPS data correction to improve accuracy enabling automated steering, precision ploughing, seeding, fertilizing and spraying Tethered control between combine harvester and grain cart enabling unloading on-the-go is used to distribute data between the components inside the Combine system Communications between the Combine and the Grain Carts is made possible using the DDSI network protocol The OpenSplice Unicast networking capability allows to run over ad-hoc wireless networks

Race Car Video Feed Successfully proved for distributing racing car video

The Standard

DDSI DCPS The DDS Standard [1/3] The DDS Specification defines a programming-language independent API and a wire protocol for high performance data distribution API Protocol Application Durability Ownership Content Minimal Durability Ownership Content Minimal UDP TCP WebSock

DDSI DCPS The DDS Standard [2/3] The DDS API standard defines the semantics of DDS entities and the abstractions provided to applications Four different sub-profiles are defined to deal with: Content Awareness Data Durability Fail-Over Basic Pub/Sub API Protocol Application Durability Ownership Content Minimal Durability Ownership Content Minimal UDP TCP WebSock

DDSI DCPS The DDS Standard [3/3] The DDSI standard defines the protocol that DDS-compliant implementations have to use in order to implement the DDS semantics The DDSI protocol requires at most a best-effort transport--it implements reliability at a protocol level API Protocol Application Durability Ownership Content Minimal Durability Ownership Content Minimal UDP TCP WebSock

DDS Standard Evolution Application Application Security X-Types DDS RMI ANSI C 2012 2012 ISO C++ DDS Java-5 DDS RMI 2012 2010 2010 2012 2006 API Scala 2004 2010 2010 201x DDSI-RTPS DDSI-RTPS Wire Protocol Best Effort Transport Protocol 2004 X-Types Security 2006

Standards: What For? Data Model DDSI Common Language + QoS Requirements Wire Interoperability Interoperability

Standards: What For? Application DDS API Standard Portability

D D S ics Ba

OpenSplice Mobile

Data Distribution Service For Real-Time Systems DDS provides a Topic-Based Publish/ Subscribe abstraction based on: Topics: data distribution subject s DataWriters: data producers DataReaders: data consumers Data Writer Data Writer Data Writer Data Writer Topic D Topic A Topic B Topic C... DDS Global Data Space Data Reader Data Reader Data Reader Data Reader

Data Distribution Service For Real-Time Systems DataWriters and DataReaders are automatically and dynamically matched by the DDS Dynamic Discovery A rich set of QoS allows to control existential, temporal, and spatial properties of data Data Writer Data Writer Data Writer Data Writer Topic D Topic A Topic B Topic C... DDS Global Data Space Data Reader Data Reader Data Reader Data Reader

DDS Topics A Topic defines a class of streams A Topic has associated a unique name, a user defined extensible type and a set of QoS policies QoS Policies capture the Topic non-functional invariants Topics can be discovered or locally defined Circle, Square, Triangle,... ShapeType Type struct ShapeType { @Key string color; long x; long y; long shapesize; }; Name Topic QoS DURABILITY, DEADLINE, PRIORITY,

Topic Instances Each unique key value identifies a unique stream of data DDS not only demultiplexes streams but provides also lifecycle information A DDS DataWriter can write multiple instances Instances color = Green color = red color = Blue Topic struct ShapeType { @Key string color; long x; long y; long shapesize;}; Instances

DDS Readers and Writers Topic Instances DataWriter DataReader........................ DataWriter Cache DataReader Cache Writer History Reader History network... DataReader............ DataReader Cache

Data Selectors

Filters and Queries Application DDS Filters allow to control what gets into a DataReader cache DDS Queries allow to control what gets out of a DataReader cache Filters are defined by means of ContentFilteredTopics Queries operate in conjunction with read operations Filters and Queries are expressed as SQL where clauses Query DataReader............ DataReader Cache Filter

DDS Application s Anatomy

Anatomy of a DDS Application Domain (e.g. Domain 123) Partition (e.g. Telemetry, Shapes, ) Topic Instances/Samples Publisher Domain Participant DataWrter Topic Subscriber DataReader

Anatomy of a DDS Application [Scala API] Domain val dp = DomainParticipant(domainId) Publisher DataWriter Domain Participant Topic Subscriber DataReader

Anatomy of a DDS Application [Scala API] Domain val dp = DomainParticipant(domainId) Session // Create a Topic val topic = Topic[ShapeType](dp, Circle ) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Publisher DataWriter Domain Participant Topic Subscriber DataReader

Anatomy of a DDS Application [Scala API] Domain val dp = DomainParticipant(domainId) Session // Create a Topic val topic = Topic[ShapeType](dp, Circle ) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Reader/Writers for User Defined for Types // Create a DataWriter/DataWriter val writer = DataWriter[ShapeType](pub, topic) val reader = DataReader[ShapeType](sub, topic) Publisher DataWriter Domain Participant Topic Reader/Writer for application defined Topic Types Subscriber DataReader

Anatomy of a DDS Application [Scala API] Domain val dp = DomainParticipant(domainId) Session // Create a Topic val topic = Topic[ShapeType](dp, Circle ) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Reader/Writers for User Defined for Types // Write data val data = new ShapeType( RED, 131, 107, 75) writer write data // But you can also write like this... writer! data Publisher DataWriter Domain Participant Topic Reader/Writer for application defined Topic Types Subscriber DataReader // Read new data and print it on the screen (reader read) foreach (prinln)

Anatomy of a DDS Application [DDS C++ API 2010] Domain auto dp = DomainParticipant(domainId); Session // Create a Topic auto topic = Topic<ShapeType>(dp, Circle ) // Create a Publisher / Subscriber auto pub = Publisher(dp) auto sub = Subscriber(dp) Reader/Writers for User Defined for Types // Create a DataWriter/DataWriter auto writer = DataWriter<ShapeType>(pub, topic); auto reader = DataReader<ShapeType>(sub, topic); Publisher DataWriter Domain Participant Topic Reader/Writer for application defined Topic Types Subscriber DataReader

Anatomy of a DDS Application [DDS C++ API 2010] Domain auto dp = DomainParticipant(domainId); Session // Create a Topic auto topic = Topic<ShapeType>(dp, Circle ) // Create a Publisher / Subscriber auto pub = Publisher(dp) auto sub = Subscriber(dp) Reader/Writers for User Defined for Types // Write data writer.write(shapetype( RED, 131, 107, 89)); // But you can also write like this... writer << ShapeType( RED, 131, 107, 89); Publisher DataWriter Domain Participant Topic Reader/Writer for application defined Topic Types Subscriber DataReader // Read new data (loaned) auto data = reader.read();

QoS

QoS Model QoS-Policies control local and end-to-end properties of DDS entities Local properties controlled by QoS are related resource usage End-to-end properties controlled by QoS are related to temporal and spatial aspects of data distribution Some QoS-Policies are matched based on a Request vs. Offered Model thus QoS-enforcement QoS QoS Publisher DomainParticipant... QoS DataWriter DataWriter QoS Type Matching QoS Topic writes Type... writes Type Topic QoS Name reads reads Name QoS matching QoS DataReader DataReader QoS... QoS QoS Subscriber DomainParticipant

QoS Policies [T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant] QoS Policy Applicability RxO Modifiable USER_DATA TOPIC_DATA GROUP_DATA DURABILITY DURABILITY SERVICE HISTORY PRESENTATION RELIABILITY PARTITION DESTINATION ORDER DP, DR, DW N Y T N Y Configuration P, S N Y T, DR, DW Y N T, DW N N Data Availability T, DR, DW N N P, S Y N T, DR, DW Y N P, S N Y T, DR, DW Y N Data Delivery LIFESPAN T, DW N Y

QoS Policies [T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant] QoS Policy Applicability RxO Modifiable DEADLINE LATENCY BUDGET TRANSPORT PRIORITY TIME BASED FILTER OWNERSHIP OWNERSHIP STRENGTH LIVELINESS T, DR, DW Y Y T, DR, DW Y Y T, DW N Y DR N Y T, DR, DW Y N Temporal/ Importance Characteristics DW N Y Replication T, DR, DW Y N Fault-Detection

Data Delivery Partition Reliability Data Delivery Presentation Destination Order

Data Availability Lifespan History Data Availability Ownership Durability Ownership Strength

Temporal Properties TimeBasedFilter [Inbound] Throughput [Outbound] LatencyBudget Deadline Latency TransportPriority

Concluding Remarks

Concluding Remarks The DDS provides a powerful and featurerich topic-based publish/subscribe abstraction This technology is widely used in mission and business critical systems and it being swiftly adopted in data-centric/big-data systems Differently from what some people think, DDS is very simple to get-started with Very good Open Source implementation are available... Good Hacking!

DDS Everywhere

DDS Everywhere! OpenSplice Mobile OpenSplice Mobile Embedded JVM DDS OpenSplice Enterprise MQTT, AMQP, JMS, REST,... OpenSplice Gateway OpenSplice Lite Embedded OS OpenSplice RTE RTOS Enterprise OS

References DDS-based Advanced Distributed Algorithms Toolkit Open Source github.com/kydos/dada OpenSplice DDS #1 OMG DDS Implementation Open Source www.opensplice.org Simple C++ API for DDS Open Source code.google.com/p/simd-cxx [C++] Fastest growing JVM Language Open Source www.scala-lang.org [Java] DDS-PSM-Java for Open Source github.com/kydos/simd-java Escalier Scala API for Open Source github.com/kydos/escalier DDS-PSM-Cxx 2010 DDS-PSM-Cxx API Standard Open Source github.com/kydos/dds-psm-cxx

:: Connect with Us :: opensplice.com opensplice.org youtube.com/opensplicetube forums.opensplice.org opensplicedds@prismtech.com slideshare.net/angelo.corsaro @acorsaro @prismtech crc@prismtech.com sales@prismtech.com