The Data Distribution Service [Universal Standard for Real-Time Data Sharing]

Transcription

1 The Data Distribution Service [Universal Standard for Real-Time Data Sharing] Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech

2 Speaker Short Bio Angelo Corsaro, Ph.D. is Chief Technology Officer (CTO) at PrismTech. As CTO, Angelo directs the technology strategy, planning, evolution, and evangelism. Angelo leads the strategic standardization at the Object Management Group (OMG), where he co-chairs the Data Distribution Service (DDS) Special Interest Group. Prior to joining PrismTech Angelo worked as a Technology Scientist within the FINMECCANICA CTO Directorate looking after group-wide middleware strategy. Angelo is a widely known and cited expert in the field of real-time and distributed systems, middleware, and software patterns, has authored several international standards and enjoys over 10 years of experience in technology management and design of high performance mission- and business-critical distributed systems. Angelo received a Ph.D. and a M.S. in Computer Science from the Washington University in St. Louis, and a Laurea Magna cum Laude in Computer Engineering from the University of Catania, Italy. Contacts: [email protected] Mobile:

3 Background

4 Data Distribution Service Introduced in 2004 to address the Data Distribution challenges faced by a wide class of Defense and Aerospace Applications Key requirement for the standard were to deliver very high and predictable performance while scaling from embedded to ultralarge-scale deployments Scale Real-Timeliness Parallelism Systemic Signal Processing Data Processing Parallel Systems Determinism Real-Time Information Processing Throughput, Availability Near Real-Time FaultTolerant Information Processing Distributed Systems Scalability, Persistence, Security Complex Information Management Copyright 2011, PrismTech All Rights Reserved. For Real-Time Systems

5 Data Distribution Service Recommended by key administration worldwide, e.g. DoD, MoD, EUROCAE, etc. Widely adopted across several different domains, e.g., Automated Trading, Simulations, SCADA, Telemetry, etc. Scale Real-Timeliness Parallelism Systemic Signal Processing Data Processing Parallel Systems Determinism Real-Time Information Processing Throughput, Availability Near Real-Time FaultTolerant Information Processing Distributed Systems Scalability, Persistence, Security Complex Information Management Copyright 2011, PrismTech All Rights Reserved. For Real-Time Systems

6 Defense and Aerospace Integrated Modular Vetronics Training & Simulation Systems Naval Combat Systems Air Traffic Control & Management Unmanned Air Vehicles Aerospace Applications

7 Agricultural Vehicle Systems Large Scale SCADA Systems Smart Cities Train Control Systems Complex Medical Devices High Frequency Auto-Trading Copyright 2011, PrismTech All Rights Reserved. Commercial Applications

8 Smart Cities

9 City of Nice -- France

10 City Service - Architecture Urbio&ca APPLICATIONS User, City agent Urbio2ca MESH network City Message BUS Opensplice DDS Data warehouse Esper Park Control Exis2ng systems

11 About Opensplice DDS Publica2on Service Filtering and reading a ContentFilteredTopic an hundred 2me is faster than reques2ng once on DB Started with OpenSplice DDS for sensor data, now using OpenSplice everywhere, even as cloud messaging on Amazon Cloud!

12 Smart-Grids

13 U.S. Army Corps of Engineers 13 Grand Coulee Dam The Grand Coulee Dam is the largest hydro-electric power plant in the United States The dam network connects a 40,000-point SCADA system controlling 30 generators and the transmission switchyard In September 2011 USACE successfully deployed using

14 Grand Coulee Dam is used as the communication mechanism for the Generic Data Acquisition and Control System (GDACS) The dam network connects a 40,000+ point SCADA system controlling 30 generators and the transmission switchyard was selected because of its unique scalability, determinism, and robustness in presence of overload situations

15 Cool Vehicles

16 16 Agricultural Vehicle Systems GPS data correction to improve accuracy enabling automated steering, precision ploughing, seeding, fertilizing and spraying Tethered control between combine harvester and grain cart enabling unloading on-the-go is used to distribute data between the components inside the Combine system Communications between the Combine and the Grain Carts is made possible using the DDSI network protocol The OpenSplice Unicast networking capability allows to run over ad-hoc wireless networks

17 Race Car Video Feed Successfully proved for distributing racing car video

18 The Standard

19 DDSI DCPS The DDS Standard [1/3] The DDS Specification defines a programming-language independent API and a wire protocol for high performance data distribution API Protocol Application Durability Ownership Content Minimal Durability Ownership Content Minimal UDP TCP WebSock

20 DDSI DCPS The DDS Standard [2/3] The DDS API standard defines the semantics of DDS entities and the abstractions provided to applications Four different sub-profiles are defined to deal with: Content Awareness Data Durability Fail-Over Basic Pub/Sub API Protocol Application Durability Ownership Content Minimal Durability Ownership Content Minimal UDP TCP WebSock

21 DDSI DCPS The DDS Standard [3/3] The DDSI standard defines the protocol that DDS-compliant implementations have to use in order to implement the DDS semantics The DDSI protocol requires at most a best-effort transport--it implements reliability at a protocol level API Protocol Application Durability Ownership Content Minimal Durability Ownership Content Minimal UDP TCP WebSock

22 DDS Standard Evolution Application Application Security X-Types DDS RMI ANSI C ISO C++ DDS Java-5 DDS RMI API Scala x DDSI-RTPS DDSI-RTPS Wire Protocol Best Effort Transport Protocol 2004 X-Types Security 2006

23 Standards: What For? Data Model DDSI Common Language + QoS Requirements Wire Interoperability Interoperability

24 Standards: What For? Application DDS API Standard Portability

25 D D S ics Ba

26 OpenSplice Mobile

27 Data Distribution Service For Real-Time Systems DDS provides a Topic-Based Publish/ Subscribe abstraction based on: Topics: data distribution subject s DataWriters: data producers DataReaders: data consumers Data Writer Data Writer Data Writer Data Writer Topic D Topic A Topic B Topic C... DDS Global Data Space Data Reader Data Reader Data Reader Data Reader

28 Data Distribution Service For Real-Time Systems DataWriters and DataReaders are automatically and dynamically matched by the DDS Dynamic Discovery A rich set of QoS allows to control existential, temporal, and spatial properties of data Data Writer Data Writer Data Writer Data Writer Topic D Topic A Topic B Topic C... DDS Global Data Space Data Reader Data Reader Data Reader Data Reader

29 DDS Topics A Topic defines a class of streams A Topic has associated a unique name, a user defined extensible type and a set of QoS policies QoS Policies capture the Topic non-functional invariants Topics can be discovered or locally defined Circle, Square, Triangle,... ShapeType Type struct ShapeType string color; long x; long y; long shapesize; }; Name Topic QoS DURABILITY, DEADLINE, PRIORITY,

30 Topic Instances Each unique key value identifies a unique stream of data DDS not only demultiplexes streams but provides also lifecycle information A DDS DataWriter can write multiple instances Instances color = Green color = red color = Blue Topic struct ShapeType string color; long x; long y; long shapesize;}; Instances

31 DDS Readers and Writers Topic Instances DataWriter DataReader DataWriter Cache DataReader Cache Writer History Reader History network... DataReader DataReader Cache

32 Data Selectors

33 Filters and Queries Application DDS Filters allow to control what gets into a DataReader cache DDS Queries allow to control what gets out of a DataReader cache Filters are defined by means of ContentFilteredTopics Queries operate in conjunction with read operations Filters and Queries are expressed as SQL where clauses Query DataReader DataReader Cache Filter

34 DDS Application s Anatomy

35 Anatomy of a DDS Application Domain (e.g. Domain 123) Partition (e.g. Telemetry, Shapes, ) Topic Instances/Samples Publisher Domain Participant DataWrter Topic Subscriber DataReader

36 Anatomy of a DDS Application [Scala API] Domain val dp = DomainParticipant(domainId) Publisher DataWriter Domain Participant Topic Subscriber DataReader

37 Anatomy of a DDS Application [Scala API] Domain val dp = DomainParticipant(domainId) Session // Create a Topic val topic = Topic[ShapeType](dp, Circle ) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Publisher DataWriter Domain Participant Topic Subscriber DataReader

38 Anatomy of a DDS Application [Scala API] Domain val dp = DomainParticipant(domainId) Session // Create a Topic val topic = Topic[ShapeType](dp, Circle ) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Reader/Writers for User Defined for Types // Create a DataWriter/DataWriter val writer = DataWriter[ShapeType](pub, topic) val reader = DataReader[ShapeType](sub, topic) Publisher DataWriter Domain Participant Topic Reader/Writer for application defined Topic Types Subscriber DataReader

39 Anatomy of a DDS Application [Scala API] Domain val dp = DomainParticipant(domainId) Session // Create a Topic val topic = Topic[ShapeType](dp, Circle ) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Reader/Writers for User Defined for Types // Write data val data = new ShapeType( RED, 131, 107, 75) writer write data // But you can also write like this... writer! data Publisher DataWriter Domain Participant Topic Reader/Writer for application defined Topic Types Subscriber DataReader // Read new data and print it on the screen (reader read) foreach (prinln)

40 Anatomy of a DDS Application [DDS C++ API 2010] Domain auto dp = DomainParticipant(domainId); Session // Create a Topic auto topic = Topic<ShapeType>(dp, Circle ) // Create a Publisher / Subscriber auto pub = Publisher(dp) auto sub = Subscriber(dp) Reader/Writers for User Defined for Types // Create a DataWriter/DataWriter auto writer = DataWriter<ShapeType>(pub, topic); auto reader = DataReader<ShapeType>(sub, topic); Publisher DataWriter Domain Participant Topic Reader/Writer for application defined Topic Types Subscriber DataReader

41 Anatomy of a DDS Application [DDS C++ API 2010] Domain auto dp = DomainParticipant(domainId); Session // Create a Topic auto topic = Topic<ShapeType>(dp, Circle ) // Create a Publisher / Subscriber auto pub = Publisher(dp) auto sub = Subscriber(dp) Reader/Writers for User Defined for Types // Write data writer.write(shapetype( RED, 131, 107, 89)); // But you can also write like this... writer << ShapeType( RED, 131, 107, 89); Publisher DataWriter Domain Participant Topic Reader/Writer for application defined Topic Types Subscriber DataReader // Read new data (loaned) auto data = reader.read();

42 QoS

43 QoS Model QoS-Policies control local and end-to-end properties of DDS entities Local properties controlled by QoS are related resource usage End-to-end properties controlled by QoS are related to temporal and spatial aspects of data distribution Some QoS-Policies are matched based on a Request vs. Offered Model thus QoS-enforcement QoS QoS Publisher DomainParticipant... QoS DataWriter DataWriter QoS Type Matching QoS Topic writes Type... writes Type Topic QoS Name reads reads Name QoS matching QoS DataReader DataReader QoS... QoS QoS Subscriber DomainParticipant

44 QoS Policies [T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant] QoS Policy Applicability RxO Modifiable USER_DATA TOPIC_DATA GROUP_DATA DURABILITY DURABILITY SERVICE HISTORY PRESENTATION RELIABILITY PARTITION DESTINATION ORDER DP, DR, DW N Y T N Y Configuration P, S N Y T, DR, DW Y N T, DW N N Data Availability T, DR, DW N N P, S Y N T, DR, DW Y N P, S N Y T, DR, DW Y N Data Delivery LIFESPAN T, DW N Y

45 QoS Policies [T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant] QoS Policy Applicability RxO Modifiable DEADLINE LATENCY BUDGET TRANSPORT PRIORITY TIME BASED FILTER OWNERSHIP OWNERSHIP STRENGTH LIVELINESS T, DR, DW Y Y T, DR, DW Y Y T, DW N Y DR N Y T, DR, DW Y N Temporal/ Importance Characteristics DW N Y Replication T, DR, DW Y N Fault-Detection

46 Data Delivery Partition Reliability Data Delivery Presentation Destination Order

47 Data Availability Lifespan History Data Availability Ownership Durability Ownership Strength

48 Temporal Properties TimeBasedFilter [Inbound] Throughput [Outbound] LatencyBudget Deadline Latency TransportPriority

49 Concluding Remarks

50 Concluding Remarks The DDS provides a powerful and featurerich topic-based publish/subscribe abstraction This technology is widely used in mission and business critical systems and it being swiftly adopted in data-centric/big-data systems Differently from what some people think, DDS is very simple to get-started with Very good Open Source implementation are available... Good Hacking!

51 DDS Everywhere

52 DDS Everywhere! OpenSplice Mobile OpenSplice Mobile Embedded JVM DDS OpenSplice Enterprise MQTT, AMQP, JMS, REST,... OpenSplice Gateway OpenSplice Lite Embedded OS OpenSplice RTE RTOS Enterprise OS

53 References DDS-based Advanced Distributed Algorithms Toolkit Open Source github.com/kydos/dada OpenSplice DDS #1 OMG DDS Implementation Open Source Simple C++ API for DDS Open Source code.google.com/p/simd-cxx [C++] Fastest growing JVM Language Open Source [Java] DDS-PSM-Java for Open Source github.com/kydos/simd-java Escalier Scala API for Open Source github.com/kydos/escalier DDS-PSM-Cxx 2010 DDS-PSM-Cxx API Standard Open Source github.com/kydos/dds-psm-cxx

54 :: Connect with Us :: opensplice.com opensplice.org youtube.com/opensplicetube

55