Christoph Emmersberger. Florian Springer. Senacor Technologies AG Wieseneckstraße Schwaig b. Nürnberg

Transcription

1 Christoph Emmersberger Universität Regensburg Universitätsstraße Regensburg Florian Springer Senacor Technologies AG Wieseneckstraße Schwaig b. Nürnberg

2 Open Source Enterprise Application Integration Introducing the Event Processing Capabilities of Apache Camel

3 Agenda Motivation and Background Introduction of EAI and Camel Excursion: Event Processing Building Blocks Use Cases and Implementation using Camel Conclusion and Q&A 2

4 Motivation and Background What is the motivation for EAI in Business? Realignment of Business Strategy New regulatory Requirements Mergers & Acquisitions Business Process Optimization Cooperation with suppliers and partners Change of product strategy e.g. Mobile Changings in business require continuous adaptation of applications Reduce Time-to-Market Cost Cutting (IT + Business) 3

5 Motivation EAI: Banking Domain New Requirements/ Business Capabilities EXAMPLE Multi-Channel Banking Regulatory Requirements Merger Globalization Diversification of Competitors Consistent information and transparent business processes over all channels, without media breaks for MCB Fast and flexible introduction of new banking products New regulatory requirements (Basel, EMIR, IFRS, etc.) Complete process automation from front to the back office Consolidation of the system landscape in the front and middle office 4

6 Motivation EAI: Logistics Domain EXAMPLE New Requirements/ Business Capabilities Direct shippment Globalization Cooperation, Diversification E-/M-Commerce Alignment to receiver Consistent customer communication via different channels Fast introduction of new logistic products Real-time Track & Trace Alignment of products to private customers as well as receiver Flexible, GPS-based optimization of transportation route Flexible models for cooperation with customers 5

7 Demand for Application Integration Researchers View Gartner Predictions 2013 for Application Integration [1] By 2016, midsize to large companies will spend 33% more on application integrations than in By 2016, the integration of data on mobile devices will represent 20% of integration spending. By 2017, over two-thirds of all new integration flows will extend outside the enterprise firewall. OVUM view [2] Global integration middleware market to hit $17.9 billion by 2018 Source: [1] [2] 6

8 Agenda Motivation and Background Introduction of EAI and Camel Excursion: Event Processing Building Blocks Use Cases and Implementation using Camel Conclusion and Q&A 7

9 EAI Introduction: A Simple Picture EAI the universal application plug 8

10 EAI Introduction: What is EAI? ( ) Interesting applications rarely live in isolation [1] Integration Layers User integration Process integration Application integration Data integration Examples Make same GUI (functionality) available in several Application (e.g. logon) Make same Processes available for different UseCases (e.g. Payment with PayPal) Make same functionality available for different UseCases (e.g. price calculation) Provide aggregated Data, e.g. Customer Data (MS CRM), Order Details (SAP), etc. for reporting purpose Enterprise Application Integration Creates business solutions by combining applications Middleware specifies a technology stack which is capable to mediate between applications with the overall goal of improving the supply-chain relationships in a distributed application environment. EAI Tools provide functionality ( EAI patterns ) to reduce effort for integration, e.g. filter, transformation, construction, etc EAI is a enabler for continious integration Source: [1] G. Hohpe and B. Woolf, Enterprise Integration Pattern: Designing, Building and Deploying Messaging Solutions,

11 EAI Introduction Use of Tooling Depends on Complexity of Integration API Apache Camel Integration Framework Enterprise Service Bus (ESB) Integration Suite low - Connectivity Routing Transformation Integration Tooling Monitoring Support BPM high Big Data Mgmt, MDM Registry Repository Rule Engine Complexity of Integration Source: 10

12 Market View - Gartner Magic Quadrant Magic Quadrant for Application Infrastructure for Systematic SOA Infrastructure Comment on Open Source Products ( ) All open source vendors are in the visionary quadrant. In some cases their offerings are more modern than the Leaders products, since they are hardly burdened with backward compatibility issues ( ) BAKUP ( ) In general the open source platforms are less expensive and easier to implement and deploy. However their offerings are generally less comprehensive than the Leaders offerings. If these offerings fit your requirements this could be an easy-to-use/low-cost SOA infrastructure for your organization. The open source platforms are a strong technology offering. ( ) Source: 11

13 Open Source EAI Alternatives Contain Integration Frameworks and ESB s SIMILARITIES DIFFERENCES Do not reinvent the integration wheel. There are a lot of great frameworks available integration Open Source Frameworks Use of EPI s (Enterprise Integration Patterns) Based on DSL (Domain Specific Language) Pipes and Filters Concept APIs few libraries necessary Transactions Concurrency Error Handling Monitoring Testability A lot of connectors available, easy to creat new ones via Maven archetypes Many B2B connectors (SAP, Sage Microsoft Dynamics, salesforce, etc.) Largest community, best concepts Easy to creat new connections Enables complex integration (tooling, monitoring, support, etc.) Many connectors/ B2b Connections available Different functionality/ tooling provided by each ESB Integration Framework ESB Source: 12

14 Apache Camel: Programming Languages Growth of Code, Comments and Blank Lines Total Lines: 1,397,548 Number of Languages: 15 Code Lines: 855,399 Total Comment Lines: 357,134 Total Blank Lines: 185,015 Percent Code Lines: 61,2% Percent Comment Lines: 25,6% Percent Blank Lines: 13,2% Source: /06/18 13

15 Apache Camel: Language Breakdown Language Break Down Source: /06/18 14

16 Apache Camel: All-Time to Monthly Commit Structure All Time 12 Month 30 Day Commits: 17,816 2, Contributors: Files Modified: 22,922 4, Lines Added: 2,253, ,145 36,476 Lines Removed: 855,364 58,641 5,495 Commits per Month Source: /06/18 15

17 Apache Camel: Contributors and Distribution Number of Contributors Commits by Top Contributors Source: /06/18 16

18 Agenda Motivation and Background Introduction of EAI and Camel Excursion: Event Processing Building Blocks Use Cases and Implementation using Camel Conclusion and Q&A 17

19 Excursion: Event Processing Building Blocks Event channel Event producer Event type Event consumer Event agent Context State Source: Etzion, Nibblet, Event Processing in Action, p

20 Excursion: Event Type and Event Object Events are a representation for something which has happened. As a computational element for the technical representation, an event object is a concrete instance of an event type. All event objects, which belong to an event type, carry the same semantics within their object structure. Source: Etzion, Nibblet, Event Processing in Action, p

21 Event Types and Objects An Object is an Instantiation of a Type Event Type (Payload) <?xml version="1.0" encoding="utf-8"?> <xsd:schema xmlns= > <xsd:element name="product" type="tns:producttype" /> <xsd:complextype name="producttype"> <xsd:sequence> <xsd:element name="name" type="xsd:string" /> <xsd:element name="size" type="xsd:string" /> <xsd:element name="weight" type="xsd:string" /> <xsd:element name="price" type="xsd:double" /> </xsd:sequence> </xsd:complextype> </xsd:schema> INSTANTIATES Event Object (Payload) <?xml version="1.0" encoding="utf-8"?> <tns:product xmlns:tns=" " xsi:schemalocation=" "> <tns:name>parcel</tns:name> <tns:size>max. 60x30x15 cm</tns:size> <tns:weight>up to 2 kg</tns:weight> <tns:price>4.90</tns:price> </tns:product> 20

22 Event Type Model Header, Payload and Open Content EVENT HEADER PAYLOAD OPEN CONTENT HEADER Type identifier Event composition Temporal granularity EVENT INDICATORS Occurrence time Detection time Event source Identity Certainty Source: Etzion, Nibblet, Event Processing in Action, p

23 Camel s Message Model Maps to the Event Type Model Exchange Exchange ID MEP Exception Properties Event Producer In Message Headers Out Message Headers Event Consumer Attachements Attachements Body Body Source: Ibsen, Anstey: Camel in Action, p.14 22

24 Excursion: Event Producer, Consumer and Event Channel Producer and consumer are both event processing elements that communicate with each other. While an event producer obtains the role for generating events, the event consumer receives the results. An event channel fulfills the mediation role between producer and consumer. It s main task is to route events from the producer to the appropriate consumer. Source: Etzion, Nibblet, Event Processing in Action, p

25 Producer, Consumer, Channel Component Concept Provides Abstraction Level Component - component 0..1 Endpoint endpoint Producer - endpoint Consumer 24

26 Camel Endpoints Referencing Component Instances SCHEME CONTEXT PATH OPTIONS.to("log://translate-route?level=INFO") SCHEME Instantiates component Examples are: AMQP HTTP JMS Log NETTY SEDA SMTP POP QUARTZ SIP CONTEXT PATH Identifies the endpoint within the Camel Context in combination with the scheme uniquely Serves as runtime reference OPTIONS Component specific configuration parameters Log component: Log Level Formatting Mail Component Host Credentials 25

27 Excursion: Event Processing Network An event processing network combines all processing elements, notably event producer and consumer, event channel, event processing agent including context and state elements into an executable application unit. This does not necessarily imply that all elements need to be executed within a single execution node. Source: Etzion, Nibblet, Event Processing in Action, p

28 Event Processing Network Camel Context and Routes Keep the Network Together Context Route Route Route Endpoint Endpoint Endpoint Endpoint Endpoint Endpoint Serves as a container at runtime level Provides the Camel core services such as Registry, Type converter, Components, Endpoints, Routes, Data formats and Languages Realizes a concrete implementation of a message flow Is executed on Camel s routing engine Multiple routes can exist within a context at the same time Endpoint connects to event producer, consumer and channel Provides the reference to a specific component instantiation CONTEXT ROUTE ENDPOINT 27

29 Event Processing Network Processors Provide Further Processing Logic Camel Context Routing Engine Route 1 Processors Message Filter Route 2 Route 3 Content-based Router Endpoints Message Aggrega. Components File HTTP JMS Source: Ibsen, Anstey: Camel in Action, p.15 28

30 Event Processing Network Example of a Spring-based Camel Context <?xml version="1.0" encoding="utf-8"?> <beans xmlns=" xmlns:camel=" xmlns:context=" xmlns:xsi=" xsi:schemalocation="..."> <camel:camelcontext id="camelcontext"> <camel:route id="camelroute"> <camel:from uri="seda://shipmentorder?multipleconsumers=false"/> <camel:to uri="log://camelcontext/camelroute?level=info"/> </camel:route> </camel:camelcontext> </beans> 29

31 Excursion: Event Processing Agents, Context and State An agent is a software component with the purpose of processing events. The main functionalities of an agent can be summarized as filtering, matching and derivation with support of context and state information. The processing of context and state is required for event reasoning which means the detection of any conditions that lead to the event generation. Source: Etzion, Nibblet, Event Processing in Action, p

32 Pattern Name Overview Pattern Definitions: Derived Pattern Structure Context Problem Solution Example 31

33 Example Process of Industry Domain: Parcel & Delivery Logistics Order Management Delivery Disposition Delivery Execution Order Order System Disposition System Tracking System Invoic. Placement of transportation orders Schedules the parcel delivery Executes the physical delivery Executes monthly based invoicing Covers multiple distribution channels Defines the size of delivery area Creates transportation routes Provides status information on pickup, sortation, transportation and delivery Supports customer care activities 32

34 Agenda Motivation and Background Introduction of EAI and Camel Excursion: Event Processing Building Blocks Use Cases and Implementation using Camel Conclusion and Q&A 33

35 Agent Implementation Realizes Common Integration Patterns Filter Agent Split Agent Translate Agent Aggregation Agent Enrich Agent Source: Etzion, Nibblet, Event Processing in Action, p

36 Filter Agent Definition Performs filtering only Has no matching or derivation steps Utilizes a filter expression to eliminate uninteresting events Processes events in a stateless manner Source: Etzion, Nibblet, Event Processing in Action, p

37 Filter Agent Pattern: Context and Problem Context Problem Automated sorting process produces many scan events Scan events contain routing information to deliver to the requested destination Produced amount of scanned data is not suitable to be distributed to all logistics nodes Flexibility of data distribution is required since delivery areas may differ based on the use of capacity 36

38 Filter Agent Pattern: Solution Messages Filter Messages Solution Events will be published to a messaging topic Depots and hubs, the event consumer, can register at the topic The nodes filter the events that have been sent to that topic by applying filter expressions derived from the calculated routing rules The processing nodes can therefore assure to retrieve only events of interest to them 37

39 Filter Agent Pattern: Example 38

40 Split Agent Definition Takes a single incoming event and emits a stream of multiple event objects Can be used for partitioning and event distribution to different consumers Source: Etzion, Nibblet, Event Processing in Action, p

41 Split Agent Pattern: Context and Problem Context Problem Customers need to support bulk processing of orders Bulk of orders support distributed manufacturing sites The assignment of the bulk to a single distribution node is not possible Orders have to be split and processed by the nearest logistics hub 40

42 Split Agent Pattern: Solution Solution Partitioning the bulk of orders into small chunks for redistribution Each order contains a single sender recipient information identified by their addresses The partitioning will be conducted by a split operation on the incoming order collection 41

43 Split Agent Pattern: Example 42

44 Translate Agent Definition Takes a single incoming event and generates a single derived event as output Is stateless Used for simple conversions to complex event transformation modifying event attributes Source: Etzion, Nibblet, Event Processing in Action, p

45 Translate Agent Pattern: Context and Problem Context Problem Conducting a technical event transformation is required to map incoming XML objects to Java POJO s Attributes need to be remapped to match the specified interface structures The original data needs therefore be transformed into a new data set 44

46 Translate Agent Pattern: Solution Solution Retrieve the initial message XML data as InputStream Instantiate a new enriched message of type Product.class Map the data from the initial message on to the enriched message 45

47 Translate Agent Pattern: Example 46

48 Aggregate Agent Definition Takes as input a collection of events and creates a single derived event Input as collection of events it is not within the meaning of a Java collection and is therefore not a single object that embraces multiple objects Operates on multiple input events and generates a single output event Sometimes requires an aggregation function such as calculating Source: Etzion, Nibblet, Event Processing in Action, p

49 Aggregate Agent Pattern: Context and Problem Context Problem Competitors change their prices It is required to respond quickly in an automated manner upon any price change Based on this information it is possible to react with it s own product prices 48

50 Aggregate Agent Pattern: Solution Solution To enable the reaction upon any price change it is necessary to aggregate the price information Calculating all market prices of competitors as soon as they are available enables an immediate reaction on those information The solution is implemented as an aggregation strategy to hide the calculation coplexity 49

51 Aggregate Agent Pattern: Example 50

52 Enrich Agent Pattern: Definition Definition Takes a single input event Uses it to query data from a global state element Creates a derived output event which includes the attributes from the original event, possibly with modified values May include additional attributes. Source: Etzion, Nibblet, Event Processing in Action, p

53 Enrich Agent Pattern: Context and Problem Context Problem To calculated an accurate bill it is required to have some product classification available The product classification relates directly to the product prices that can be added to the bill Since the information is not directly available it needs to be derived from an additional data source 52

54 Enrich Agent Pattern: Solution Enricher Initial Message Enriched Message Resource Solution The initial message needs to contains some reference data to access an additional resource The resource provides additional information to enrich the original message As a result of the enrichment, the output message contains additional information in contrast to the original message 53

55 Enrich Agent Pattern: Example 54

56 Agenda Motivation and Background Introduction of EAI and Camel Excursion: Event Processing Building Blocks Use Cases and Implementation using Camel Conclusion and Q&A 55

57 Conclusion and Outlook Conclusion Apache Camel supports the event processing concept It provides all features to realize: Event types and objects Event producer, consumer and channel Event processing networks In case of missing functionalities, e.g. statistical reasoning, the framework is capable to be easily extended Outlook Apache Camel has the potential to serve as integration component for the data cloud It supports already a huge amount of modern technologies Adopting Camel for your project needs reduces the risk of repeating work DON T REINVENT THE WHEEL 56

58 Q&A 57

59 Recommended Reading 58

60 Thank You for Your Attention Christoph Emmersberger Universität Regensburg Universitätsstraße Regensburg Florian Springer Senacor Technologies AG Wieseneckstraße Schwaig b. Nürnberg