Oracle SOA Suite/B2B as a Critical Mission Hub for a High Volume Message Use Case Introduction Stop. Think. Ok, in the meanwhile 2 seconds has passed and 250 messages more were processed by a mission critical hub built with Oracle B2B and SOA Suite which connects thousands of trading partners and processes millions of messages per day, handling 40% of Global Air cargo Traffic. In this article, you will find described high availability solution architecture, covering B2B and core SOA Suite components as BPEL, along with Business Rules, Mediator and BAM integration, as well as lessons learned in conducting such complex and mission-critical project starting from a set of legacy applications. Imagine now how many messages were processed when you finish reading it. Mission and scope The mission was to build a high available and performant message hub between different intervenients in the air cargo industry integrating 15 thousand trading partners exchanging 3 Billion messages every year and execute complex document validation, multi-factor identification, correlation and batching, dynamic routing, transformations and monitoring. The document standards involved elevated B2B core product for the implementation due to its enhanced features available for document management. Main document standards are Air Cargo market defined standards as IATA - Cargo Interchange Message Procedures (CIMP), EDI CargoFact and CargoXML, as long other customer tailored documents based in FlatFile, XML and other custom format totalling 238 different document types. Another important and distinctive requirement was the relation between the envelope types in the exchanged messages. In this respect, except the
standardize formats EDI, XML that implicitly define the envelope type to follow, all the others, including CIMP, are decoupled from the type of envelope that can wrap the message, and we are talking about 99 different and customizable envelope types built to be used with any document format. The transportation channels that can be used for message exchange are quite vast as well: different trading partner can communicate with the platform using 8 different transportation protocols, both for inbound and outbound. All of these enhancements were done, never neglecting the flexibility that allows evolving the platform for on-boarding customers, new message and envelope types achieving the desirable expansibility needed in a constant change market wherein the velocity to add capabilities is preponderant. Oracle B2B and SOA Suite Nice to meet you Due to its strong features, Oracle B2B was elected as the core product for this type of implementation together with SOA suite components as BPEL, Mediator and Business Rules for back-ended operations. It thus provide a centralized platform for document, partners and agreements management as a reliable platform for document exchange between business partners. The following features are per se the reasons to elect B2B as centre of this implemented architecture: Multiple document protocol support as EDI, XML, Flat File and Custom-Defined Document definition tools available with possibility to implement rich and complex validation rules - Oracle Document Editor + Custom Editors for custom documents Document definition design time testing Document translation and validation Ability to set document callouts to handle custom document
Ability to set Channel callouts for custom inbound/outbound message processing Document batching Message resubmission capabilities Message store-and-forward capabilities Complex message identification and correlation Complex sender and recipient identification based in different envelope types Automatic message retry Support for large documents Message acknowledge support Messaging security enablement (EDI) Multiple envelope support Envelope generation Transport management Multiple transport protocols supported HTTP, SMTP, SOAP, TCP, JMS, MQ, FILE, FTP Trading partner management Trading partner downtime Native integration with Oracle SOA suite and Oracle BAM API for message and Trading Partner metadata access E2E Message monitoring and auditing Message metrics dashboards and reports Partner Agreements User Access management Self Service Provisioning Exception handling On the document management side, it s the ideal platform to leverage business standard documents and manage agreements. However, Oracle B2B acts as a gateway requiring architecture to manage the complete end-to-end business processes. This is where Oracle SOA Suite Service Component Architecture (SCA) components come into picture. Oracle BPEL, Mediator and Business Rules play the major role as back-end pieces in the global architecture. Oracle B2B is in fact a component part of Oracle SOA Suite what makes the integration quite simple and efficient. The integration between both components is ensured via out-of-the-box adapter implemented as binding component in a SCA. All of this leveraged by the use of SOA Metadata Services repository (MDS) allowing the sharing of different artefacts, as XML schemas or WSDL files among different architectural components as central repository for B2B artefacts and configuration. Service Composite Applications components were ultimate to implement the following capacities: Routing and document handling business rules Service orchestration Message enrichment Message canonicalization/normalization Message duplication Mapping and transformations
Document conversion Message splitting Message rejection Message deferring Message acknowledge Legacy system integration Exception handling Oracle B2B and SOA Suite, together provide a natural and integrated architecture that enables a unified platform with end-to-end instance capabilities, empowering standardization, governance, and security. Solution architecture B2B as a hub assumes that it is acting both as inbound and outbound gateway sharing the same configuration for both directions. Trading partners are connected via inbound listening channels and trading partner channels for delivery. In the message exchange process, B2B is responsible for identifying both the sender and recipient(s) of the message, identifying the message type and verifying if the sender is active to send such type of message, if the recipient is able to receive it and in which format it is able to. Message validation and parsing (from raw to XML) for inbound messages and message construction (from XML to raw) for outbound are also performed on B2B domain. The host trading partner will be the virtual receiver of all incoming documents and the virtual sender of all outgoing documents eliminating any point to point relation between trading partners and documents. All of this consolidates the possibility of document normalization, allowing the definition of message classes. This facilitates the usage of common objects between message types and formats implemented in a canonical model eliminating point-to-point transformations and allowing putting in place a mechanism of global mappings from the message format to canonical structure and from canonical structure to the message format.
A native integration between Oracle B2B and Oracle SOA SCA allows covering all the necessary requirements. In the picture bellow it s shown a correspondence between components domains and functionalities succeeding the message processing timeline. Apart of the functionalities mapped to B2B and SOA SCA, its integration across the SOA platform allows to naturally implementing an end to end process tracking and exception handling having all exceptions being delivered to SCA to be handled in properly. In this case in particular, the solution is covering acknowledge to the sender that the message failed and the reasons behind the exception. This is an extremely valuable solution feature, since most of the exceptions are typically generated from invalid documents. This allows the sender to correct and fix the messages interactively reducing the time needed to implement such corrections. This is also helps the customers to fit to the market standards reducing custom made message definitions. Inbound Transport callouts are assigned to Inbound/Outbound B2B channels responsible for parsing the envelope to a B2B Internal Properties native structure. Then, two possible paths can be followed for document parsing: Native xengine used for supported documents Document Callouts for custom documents processing
This functionality offers the possibility to implement custom classes to deal with document types that are not covered out-of-the-box. In this implementation, a custom parser was implemented to generate XSD and JAVA classes from standard language Augmented Backus Naur Form i (ABNF) that can be associated as B2B Document Callout to dynamically execute parsing, construction and validation of documents. Please, bear in mind that a good architecture of this callouts is paramount to guarantee full performance answer when processing such documents. SCA Core processing The message is delivered via B2BAdapter to a composite and so entering the core domain of the solution. It is then in this domain that the message is handled and all the necessary core functionalities are provided: Mapping from message type structure to canonical structure. Normalizing the message allows to reuse functionalities that become agnostic to every different message type to be processed. Those mappings are implemented using XSLTs that are read dynamically from the MDS Business rules are verified and deliver an outcome list of functionalities that affects the message exchange processing The message is orchestrated to the different core functionalities guided by rules Communication with the B2B repository for enrichment and operations using the API provided The message is mapped from canonical structure to the message type structure and delivered via B2BAdapter to B2B Outbound Similar to the inbound. the message and envelope are constructed using the xengine or Document callout and delivered to the final recipient using the configured trading partner channel. BAM Integration The integration between the components and BAM is established on the B2B side by native integration based in Advance Queue and from SOA SCA via sensors. Exception Handling The exception handling implemented transversally allows end-to-end coverage of exceptions. B2B exceptions are communicated natively to a SCA via B2B exception handling queue. EDN infrastructure together with B2B Exception handling and SOA Fault Handling Framework is used for exception handling purpose guaranteeing 100% message reliability on the platform. What about performance? For performance, a full implementation of the message processing without the need to set any database connection was defined leveraging all the configuration and persistence on the Oracle SOA Infra; This significantly reduced the interactions with the DB.
Cherry picking the components to reduce dehydration and persistence allowed to maintain reliability and adequate performance. On composites side a one-way-delivery policy exchange pattern was defined as a core pattern allowing the composites to be fine-tuned in order to reduce the persistence and improve performance. Since B2B and SOA Suite SCA were running in the same domain, the integration adapter between B2B and SCA was configured to use in-memory methods; a very effective option in terms of performance and reliability. Other options as JMS or AQ can be used in case B2B and SOA are not running in the same domain. Bottom line This was a demanding project but represents an important challenge for a platform that excels in all of the full capabilities of Oracle B2B. It was a winning strategy since most of core functionalities were presented and thanks to a close collaboration between implementation team and Oracle s Product Management and Development teams elevated B2B to a more complete solution meeting recent market demands. Success Factors: Standardization of messages supported by the platform permitted reduction of costs compared to custom tailor made messages oriented solution Reducing consumption of invalid messages allowed freeing up of physical resources resulting in a more scalable platform targeted for a greater return on investment Flexibility became key and the users are now empowered using graphical tools for provisioning new documents definitions using Oracle B2B Document Editor and XSLT to perform the canonicalization of those messages Bringing standardization and normalization on the messages allowed the customer to agilize the process of onboarding and reduce costs, providing simultaneously better service to their clients The platform is 100% Cargo-XML ready IATA is pushing to use this format rather than old fashioned CIMP standard; an IATA Cargo-XML Task Force initiative now exists to force this move Final remarks Access all of these Oracle B2B functionalities on Oracle SOA Suite 12c or, for the version 11g, by downloading and installing the SOA bundle Patch 19190139 11.1.1.7.5 or the merge patch 18952479 via My Oracle Support. For more details and guidance on how to use some of this B2B functionalities feel free to have a look at contents in one of our blogs: http://fusionbpmsoa.blogspot.co.uk/
About the Author Bruno Neves Alves, Partner, Technical Director - Member of the CTO Office, eproseed UK Working with Oracle Fusion Middleware technologies for 10 years across Europe and South America, Bruno holds several Oracle certifications and is deeply experienced as SOA/BPM Senior Consultant, Solution Architect, Team Leader and Presales. Currently performs the role of Technical Director as part of the CTO Office and is Partner of eproseed UK. Bruno can be contacted via bruno.neves.alves@eproseed.com or +44(0)7765190419 i ABNF Meta-language often used as definition language for rules associated with document exchange http://en.wikipedia.org/wiki/augmented_backus%e2%80%93naur_form