Supportability and Operation of Microsoft BizTalk Server

Supportability and Operation of Microsoft BizTalk Server

2012 Motion10 motion10.nl Table of Contents 2 4 Introduction 5 Requirements Phase 7 Design Phase 13 Planning Phase 14 Deployment Phase 15 Validation Phase 18 Operation Phase 22 Tools 25 Checklist 26 Conclusion 27 The Author / Motion10

3 Supportability and Operations of Microsoft BizTalk Server Steef-Jan Wiggers December 2012 Having a stable, robust, and highly available BizTalk Server environment is important. Many times when BizTalk Server is installed and solutions are architected, designed and developed, the environment has not been implemented with supportability and operations in mind. System boundaries are reached fast and solutions do not perform well or make BizTalk throttle, as a result of excessive memory use or flooding of the system with messages, etc. Furthermore, it is often hard to find the root cause of problems arising in production environments. There are many ways to prevent these kind of situations. For example by using the right architecture, design and development approach and by using the available validation and monitoring tools. This, together with applying some of the best practices provided by Microsoft and the effective collaboration between system, network, database and BizTalk administrators with the integration team can greatly help increase supportability of BizTalk Server based integration solutions.

Introduction 2012 Motion10 motion10.nl 4 Thinking about supportability of BizTalk from the very beginning should be one of the best practices that organizations need to implement. Because the team responsible for supporting the BizTalk Server environment is not involved at the beginning, during the deployment of the environment or are pulled into a project at the very end, in many projects the supportability is overlooked or not part of the initial project plan. This will lead to increasing costs of supporting the BizTalk environment as the administrators are ill prepared, poorly trained or may lack the necessary skills, and even worse given a poorly performing, not well designed and/or unhealthy BizTalk environment to support. BizTalk environment from becoming a SPOF (single point of failure) and it also enables you to better prepare your administrators. They will have to support the BizTalk environment and its infrastructure in the future. To prepare your integration middleware environment for supportability, you have to go through the following steps: Requirements Design Planning Deployment Validation Operation This white paper will provide you with a process you can follow up, the necessary set of tools and guidance to implement a sustainable and robust BizTalk environment that can be efficiently supported by your technical staff. Knowing your environment and its boundaries will provide insight in how to prevent your Requirements In the next sections details for each of the process steps will be described. The process involves collaboration of system, network, database and BizTalk administrator, each with his/her own responsibilities, and however combined, they are responsible for the complete BizTalk environment. The Operation phase is crucial for the supportability of BizTalk Server. Design Planning Figure 1 To prepare your integration middleware environment for supportability, you have to go through these steps Deployment Validation Operation

5 Requirements Steef-Jan Wiggers December 2012 Phase When a new BizTalk project kicks off, the business analyst typically focuses on the functional side of the project. There will normally be very little focus on the non-functional requirements like supportability, Disaster Recovery (DR), high availability, operations, monitoring, etc. As soon as the requirement analysis is started for your BizTalk project, it s good practice to involve one of your BizTalk administrators and start thinking about the non-functional requirements. Setting up a robust, highly available BizTalk environment will be a time consuming process. Think about the hardware procurement process, cluster and load balancing setup, certificate requests procedures, etc. The focus of this first process step is to assess all the requirements, having a clear view and understanding of them. This will in the end enable the deployment of the required BizTalk environment that meets expectations. Here s what you need to think about: Expected BizTalk work load(s) in messages (size, frequency) Non-functional / QoS (availability, scalability, performance, etc.) Licensing (software) Hardware Virtualization Storage and networking Types of monitoring and monitoring tools Different types of documentation Procedures checklist Development, Test, Acceptance, and Production (DTAP) environment Tracking/Tracing/Logging Data retention, archiving and purging policies Hosting of the BizTalk environment (inhouse, hosted or IaaS in the cloud) Security Exception Management Supportability Operations Management Application Management Incident Management During workshops you will discuss the BizTalk architecture (topology and patterns), integration landscape, functional and non-functional requirements with your stakeholders. You will have a session with the business first and subsequently have one or more with the IT department depending on the size and structure of the (IT) organization. Each enterprise will have a different set of requirements for integration of data and processes than the other. Yet, the topics will remain the same as listed below: Accessibility of data Manageability of processes Process automation Integration of systems and applications Supportability Documentation Procedures checklist As a follow-up to the business requirements you can easily assess the workload for the BizTalk environment and what components are required (i.e. Orchestration, Adapters, Business Activity Monitoring, Business Rules Engine, and so on).

2012 Motion10 motion10.nl 6 An important factor to reckon with when talking to the business is risks associated with transitioning to the BizTalk platform. This risk can be of a technical, operational, political, and financial nature. Therefore it is important that the business is aware and accepts the costs that come with the investment. This should include investment in supportability of BizTalk Server. With IT, you focus more on the technical side of the BizTalk Environment such as, What message types, sizes, formats, and encodings are sent to the BizTalk system or what does it need to output?, Maximum amount of messages?, Or Are there peak loads during the day? You should consider security around it, especially when information going to or coming from external trading partners is confidential. Encryption and decryption of data, signing of messages and transport security are going to be discussed. Besides the exchange of message you will also discuss process automation: What processes that are going to be automated need to interact with internal and external systems? Or supportability matters like: How are you going to monitor messages that are going in and out?. As part of the requirement analysis phase, you should also pay attention to the number of non-production QA environments. You should also focus on how the solution is going to be deployed in multiple QA environments before getting into production. It s better to think about deployment automations in the early phase of the project and develop and improve your deployment process along with your main solution. The results of the workshops can be processed in an Architecture and High Level Design, which can be discussed with IT- and Business Stakeholders. When stakeholders agree upon this design you can proceed to create a detailed design that can be fine-tuned with IT.

7 Design Phase Steef-Jan Wiggers December 2012 Analyzing requirements and creating a detailed design for the BizTalk landscape is the next step forward before planning and installation. With the gathered requirements, you can make decisions on how to architect and design a BizTalk environment. If BizTalk is used for the first time or when migrating to a newer version, an enterprise environment capacity planning and server allocation is something to focus on. You need to ensure that enough hardware is available. Once you have gathered the requirements, you will have a clear picture of where the platform will be hosted and/or whether it needs to be scaled up or out. If everything gets placed on one big server, it will introduce a serious single point of failure. The same applies to running multiple virtual servers on one big physical server with the intention to implement a failover scenario. This should be avoided. Separating your BizTalk Server instance(s) from the SQL Server(s) is considered a best practice; therefore, this should be the first thing you will put down in your design. Preferably each BizTalk Server instance and the database server(s) should be installed on a separate piece of hardware. In the next few paragraphs some of the critical design decisions are described. One of the decisions will be on setting up efficient and effective support for the BizTalk environment. Availability Depending on availability requirements, you will probably cluster the SQL Server. This SQL Server will run on a separate piece of hardware. Besides that, you can choose to scale out BizTalk into a multi-server group, because of availability requirements (see next paragraph). If the expected load cannot be handled by one BizTalk instance, you will need more to handle the load. And when you need high availability for pull adapters like FTP, MSMQ or POP3 than the (Windows) clustering of hosts is required as well. This means that for high availability you will need at least two BizTalk instances (not necessarily clustered instances). Also, in case you require load balancing of incoming HTTP traffic you will need at least two BizTalk instances. Note here that in a highly available environment you will need four servers if you are using Microsoft Cluster Server (MSCS) and Network Load Balancing (NLB): Two for the windows cluster Two for NLB, due to the fact that Microsoft doesn t support NLB and MSCS running on the same servers. Scaling The first thing you should decide is whether or not you want to virtualize your environment, by doing so you can lower infrastructure costs by taking advantage of Windows Server 2008 s Hyper-V Virtualization capabilities to run unlimited virtualized instances of BizTalk Server and also have some flexibility and functionality like the ability to define multiple distinct logical

2012 Motion10 motion10.nl 8 security boundaries on a single physical computer, fault tolerance support through Hyper-V clustering, ease of scale-out and so on. BizTalk environments provide various options for scaling when your load increases. One of the starting points will be to cluster the SQL server layer with multiple SQL instances, which provides both high availability as well as load spreading across multiple instances for the different databases needed by BizTalk Server such as BAM and the MessageBox. When validating the BizTalk environment you will assess the load by performing a benchmark. During the design you can opt for installing BizTalk and SQL separately first and then scale-out after performing the benchmark tests. Scaling can be done vertically (scale up) by increasing the number of processors and the amount of memory each server uses, or horizontally (scale out) by adding more servers to your BizTalk Server configuration. Other options you can consider during your design are: Separating the BizTalk databases, which you will need to do when having multiple MessageBox database or when you rely more on tracking capabilities of BizTalk (separate Tracking and BAM from the MessageBox) Adding multiple MessageBox databases, when the primary message box becomes a bottleneck in high throughput scenarios (this usually makes sense when deploying more than 4 BizTalk Server instances) Separating hosts to tune for low latency, performance and high throughput when specific requirements in that area need to be met per BizTalk Application. Based on the requirements you can consider isolating the BizTalk hosts to be able to better manage and monitor the BizTalk applications and divide the load. By separating send, receive, tracking and processing functionality in different hosts you will benefit from better memory and thread management, capabilities to alter thresholds, and the possibility to quickly stop receiving, send, orchestration or tracking. Security Another critical requirement within a BizTalk environment is security. BizTalk security can be enhanced by deploying Secure Socket Layer (SSL), IPSec Tunneling, Forefront Threat Management Gateway and certificate services included with the Windows Server operating system. With BizTalk Server you can apply access control and implement least rights to limit access, and provide integrated security through Enterprise Single Sign-On. Furthermore you can protect and secure applications and data by authenticating the sender of a message and authorizing the receiver of a message. In case dedicated hosts for send, receive, processing and tracking are setup, individual host instance accounts are required. This prevents that for instance send hosts can interfere with hosts on the receive side. Microsoft provides extensive guidance on securing your integration middleware, see Securing BizTalk Server http://msdn.microsoft. com/en-us/library/aa561593.aspx Tracking Tracking in BizTalk of messages can be useful to be able to see what messages come in and go out of the system, or for auditing, troubleshooting or archiving purposes. Tracking of messages within BizTalk is a process by which parts of a message like body, properties, and metadata are stored in a database. These parts can be viewed by running queries from the Group Hub page in the BizTalk Server Administration console.

9 There are some considerations to make regarding tracking. Tracking everything is not the smartest thing to do, as each time a message is touched in BizTalk; a copy is made and stored in the database. For example, focus on scope by activating tracking only on a specific (set of) port(s), which is better for performance and keeps the database uncluttered. For the latter it is important that the DTA Purge and Archive job is configured properly. It is worth considering having a dedicated host for tracking for each MessageBox database and one extra in a multi-server environment. When introducing BizTalk into an environment Tracking can be critical. There may be political issues with BizTalk being the Middleware. Sometime it seems like everyone blames the guy in the middle. So a consideration may be that over-tracking during an initial deployment may be beneficial and once the solution is in place and stable the tracking may be reduced. This can be a delicate area to discuss, especially with the database folks. However you can over-track early on and then tune it down to find a good balance. Another critical consideration you must decide on is the type of tracking you want to implement: regular tracking (BizTalkDTADb), Business Activity Monitoring (BAM) or both. There are several reasons why you should consider using BAM as your primary tracking source, rather than the regular tracking: The BAM infrastructure allows purging and partitioning of tables while updating views allows you to access the data from all tables Filtering for promoted properties in a large Tracking database most often results in a database time out exception The Tracking database does not come with any support for Analysis Services Enabling BAM Tracking does not take much effort BAM can be used for multi-dimensional tracking purposes Steef-Jan Wiggers December 2012 A combination of the two technologies can also be implemented. Using DTA for tracking message bodies (if necessary) and meta data and using BAM for tracking milestones and specific message elements and combining the two in one dashboard is done quite often. Fault Tolerance and Load balancing Fault tolerance and load balancing for BizTalk can be achieved through clustering and separating hosts as described earlier. Implement a storage area network (SAN) to house the BizTalk Server databases, cluster Enterprise Single Sign-On (SSO) master secret server, and configure the Internet Information Services (IIS) web server for isolated host instances and the BAM Portal Web page to be highly available using Network Load Balancing (NLB) or other load balancing device. Hosts running the receive handler for the HTTP, SOAP, and BizTalk Message Queuing (MSMQ) adapters require a load-balancing mechanism such as Network Load Balancing (NLB) to provide high availability. This would mean two servers for supporting NLB that do not have MSCS installed. Host cluster support is provided to support high availability for integrated BizTalk adapters that should not be run in multiple host instances simultaneously; such as the FTP receive handler, MSMQT or, under certain circumstances, the POP3 receive handler. Everything else just does not require any additional clustering or load-balancing mechanism because BizTalk Server automatically distributes workload across multiple computers through host instances, unless you have made (faulty) changes to the hosts themselves (throttling). Licensing An important factor in any BizTalk Server deployment is cost. Licensing has a large impact on cost and is mainly driven by the non-functional requirements an enterprise has. For BizTalk you can choose Branch, Standard

2012 Motion10 motion10.nl 10 or Enterprise Edition (Developer Edition may only be used in a non - production environment so you can use this free version in Quality environments as long as it is not used in a (back-up) production scenario). processor is assumed to have the same number of cores as the physical processor. Using less than the number of cores available in the physical processor still counts as a full virtual processor. The editions differ in price, but also in functionality. With the standard edition it is not possible to support scenarios for high availability (because it can only run on one machine) and it is limited to two CPUs (in the same machine) and the maximum number of applications is 5. The latter does not often seem like a problem, but it often severely hampers you in supportability. If you do not have an SLA with 99.99% uptime, but only the continuity of the service, it is in some scenarios possible to use a standard edition and have an identical machine in cold stand-by mode in case of failure or shutdown. The Branch edition has even more limitations and is designed for hub and spoke deployment scenarios including RFID. With any version you probably want to consider whether or not to virtualize. With Virtualization in mind licensing can be difficult. With standard edition you need a license for each virtual processor used by the virtual OS Environment, regardless of whether the number of virtual processors is less than or greater than the number of physical processors on the server. With Enterprise Edition, if you license all physical CPU s on the server you can run any number of instances in the physical or virtual OS Environment. With both of these, a virtual Support Last but not least you need to consider how to support your BizTalk environment. Monitoring is one aspect of supporting your BizTalk environment. There are however other factors that are important to have a healthy support for BizTalk. These factors will be described in the following paragraphs. Monitoring Good monitoring practices are important not only to keep the system healthy but also its applications. However, this is a complex and knowledge-intensive operation. Mostly because of the fact that an integration middleware infrastructure is built on top of different products such as Windows Server, IIS, SQL Server, BizTalk Server and so on. The BizTalk Server infrastructure can vary from organization to organization, from the use of a single server (standalone station) or robust platforms with the use of server clusters for Biz- Talk Server and SQL Server. The figure below gives an overview of the different technical layers that you need to monitor in this kind of platform. Services and Applications Web Services, WCF, BizTalk Applications BizTalk Server Platform BizTalk Server, SQL Server, ILS, WSB, Active Directory Infrastructure Figure 2 An overview of the different technical layers that you need to monitor in this kind of platform Operating System Server Hardware Firewall, MS DTC, COM+ Network access Physical or virtual, memory, storage, cpu Network or storage

11 It s worth considering System Center Operations Manager to monitor your BizTalk environment using management packs for SQL Server, Windows Server and BizTalk Server. The Management pack for BizTalk Server provides two views, one for the enterprise IT administrator and one for the BizTalk Server administrator. The Enterprise IT Administrator will be monitoring the state and the health of the various enterprise deployments on the machines hosting the SQL Server databases, machines hosting the Enterprise Single Sign-On (ENTS- SO) service, host instance machines, IIS, and network services. And he is interested in the overall health of the physical deployment of a BizTalk Server setup. The BizTalk Server Administrator will be monitoring the state and the health of various BizTalk Server application artifacts such as orchestrations, send ports, receive locations, and he is interested in monitoring and tracking BizTalk Server health. If necessary he can carry out corrective measures to keep applications running as expected. If SCOM is not already used in the organization, it may be a bit overkill to invest in SCOM purely for BizTalk environment monitoring. In situations like this it may be better to take advantage of the existing monitoring solutions available in the organization like HP Operations Manager or IBM Tivoli or purchase something like Biz- Talk360 which is purely designed for monitoring BizTalk solutions and easy to implement. A BizTalk application can be monitored (from a functional interface perspective) using Business Activity Monitoring (BAM). BAM will give you end-to-end visibility into your business processes, providing accurate information about the status and results of various operations, processes and transactions. The BAM feature is included with BizTalk Server and provides a collection of tools that allow you to manage aggregations, alerts, and profiles to monitor relevant business metrics (called Key Performance Indicators, or KPIs). Steef-Jan Wiggers December 2012 Recoverability Microsoft BizTalk Server databases and the health of the databases are very important for a sustainable BizTalk Server messaging environment. The BizTalk database jobs play an important role in keeping the BizTalk databases healthy. Besides these jobs there are many other factors that you have to consider like disaster recovery, tracking, and availability (e.g. Clustering). Disaster Recovery needs to be in place in case you need to ensure that the BizTalk group can be up and running again in a certain time frame based on the agreed upon SLA. Staff The people that will need to support the BizTalk environment need to be well trained and experienced. This can be a difficult task as many seasoned BizTalk professional are not keen on taking a supporting role. Often the SQL Server Administrator is assigned the task to support the BizTalk Server environment as well. Another challenge is finding a professional with some experience under the belt. Recruiting can be challenging, however there are other options. You can train people by hiring an external company that provides training for administrators, or you could hire a senior BizTalk consultant that temporarily provides guidance and support. The BizTalk platform Administration team should normally consist of the following roles: Windows Server administrator Storage Administrator BizTalk Server administrator Database administrator Monitoring system (tool) Administrator (optionally) Functional administrator Network administrator Process To ensure that the BizTalk environment can be supported in an effective way you need a well defined process. A process that will define how

2012 Motion10 motion10.nl 12 issues need to be handled by which technician. The process can be captured in a support model. A support model is an overview of a process that defines how BizTalk is supported when issues arise and how to operate it. It will show which roles are involved to support BizTalk. This model needs to be supported by a health model, a supportability matrix and a skills matrix. The process, its models and matrixes will be discussed later in this document, the Operation phase. What you have read so far are considerations that are useful when analyzing requirements and preparing your architecture and design. You will need to take a considerable amount of time for analyzing requirements to be able to create a solid design for your BizTalk environment. It will be worth investing time now as you will lose a lot time and money if your applications do not perform or the system cripples under the load it s supposed to be handling.

13 Planning Phase Steef-Jan Wiggers December 2012 After having your design set and written down you can start planning the deployment of the BizTalk Server environment. You will need to have plans in place when to deploy the environment, when to order (new) hardware and software, and how to identify what resources are required for the deployment. Based on the capacity of your datacenter or the datacenter that hosts your IT infrastructure you will need to assign or order hardware. The hardware required for your BizTalk environment can be found in the Microsoft installation documentation or the BizTalk Server website (Product details > System requirements). Sizing The nonfunctional requirements (NFR s) in your design drive the sizing for the BizTalk environment. Throughput and latency are key requirements that impact sizing of the environment. How many messages must the BizTalk solution process within a given time interval. Factors that affect throughput are message size, format, adapter, load, orchestration and tracking. These factors should be documented in the design as considerations and should result in a decision for what the infrastructure design of the BizTalk environment should look like. The throughput will also determine the scale and performance of the BizTalk environment. The design will be the input for how to size the environment and the dimensions for the infrastructure concerning CPU, Memory, Disk and Network. Software (licenses) Depending on which edition you require for your BizTalk environment you will have to purchase licenses. An enterprise license will cost more than a standard license. However, the enterprise will support more features and is applicable in scenarios where you require failover, clustering and high availability. The number of licenses depends on the nonfunctional requirements set in the design. Resources To deploy, validate and operate BizTalk the appropriate professionals need to be allocated to perform the tasks during each phase. A system engineer, BizTalk administrator or developer, network administrator, and database administrator for instance may be required to perform tasks in each phase. Therefore, resources need to be allocated within the enterprise to fulfill these tasks. Resources can be professionals within the enterprise or hired from a third party. Timelines During the planning phase it is very important that there will be a clear and concise record of milestones for installing, validating and deploying the BizTalk environment. Setting specific milestones will help in getting a clear insight in the lead time to having the BizTalk environment deployed and ready to operate. A plan should contain a diagram that visually represents dates, milestones, and tasks associated with the BizTalk Environment. The milestones should be agreed upon by all the stakeholders involved and reviewed regularly to prevent increased lead time and costs.

2012 Motion10 motion10.nl Deployment Phase 14 The Deployment phase of a BizTalk environment starts when all the hardware is in place, operating systems are installed, patched and configured with appropriate features like for instance Clustering Services and Internet Information Services. You now need to collaborate with system administrators to arrange different Active Directory Groups and Accounts for each environment (Develop, Test, Acceptance and Production) and network administrators to open network ACLs between the different servers and environments. Besides that you will discuss a strategy for applying new patches and cumulative updates. It is key that if downtime is not an option, the infrastructure needs to be setup as such. Availability will be a driving factor and costs will increase. Finally you can consider and discuss with system administrators on how to perform repeatable deployments via Scripting or PowerShell. Installation of BizTalk Server needs to be carried out according to the guidance Microsoft provides for each edition and target Operating System. The installation documentation is available online http://www.microsoft.com/ en-us/download/details.aspx?id=11503. The installation and configuration need to be documented. What components have been installed and how the BizTalk instances are configured (which accounts are used, machine names, and so on). This document will be important for BizTalk administrators as they will learn how the environment is set up.

15 Validation Phase Steef-Jan Wiggers December 2012 When you deploy BizTalk on the determined hardware and software, you will be able to validate, test, and tune it using the following: DTCPing DTCTester BizTalk Best Practices Analyzer (BPA) MessageBox Viewer (MBV) BizTalk Benchmark Wizard (BBW) Performance Analysis of Logs (PAL) These tools enable you to validate your BizTalk deployment. The DTCPing and DTCTester tools enable you to verify the DTC settings and configuration. This is important because BizTalk Server heavily relies on the Distributed Transaction Coordinator. BPA checks your BizTalk configuration against a known set of best practices. Similarly, the MBV checks your configuration and will detect if there are major issues. With the BBW and PAL you can establish a performance baseline and have a rough estimate of the system boundaries. Both tools can also aid in tuning your environment for performance. The Benchmark Wizard can also be very useful to perform load tests. Following paragraphs will describe each of these tools. DTCPing The DTCPing tool can be used to assist with troubleshooting Microsoft DTC Firewall issues, which you ll often see in a multi-server BizTalk deployment. DTCTester The DTCTester utility is to verify transaction support between two computers, if SQL Server is installed on one of the computers (multiserver BizTalk environment). The DTCTester utility uses ODBC to verify transaction support against a SQL Server database. BizTalk Best Practices Analyzer (BPA) BizTalk Best Practice Analyzer (BPA) examines a BizTalk Server deployment and will generate a list of issues pertaining to best practice standards for BizTalk Server deployments. This tool is designed to assess the configuration of a BizTalk installation. It is very valuable when it comes to validating your environment. BPA performs configuration-level verification by gathering data from different information sources, such as Windows Management Instrumentation (WMI) classes, SQL Server databases, and registry entries. It will present a comprehensive report to the user. Under the hood it uses the data to evaluate the deployment configuration. It does not modify any system settings and is not a self-tuning tool. The tool is there to deliver support in achieving best suitable configuration and it will report issue or possible issues that could potentially harm the BizTalk environment. MessageBox Viewer The BizTalk Message Box Viewer (MBV) was developed by Jean-Pierre Auconie. It retrieves information from a BizTalk System and identifies all possible issues that could be critical or need attention, and presents them in a user friendly format. Like BPA is a health check tool that generates reports of an existing BizTalk System. BPA and MBV are complementary to each other. The MBV can help administrators

2012 Motion10 motion10.nl 16 very quickly in acquiring information of a production system with hardly any impact on the running environment. It can help administrators identify possible issues, which could be critical or need attention. MBV has one executable to fire up a console application and another to fire up a Windows Forms application. Both use the same engine, reading only in the BizTalk databases and Server s registry to retrieve useful lists of information. Both the console application and the Windows Form application can produce three types of reports: HTML report: the first to open to see a well formatted report of the health check History log file: this contains in textual format the history of all reports Status log file: this contains detailed status of each query execution and possible errors encountered execution of the tests, performance counter information is collected and benchmarked against collected statistics relevant to your BizTalk Server environment. Microsoft provides extensive guidance on performance optimization through the performance optimization guide that provides in-depth information for optimizing the performance of a BizTalk Server. Although the BizTalk Server 2010 Optimization Guide http://www. microsoft.com/download/en/details. aspx?id=10855 is very useful when designing a BizTalk system, it does not provide any expected performance numbers related to specific environments. This is where Benchmark Wizard comes into the picture. After having configured the BizTalk group and applied some of the best practices through using BPA you can start testing your BizTalk configuration. The status log file is important to read if some errors were found during the collect process. It displays the start time of each query. Both tools also offer the option to generate an XML report. BizTalk Benchmark Wizard (BBW) The BizTalk Benchmark Wizard is another tool to validate a BizTalk environment. It is built by Mikael Håkansson (BizTalk Server MVP) and Ewan Fairweather (Microsoft). It is intended to verify your BizTalk installation and is a useful tool to investigate and confirm the performance characteristics prior to deploying the first solutions. You may also want to use it before you re about to scale out your environment, to make sure you re really using all resources before investing in additional hardware and licenses. The benchmark wizard generates load to BizTalk Server in relation to specific scenarios. During the The tool is not a load generation or analyzing tool; it doesn t give advice or hints like BPA. However, if the test fails you can analyze the data using the PAL tool. You can run the tool on either a single- or a multi-server installation. Regardless of the number of BizTalk servers in your group, you should not run it with more than two active servers http://www.microsoft. com/download/en/details. aspx?displaylang=en&id=2290, as it will otherwise not be covered by the benchmark values. The tool should be used after BizTalk Server has been installed and before any solutions are deployed in the environment. This will ensure that you are getting consistent and clean results from the BizTalk Benchmark Wizard.

17 Steef-Jan Wiggers December 2012 Performance Analysis of Logs (PAL) PAL (Performance Analysis of Logs) tool is a powerful tool that reads in a performance monitor counter log and analyzes it using known thresholds. It reads in a performance monitor counter log (any known format) and analyzes it. Basically it automates the analysis of performance counter logs, for instance one that is generated by the Benchmark wizard. In the end you can generate an HTML or XML based report that graphically charts important performance counters and throws alerts (in red), when thresholds are exceeded. The thresholds are originally based on thresholds defined by the Microsoft product teams, including BizTalk Server, and members of Microsoft Premier Support. Before PAL it was quite hard to analyze logs; it had to be done manually and required quite extensive knowledge of Windows architecture. It could require a lot of time to analyze the logs and therefore required significant investment during project execution. There are tools that could help in analysis like SCOM, but that might not collect all critical data necessary. There was not really a tool that could analyze log file(s) easy and fast. PAL does just that, it analyzes logs and doesn t require much time and is free. The tool has the following benefits: Consolidated guidance: a central repository gathered from multiple sources Log File Data Access Layer: using the Microsoft Log Parser Analyze More Data Points: the data is broken down into smaller slices and analyzed individually Dynamically Changing Thresholds: a learning environment asking questions Reusable: the code is open source Extensibility of Thresholds: add, edit or delete thresholds

Operation Phase 2012 Motion10 motion10.nl 18 The last step in the process comes down to maintaining the BizTalk Environment. Supportability will be the main focus in this step. To have efficient supportability for your BizTalk environment you will need to have a health model of your environment, a support model, a supportability matrix and a skills matrix. Besides the models and matrixes you as an administrator will need to have the right set of tools to maintain a healthy BizTalk environment. These tools consist of the out of the box BizTalk administration console, BPA, BizTalk Message- Box Viewer and monitoring product(s) as described earlier. The following paragraphs will further discuss these aspects. BizTalk Administrator(s) that operate BizTalk should primarily work from documented processes and procedures and should be responsible for: Monitoring all layers of the platform, System Health checks, Backup and Restore and produce or update existing documentation. Is also important to note that in BizTalk Server there are two administrative roles: BizTalk Server Administrator: Is a high privilege role with full access to the environment (configurations, tracking data, etc.) BizTalk Server Operator: Is a low privilege role with access only to monitoring and troubleshooting actions, i.e. this role enables members to monitor BizTalk Server for errors, query suspended messages/ instances and view configurations but prevent them from changing BizTalk configurations. The role of operations is often confused with that of incident management. Unfortunately incidents can never be prevented 100%, this is why the administration team must offer a second or/and third line support focus on resolving incidents as soon as possible. However Operations should focus on preventing incidents on the BizTalk Server infrastructure, being more focused on and concerned with events generated by the infrastructure and not focused on incidents reported by users. Normally organizations have a tendency to subdivide the operations into different groups, each one with a set of roles and functions clearly defined: Operations management: focus on preventing incidents on the BizTalk Server infrastructure Incident management: focus on resolving incidents reported by users, business and/or customers as soon as possible Problem management: focus on resolving platform incidents (like not enough network bandwidth, bottleneck in disk I/O) Application management: focus on processes to ensure that standardized methods and procedures are used for deployments of and updates to BizTalk applications for efficient and effective implementations. Each of these groups must have their own well-defined processes and plan models (see Support Model). Health model A health model defines clearly how and what

19 we Support model Steef-Jan Wiggers December Technical 2012 support consists of a range of Technical support is often subdivided into tiers, or levels, in order to better serve a Business Integration Suite. In general, technical business or customer base. The number of levels a business uses to organize their support in this context attempts to help the technical support group is dependent on the business needs or desires as it revolves business and/or customers by solving specific around their ability to sufficiently serve the customers or users. The reason for providing a multi-tiered support system instead of one general support group is to provide the best problems with a solution or service running in possible service in the most efficient way. The success of the organizational structure is its integration landscape. A support model aids depending on the technician s understanding of their level of responsibility and in defining the support process and in providing commitments, their customer response time commitments, and when to appropriately escalate an issue and to which level. A common support structure revolves around a the right service in case of required assistance. three-tiered technical support system. Most organizations offer different channels to deliver assistance, which can be over the The first tier is the initial support level responsible for basic customer or technical issues. This tier is commonly known as first line support or level 1 support. It is basically the entry telephone, online by email or a website. point for issues. The responsibility of the first tier is to analyze the issue by analyzing the symptoms and figuring out the underlying problem as far as possible. When a first line Technical support is often subdivided into tiers, support technician is analyzing the problem it is important to identify what BizTalk is or levels, in order to better serve a business or trying to accomplish so that time is not wasted on "attempting to solve a symptom instead of a root cause." As soon as the problem is identified the technician can start with sorting customer base. The number of levels a through possible solutions or forward the issue to second tier. Steef-Jan Wiggers December 2012 want to monitor, diagnose, and recover in business uses to organize their technical any application, device, or service that has support group is dependent on the business been defined in an integration model for your needs or desires as it revolves around their environment. Based on how your environment ability to sufficiently serve the customers or is setup, i.e. what configuration has been users. chosen, and what non-functional requirements account for solutions running in your environment The reason for providing a multi-tiered support such a model can be created. In short system instead of one general support group is your BizTalk environments need to adhere to an to provide the best possible service in the most agreed upon Service Level Agreement (SLA). To efficient way. The success of the organizational meet the SLA(s) a health model, supported by structure is depending on the technician s system-checks using a set of (specialized) tools understanding of their level of responsibility and that are run periodically, can be very valuable to commitments, their customer response time your integration services organization. commitments, and when to appropriately escalate an issue and to which level. A common support structure revolves around a three-tiered technical support system. services providing assistance with technology products found for example in the Microsoft The first tier is the initial support level responsible for basic customer or technical issues. This tier is commonly known as first line support or level 1 support. It is basically the entry point for issues. The responsibility of the first tier is to analyze the issue by analyzing the symptoms and figuring out the underlying problem as far as possible. When a first line support technician is analyzing the problem it is important to identify what BizTalk is trying to accomplish so that time is not wasted on attempting to solve a symptom instead of a root cause. As soon as the problem is identified the technician can start with sorting through possible solutions or forward the issue to second tier. Figure 3 A technical support model Copyright - motion10 Page 22 of 31

2012 Motion10 motion10.nl 20 Second tier or level 2 support is a more in-depth support level, where more experienced and trained professionals reside. They are responsible for assisting the first tier personnel in solving basic technical problems. Besides that, they investigate elevated issues by confirming the validity of the problem and seeking for known solutions related to these more complex issues. Before the second tier technician will start with troubleshooting, he will review the work of the first tier technician to see what already has been accomplished. If an issue is new and/or the staff in this group cannot determine the root cause or come up with a solution, they are responsible for raising this issue to the third tier support group. Third tier or level 3 support is the highest level of support in a three-tiered technical support model responsible for handling the most difficult or advanced problems. Technicians in this group are experts in their fields and responsible for not only assisting both first and second tier personnel, but also with the research and development of solutions to new or unknown issues. They also will have to review the work done by first and second tier technicians before starting with troubleshooting. The third tier technician might work to solve the problem with the customer as it may become apparent that the first and/or second tier technicians simply were not able to discover the proper solution. Supportability matrix The supportability matrix identifies the roles and responsibilities the different people can have in a support model. Each individual that plays a part in the support process within the organization needs to understand its role and responsibilities. Depending on the size of the IT department and infrastructure the roles and responsibilities may overlap or can be completely separated. The clearer the roles and responsibilities are separated, the better the support process will be executed. Skills Matrix Skills Management is the practice of understanding, developing and deploying people and their skills in the most efficient way. Well-implemented skills management should identify the skills that job roles require, the skills of individual employees, and any gap between the two. This will help support people to do their jobs in a complex IT landscape with MS Business Integration Suite components. Managing skills inside an organization is a complex process, and a skills matrix can be helpful. This consists of a list of skills, and a grading system, with a definition of what it means to be at a particular level for a given skill. To be most useful, skills management needs to be conducted as an ongoing process, with individuals assessing and updating their recorded skill sets regularly. These updates should occur at least as frequently as employees regular line manager reviews, and certainly when their skill sets have changed.

21 Support Steef-Jan Wiggers December 2012 role Responsibilities and necessary skills BizTalk Administrator Deploy applications, monitor UAT and production, troubleshoot issues First Tier Support Technician Identify issue, route issue to appropriate support group Second Tier Support Technician Troubleshoot (advanced) issues, technical BizTalk and.net skill set Third Tier Support Technician Troubleshoot advanced issues, deep technical BizTalk and.net skill set Functional Technician, Business Analyst Knowledge of functional domains and processes, identify functional issues Support roles in an organization where MS Integration Suite components have been deployed should have the following skills: Role Skills BizTalk Administrator SQL Server, PowerShell, VBScript, BizTalk Server (Certified), MCSA Certified, SCOM, XML First Tier Support Technician BizTalk Server (Certified), Domain knowledge, Overview on processes, XML Second Tier Support Technician BizTalk Server (Certified),.NET, Visual Studio, XML Third Tier Support Technician SQL Server, PowerShell, VBScript, BizTalk Server (Certified), MCSA Certified,.NET, Visual Studio, Troubleshooting Tools, XML Functional Technician, Business Analyst Process knowledge, domain knowledge, XML, Business

Tools 2012 Motion10 motion10.nl 22 To perform troubleshooting, experience and knowledge is required. However, tools to support with troubleshooting are mandatory as well to effectively assess and resolve the issues. This chapter discusses the most important tools used by support personnel in an integration middleware environment based on the Microsoft Business Platform. indication about an error that has occurred in BizTalk. BizTalk Server writes errors to the application event log each time an error occurs. Error information in the event log contains information on when, why, and where the fault happened. Through the BizTalk Administration Console, the event log can be viewed and the administrator or operator can start investigating the problem within the same snap-in using the console s Group Hub page. The BizTalk Administration Console The BizTalk Server Administration Console is a Microsoft Management Console (MMC) snap-in that you can use to manage the BizTalk Server landscape. It is less suitable for monitoring your environment as it lacks features like notifications and alerts. These features are present in monitoring products like SCOM and solutions based on BAM. The Management Console can make the state of BizTalk artifacts visible, however to make the current state visible you will have to refresh by pressing F5. The administration console however is an excellent tool to use for deploying and managing your BizTalk Server applications. Administrators and/or BizTalk operators will use this tool for numerous tasks as all the settings in the BizTalk group can be controlled in a single user-interface similar to other MMC snap-ins such as those for IIS and certificate management. Often the BizTalk Administration Console is the first place the administrator will look when identifying errors or issues in BizTalk and determining how to resolve them. Other tools which can aid, or are better suited to do so, are the Windows Event Log (which is also accessible through the BizTalk Administration Console), SQL Server, and other external applications. The Event Viewer can give a first Monitoring Monitoring is an important aspect of operations. Now what does monitoring mean in context of BizTalk? What to monitor or monitoring generally means is to be aware of the state of your BizTalk group. As a BizTalk administrator you want to be aware of the state of the integration solutions running inside the BizTalk group. You will need to observe your running solutions using a monitor and measure facts i.e. Performance counters, volume, frequency and transactions. As an administrator you are responsible for keeping the environment running healthy, therefore you need to be in control. If you are not in control you cannot manage your solutions and environment properly, and they will become unavailable, fail or cripple other applications or systems. The business ultimately will lose money, a lot of money because it depends on an available, healthy and well performing BizTalk environment. Challenges Monitoring a BizTalk environment can be challenging. Problem areas and challenges an enterprise can face are as follows: No well defined monitoring requirements Unclear Roles and responsibilities No proper tools to drill down into issues Getting access to the problem context

23 Steef-Jan Wiggers December 2012 Proactive versus reactive monitoring Accelerators like HL7, SWIFT Supporting many EDI formats and protocols Monitoring sources Noise (too many uncorrelated events in logs) Reporting tools Exception management Resolution SLAs Response times Technicians monitor systems and applications and have certain responsibilities in their enterprise. It is sometimes difficult to assess what these responsibility entail. Is it the technical side and/or application side or both? Does the enterprise have the right people with defined roles in place or are there other parties involved like an offshore team that does support or a partner that delivers infrastructure and (2nd or 3rd tier) support. The best way to approach this problem is: To have a clear process in place, like a support model A clear view on the responsibility and roles through a supportability matrix and a skills matrix To have the proper support tools The combination of the above will also help to gain a good consolidated overview of the health of a system or landscape, which nowadays can be a challenge and may not be easy to achieve. This will become more apparent in the paragraphs to follow. Another challenge is when there is a problem or issue that needs to be resolved it can be difficult to gain insight in the proper context. An error message can lack enough descriptiveness to be able pinpoint what exactly causes the problem. Besides context, dealing with problems pro-actively or reactively is another challenge. When a problem occurs the follow up is a reaction to it, which is a reactive action. Proactive is to get notified of issues before a support person find out manually. If problems are common or occur quite often a monitoring rule or process can pro-actively deal with it. It can be a difficult assessment within support teams how to deal with certain issues. This partially depends on how monitoring requirements are defined and what the non-functional requirements are for certain systems and applications. A knowledge base (i.e. Wiki), automated rules, alerts and notifications, forums can help to face these challenges. There are many sources that can aid in monitoring, such as the Windows event log and other application specific log files or databases, which can lead to confusion or a lot of noise. Applications and systems can generate a lot information that can be overwhelming and in case of a problem or situation it can be difficult to filter out the exact cause in a timely matter because the information is spread over different locations and can be hard to correlate. Having a lot of noise will make effective monitoring a hard task. Appropriate filtering of information and consolidating information in one view can be good measures to tackle this problem. It will also aid in having a clearer overview of systems and applications you monitor. Another challenge is the interaction between the different members of the administrative team: System administrators, DB administrators and BizTalk administrators need to be in constant communication and validate the support process they intend to implement, respecting the functionalities provided by the specific server products. For example, to back up the BizTalk databases, the jobs shipped with

2012 Motion10 motion10.nl 24 the product should be used and not the standard maintenance plans that a DB administrator is accustomed to. other software vendors make these management packages available through the System Center Marketplace. System Center Operation Manager (SCOM) Microsoft System Center Operations Manager (SCOM) is a component that belongs to the System Center Suite. It can be installed as a standalone product. The latest version is SCOM 2012, which enables you to do end-toend service management. This means that with SCOM you can monitor your entire Microsoft infrastructure including the Windows Server (OS), application servers and for example Microsoft Exchange. Through a single UI, an administrator and/or engineer can monitor the health, performance, and availability of data center environments across applications, operating systems, hypervisors, and even hardware. What happens when you use SCOM is described in this section. Each machine which needs to be monitored has an agent (that is, opsmgr health service) running. This agent watches several sources on that computer, including the Windows Event Log, for specific events or alerts generated by the applications running on the monitored computer. When a specific event or alert occurs, the agent will detect it and forward it to a SCOM Server. The SCOM application maintains a database that includes a history of alerts and events and applies filtering rules to alerts/events as they arrive. Each rule can trigger some sort of notification, like an email to be sent to the operator or start a workflow intended to correct the cause of the alert in an appropriate manner. SCOM uses the term management pack to refer to a set of filtering rules specific to some monitored applications such as the BizTalk Server. Microsoft and other software vendors make management packages available for their products. SCOM also provides for authoring custom management packs. Microsoft and Alternative Monitoring Solutions To use SCOM solely for BizTalk would lead to too much overhead, require training for BizTalk administrators, and it can become quite expensive because of the increasing costs of operation. However, SCOM can be a valuable asset, yet it will require an investment from IT. If you need to monitor a complete stack of Microsoft server products, it makes perfect sense. There are third-party products available which can provide a monitoring solution for the BizTalk environment. For instance, BizTalk360 can serve as a good alternative and delivers a web-based UI, which has been a long demanded feature by administrators. It also addresses some of the issues one may have with the BizTalk Administration Console. The BizTalk management tool console is more focused towards administrative tasks than monitoring your BizTalk group. BizTalk360 is an administration and monitoring tool that proves valuable in small to mid-sized BizTalk implementations. Besides BizTalk360, there are other third-party monitoring solutions such as HP OpenView, Minotaur, IPM, FRENDS Helium and AIMS. Some of these products have been developed by a member of the BizTalk community in the early days of BizTalk. Subsequently all of them have turned into a full featured product, with support and a licensing model offered through a company. Most companies also have a partner program.

25 Checklist Steef-Jan Wiggers December 2012 Below you will find a checklist for each step in the process to establish a robust and sustainable BizTalk environment that is ready to be handed over to the support team. Checklist Task 1. Workshops with the business 2. Workshops with IT 3. Create High Level Design 4. Present HL Design Design Phase 1. Process information 2. Create Detailed Design 3. Present Detailed Design Planning Phase 1. Create Project Plan 2. Allocate Resources 3. Order hard-/software Deployment Phase 1. Deploy the (virtual machines) 2. Setup 3. Configure Domain Controller, Windows Group and Service Accounts 4. Install OS 5. Install prerequisites 6. Install BizTalk 7. Configure BizTalk Group 8. Create config documentation Validation Proces 1. Use DTCPing/Tester 2. Use BPA 3. Use MBV and schedule it 4. Use BBW 5. Use PAL if required, rerun BBW 6. Create Documentation Operation Phase 1. Create a Health model 2. Create a Support model 3. Create a Supportability matrix 4. Create a Skills matrix 5. Train support staff 6. Implement staff 7. Setup monitoring 8. Setup and test DR 9. Transition to support Comments The checklist can be used for guiding the process of preparing your BizTalk environment for operation. Following through the checklist will ensure that your BizTalk environment is built properly according to requirements with supportability in mind.

Conclusion 2012 Motion10 motion10.nl 26 Implementing BizTalk Server means creating a business critical integration platform that is robust and sustainable to meet your requirements now and in the future. It also means that your IT support team has to be able to support the integration middleware and functional interfaces deployed. To have supportability in mind during architecture, design and development is the key to success. To achieve this goal a plan has to be prepared and followed through. In this white paper you have been provided with the tools and guidance to have this in place before you start. The checklist at the end can greatly help you in the preparation, implementation and set-up of a robust, supportable BizTalk landscape.

27 The author Steef-Jan Wiggers Steef-Jan Wiggers December 2012 Steef-Jan Wiggers (steef.jan.wiggers@motion10.com) is a Principal Consultant at motion10, specialized in Integration. He has experience in architecting, designing, developing and supporting sophisticated and innovative solutions using many different Microsoft technologies and products. Besides being very active in the BizTalk community as a blogger, Wiki Ninja and public speaker, he is also the author of the BizTalk Server 2010 Cookbook (Packt Publishing). Steef-Jan has been awarded Microsoft Integration MVP 3 times in a row for his many contributions to the worldwide adoption of BizTalk Server and Windows Azure. Many thanks go to the following people for reviewing this paper in their precious (spare) time; colleagues Marnix Cox, Andre Ruiter and Sander Nefs, fellow Microsoft Integration MVPs Gijs in t Veld, Sandro Perreira, Nino Crudele, Saravana Kumar and Kent Weare, and dedicated BizTalk community members Tord G. Nordahl and Hendrik Roth. Motion10 specializes in integrating and presenting information from different sources. The solutions implemented by the company link enriching and disseminating information from business-critical applications and ensure that information is easily accessible, thus bringing more value from the information obtained. Motion10 has extensive experience with complex integrated environments (hundreds of implementations) and can thereby eliminate the complexity of IT environments. Motion10 s solid, reliable IT approach has established that all employees have the personalized skills and technology to help implement solutions based on the Microsoft technology platform utilizing Windows, Azure, SQL Server, BizTalk and SharePoint. For more information: www.motion10.com

www.motion10.nl