BizTalk 360. BizTalk Monitoring Product. Detailed Feature Review

Transcription

1 BizTalk 360 BizTalk Monitoring Product An efficient way to monitor your BizTalk Environments Detailed Feature Review Page 1

2 Contents Introduction... 4 Author... 4 Security & Audit... 5 User Access Policies Operational Governance & Auditing Avoid direct RDP access to Servers Avoid direct access to Databases Operations... 7 Productivity Tools... 7 Advance Event Viewer Integrated Knowledge base Search Design time artifacts See what's running inside your Host Graphical Message Flow Viewer Message Box Viewer (Integration) Backup/DR Visualizer Throttling Analyser Tracking Manager Reality Data in One place Page 2

3 Message Box & Tracking Data BAM Portal ESB Portal Any SQL Data Business Rules Monitoring BizTalk Applications Monitoring BizTalk/SQL Servers Monitoring Health Roll Up for BizTalk Deployment Valid Scenarios Summary Page 3

4 Introduction This is a review, based on the features, listed on the BizTalk 360 website, and documentation about what BizTalk 360 actually does. We will go through each of the features and provide a Reality version of what it really means, does and delivers. Whilst we do know the author of BizTalk 360, and for all purposes he is a good guy, and has put a large degree of effort into BizTalk 360, as if you have no monitoring of BizTalk, we all know it s not going to be a happy BizTalk installation in a very short amount of time. You need to monitor, you need to watch, and you need to be proactive. If you architect your BizTalk solution property many of the tried and failed methods will not catch you, it comes from this point that you can make the right decisions to make your deployment enterprise grade. Sadly not everyone does this, and many inherit what some 3 rd party decided was a good thing to do, and turned out not to be the best practice or anything close to it. I do see this often. In BizTalk there are always 1000 different ways to implement your solution, given the scenario and requirements there are about 1-2 of these that will work for you. When you add the need to actually monitor this going forward, and see not just what is not working, but what is, and how much, how long it is taking and what transactions are flowing. At the end of the day you will need to formulate your own opinion and view of the product if it fits into your rollout of BizTalk. There are valid reasons to use this product, but make sure you don t do things twice or cut corners. Author The author of this document is Paul Somers, a seasoned BizTalk Specialist with over 14 years experience of BizTalk. Deep ties to the product development of BizTalk as an enterprise product as well as the new BizTalk services in Azure, more to come soon... Having been involved in some of the largest and most complex BizTalk implementations across the world, if there is anything you need to do with BizTalk, the foundation, basis and architecture behind this, may have already been done by Paul. The main rules of thumb with BizTalk, there are 1000 different ways to do anything in BizTalk, there are only a few that fit both your scenario and objectives. If you have done it wrong the first few times you learn, if you have worked many, many, many BizTalk implementations you learn what works and more importantly what doesn t. It s this knowledge you can t just place an order for, it s this knowledge that gets you a successful BizTalk implementation first time round. If you have any feedback or follow up questions about BizTalk: paul@somers.com or paulsomers.blogspot.com Page 4

5 Security & Audit User Access Policies Define your own rules; you are not restricted just to the standard "BizTalk Administrator" or "BizTalk Operators" groups. Yes you can define your own rules/access, based on an BizTalk Application specific architecture. This is good; it does depend on your solution/architecture. If you made shared, service bus applications, that other applications can call, or your application structure was more of a shared global structure, where function themes made up the applications, rather than end point systems. It is dependent on you setting up the application structure to suite, but if you did anything shared, one user would have access to other applications transactions, not necessarily the ones they are supposed to be looking at. I don t know a single environment where there was not shared application(s) that others didn t use/send requests to and get responses from. The transactions on share applications have no ability to limit what they can see, and it s next to near impossible to do this in this context. Operational Governance & Auditing Track all your support people activities, "Sam restarted Receive Location at 10:45 am". Yes, this is fantastic; it s really good to see, I wish the admin tool had this, in a test or staging environment where many people will be getting on the box and messing around it s a real problem. The reality is, they would most likely also have access to the admin console here, even RDP to the box, I don t like it, and really am against people RDP ing and making changes, but you can make changes in BizTalk, and have them tracked, but if they change a config file or other setting on the actual box, this is not recorded. There is the event log, who logged in, what bad things happened after this, and it s all audited, you will be found out, but rather than who did what, don t do it, or fix it right away. In production, no one will be messing around only one or two people would even have access to do this, as BizTalk administrators are usually a very limited group. This does not track a config file change, again the event log, will track a log on to the machine, and if really needed, and you can pull this up. To audit here is valuable but when it s only one or two people is it necessary/useful? Page 5

6 Avoid direct RDP access to Servers Support team has access to all the resources directly via BizTalk360 web console, eliminating physical access to servers. Yes, it does. You should not RDP to the box in any event. The BizTalk administration console can, and is often run from the operations people s machine, and points to the various environments, and they do not RDP, they can do everything from here. There is also the BizTalk Operators Group, which provides limited access to BizTalk, again users in this group would not even have RDP access, but can see what s happening, and have limited abilities. Avoid direct access to Databases Your DBA's will love this feature! BizTalk360 allows support team to store and execute queries directly from its web console in a secure way. It does do this. The ability, or even querying the BizTalk databases, should never be done, these databases run BizTalk, and all the crucial stuff is exposed via the performance counters, such as spool size. Every single BizTalk host is querying the message box every 500 ms for work, you do not just run a adhoc query here. DBA s will hate this, as they don t want users and they don t want anything that s not supported querying the databases, BizTalk 360, allows someone to create adhoc Queries, which a DBA has no say in, and could do not great things... do you want to have to deal with the fallout from this? In any event direct access to the database is NEVER required, NEVER granted, and NEVER done. BizTalk manages itself. Page 6

7 Operations Productivity Tools Advance Event Viewer The standard windows event viewer is not good for diagnosing BizTalk problems, especially when you have environments with 2 or more BizTalk servers. Correlating events between servers is tiring work. BizTalk360 comes with advanced event viewer designed to make your diagnosing capabilities super productive. Very true, your BizTalk environment usually has several machines, with events firing all over the place, it s a total pain if you have to go look through all of these manually. So it does this quite well. Any operations product you have will also do this, such as Microsoft System Center Operations Manager, which are usually there already monitoring the very system you are probably using BizTalk to interface with. If you do not have such a system, then you will need one, BizTalk 360 is a cheaper option, and will do the job. However not to state the obvious, what is monitoring those other systems that BizTalk is probably talking to? You are going to need an enterprise grade monitoring system; BizTalk 360 is not this system. When you do move to such a system, there are countless WMI events BizTalk exposes and raises, countless performance counters, and all of those many event logs, they will all get sucked up into this. See further on about this under monitoring. Integrated Knowledge base There will always be some known problems during your day-to-day support. Example: Contoso Inc occasionally sends with some junk ASCII characters that breaks. BizTalk360 helps to associate KB articles with error codes, when the same issue happens next time you'll see the associated KB avoiding to diagnose the problem from scratch. This is crucial, you need to have common repository, other than Google, to help you help yourself. If you don t happen to have a documents portal like SharePoint, or office 365 which can do this as well, then BizTalk 360 is what will help you do this. Saying that a L:\ drive on the network will also work. Page 7

8 Search Design time artifacts Do you have 100's of applications, send ports, receive locations, orchestrations etc and managing them gets challenging? BizTalk360 rich design time artifact search can help to discover and take action on them instantly. This is a real thing that occurs, you end up with 100 s of these, in several applications, all doing their thing, generally they don t change unless you have a deployment occurring, but it would be good to document these, and know what s what, and also be able to search who is using a certain port. Which host runs which orchestrations, etc. The great person (Jason B) who originally made the BizTalk Documenter ( and the people who have maintained it over the years already thought of this, and with each deployment, a fresh profile can be generated, stored in your KB repository, and enabled you to create a.chm file with hyper links, to click on a port, host, orchestration and see what s what, even view the actual orchestration, map, schema in full detail, stepping into the code within these. Much more than BizTalk 360 provides, but this is very valuable information either way. See what's running inside your Host Host is a logical container. Great! But do you know what's configured to run inside the host, it will take couple of hours for you to figure out using BizTalk administration console. BizTalk360 host container view makes it piece of cake. This is really neat, and works, the BizTalk Admin console does not allow you to filter on what s configured, just what s running in a host, this can be very useful. You don t get to view what configured to run in your host anywhere other than the BizTalk Documenter, which you would have run as part of a deployment/kb creation. You can of course use the BizTalk Management console to run many more queries even more complex queries against what s really happening in your environment, by host, application, service, status etc. Page 8

9 Graphical Message Flow Viewer How does a message travel from "A" to "B" inside our BizTalk environment? Well you need to ask Mr.Joe who is our BizTalk expert. You don't need Mr.Joe to tell you that information, BizTalk360 graphical message flow viewer will visualize your message flow in your environment without any custom coding. Personally this is one of the best features of BizTalk 360, its great, to graphically go into the tracking database, see what called what, where did the message go, and get to the level of detail it provides, hats off to the developers of that. We do have this via message flow of any message, and you can follow it, but nothing like what s available here. It s a really good feature and a pretty Picture... Message Box Viewer (Integration) MBV comes with 400+ rules that validates various things in your BizTalk environment and reports whether your environment is in a healthy state or needs some attention. MBV is fully integrated into BizTalk360, you can schedule to run MBV on a periodic way and see the reports directly in the UI, in addition you can also set alerts based on MBV reports.. Yes it does, much the same as the BizTalk best practices analyser, you can t click on them to tell you want to do yet... it s neat, it gives you a lot of detail... In reality no environment would NOT have some detail here. Every environment is different, and you can strive for best practices, but sometimes it s not needed. It would be good to ignore some of these. It s neat, the only problem is this run s against the message box database, a whole bunch of queries, and it does this on schedule, as frequently as you configure it. This is not a database you want to be running this level of detailed queries against in a production environment. Backup/DR Visualizer The only supported way of doing backup and DR in BizTalk server environment is by using the standard SQL jobs and log shipping capabilities. BizTalk360 provides a single page visualizer to check the health of your backup/dr configuration. This looks at the SQL Agent jobs, when they were last run, and what the outcome was. It s very important to set this up, and configure it. This does provide you the ability to monitor this. Any operations monitoring product worth its money will also be able to do this, if you don t have one yet, you WILL need one, and BizTalk 360 will do this for BizTalk events. What about every other event? Page 9

10 Throttling Analyser No one understands Throttling!! you need a super BizTalk expert to understand various throttling counters and make sense of the values it displays. BizTalk360 comes with a simple dashboard to understand your environment throttling conditions. You don't need to be an expert. This didn t work for me, I had throttling, but the dashboard didn t display them correctly, I had to drill down to the host, and then it didn t display it very well... Throttling is something that VERY difficult to see, diagnose, and fix, you will need a BizTalk expert somewhere here to do this. Personally I create BAM profiles, to record timing, what s taking a long time, which parts of the process, usually a 300 second delay is a throttle, and throttling is not a bad thing, it s more where its making your request response request take 610 seconds because 600 of those seconds was a throttling delay. The more you drill into what really happening in your environment, the more you will see BAM is the master of this kind of report, and much much more... Tracking Manager You track information at various places - pipelines, orchestrations, schemas, global tracking etc. Tracking manager from BizTalk360 helps to visualize all these data in a single page at application level, making you super productive. Reality Yes it does this on an application by application basis, however not on the whole. Whereas the BizTalk admin console already does all of this, and more. Filter for tracked message events, and tracked service events. Where host name is x, or service name is y, none of this is possible in BizTalk 360, and kind of redundant. You can see much much more inside the BizTalk admin console, you cannot see much in here at all. Page 10

11 Data in One place Message Box & Tracking Data Access your runtime and tracked data directly from BizTalk360 web console with rich query builder. You can also save and open queries centrally. You can query the message box, and you can kind of... see the tracking data, these queries run directly against the message box, the support of running queries like this, is questionable, consult Microsoft if they support this occurring. Ask you DBA if they are comfortable with queries being run directly against the BizTalk Databases? There is a supported method, the BizTalk Administration console which can do this. There is the BizTalk Operator role/group which provides restricted access to be able to do this as well. BAM Portal Your support team can view data stored in BAM activities and views directly via BizTalk360. It provides the BAM views; it removes and bypasses all security of these. Enabling ANYONE the ability to view any BAM view. You typically create a bunch of views, they are then limited to certain user groups, and some can contain confidential information, security here is crucial. There are also views with aggregations. I admit a separate product, of the better BAM portal is needed, as the one in the box is very outdated. However when you realize, maybe reporting services and SharePoint can also display the same data/view in more of a dashboard context, then you can see the real power behind BAM. People generally use the BAM data to create that customer facing branded view of what s happening, customized to their requirements, drive reporting to customers/clients of their view of the world. BizTalk 360 is not great here. ESB Portal If you are using ESB Toolkit exception capabilities, you can access the ESB exception data directly via BizTalk360. Whilst the ESB Portal has been removed from BizTalk, it s not a direct install, you can get it going but it s a pain. This does SOME of this, and not as nice as the ESB Portal, so if you are using the ESB framework, MAKE SURE you are using it for the right reason, and know the pain you are about to endure, it better be worth it. This is one way of seeing that valuable exception data that can get logged as a fault to the ESB databases. Page 11

12 Any SQL Data Any custom SQL query can be saved and executed via BizTalk360 directly. No need to have access to SQL management studio. Yes you can do this it works, some of the out of the box queries are nice, some of them are already available via the Performance counters, and a query is perhaps not the best of ideas. If you need to get to the point of running custom queries, your DBA better be VERY aware of these, and perhaps a more reporting nature of recording these queries, such as reporting services or SharePoint would be a better option than adhoc queries. Business Rules Access all your BRE policies, Rules, Conditions and Actions all within BizTalk360 web interface. If you are using business rules, it s important to know what rules you have deployed, what rules are in operation, and no one should be going into rules composer to check this in production. It s a useful feature to see this. However these rules are not changing often, as in every hour? Maybe on each deployment they change, but it s a change event. Part of the BizTalk Documenter ( is to document all of the Rules in the rules engine. Page 12

13 Monitoring BizTalk Applications Monitoring Receive Locations Monitor Receive locations states (enabled, disabled). Send Ports Monitor send port states (started, stopped, enlisted etc). Orchestrations Monitor Orchestration states (started, Stopped, Enlised, etc). Service Instances Monitor Service Instances state (Suspended, Active, Dehydrated etc). BizTalk/SQL Servers Monitoring Disks Make sure your disks are healthy (% free space). Event Logs Keep an eye on various event log alerts. NT Services Make sure NT services are in correct state (started, stopped etc). CPU/Memory Keep an eye on CPU (Max utilization) and Memory ( % free). BizTalk Host Instances Monitor BizTalk Host Instances including clusters. Web End-Points Monitor HTTP/SOAP/REST end points for expected return codes. Message Box Viewer Statistics Ensure Message Box Viewer reports are healthy. Get alerted if there are too many critical errors reported by MBV. Database Queries SQL queries that return scalar values can be monitored. Ex: Depth of the spool table, count > 100 : warning, count> 500 : error. SQL Jobs State Monitor state of SQL Jobs a for healthy BizTalk environments. SQL Jobs Errors BizTalk360 alerts when configured jobs doesn't execute as expected and results in errors. Page 13

14 If you think you can get away with not monitoring all of this, then you are not thinking correctly. You need to monitor all of this. BizTalk 360 can do this. Any operations product you have will also do this, such as Microsoft System Center Operations Manager, which is usually there already monitoring the very systems you are probably using BizTalk to interface with. If you do not have such a system, then you will need one, BizTalk 360 is a cheaper option, and will do the job. However not to state the obvious, what is monitoring those other systems that BizTalk is perhaps accessing? You are going to need an enterprise grade monitoring system; BizTalk 360 is not this system. That will also monitor BizTalk to provide the ONE view. As a window into other products about what they monitor, pretty much everything that BizTalk 360 does, and more. Microsoft System Center Operations Manager 2007 R2 or 2012 R2 The BizTalk Server Management Pack discovers the object types described in the following table. ( Name Category Object Type BizTalkGroup Logical entity Application view BizTalkHost Logical entity Application view BizTalkApplication Logical entity Application view ApplicationArtifact Logical entity Application view HostedApplicationArtifact Logical entity Application view ReceivePort Application artifact Application view ReceiveLocation Application artifact Application view Orchestration Application artifact Application view SendPort Application artifact Application view SendPortGroup Application artifact Application view BizTalkGroupDeployment System.Service Deployment View BizTalkInstallation Windows.SoftwareInstallation Deployment View Page 14

15 Name Category Object Type ServerRole Windows.ComputerRole Deployment View BiztalkRuntimeRole ServerRole Deployment View RulesEngineRole BizTalkServerRole Deployment View BAMRole Logical entity Deployment View BAMRuntime Logical entity Deployment View BAMAnalysis Logical entity Deployment View BAMAlerts Logical entity Deployment View BAMPortal Web site Deployment View BizTalkApplicationService Windows.ApplicationComponent Deployment View BizTalkHostDeployment Logical entity Deployment View Health Roll Up for BizTalk Deployment The following diagram shows how the health states of BizTalk Server deployment rolls up in this management pack. Page 15

16 Valid Scenarios There are some valid scenarios for using parts of BizTalk 360 and an enterprise monitoring solution. 1. The monitoring side should ideally be handled by an enterprise monitoring solution/product, if you do not have one of these, this can provide the stop gap until you get one and are up and running, so as you don t delay the deployment of BizTalk to production. 2. If you production environment is totally isolated, in that your support staff do not have access from their machines to access the environment via a one way domain trust even, this would prevent them from being able to run any local management consoles pointing at the BizTalk servers. In this scenario, the management abilities of applications hosts, orchestrations and ports would come in handy. The BizTalk 360 website would need to be exposed however, along with the BAM Portal website. It is a website, and via the magic of firewall rules, you could expose this. However if they are willing to expose a website they might via the same firewall rules, be able to expose more, it does depend on the location of your environment, if it is over the web, there is also ssl vpn that would also enable you to get a secure link to your prod environment. While you are working all of this out, as your lay administrator person might struggle with this, this again becomes a viable option. 3. You have too much money, and/or are too lazy. If you absolutely need to have it, and can t be bothered trying to figure out how to configure your operations management/monitoring tool, spend the money, I don t control your budget you do. At the end of the day you need to be monitoring BizTalk, and this does do that. If you are not monitoring BizTalk... you are running an environment at your own risk, add that to the risk register of your project, development 100% complete, go Live, 100% risk factor of ultimate failure... not a great situation to be in. Please sort it out. Page 16

17 Summary BizTalk 360 does work, it does provide a viable interface to BizTalk, it looks pretty has some nice features. If you are NOT monitoring BizTalk, then you need to, and this will do that. There is a whole bunch of other functionality that comes with the product, that there are already existing parts of BizTalk that perform these tasks, and in many ways are better. Some of the functionality is not such a good thing for a production BizTalk Environment, such as adhoc SQL queries against the databases in production. This is never a good idea. There is a wealth of detail and information you can gather from simply architecting your solution in BizTalk correctly, and utilizing BAM to gather all of this very valuable information. If you are NOT using BAM, then you need to, I will not pull any punches, if you are not using it, you really have no idea, quite literally, of your own environment, transactions, volumes response times, individual process detail and throughput. Once you have all of this in place, you will need to monitor the solution for health; the monitoring part of BizTalk 360 does this. However the question is always asked what are you using to monitor all of your other systems? If you don t have one then you will need one. But that, as any operations monitoring product WILL/MUST/SHOULD be able to monitor BizTalk, given the countless performance counters, and WMI methods that BizTalk exposes out of the Box, of which things like the free MOM/SCOM pack expose, however just pointing at a BizTalk environment will enable these operations tools to self discover. You ALWAYS need to configure this, even in BizTalk 360. If you have the Budget, and not much sense, buy both of these products, however the one to rule them all does come into play, when it comes to operations monitoring if it all filters and spills out from the single point, it s a single infrastructure and a single point of management, and a single set of skills. At the end of the day you need to monitor BizTalk, if you have a low budget and no enterprise monitoring tool, then BizTalk 360 is for you. However read the whole document and make an informed decision, BizTalk may be something you have never seen before, don t under-estimate it. Page 17