Monitoring with Optimize for Infrastructure. Stewart Loewen Chief Solution Architect, Software AG

Transcription

1 Monitoring with Optimize for Infrastructure Stewart Loewen Chief Solution Architect, Software AG

2 Objectives of Monitoring Predict future issues Get alerted on potential issues before they become costly outages Take actions Take action automatically on applications when rules violated Make informed decisions Utilize real data to make decisions on optimizing your applications Monitor current state Get real time visibility on infrastructure performance Understand your infrastructure Discover key infrastructure components that support business processes 27 June 2011 ProcessWorld

3 Optimize for Infrastructure Comprehensive View of IT Environment An automated searching facility quickly identifies deployed webmethods components throughout your network and allows easy identification of appropriate KPIs to monitor. In addition to webmethods components, any SNMP-enabled component can also be monitored. 27 June 2011 ProcessWorld

4 27 June 2011 ProcessWorld

5 Integration with Mashzone 27 June 2011 ProcessWorld

6 Optimize for Infrastructure - Key Features Intelligent Self-Learning and Alerting webmethods Optimize for Infrastructure monitors each KPI and learns what is normal based on historical patterns. Understanding what normal behavior is allows the user to created behavioral based alerts that align with the business needs and reduce false alerts. Real-Time, Continuous Process Monitoring and Analysis Use built-in reporting and analysis capabilities to examine root causes of specific issues and uncover trends and patterns related to underlying process inefficiencies. Innovative, Open Standards-Based Technology Platform webmethods Optimize for Infrastructure leverages the integration capabilities of the webmethods suite to quickly access information about mission-critical business processes within and across heterogeneous IT systems. Full integration with preexisting LDAP infrastructure reduces setup time and speeds time to value. Integration with ARIS MashZone Display any Optimize for Infrastructure KPI in MashZone for a real-time dashboard view of server performance. 27 June 2011 ProcessWorld

7 Managing Operations for Large Scale webmethods Implementations Neil Prater Distributed Systems Architect / Fiserv

8 About Fiserv Fiserv, Inc. is the leading global provider of information management and electronic commerce systems for the financial services industry. We are trusted by more than 16,000 clients worldwide including: Banks Credit Unions and Thrifts of All Sizes Mortgage Lenders and Leasing Companies Telecommunications and Utility Companies Brokerage and Investment Firms Healthcare and Insurance Providers Retailers and Municipalities 27 June 2011 ProcessWorld

9 How Fiserv is Organized Fiserv is organizationally split based on the type of solutions sold in the industry: Digital Payments Solutions (19 solution sets): Productions related to all aspects of payment related transactions Depository Institution Services (27 solution sets): Management and Processing of account based transactions Risk and Compliance Group (11 solution sets): Risk prevention, Market Investment, Loan Management and Processing type products Enterprise Technology: Information Technology service provider to the above operating groups 27 June 2011 ProcessWorld

10 webmethods at Fiserv We have several Service Buses that comprise a logical Enterprise Service Bus. This is because: Not all Fiserv products need to integrate with all Fiserv products Fiserv maintains several operational maintenance windows to serve the needs of our customers across the globe Limits the exposure to the business in the event of a localized outage Audit requirements 27 June 2011 ProcessWorld

11 webmethods Statistics at Fiserv 7 Major Operating Datacenters located in the US 7 Operating regions (Performance Lab, Development, QA, Customer QA, Customer Certification, Production, DR) 300 Integration Servers 90 Broker Servers 85 MWS Servers 30 BAM/BPM Servers 3 Central Management Infrastructure Environments Integration Server, Analytic Engine, Broker 27 June 2011 ProcessWorld

12 Management 27 June 2011 ProcessWorld

13 How Do We Manage Such a Large Number of Servers? The key to managing large scale implementations of any technology is automation and standards, standards, standards. To efficiently design, deploy and manage the Fiserv webmethods footprint, a dedicated team of system, application, database, security and network engineers came together to form a standard webmethods footprint: Build Standards Deployment and Configuration Standards Support and Monitoring Standards Patching and Upgrade Process Standards 27 June 2011 ProcessWorld

14 OS Build Standards All components receive the same Operating System and patch levels Systems are all silently provisioned utilizing Microsoft System Center Configuration Manager (SCCM) OS is deployed Supporting patches and software packages are deployed. Any rogue software installations not registered in SCCM are flagged System security is enabled through Active Directory users and groups Group policy is applied every 15 minutes; only operational support groups have access to Production installs of webmethods 27 June 2011 ProcessWorld

15 webmethods Deployment Process A single image, comprised of all webmethods products for a particular version (i.e. 8.0) is built. This image is added to the Microsoft SCCM repository and available via a NAS fileshare BUILD webmethods Installation Image DEPLOY Text Response Files CONFIGURE* Powershell Configuration Files *All webmethods servers retain the same configuration structure. Extended settings and Adapter installations are the same on all IS s; Broker stores are all the same size and the Broker Instances follow a naming standard. 27 June 2011 ProcessWorld

16 Deployment Code Sample set service_account_username="ec\webmethsvcc" set service_account_password= accountpassword "D:\program Files\Java\jre6\bin\java" -jar -Xmx256m "\\webmethods- NAS\SoftwareAGInstaller80SP2.jar" -readscript "\\webmethods-nas\ 8.0\Deploy\webMethods80-IS.txt" xcopy "\\webmethods-nas\8.0\ejb" D:\webMethods80\IntegrationServer\packages\WmEJBAdapter\code\jars xcopy "\\webmethods-nas\8.0\jdbc" D:\webMethods80\IntegrationServer\packages\WmJDBCAdapter\code\jars sc config "softwareagwebmethodsintegrationserver_8.0" obj= "%service_account_username%" password= "%service_account_password%" sc config "SAGCTP80" obj= "%service_account_username%" password= "%service_account_password%" 27 June 2011 ProcessWorld

17 webmethods Database Deployment and Management All webmethods databases are created by the webmethods Database Configuration tool. Every group of IS s get their own database schemas as listed below: Archive IS Core IS Process All IS databases are archived following Software AG s database archiving stored procedures. For non-production, we keep 7 days of data in the IS Core tablespace. For production, we keep 30 days of data in Production. Any data older than 6 months is removed from the Archive database. Since the database platform is on SQL Server, we rely on Microsoft tools to manage tablespace growth and diskspace consumption. 27 June 2011 ProcessWorld

18 webmethods Management All webmethods servers follow a naming standard that allows us to quickly determine: Datacenter location, Network zone, OS platform, Environment and webmethods component (wmis wmbrk wmmws) Due to the lack of a webmethods management tool in v8.0.1, we are managing server service details in a spreadsheet. Efforts are underway to create code to pull data from the webmethods component and put that information into a database. 27 June 2011 ProcessWorld

19 webmethods Security Management All webmethods components are configured to use Active Directory through the MWS LDAP service. Due to the various industry standards we are required to follow, we prohibit the creation and use of any local webmethods accounts: Active Directory Groups are created and Active Directory users are added to these groups The AD groups are applied to various MWS and IS roles SSL authorization is done via service accounts in AD Access to Production environments is limited to the Engineers and Operational support people who support the platform. This ensures no changes are made while the system is taking live traffic To ensure proper auditing of the webmethods components, we ve enabled security auditing within IS. 27 June 2011 ProcessWorld

20 webmethods Code Management and Deployment All IS code is stored in a Subversion repository and all Integration Servers are connected to the repository. The repository is configured to reflect source trees for each product and for each operating region. webmethods Deployer is utilized to deploy code to various IS s in all regions. 27 June 2011 ProcessWorld

21 27 June 2011 ProcessWorld

22 Monitoring 27 June 2011 ProcessWorld

23 webmethods Monitoring Monitoring of the OS Layer: Achieved through the usage of 3 rd party tools such as Microsoft System Center Operations Manager and Computer Associates Management Suite OS health / Memory and CPU availability webmethods service up down Disk space availability Monitoring of the webmethods product stack: Achieved through Optimize for Infrastructure Separate central management infrastructures for Non- Production, Production and Production DR All Integration Servers and Brokers are added to their respected CMI 27 June 2011 ProcessWorld

24 webmethods Monitoring, Continued Rule of Thumb: Less is more. As a result, Fiserv has standardized on the creation of 12 custom rules that get applied to every Fiserv solution that utilizes the Integration Server and/or Broker. Integration Server IS Down IS Broker Transport Down IS HTTP Port Down IS HTTPS Port Down IS Package Down IS JVM Free Memory Low Broker Server Broker Down Broker Server Down Broker Server Disk Low Optimize Analytic Engine Down Anayltic Engine Processing Time BPM/BAM Flow Failed 27 June 2011 ProcessWorld

25 webmethods Monitoring, Continued While we are collecting KPI s for more than we are logging, we only use them for reference. We learned that too many rules became hard to manage and created a lot of false positives. All application and service logging is expected to be handled within the application itself and not by webmethods. Fiserv has strict development standards based on application/service logging. 27 June 2011 ProcessWorld

26 27 June 2011 ProcessWorld

27 Applying Patches 27 June 2011 ProcessWorld

28 webmethods Patching Proactive Patching To maintain industry compliance (SAS70/SSAE16, PCI, etc) requires that Fiserv stay current with all vendor security patches. To assist in vendor support best practices, Fiserv subscribes the webmethods platform to an internal Patch and Update Process: Proactive Patching Once a quarter Only core fix packs are applied All servers are put on a schedule to receive their respective patches 2-3 weeks of regression testing If any regression testing fails, entire patch process is put on hold 27 June 2011 ProcessWorld

29 webmethods Patching Reactive Patching Reactive Patching Performed when a critical security fix has been presented or when an issue has been exposed Determine if the fix is part of a Fix Pack or if it s a singular patch All servers are put on a schedule to receive their respective patches 2-3 weeks of regression testing If any regression testing fails, entire patch process is put on hold 27 June 2011 ProcessWorld

30 How we patch webmethods Existing Strategy: STAGE CREATE DEPLOY TEST Download Patches to a NAS Share Powershell Deployment Files Powershell on Affected Servers Regression Test Go-Forward Strategy: STAGE DEPLOY TEST Create Patch Image with Update Manager Image with Update Manager Regression Test 27 June 2011 ProcessWorld

31 Upgrades 27 June 2011 ProcessWorld

32 webmethods Product Upgrades Fiserv has standardized on building new infrastructure and migrating solutions over to it. This facilitates: A backout plan in the event of a migration failure Revamping of the OS level Revamping of hardware A clean slate 27 June 2011 ProcessWorld

33 Thank you!