Reducing the Operations Burden and Increasing QoS with WSO2 Platform Management and Administration Tactics

Transcription

1 Reducing the Operations Burden and Increasing QoS with WSO2 Platform Management and Administration Tactics Charitha Kankanamge Chamith Kumarage

2 Agenda Common Deployment Patterns Deployment Synchronization Registry Mounting Setting up a worker-manager separated ESB cluster Demo/Hands-on Deployment Automation Server Monitoring Production Deployment Guidelines Common issues in Production Setups

3 WSO2 Product deployment patterns Standalone Clustered Cloud based

4 Standalone Deployment User engages with one or many independent nodes

5 Clustered Deployment User engages with the load balancer Highly available Concept of worker and manager nodes

6 Cloud Based Deployment Extension of the Clustered deployment Highly scalable Span across multiple IaaSs

7 Deployment Synchronization Distribute deployment artifacts and related meta-data across the cluster Make the cluster nodes consistent Automated synchronization based on SVN or Git Management nodes commits and worker nodes check-out

8 Registry Mounting An approach to share registry space across cluster nodes Local Data repository Configuration registry Should not be shared Shared across multiple nodes of the same product Governance registry Shared across product platform

9 Worker-Manager Separated Cluster Demo/Hands-On Setting up user management and registry databases Configuring the Elastic loadbalancer Configuring WSO2 ESB as a management node Configuring WSO2 ESB as a worker node

10 Deployment Automation Abstracted configuration Polymorphised product modules One file to control the whole deployment Same product module in different forms Configuration management Patch distribution Orchestration Powered by Puppet

11

12 Server and Health Monitoring JMX based monitoring WSO2 BAM for service monitoring Message Tracing Tools Wireshark, tcpmon System monitoring tools JConsole, Graphite Ganglia, Cacti, Nagios Alerting , SMS, Phone-call alerts

13 Production Deployment Guidelines Changing the default administrator credentials Replacing SSL certificates/keys Changing the default ports Hardening the Operating System Turning off unused services/ports Switching the default H2 registry database to a production ready database Using secure vault to encrypt passwords in configuration files Tune database connection pools

14 Production Deployment Guidelines contd.. Tuning nhttp/pass-thru properties Disabling HTTP access logs Tuning heap memory Tuning GC Running servers as background processes nohup, YAJSW Connecting to an external cassandra in case of MB/BAM Avoid using full level logs in ESB log mediator

15 Patching Follow the recommended procedure Automate the patch applying process Test the patch in dev and staging environments Automated process is a must Update production servers and restart in a RR manner Maintain a log for patches

16

17 Common errors in production IO errors - Connection reset by peer, Broken pipe, Too many open files, I/O reactor has been shut down Database connection pool errors Automatic server restarting due to starting of servers using daemon.sh/wrapper OutOfMemoryErrors SSL handshake errors due to misconfigurations Wrong use of message builders/formatters - com.ctc.wstx.exc. WstxIOException: Invalid UTF-8 start byte 0xff

18 Production Troubleshooting Tips carbondump WSO2_HOME/bin/carbondump.{sh,bat} Replicate the issue in staging Use of wire-logs to capture complete request/response flow Report JIRAs with detailed information Upload carbondump output to the provided FTP server Attach the relevant deployment artifacts

19 Charitha Kankanamge Chamith Kumarage Blog: Blog:

20 Thank You