Reducing the Operations Burden and Increasing QoS with WSO2 Platform Management and Administration Tactics Charitha Kankanamge Chamith Kumarage
Agenda Common Deployment Patterns Deployment Synchronization Registry Mounting Setting up a worker-manager separated ESB cluster Demo/Hands-on Deployment Automation Server Monitoring Production Deployment Guidelines Common issues in Production Setups
WSO2 Product deployment patterns Standalone Clustered Cloud based
Standalone Deployment User engages with one or many independent nodes
Clustered Deployment User engages with the load balancer Highly available Concept of worker and manager nodes
Cloud Based Deployment Extension of the Clustered deployment Highly scalable Span across multiple IaaSs
Deployment Synchronization Distribute deployment artifacts and related meta-data across the cluster Make the cluster nodes consistent Automated synchronization based on SVN or Git Management nodes commits and worker nodes check-out
Registry Mounting An approach to share registry space across cluster nodes Local Data repository Configuration registry Should not be shared Shared across multiple nodes of the same product Governance registry Shared across product platform
Worker-Manager Separated Cluster Demo/Hands-On Setting up user management and registry databases Configuring the Elastic loadbalancer Configuring WSO2 ESB as a management node Configuring WSO2 ESB as a worker node
Deployment Automation Abstracted configuration Polymorphised product modules One file to control the whole deployment Same product module in different forms Configuration management Patch distribution Orchestration Powered by Puppet
Server and Health Monitoring JMX based monitoring WSO2 BAM for service monitoring Message Tracing Tools Wireshark, tcpmon System monitoring tools JConsole, Graphite Ganglia, Cacti, Nagios Alerting Email, SMS, Phone-call alerts
Production Deployment Guidelines Changing the default administrator credentials Replacing SSL certificates/keys Changing the default ports Hardening the Operating System Turning off unused services/ports Switching the default H2 registry database to a production ready database Using secure vault to encrypt passwords in configuration files Tune database connection pools
Production Deployment Guidelines contd.. Tuning nhttp/pass-thru properties Disabling HTTP access logs Tuning heap memory Tuning GC Running servers as background processes nohup, YAJSW Connecting to an external cassandra in case of MB/BAM Avoid using full level logs in ESB log mediator
Patching Follow the recommended procedure Automate the patch applying process Test the patch in dev and staging environments Automated process is a must Update production servers and restart in a RR manner Maintain a log for patches
Common errors in production IO errors - Connection reset by peer, Broken pipe, Too many open files, I/O reactor has been shut down Database connection pool errors Automatic server restarting due to starting of servers using daemon.sh/wrapper OutOfMemoryErrors SSL handshake errors due to misconfigurations Wrong use of message builders/formatters - com.ctc.wstx.exc. WstxIOException: Invalid UTF-8 start byte 0xff
Production Troubleshooting Tips carbondump WSO2_HOME/bin/carbondump.{sh,bat} Replicate the issue in staging Use of wire-logs to capture complete request/response flow Report JIRAs with detailed information Upload carbondump output to the provided FTP server Attach the relevant deployment artifacts
Charitha Kankanamge Chamith Kumarage Blog: http://charithaka.blogspot.com Blog: http://gonesimple.org Follow: @charitha_ka Follow: @gnuchami
Thank You