Proposed Solution. VoiceXML. Solution Architecture. Solution Architecture

Transcription

1 About us Developing Interactive Voice Response applications on JBoss Enterprise Middleware: A Case Study Ravi Srinivasan, Hector Hugo Gonzalez Hewlett Packard Consulting & Integration Group HP Open Source Middleware - Consulting Services Linux/Apache/Tomcat/JBoss Migration Services Performance Tuning & Optimization Feb Outline Introduction & Overview Solution Architecture Deployment Strategy Performance Summary Introduction & Overview

2 Customer Profile One of the largest providers of mobile services in Mexico More than 45 million subscribers and growing! 70 million+ subscribers all over Latin America Majority market share in multiple countries in Latin America Interactive Voice Response Systems (IVR) Handle large call volumes efficiently Efficient segmentation/routing based on caller profile Extended business hours Cost-effective The old system Not scalable Software approaching EOL Proprietary - integrating with other systems difficult A solution was needed and fast! High performance and meeting SLA s High Availability/fail-over capabilities Open and scalable platform Integration capabilities Standards based Simplified Monitoring and Management Failing to meet SLA losing revenue!

3 Proposed Solution Replace existing prepaid IVR functionality with VoiceXML applications developed on HP s OpenCall Media Platform (OCMP) and JBoss Enterprise Middleware VoiceXML W3C standard for interactive dialog between human and computer Interpreted by a Voice Browser OCMP provides a new IVR system built on industry standards that can act as a foundation for future Voice & Media services JBoss Enterprise Middleware Lower TCO. Zero license fees Open and Scalable platform * Source: Wikipedia Solution Architecture HP OpenCall Media Platform Legacy / Back office / Prepaid Systems as VoiceXML gateway Solution Architecture Telephony Network TCP/IP TCP/IP VoiceXML HTTP Application Server Database Called number 333 URL

4 Application Server IVR Application Design Talking Points OCMP Apache Web Server Static VXML documents TCP/IP Sockets To EJB or not to EJB VoiceXML Browser http request VoiceXML document JBoss Application Server controller view model XML XSL DAO Controller DAO Servlets XML-RPC JDBC TCP/IP sockets CSV Multiple legacy back-ends CSV formatted results over TCP/IP sockets Multiple third party external systems XML/RPC JDBC HTML Browser http request HTML JSP Java Virtual Machine DAO Coarse grained Stateless services Easier to provide HA facilities Performance achieved by scale out Document Types Static Documents (Apache) VXML documents that will not change, for instance: menus, application logic, etc. Audio files. Grammar files (used for user input validation). Deployment Strategy Dynamic Documents (JBoss) VXML documents built on-the-fly (XSLT), for instance: Balance Enquiries and Credit Replenishment results. Data fetched from back-end (RDBMS, Legacy etc) low level re-usable libraries to each of the back-ends

5 Central idea Redundancy Redundancy across all layers OCMP App Server Backend Hardware Redundancy RAID NIC bonding ( teaming ) Redundant PSU s/routers/switches Approach 1: HA / Failover using JBoss Clustering Approach 2: HA / Failover using HP ServiceGuard Freely available out of the box. Very flexible (dynamic cluster membership) HP ServiceGuard is an enterprise grade HA solution that is widely used for mission critical IT infrastructures Stability and performance issues with mod_jk Migrates IP of failed node to a Standby Active/Passive node Moving to Apache mod_proxy/mod_rewrite. Better stability and performance Flexible cluster configuration (Active-Active, Active-Passive etc)

6 HA using HP ServiceGuard CLUSTER HP Service Guard Disk Array (Raid 1) Management & Monitoring HP OpenView for monitoring (high-level) Plugins for JBoss, Tomcat, Apache & MySQL available for free! Custom MBeans exposes some run-time data AppSvr1 Red Hat Enterprise Linux AS 4 JBoss1 (Active) Intra-cluster communication AppSvr2 Red Hat Enterprise Linux AS 4 JBoss2 (Active/Passive) HP SIM (System Insight Manager) for Rack/Blade level monitoring Customized Redhat OS Customized software appliance like stack Slimmed down version of Redhat Linux AS 4 Quick & Easy Provisioning Image based (for homogeneous hardware) Kickstart LinuxCOE Patches and Updates on a need only basis Customized JBoss Customized for fast startup (15-20 seconds) Custom Logging policies Minimalist jboss-service.xml Synchronized deploy directories via NFS/FTP Patches and Updates on a need only basis

7 Performance focus areas Performance Apache/JBoss OS & Hardware JVM Application Source Code Apache/JBoss Tune for higher concurrency MaxClients ServerLimit maxthreads (JBoss/Tomcat) Disable unwanted modules/services Access logging separate partition (mount noatime ) Offload SSL processing to dedicated hardware Prefer mod_proxy_* over mod_jk for Apache 2.1/2.2 Apache KeepAlive timeout very low or turn OFF Note on JBoss Logging Log Levels prefer ERROR or WARN (in Production) Configure log file to rollover after it reaches a preset size Avoid using NFS for log directories Separate partition mount with noatime option Log4j jars inside your WAR can cause problems use scoped logging config use JDK java.util.logging

8 OS & Hardware Hardware Fast CPU s with large L2 caches 15K RPM SCSI disks, plentiful RAM Gigabit network switches File descriptors fs.file-max, ulimit n (per process) TCP stack tuning rmem_default, rmem_max, wmem_max, wmem_default tcp_sack, tcp_timestamps OFF (YMMV) JVM Aim for stability first! Server side apps tune for throughput (mostly) Important params (YMMV) -Xmx, -Xms, Eden space/ Survivor space sizing UseParNewGC (for Eden), CMS collector (for tenured) PermSize (to avoid OOM errors) JVM Tools VisualGC/jconsole (Sun JVM) JRockit Mission Control (BEA JRockit) HPJTune (Free!) Detailed Code/JVM Profiling Yourkit JXInsight Last but not least.application source code Profiling revealed bottlenecks in the DAO->backend layer Socket creation and teardown very expensive Thread dumps (SIGKILL) showed many threads waiting - contention on shared resources Socket pooling improved performance tremendously Processing Read only Transactions from a separate dedicated back-end node also improved performance tremendously

9 Performance numbers approx. 24 million requests served per day Stability of the middleware platform has been excellent Summary < 30 seconds fail-over and recovery (Active-Passive) 30 million subscribers migrated over with minimal disruption In conclusion Stability and Performance of the system has resulted in customer s SLA being met Mission critical IVR applications can be scaled on commodity hardware and open platforms like the JBoss Enterprise platform Reduced TCO for the customer by using open source software and commodity hardware The open and standards compliant nature of OCMP and JBoss makes it easy to add new functionality and services Next Steps We are already in the process of implementing new services on the JBoss Enterprise platform We expect technologies like Seam, JBoss Messaging and ESB to play a very important role in the future Evaluating jbpm and Drools for workflow services We are planning to implement JAX-WS style web services for interoperability with 3 rd party systems

10 Questions? hp.com Gracias!