Panaya Inc. 7 Best Practices When SAP Must Run 24 x 7 Maintenance windows are all but non-existent and infrastructure upgrades are a fact of life, but you can still keep SAP running around the clock by following these best practices. For questions and additional information e-mail info@panayainc.com or visit Page 1
As companies come to rely on SAP for more and more applications they have to come to grips with a harsh reality: shrinking maintenance windows. Indeed, for a global company with operations in different parts of the world, or a firm that has SAP handling customer transactions from a public Web site, that window may be non-existent. In effect, these companies have to keep SAP running 24x7, 365 days a year. That complicates tasks such as installing new software or updating infrastructure such as hardware, operating systems and databases. The good news is that automated tools exist from companies such as Panaya, Inc. to help with the thorniest problems, such as determining what needs to be tested when you install a Support Pack or Enhancement Pack. And there are ways to tackle most other issues with little to no downtime. We ve assembled seven of the best ideas from SAP experts, along with a few bonus tips that can help keep any SAP environment running more reliably. 1. Virtualize. Virtualization technology is a boon for companies that have to run SAP around the clock. Virtualization enables companies to create multiple instances of their environment and take one of them down as needed while the other is in production, says Greg Leiner, SAP Solutions Engineer with Rimini Street, a third-party provider of support services for SAP and other ERP systems. Once you conduct any required maintenance on the primary gold machines, you can swap back over. You will need a tool to identify what data has changed while the gold machines were offline. Vendors of such tools include BackOffice Associates and SAP, with its NetWeaver Master Data Management, Leiner notes. Having multiple virtual SAP environments can also protect a company in the event of a failure of the primary SAP instance, says Frank Powell, chief operating officer for Symmetry Corp., an SAP hosting provider and consultancy. VMware, in particular, has a good high-availability solution with its vmotion technology, which allows for the live migration of virtual machines. With vmotion, we re talking very minimal downtime, less than a minute in some cases to move from one server to another, Powell says. Page 2
Virtualization technology is a step beyond clustering technology, he notes. A lot of people assume a cluster means you re up 100% of the time, but that s not true. If one side of the cluster dies and you need to go to the other side, it s just like rebooting the machine, essentially, he says. The database has to restart, perform a rollforward/rollback of transactions, then SAP has to restart/reconnect, all of which can take 3 to 5 minutes. It s still a pretty good highavailability solution, but there is that caveat, he says. 2. Employ a shadow database. Axel Angeli, founder of the Logosworld network of SAP and SOA consultants, says the simplest solution to dealing with unplanned SAP failures is to run a shadow database. Every write to a database triggers a similar update in a database in a remote place, he says. If the main database fails the system will switch to the shadow database, while buffering the activities until the original database is up again. Leiner agrees that approach can be helpful but says it s best to use data management tools to help with the buffering, to ensure all transactions are captured and updated once you bring the failed database back online. 3. Break it down. Probably the most challenging aspect to keeping SAP up around the clock is upgrading SAP itself and applying Support Packs. The solution is breaking the application inoto smaller entities and components, Angeli says. He advises running different instances of various SAP application areas, such as Financial Accounting (FI), Materials Management (MM) and Production Planning (PP) on different servers one application area, one box. If something goes wrong during an upgrade or patching process, it will then only affect that single application. Even better, For middleware and SOA-based application servers like SAP PI, he suggests running several servers in parallel. Then you can apply fixes to one server and redirect all transactions to that server. If the fix is unstable, you can switch back to the unpatched server farm to to handle transactions. Services such as Panaya Upgrade Automation and SAP Support Package Automation can further simplify the task, by pointing you directly to the fixes that you need to apply and the exact code that needs to be changed. Page 3
4. Use your DR site. TiVo, Inc. is one company that has to keep SAP running around the clock because it s used to handle all customer-facing billing and ordering systems from its Web site, says Richard Rothschild, Senior Director of IT, Facilities & Security for the company. To deal with major infrastructure upgrades, he takes advantage of the firm s disaster recovery site. If we need to be down for more than a few hours, we can fail over and run SAP on that site, he says. We re planning to upgrade to newer version of SAP. That s a fair amount of downtime, so it s likely we ll run out of the other site to do that. He also plans to carve the upgrade into as many pieces as possible, rather than try to upgrade everything at once, and do as much testing up front as possible. (For more tips from TiVo, click here.) 5. Keep track of changes. The vast majority of downtime that occurs is due to unexpected problems because companies don t have good change management and maintenance processes, Powell says. Perhaps a Support Pack that worked fine on a development system caused a problem on a production system because the two systems had different operating system or database patches. Maintaining identical levels of patches and applications in dev, QA and production at all times will save you unanticipated downtime, he says. When we get a new customer, our first step is to get those three systems in sync, so they re at the same OS, database and SAP patch levels. Keeping all the systems in sync requires following good change management procedures, which includes keeping detailed documentation. Following a standardized change management process such as ITIL is ideal, but at the very least companies should have a change management board or peer review process that takes their SAP architecture into account. If you have SAP ECC, BW and CRM, it s very likely changes in one of those systems may require changes in the other two, Powell notes. So part of the change management process is interrogation, which entails involving the right people to assess the scope of the change and what applications it impacts. Having a change board comprised of enough members that they have the scope of the entire landscape is very important. Page 4
Here again, Panaya can help automate the process with its online service that points customers to the exact changes that have to be made to each system for any given upgrade. 6. Monitor maniacally. Another aspect of proper system maintenance is monitoring, Powell says. Keeping an eye on statistics such as the rate of growth of your central SAP database can help you head off unexpected problems. The unexpected problems are the ones that have the biggest impact, he says. When you have a 2-hour outage on a Tuesday afternoon after your archive directory filled up because you weren t keeping an eye on it, that s a big, big problem. SAP Solution Manager is a solid tool for monitoring the SAP environment and sending alerts when predefined thresholds are reached. It s a tool we don t see many customers leveraging, Powell notes. Beyond monitoring core environmental aspects such as databases, Solution Manager can also monitor specific business processes, such as order entry. If orders are queuing up faster than the system can fulfill them, that can raise a flag. It might not be an outage, but could be having a severe impact on a major portion of your business, he says. Another SAP tool that Powell says few customers take advantage of is the SAP EarlyWatch Report. EarlyWatch is a reporting service where SAP will collect information on your SAP applications, databases and operating systems, based on parameters you configure in Solution Manager, and deliver a report that paints a picture of the overall health of your system and identifies potential problems. It shows if performance is suffering and in what particular application, gives you database stats and lots of other statistical data, Powell says. Such information can be valuable in alerting administrators to changes in the business that they may not be aware of. A business unit whose database was growing at 2 GB per month may suddenly shoot up to 10 GB per month because of some change to a business process that wasn t communicated to IT. An EarlyWatch Report will bring that to your attention and prevent you from burning through your disk space, he says. 7. Test, test, test. When you need to make changes to your SAP implementation whether a Support Pack, Enhancement Pack, database update or the like - it s crucial to follow a good test Page 5
procedure. Have a checklist of what needs to be tested, along with the specific business units and people who should be involved, Powell suggests. There have been a lot of upgrades where, Monday morning a certain business function that wasn t tested properly fails, he says. One of his clients failed to test the Web interface for their online ordering system, for example. It took them 2 to 3 days to work through that. In addition to pointing out what changes are required during an upgrade, Panaya s tools also identify what needs to be tested. The solutions scour your SAP implementation and provide detailed testing plans, even highlighting the exact lines of code that will be affected during an upgrade or Service Pack installation so they can be fixed prior to testing. Bonus Tips for Optimal Performance SAP system uptime goes hand in hand with proper maintenance and a deep understanding of the business requirements the system needs to meet. With that in mind, we offer these four tips from experts at Symmetry Corp., an SAP hosting provider and consultancy, to help ensure your SAP implementation continually runs with optimal performance. 1. Solid, redundant environment. A good environment is a good way to prevent downtime, says Nick Miletich, client manager with Symmetry. The last thing you want is a server overheating or a power outage happens and you don t have UPS backup or redundant power. By the same token, servers should have redundant network connections along with redundant channels between servers and disk. While providing such redundancy does cost money, it s a marginal additional cost that will pay for itself many times over if its saves you even an hour of downtime. 2. Scheduled reboots. Over time, as you make changes to SAP applications and databases, you need to ensure that the system will restart properly in the event of an outage. A scheduled reboot is the best way to provide that assurance, Miletich says. Page 6
We ve had clients who didn t reboot for 2 years, he says. So no one remembers what nuances changed on a server. And suddenly you have a 4-hour outage while everyone s trying to figure out what happened. Scheduled reboots are especially important in the Unix world, since you can make changes to a Unix system while it s running and without rebooting. But to make the change permanent you should also edit a text file so that the same parameter takes affect at boot time, says Frank Powell, chief operating officer at Symmetry. If in the heat of the battle you forget to edit that text file, and then you don t reboot for 14 months, you ll forget what the change was. When conducting any system maintenance, a reboot should be part of the plan, he says. 3. Tune for the worst. Certain businesses have a specific time period during which their SAP system experiences far greater stress than usual. Symmetry works with companies in the magazine distribution and grocery delivery businesses, for example. Such firms take orders all day, putting little stress on SAP, but process the orders in the middle of the night in order to get delivery trucks ready to roll early in the morning. A couple of these companies didn t size for this short window where the system is taxed at a high level, Powell says. They go live and all of a sudden the 2-hour processing window they planned for, from midnight till 2 a.m., was stretching till 6 a.m. 4. Understand business unit requirements. Avoiding such issues requires a deep understanding of the requirements for each of your business units and when their most critical periods of SAP usage are. That s a particular challenge for a global organization, but one they can often use to their advantage, Powell says. For example, you don t want batch jobs interfering with the performance of real-time transactions. In a global environment, perhaps the servers in Europe can be used to support batch jobs during business hours in the U.S. SAP has great tools to allow you to turn on batch processes on different servers and dictate where they run, he says. Page 7
About Panaya Panaya's Software-as-a-Service solutions enable companies that use SAP to save up to 50% of their application lifecycle costs and minimize the risks associated with system changes. Utilizing cloud-based simulation to analyze the impact of pending changes, Panaya automatically pinpoints which custom programs will break as a result of an upgrade or support package implementation and automatically fixes most of these problems. Panaya provides a complete solution for managing these changes, explaining how to fix the anticipated issues, fixing most of them automatically, suggesting the most efficient test plan, and calculating required project budget and resources. It s the only upgrade automation solution that automatically fixes custom code issues and generates test scenarios. To learn more, visit Panaya at. SAP is a registered trademark of SAP AG. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. Page 8