Agresso: Accidental ERP Disaster Forgiveness A New Type of Cloud Deployment Insurance Executive Summary

Transcription

1 Inside: - Executive Summary - Deep Inside Agresso s Forgiveness Center - Forgiveness in a Multi-Tenant Cloud World - Why Forgiveness is Hard for Some but Easy for Agresso - FAST Forgiveness A Business Necessity Agresso: Accidental ERP Disaster Forgiveness A New Type of Cloud Deployment Insurance Executive Summary We recently reviewed several cloud application software solutions to better understand how they handled a certain kind of recovery problem: the accidental, or purposeful, destruction of a number of transactions, table entries or other critical information in a production cloud system. The results were disappointing and eye-opening to say the least. We observed that: - Recovery from this specific type of cloud disaster which is rarely discussed or warned of publicly beyond small-print contract exception clauses -- can be expensive, time consuming and potentially fatal to a business. - Many software providers have disaster recovery technology and procedures to protect their cloud solutions and data centers. But scant little capability exists to help an individual enterprise resource planning (ERP) or other business software application customer when a business technology disaster is caused by one user within that system. - Many cloud application software companies routinely delegate the planning for and handling of this potential data disaster scenario to the ERP buyer, recommending that customers acquire their own backup and recovery technology. Meaning: this type of cloud deployment insurance is, for some ERP cloud deployments, not provided by the vendor or, if it is, it is expensive and takes too long to complete. We then reviewed the capabilities of one particular cloud ERP software solution provider: UNIT4 Agresso. The differences turned out to be significant. Agresso appears to be particularly adept at handling these data disaster recovery situations for its cloud software customers. They do so very quickly and at no additional cost. Read on to see how other vendors are failing to serve their cloud application customers. Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 1

2 Forgiveness: A Capability Every Cloud ERP Buyer May Need Someday Every software user has had that OH NO!!!!! moment. It s that instant when you realize you ve accidentally erased some/all of your files, re-formatted a hard drive, failed to save your work, or worse. It s a bad feeling, and you ll probably remember it for years. It s bad enough when it happens to your personal computer and your personal files. But, when it happens at work, it could be a business disaster at worse, or a career limiting move at best. Accidents are the reason why restart/recovery utilities and disaster recovery plans exist for business systems. Users of on-premise ERP solutions anticipate this situation and acquire appropriate disaster recovery tools. They re there for a good reason. Yet, if you thought that a cloud software solution automatically comes with this, think again. Accident forgiveness may be a much promoted aspect of several automobile insurance companies policies, but, there are any numbers of situations when business software users can accidentally destroy or misclassify business data, processes or critical reporting. These include situations like deleting or incorrectly classifying a legal entity, a subledger, a customer/prospect/partner master file and all of the associated critical tables or databases. When this happens, and it happens frequently and regularly, both the employee and employer could be in real trouble. How much trouble? Deep Inside Agresso s Cloud Customer Forgiveness Center Agresso, a multi-tenant cloud ERP software provider, has a support center that frequently gets a number of interesting calls. These aren t the usual support calls where someone has a question about how to enable a specific bit of functionality. No, these are the calls where people have screwed up big time. These people need help fast because their goof has caused their firm s ERP software to no longer work or work correctly. What kinds of mistakes did these people make? Well, here are just some of the ones I learned about and how they were resolved: A university erased all of their suppliers (The data was fully restored half an hour later.) A real estate company made changes to their accounting rules. This triggered an error that over time spread to the whole accounting system. (Agresso had old backups to fall back on once the customer figured out their mistake a couple of months later.) An entire legal entity was erased (and all data associated with it) from a multi-company setup. After confirming multiple warnings from the system that this couldn't be undone, the administrator realized that the wrong company had been erased. (The deleted company s system was back in business within an hour.) Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 2

3 A project administrator at a professional services company updated the price lists for all projects with an increase only intended for one project. (Recovery of the correct information was achieved within 2 hours.) A financial administrator in a commercial organization accidentally imported and posted a large number of general ledger adjustments to several accounting periods in the production ERP system s environment instead of the test environment. (Recovery of the full production environment was achieved within 2 hours.) An accounting firm made a bulk import from an Excel spreadsheet that contained faulty data. (Although this could have been resolved by the customer manually, a restore of the most recent database turned out to be the quickest solution for the customer.) A company within the healthcare sector requested a restoration after accidently closing a salary period and starting a new one, without the possibility for a rollback. (The solution here was to restore a copy of the database made the previous day.) You re probably thinking: Don t all systems do this? Our old on-premise system is setup to handle these kinds of things. In the cloud world, especially the multi-tenant space, it isn t that easy or common. ERP accident forgiveness, in the cloud world, is a rare thing. State of Forgiveness in Multi-Tenant Cloud Solutions How much forgiveness is out there? Not a lot, it would seem. Software vendors have created a number of capabilities that protect all of their customers should the vendor s data center or a specific server experience the more talked about disaster recovery scenarios of fires, floods, earthquakes, power outages, system-wide failure, etc. These vendors often commit to terms such as: - Rolling customers over to another server or data center with minimal disruption - Maintaining mirrored servers in their production and failover data centers - Guaranteeing up-time rates of 99.5% or better - Having applications up and running within 24 hours after a major disaster - Creating local backups every 30 minutes - Generating data backups to a remote disaster recovery center every 2 hours - Preserving at least 24 hours of transactions in redo logs The main disaster recovery focus of most vendors is to prepare for problems that affect their data center and their computer hardware. It does not appear to be to protect their users or more importantly their users data and processes -- from the users own, either well- or ill-intentioned, mistakes. Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 3

4 Specific to the forgiveness issue, we learned that: - Many vendors maintain a mirror copy of your transactions; however, this information is principally used by the vendor for their disaster recovery needs (not yours). - Some vendors offer a limited forgiveness solution but it can be pricey and take too much time to complete. For example, one major cloud solution vendor stated that: o your firm must exhaust all other options before they will attempt to restore your system o the fee for this restoration will be a minimum of $10,000 (USD) o the restoration will take a minimum of 15 business days - Some vendors allow users to export their data to comma delimited/separated files or other file formats but this type of disaster recovery data reclaim would still require re-processing via an upload program. Through telephone calls we conducted with backup software vendors and other sources, we learned that the elegance and ease of these solutions appears to be dubious. Notice how all of the arrows go oneway. The recovery processes and tools are mostly absent. Third party products are just that: products not supplied by the application vendor. All of these costs as well as costs for local backup media are borne by the customer. The Vendor s databases and backups are predominately for the vendor and their backup/recovery needs. The CSV file may or may not contain all the necessary data elements and may be infrequently generated. It may also require interpretation to understand what should/shouldn t be reprocessed. If you really want ERP cloud solution protection and forgiveness, your organization may need to lean on some third parties that have offer special solutions to fill some or most of this protection gap. We observed that: - Many backup utilities exist to move data from a cloud solution to another location for an added fee. While some of these are quite affordable, the tools mostly work in one direction. They take Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 4

5 data from the cloud solution s databases and move it to another cloud or to an on-premise file or database. The key value of these tools often is to create another data store so that a customer can run reports or do other processing not to restore a full ERP application in context - These add-on products may provide a faster path to recovering data than many vendor provided options. Users may be able to see their data with these third-party tools but how this information is uploaded and re-applied to the ERP cloud solution is not always apparent. As a result, what customers need is a true two-way solution. They need either a solution that can make a copy of transactions in real-time and re-apply them as needed in a straight-forward and easy fashion; or they need a software vendor who can provide this capability as part of their cloud application service. Agresso provides this service as part of their monthly subscription fee and does not charge an additional fee to restore their customer s systems. Why Multi-Tenant Cloud ERP Forgiveness is Hard for Some, But Easy for Agresso To err is human; to forgive, divine. Alexander Pope wrote those words almost three hundred years ago. As we have already noted, human errors are still occurring in ERP systems but for cloud-deployed ERP solutions, forgiveness is scarce. Why is the forgiveness so tough? Recovering a system to a prior state is a much documented function oft practiced as part of a firm s disaster recovery procedures. Prior to cloud solutions, virtually every computer data center had its own software, hardware, processes, redundant technologies and strategies to handle either disasters (e.g., a fire) or a user error. The approach for recovering or restoring cloud software to a prior state would look very similar to that used by older on-premise solutions assuming that the cloud solution is a single-tenant or private cloud product. In those situations, the application and the databases are not shared by other software customers. The vendor or the customer simply shuts down their instance of the software and either restores it to a prior version (and re-applies many of the desired transactions) or establishes another instantiation altogether. Other software customers would, obviously, be unaffected. In a multi-tenant cloud software world, however, this process can get tougher. In a multi-tenant environment, the application code is often shared across all customers, and the data may only be logically (not physically) separated from that of other customers. Stopping all customers usage of the same physical database just to fix one customer s mistake would be highly disruptive to the other customers. Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 5

6 Typical Agresso Recovery Timeline 3:20 pm Distracted employee inadvertently tries to delete much of the supplier database 3:22 pm After multiple system warnings to the user advising against this action, worker proceeds with deletion 3:25 pm Co-workers report inability to process accounts payable invoices to IT 3:40 pm IT contacts Agresso help desk 3:45 pm Agresso stops customer from processing more transactions and investigates issue 4:00 pm Agresso locates time of unfortunate act and determines appropriate fallback point, restoration media to be used and recovery method 4:15 pm Customer s system restored to prior state (approx. 3:00 pm that day) 4:25 pm Agresso reapplies transactions up to the incorrect action 4: 45 pm Agresso selectively applies transactions posted after the inappropriate event but before system was shut-down 5:00 pm Agresso re-opens the customer s system to users To restore some systems in such a business disaster recovery scenario, ERP customers may not only have to restore databases to a prior state, but may also need to: - Back out transactions processed after the error - Re-apply transactions up-to (and sometimes after) the unintended or malicious act - Re-establish linkages between data elements - Re-process table updates - Restore and re-apply updates to related systems (cloud or on-premise) These additional tasks can be especially confounding for a multi-tenant ERP cloud solution provider that did not build in a forgiveness capability into the solution from the beginning. Software vendors may have designed a disaster recovery process for themselves, but did they also create contextually-rich recovery systems for their customers to use? Did they make backup and recovery a part of their original offering and support desk function or is it an expensive afterthought? Agresso s multi-tenant ERP cloud solution, like its other single-tenant ERP cloud option, keeps all customers data physically separated via their own unique data stores. A simple table setting points the data to the appropriate shared version of the software that the customer uses. By segregating the data this way, versus the more common multi-tenant cloud ERP strategy of a single database with partitions, individual customer environments can be stopped, reset and recovered without impacting other customers. Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 6

7 When an Agresso customer encounters a problem, they are directed to contact the Agresso support center. In most situations, the solution is completely recovered to full-working context, within two hours. The Agresso support team relies on a variety of data backups depending on how far back the customer needs to go to reset the solution. Agresso does a full customer database backup (not a simple snapshot) every two hours. These bi-hourly backups are retained for 3 days and daily backups are kept for 30 days. Monthly backups are kept for a year and annual backups for 10 years. Agresso keeps these backups in an encrypted form in two vaults (one on-site and one off-site). The on-site backups are generally the ones used to perform these restoration actions. Why is Agresso so much better at recovery than other multi-tenant application software vendors? We observed that Unit4: - Designed this recovery capability into the Agresso product from the beginning. - Anticipated these kinds of ERP errors would occur, and interestingly enough, assumed other vendors would also be doing this. - Expected Agresso ERP customers and prospects would demand this as part of their SaaS monthly fee. Agresso executives seemed surprised to learn that other software firms recommend customers create or license their own separate backup and recovery utilities. - Designed their Agresso ERP cloud solution so that individual customers could be immediately and easily taken offline, allowing their damaged databases to be repaired and restored without interrupting other customers in the UNIT4 cloud domain. FAST Forgiveness for ERP Cloud Mistakes is a Business Survival Necessity A 15 business day minimum recovery period is, for many firms, a death sentence. Few firms that depend on their cloud accounting, CRM, HR or other systems could operate without timely access to their information or the ability to pay employees, pay suppliers, process orders, etc. In a world where complex global transactions can be completed electronically in a fraction of a second, a three week wait is unacceptable. Larger firms or firms that are subject to regulatory or compliance requirements (e.g., SarbOx) regarding disaster recovery should take heed. A more rigorous or expensive audit may occur if the company cannot show: - That its cloud backup files are complete in all aspects. These files should include, at a minimum, transaction data and events, but should also include additional data such as time/date stamps, user information, table updates, etc., as well as unique business processes and workflow. - How backed-up data can actually be restored to the cloud solution. Where are the batch upload utilities that can do this work and have they been tested to ensure that these utilities work with the current version of the software? - That it periodically creates test sandbox environments and tests its recovery programs and procedures against these. Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 7

8 Businesses can try to button down their cloud-deployed ERP systems and controls quite tightly, but user errors may still occur. No system can prevent all of these accidents (or malicious acts in some cases) from occurring, but every business should have a software vendor solution that will allow it to recover fast, effortlessly and at little to no additional cost. Exercise: Determine the economic hit your organization would incur if its core systems went down for 1 hour, 1 day, 1 week or 2 weeks. Is this number sufficient enough to get your attention on the forgiveness issue? For example, let s assume your firm had revenues of $100 million (USD) and generated net profits of $2 million annually. Let s also assume your firm suffered an error that took down your ERP system for three weeks (i.e., 15 business days) like the example discussed earlier in this paper. Missing three weeks of revenue would diminish annual revenues by possibly $5.7 million and possibly triggering a loss of over $3 million. Forgiveness Checklist Overall Does the vendor offer a rapid recovery service or is recovery a labor-intensive activity for the customer? Is there a specific backup and recovery capability/service automatically included in the cloud service and/or the monthly subscription fee? Procedures Does the vendor provide a documented set of procedures that both of you must follow when a recovery is needed? Does the vendor identify which specific files, tables, databases and linkages will be reprocessed for a given failure? Can a customer call a support desk to request an immediate rollback and recovery? Can most recoveries be completed within an hour of their request? Are vendor and customer data recovery responsibilities clearly documented and communicated? Cost What, if any extra cost does the customer incur should a recovery be required? Does the customer have to contract with a third party (e.g., systems integrator) to effect the backup? Vendor Supplied Utilities Does the vendor provide both backup and recovery programs? Does the customer have to manually apply any data? Will customers have to re-key transactions? Will the customer need to create special upload programs? Will the vendor keep these utilities current release after release? Do vendor utilities re-apply data linkages, user extensions, table updates, etc.? Third Party Is third party backup software required as part of a forgiveness strategy? Is third party hardware required as part of a forgiveness strategy? Is the customer encouraged to acquire on-premise backup hardware or systems hardware to store backup data? Is the customer encouraged to acquire cloud backup capacity to store backup data? Customer Specific What additional data dependencies exist to complete a recovery? Backups How are backup files created (e.g., CSV files, tape backups, database images or other formats)? Does the vendor provide opportunities for customers to test their 'forgiveness' recovery programs and procedures? Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 8

9 About Vital Analysis Vital Analysis is a very different kind of technology research organization. We are the intersection set where exceptional technology market knowledge meets the executive suite. Where other analysts replay vendor press releases, we give you the: impact new technologies will have on your business reasons why you should care about specific emerging solutions business justifications why you may want specific solutions Vital Analysis was carved out of TechVentive, Inc. in 2007 as a new, but complementary business. As designed, Vital Analysis is the publishing, research and analytical arm of that company. Our reach, like our blog readership, is truly global. We ve consulted with top technology executives in Australia, Brazil, Canada, the United Kingdom and the United States. We ve been briefed by technology providers from virtually every corner of the planet. About the Author Brian Sommer is the CEO of TechVentive, Inc. - a market-strategy and content firm. Brian closely follows what C-level executives think, feel and need. Brian also publishes a blog on the intersection of application software and professional services ( He welcomes your thoughts and invites you to contact him at [email protected]. Reproduction of this publication in any form without prior written approval is forbidden. The information in this report has been obtained from sources believed to be reliable. TechVentive, Inc. disclaims all warranties as to the accuracy, completeness, or adequacy of such information and shall have no liability for errors, omissions, or inadequacies in the information contained herein or for interpretations thereof. The reader assumes sole responsibility for the selection of these materials to achieve its intended result. The opinions expressed herein are subject to change without notice. To purchase reprints of this document or to quote passages within, please [email protected]. Copyright 2013 TechVentive, Inc., All Rights Reserved - Unauthorized reproduction, storage, transmission or quotation strictly prohibited. 9