1 HOW TO AVOID A DISASTROUS WEBSITE LAUNCH LIKE HEALTHCARE.GOV The Six Critical Mistakes Made and What We Would Have Done Differently By PointClick Technologies, a Managed Cloud and Hosting Provider
2 INTRODUCTION A complex system with many different, interconnected parts, HealthCare.gov didn't receive the detailed attention, planning, testing and management to prevent everything that could go wrong from going wrong. At PointClick Technologies, we know that it is no easy task to launch a complex website. We are a Cloud and Hosting Provider that knows what it takes to make web applications go live and help you launch your website successfully the first time-around. In this report, we outline six big mistakes with HealthCare.gov, and what we at PointClick would have done to anticipate and address those issues before going-live. The launch of HealthCare.gov, will forever be a lesson in what not to do when launching a website.
3 M I S TA K E 01 SLOPPY SOFTWARE AND USABILITY CHALLENGES Before users could even browse HealthCare.gov, they were required to create an user account even if they weren't purchasing insurance through the website at that moment.
4 01 Part of the account-creation process involved identity-checks and inputting personal information. Problem was, the security questions did not always appear in the drop-down tools effectively leaving many customers with absent mandatory questions to fill, and effectively no ability to create their account. This is just one example of how the system was built on a sloppy software foundation. The system then got bogged down by the repeated attempts from unsuccessful and unhappy users. MISTAKE NO. 1: SLOPPY SOFTWARE AND USABILITY CHALLENGES
5 PointClick SAYS: Imagine if Amazon required you to create an account including all your information just to see their products. To make things worse, the functionality didn t even work properly. No one really thought this through. Our approach would have been to collaborate on the requirements and expectations and draw up a plan for the staging environment that mimics the production environment. After hearing about the business requirements and properly testing performance and usability, our team would have ensured the errors were known and addressed prior to launch.
6 M I S TA K E 02 POOR USER LOAD-BALANCING The second glaring mistake that compounded this sloppy software foundation is the failure to effectively load balance, resulting in blockages and backlogs.
7 02 The Centers for Medicare and Medicaid Services (CMS), the agency that oversaw the implementation of HealthCare.gov, predicted that sign-ups in the first weeks of a six-month enrolment period would be slow. According to U.S. Chief Technology Officer Todd Park, the website was expected to draw around 60,000 simultaneous users but instead drew many more, around 250,000. New reports show, however, that a day before the site s launch, without the proper load balancing servers in place, the website could only handle about 1,100 users before response time crawled to snail s pace. MISTAKE NO. 2: POOR USER LOAD-BALANCING
8 PointClick SAYS: Whether you anticipate 1,100 or hundreds of thousands of users as was expected by Healthcare.gov, your hosting provider should take an active role in helping you prepare for the potential onslaught of users and ensuring a happy and smooth experience. Load testing isn t like it used to be it isn't hard, time consuming or expensive today. You just need to recognize that it is something that needs to be done. It s not optional anymore. - Ragnar Lonn, Founder and CEO, LoadImpact This would involve various critical tasks like performance optimization, load balancing, backups and disaster recovery. One of our own key business partners is LoadImpact.com who helps us work with our clients to perform rigorous load testing in many types of user scenarios.
9 M I S TA K E 03 SECURITY HOLES One of the most glaring and shocking - issues with HealthCare.gov is the fact that the website had many security holes as the result of inadequate security testing.
10 03 Imagine this: A complex website, which is meant to verify your identity, record this information, match it with insurance options, and enroll you into a plan, has significant security holes. According to experts, virtually no security testing went into the site s rollout. According to an internal memo obtained by a leading U.S.-based newspaper, security tests were not conducted by HealthCare.gov to identify any automated attacks against contact forms or verify if website cookies are not stored in a readable form, making it vulnerable to attacks. MISTAKE NO. 3: SECURITY HOLES
11 PointClick SAYS: We work with several 3rd party companies that provide penetration testing and website vulnerability scanning. Incapsula, a provider of cloud-based website security and performance services for online businesses, is one of our key business partners and a trusted asset for our clients. Their enterprise-grade WAF (Web Application Firewall) and global CDN (Content Delivery Network) make sure that our clients websites are always secure, fast and available. PointClick does not launch any major website application without Incapsula s Enterprise plan, featuring their always on DDoS Protection service and 24x7 support from their team of security experts. A prominent website like Healthcare.gov would be a prime target for hackers and DDoS attacks from all over the world. Incapsula offers a proven, best-of-breed security and performance solution to mitigate these risks. Our crowdsourced big data security approach leverages our customer base for 360-degree visibility into today s dynamic attack landscape. Protecting thousands of customers and subjected to hundreds of penetration tests and millions of attacks every day, Incapsula's service is aligned to meet the most stringent enterprise-grade security criteria. - Marc Gaffan, Co-Founder, VP Marketing and Business-Development, Incapsula
12 M I S TA K E 04 POOR SITE PERFORMANCE HealthCare.gov s disastrous launch resulted in potential users experiencing website errors or being locked out entirely. For those that did manage to get in, the website was agonizingly slow to load and incredibly frustrating to maneuver.
13 There are three kinds of testing that every site should undergo prior to launch Performance Testing: We're testing how fast the system can carry out "normal" commands while remaining stable. It's about response time and meeting specs. Load Testing: We are subjecting the site to a typical load to determine behavior under normal conditions. It's about reliability against performance. Stress Testing: We place an unseasonable amount of load while also denying resources to determine the result of the failure. We want to make sure the data does not get corrupted. We wonder how much of each testing Healthcare.gov undertook prior to launch?
14 04 CRN published an article entitled Heathcare.gov website is not HIPAA compliant. HHS (Health and Human Services) commented the website did not require HIPAA compliance. This was very surprising to read because as the department that oversees HIPAA (Health Insurance Portability and Accountability Act - privacy compliance regulations on the disclosure of protected information, such as medical records or personal information), you would think they would practice all the requirements to keep the data going intohealthcare.gov secure. Would you agree? MISTAKE NO. 4: POOR SITE PERFORMANCE
15 PointClick SAYS: During load testing of the website, we normally recommend real-time human testing of the application, which usually reveals any problems that could arise during heavy loads on the website. PointClick will assign a couple of our staff members to help with the human testing during load testing. Using proxy services like Incapsula's hosted WAF (Web Application Firewall) and CDN (Content Delivery Network) also helps boost performance, while blocking unwanted and malicious traffic. Their CDN delivers full site acceleration using intelligent caching and content optimization techniques that ensure fast response times. In addition, the WAF s advanced bot filtering distinguishes between good" and "bad" bot traffic, blocking the bad bots that often account for up to 50% of a website s traffic.
16 M I S TA K E 05 DESIGN AND DEVELOPMENT Unlike most web applications built today that are agile, open, and iterative, the government took a siloed and waterfall IT approach.
17 05 With constantly changing guidelines and requirements, there was not enough time to discover, much less test all the bugs that riddled HealthCare.gov. Different components of the website were built by different contractors. As such, there was a clear need for coordination and communication in order to ensure that all parts of the web application were well-integrated and performed adequately, which did not happen. The lack of communication and coordination between development groups and contractors meant that this complex, inter-connected website, was fraught with faulty breakpoints throughout the site. MISTAKE NO. 5: DESIGN AND DEVELOPMENT
18 PointClicks works with R2Integrated, a Digital Design and Development Agency, on your projects end-to-end. This ensures all critical pieces not only work together but perform at their best. HealthCare.gov is a highly exposed website that has a critical purpose, with security, vast amount of personal identifying information and complex workflows, business rules and external system integration functionality. We would architect the system using proven foundational components and frameworks, segmenting the project into incremental workable modules developed in an agile methodology environment. - Chris Chodnicki, Co-Founder, Exec. Dir. Strategic Partnerships & Alliances, R2Integrated
19 M I S TA K E 06 NON-EXISTENT PROJECT MANAGEMENT & LEADERSHIP Although there were serious technical issues that were factors that contributed to the failure of the website, the fundamental issue with HealthCare.gov was the extremely poor project management and (un) strategic coordination between the contractors, the client and the hosting provider. Given the size of this project and the importance of HealthCare.gov, there was a disturbing lack of vision, partnership and strategy when it came to building, testing, and deploying this complex website.
20 06 According to an article published by the Wall Street Journal Online quoting a Federal report, "inadequate management oversight and coordination among technical teams prevented real-time decision making and efficient responses to address the issues with the site". HealthCare.gov was fundamentally a compromise between government bureaucrats and software developers a mix that resulted in a complicated mess that was underpinned by a lack of communication and a lot of finger-pointing when it came to accountability for the website s failed launch. The WSJ article goes on to state that the contractor, the Centers for Medicare and Medicaid Services, "did not anticipate the degree of problems in the system". References: MISTAKE NO. 6: NON-EXISTENT PROJECT MANAGEMENT & LEADERSHIP
21 PointClick SAYS: Anyone can provide hosting infrastructure. What is most important is the management, consulting and depth of understanding of your business requirements that most hosting providers lack. During our customer intake process, we work with the client to understand all the groups involved in your website project, whether it be your design team, application development or any other third-party provider. We'll even work with your security auditors to help you become compliant and give your customers the assurances they need so that you can further your business. During your entire project, we work closely with you and all your partners, even bringing to light issues that may fall outside the realm of hosting we do it because we see ourselves as your trusted advisor and feel that it s a part of our job to identify something that you may have not thought about or did not anticipate.
22 SUMMARY Don t make the same critical mistakes made in the launch of Healthcare.gov. Here is a summary of the critical issues of the site and why working with a Managed Cloud and Hosting provider like PointClick Technologies could save you the pain and heartache of a disastrous web site launch:
23 AREA Critical Mistakes Made by Healthcare.gov What PointClick would have done: Software Poorly architected software Understood business requirements and provided technical guidance Load Balancing Inability to handle massive loads Architected a system with proper load balancing and redundancy Security Security flaws Conducted penetration testing and website vulnerability scanning Performance Outages and downtime Performed ample performance, load and stress testing Design & Development Poor design and development Collaborated with system and technical engineers to build robust infrastructure Project Management Lack of project management Worked with all partners and parties involved to ensure launch success
24 Copyright 2014 PointClick Technologies. All rights reserved. PointClick Technologies and the PointClick logo are registered trademarks of the company. All other trademarks are the property of their respective owners.