Software validation of applications deployed on Windows Azure. Sidharth Subhash Ghag, Divya Sharma, Trupti Sarang

Transcription

1 Software validation of applications deployed on Windows Azure Sidharth Subhash Ghag, Divya Sharma, Trupti Sarang May 2011

2 Abstract As a very popular Infosys adage goes, In God we trust, rest all we test, it reflects the importance of testing in the software development life cycle. According to Wikipedia 1, Software Testing can be stated as the process of validating and verifying whether a software program/application/product one, it meets the business and technical requirements that guided its design and development; two, works as expected and three, can be implemented with the same characteristics. As software programs get complex, testing becomes vital and error prone. Testing an application on cloud is quite different from testing an application in the traditional on-premise environment. Microsoft Azure TM is one such cloud platform that runs applications on the cloud. From a testing perspective, Test environment setup on Azure TM is quick and easy which can save considerable time and effort as compared to test infrastructure setup and administration. However by design, the Azure TM platform imposes certain restrictions on services deployed on it. Thus testing an application hosted on Azure TM becomes difficult and complex as compared with their traditional on-premise counterparts. In this paper we shall explain challenges along with approaches to handle different aspects of testing on Azure TM. Also we shall provide an insight into the end-to-end testing lifecycle involved for testing applications to be deployed on Windows Azure TM. 2 Infosys White Paper

3 Table of Content Introduc on... 3 Role of tes ng in development of Azure cloud applica ons... 5 Comparison between prac ces of tes ng tradi onal on-premise vis-a-vis Azure applica ons... 5 Testing Azure applica ons Unit Tes ng Integra on Tes ng System Tes ng Performance Tes ng Security Tes ng Compliance & Regulatory Tes ng User Acceptance Tes ng Benefits realized a er applica on valida on with Azure Challenges of tes ng applica ons on Azure Organiza on Policies Are Not Implicit For Hosted Applica ons Development Challenges Connec vity Summary Reference Infosys White Paper 3

4 Introduction Testing an application is a very important aspect of software development and so is the case with the applications deployed on Microsoft Azure TM. Similar to the development and testing tools available for on-premise applications, Microsoft offers tools for application development and testing on Azure TM Platform as well. Microsoft offers a simulation of the cloud environment for developing applications locally through the development fabric. Unit and integration testing can be performed in a manner similar as it is performed for an on-premise application. But, testing after deploying an application on Azure TM is different from the approach of testing an on-premise application. Applications deployed on Azure TM are managed, governed and controlled by the Windows Azure TM Fabric controllers. The controllers impose certain infrastructure restrictions, defined in the form of role definitions, at different levels of the platform and which limit flexibility otherwise available with on-premise systems. The levels of flexibility available to account owners on Azure TM would differ based on the Azure TM role definitions Web, Worker or VM (Virtual Machine) Roles. Broadly these allow Azure TM to offer cloud service model capabilities such as Infrastructure as a Service (IaaS) and Platform as a Service (PaaS). To offer the IaaS service model, Microsoft announced VM role in PDC 2010 with enhanced control and flexibility on a virtual machine instance. VM role is offered to support development and migration of Windows server applications which would require the creation of a custom Windows server image, installing required software, applications and managing it as well. The VM role can be used to realize scenarios where legacy applications built on design practices more suited for onpremise deployments that needed to be moved to Azure TM Cloud. Whereas In the PaaS service model offering, Azure TM offers a ready platform for directly deploying and hosting applications without having to worry about installing or managing the software required to run these applications. In this model, the platform provides an abstraction layer comprising of application runtime services and tools over the underlying infrastructure. This layer offers a more managed and standard environment on which applications that would run with the available IaaS service models. One can make use of the web and worker roles to leverage the PaaS service models offered by Azure TM. However one must note that with high abstraction levels offered by the web and worker roles, Azure has access controls in place which may restrict the free movement of applications from on-premise to cloud. These restrictions would also apply while validating applications deployed under this service model on Azure TM. When one needs to test an Azure application, it should be tested on both development and test environments. In the development environment users can test an application on the development fabric and development storage, using Visual Studio test suite capabilities. Once the application is hosted on the Microsoft Azure TM Platform, the scope of testing becomes quite restrictive, which is enforced by the design of Azure TM Platform. At the same time, a ready-to-use platform saves time and additional cost in setting up various test environments. Setting up environment and scaling it as desired is only a few clicks away. In this paper, we shall discuss the methodology and approach to test next generation cloud based applications on the Windows Azure TM platform. The scope of the discussion would be limited to software validation of applications developed on PaaS specific offering of the Azure TM platform i.e. apps leveraging the web/worker role models. We highlight the key challenges faced with developing PaaS applications along with some approaches to overcome them. IaaS - style applications, developed using Azure TM Virtual Machine roles, will not be in the scope of this paper. 4 Infosys White Paper

5 Role of testing in development of Azure cloud applications In a typical Software Development Life Cycle (SDLC), each process has its corresponding verification/validation process to ensure that the system works as required. SDLC of Azure TM applications also follows the same verification/validation approach as in standard SDLC. Testing is an essential part of SDLC to identify any defects and bugs early in the project. If a bug is detected later in the SDLC, its impact on the project will be higher and fixing it would require more effort and hence cost more to fix. The diagram below shows the correlation between the SDLC phases and their corresponding testing steps. Figure 1: Correlation between development and validation phases Proper testing of an Azure TM application becomes even more imperative since the owner of application has less control over the application and data once the application is hosted on Azure TM. Comparison between practices of testing traditional on-premise vis-à-vis Azure applications Though both on-premise and Azure TM application validation process will follow similar phases in the testing life cycle, as depicted in the diagram below, the execution of these phases will have few significant differences which we shall highlight in the subsequent sections. Figure 2: A typical Application Validation life cycle Infosys White Paper 5

6 As traditional application testing requires various test environments to set up e.g. development, system test, performance test, quality assurance, user acceptance test, production etc. Whereas, since Microsoft Azure TM platform offers Platform as a Service, it gives liberty to application developers to host applications on a ready to use hosted platform without worrying about infrastructure management. However, they get limited privileges on infrastructure. These features affect the approach of testing Azure TM applications as compared to that with the traditional application testing methods. The following diagram shows different deployment environments used and roles involved in the typical testing life cycle of a traditional on-premise application compared with that of an Azure TM application. These figures are representative and actual scenarios may differ based on project requirements. Func onal Tes ng Non - Func onal Tes ng Performance Testers Developers Unit Tes ng Developers System Testers Performance Tes ng Client End User Unit Tes ng... Integra on Tes ng System Tes ng Security Testers User Acceptance Tes ng Produc on Environment Unit Tes ng Development and Test Security & Compliance Tes ng Produc on Figure 3: Testing approach for an on-premise application The above figure shows the environment setup required for various stages of an end-to-end testing lifecycle for any traditional on-premise application. As you can observe one would be required to set up separate test environments with required hardware and software installations for the various types of testing. This adds as overhead in terms of efforts and cost involved in managing various test environments. 6 Infosys White Paper

7 Figure 4: Testing approach for an Azure TM application There are two deployment regions in one hosted Azure TM instance, i.e., Azure TM service viz. staging and production. Any of these two deployment regions of an Azure TM service can be used for testing the application based on the availability. Production and staging are exactly identical with respect to hardware and installed software. In the above diagram, we have considered a scenario where we have used two Azure TM services in a project life cycle one for testing and another for release. As depicted in the illustrative scenario above, testing on Azure TM environment saves considerable effort and time by eliminating the overheads in test environment setup and administration. In the traditional testing approach, test environment setup would require setting up several machines with required software, whereas in Azure TM it will only require provisioning an Azure TM account. Creating new Azure TM service will implicitly allocate hardware resources from a shared pool along with the required software installations required to run your application in a matter of minutes. A comparative view highlighting the differences of the effort required for testing on traditional on-premise development visà-vis Azure TM cloud development is shown below: Test plan Traditional Application Prepare test strategy Plan for test environment Azure Application Prepare test strategy Plan for test environment Test environment setup Procure hardware of desired configuration and required software licenses Identify test lab (Physical Location) Infrastructure management team will setup hardware and install software Perform a quality check of environment setup On-demand provisioning of Azure account Not Applicable (NA) Not Applicable (NA) Not Applicable (NA) Infosys White Paper 7

8 Application Setup Deploy application Prepare deployment scripts for application and database to be deployed MSI s Prepare installation manuals to guide system administrators Publish application to target environment Setup master data (if any) Deploy application Not Applicable (NA) Not Applicable (NA) Publish deployment package and database scripts to target Azure TM instance Setup master data (if any) Test Design Write test cases for all types of testing to be performed Generate test data Write test cases for all types of testing to be performed Generate test data Unit Test Unit Testing Debugging/ trouble shooting Bug Fixing Regression Test Unit Testing Debugging/ trouble shooting Bug Fixing Regression Test Integration Test Integrate and deploy the application on integration test environment Smoke Testing Integration Testing Debugging/ trouble shooting Bug Fixing Regression Test Integrate and set up application on dev fabric Smoke Testing Integration Testing Debugging/ trouble shooting Bug Fixing Regression Test Setup system test environment. Repeat steps with mentioned in the Test environment setup Use staging deployment region of Azure TM service 1 (procured for testing) System Test Deploy the application on system test environment Smoke Testing System Testing Debugging/ trouble shooting Bug Fixing Regression Test Deploy the application on staging environment of Windows Azure TM subscription for development Smoke Testing System Testing Identify bugs with code instrumentation and by capturing application data Bug Fixing Regression Test Performance Setup performance test environment. Repeat Use production deployment region of Azure TM 8 Infosys White Paper

9 Test steps mentioned in the test environment setup service 1 (procured for testing) Deploy the application on performance test environment Develop Performance Test Scripts using the identified Performance Test Tool. Deploy the application on production deployment region on Azure TM service 1 Develop Performance Test Scripts using the identified Performance Test Tool. Perform Dry Runs Setup Performance Test Data Run Performance Tests Capture performance metrics Analyze output and find out parameters to improve Modify application and deploy again Run Performance Tests run post performance defect fix Perform Dry Runs Setup Performance Test Data Run Performance Tests Capture performance metrics through Azure TM diagnostics Analyze output and find out parameters to improve Modify application and deploy again Run Performance Tests run post performance defect fix Security Test Setup security test environment. Repeat steps mentioned in the test environment setup Deploy the application on security test environment Security Testing Find out any security threats, system vulnerabilities Modify application and deploy again Security Tests run post defect fix Use staging deployment region of Azure TM service 1 (procured for testing) Deploy the application on staging deployment region of Azure TM service 1 Security Testing Find out any security threats, system vulnerabilities Modify application and deploy again Security Tests run post defect fix Compliance Test Compliance Testing Find out any non-compliances as per the organization and Government policies such as data location, security audit etc. Modify application and deploy again Conduct Audits post defect fix Compliance Testing Find out any non-compliances as per the organization and government policies such as data location, security audit etc. Modify application and deploy again Conduct Audits post defect fix User Acceptance Testing Setup user acceptance test environment Deploy the application on user acceptance test environment Smoke Testing User Acceptance Testing Find out any defects Modify application and deploy again Use staging deployment region of Azure TM service 2 (procured for release) Deploy the application on staging deployment region of Azure TM service 2 Not Applicable (NA) User Acceptance Testing Find out any defects Modify application and deploy again Infosys White Paper 9

10 Conduct tests post defect fix Conduct tests post defect fix Upgrade to Production Setup production environment Setup datacenter real estate, utility services, communication services, security management, government and environmental approvals, etc. Align teams- infrastructure, database, application administrators, developers Procure and install hardware and network resources- switches, routers, servers, cables, racks, storage, H/W load balancers Procure and setup software licenses operating system, application server, database, firewalls, antivirus, patches, etc. Validate the production environment for security threats and compliances Harden production environment setup Install operations tools for infrastructure monitoring and management Prepare deployment scripts for application to be deployed MSI s Deploy application to production environment Use production deployment region of Azure TM service 2 (procured for release) Not Applicable (NA) Align teams developers, application administrators Not Applicable (NA) Not Applicable (NA) Not Applicable (NA) Not Applicable (NA) Subscribe to cloud management vendors for application management and monitoring Prepare deployment package for application to be deployed on Azure TM - cspkg Swap deployment from staging to production Table 1: Comparison of efforts in testing on-premise and Azure TM applications Apart from test environment setup, there are dissimilarities in the processes of testing life cycles as well. The table below describes the differences in each phase of the testing life cycle for a traditional on-premise application and an Azure TM application. Traditional Application Azure Application Test Plan Prepare the test plan Prepare the test plan Test Design Define test strategy, goals and approach Define test strategy, goals and approach Test Execution Separate test environment setup needs to be done for development / Test / QA / Production Production size instances to be provisioned for different test environments 10 Infosys White Paper Hardware and Software are to be maintained Desired software has to be installed Exploring database for test verification is comparatively simple Hardware and Software details are abstracted and a computation and storage services are provided. Standard MS software components such as IIS,.NET framework etc... are pre-installed with the Azure TM compute services However installing third party components would have to be done by the test engineer by enabling Administrator privileges on the cloud services. Although in latest releases MS has provided many aiding tools, accessing and exploring Azure TM

11 Scaling the environment will require extra hardware and software setup taking either the vertical or horizontal scaling route. Storage or SQL Azure is complex as compared to their on-premise counterpart. For SQL Azure certain features of SQL Server and management studio are not available and for Azure storage, developers may need to use third party tools such as Cloudberry, Azure storage explorer, Cerebrata s Cloud storage studio etc. Azure can be scaled dynamically (both vertically as well horizontally) by modifying number of running instances without any overhead Test Analysis Debugging is easier than cloud applications Debugging cloud applications is a challenge Functionality rich tools are available for defect analysis, debugging, trouble shooting Being a nascent technology, the testing tools available are limited Table 2: Testing life cycle for on-premise and Azure applications Testing Azure applications This section explains how various testing cycles are performed on Azure applications. Testing is carried out in both development as well as the hosted Azure instance; following sections describe the overall process of testing an Azure application: Unit Testing Unit testing is a verification and validation process for a unit of source code, carried out by the developers to ensure the code is working as intended. It is performed for an Azure application in the same way as it is done for a traditional application. The diagram below shows the steps followed in each phase of unit testing. Figure 5: Steps in Unit Testing Azure applications can leverage the test feature of Microsoft Visual Studio to auto generate unit test cases from the Azure source code. Testers can provide input and expected result values to run tests and verify the application logic. Infosys White Paper 11

12 Figure 6: Comparative View for unit testing These test cases are executed on development fabric which is a simulation environment of the Windows Azure to simulate the Azure fabric locally. Unit testing of Azure applications doesn t require any additional setup. The diagram below shows a comparative view of people, environment and tools required for unit testing an on-premise application vis-àvis that of an Azure application. Integration Testing Integration Testing is performed on integrated modules comprising of individual unit tested modules. Integration testing ensures that the modules communicate right with each other and integrated modules work as specified in high level design. The diagram below shows the steps followed in integration testing. Figure 7: Steps in integration testing 12 Infosys White Paper

13 The high level design is input of integration testing and integration test plan is prepared based on this. The individual unit tested modules are integrated based on the approach chosen to integrate, i.e., top down (the top module is tested first and then its subordinate modules), bottom up (the low level modules are tested first and then the top level ones calling them) and tested to ensure conformance with high level design. For an Azure application, various unit tested modules will be integrated on development fabric and integration testing is performed on development fabric using the visual studio web tests and/or manual tests. It ensures that application interfaces are working properly and integration did not cause any issues in different modules. This is a part of functional testing of application. Once it is confirmed that application is working fine on development environment then it is moved to Azure cloud instance. The diagram below shows a comparative view of people, environment and tools required for integration testing an on-premise application vis-à-vis that of an Azure application. Figure 8: Comparative View for integration testing System Testing System testing is performed on an application with all its disparate modules integrated to form the system as a whole. Systems tests are carried out to ensure that the system fulfills all the specified functional requirements which the application is expected to meet. In the traditional on-premise model, for system testing of an application, a separate system test environment is commissioned, pre-requisite hardware and software infrastructure setup is installed.. Whereas for system testing of applications deployed on Azure Cloud, test environments need not be commissioned separately. A single cloud computation service provisioned on Azure comprises of a production and a staging environment and the system test is performed in the staging region of this service. The diagram below illustrates the steps followed in system testing of an Azure application. Infosys White Paper 13

14 Figure 9: Steps in system testing System testing is to extensively test the functionality of application. Once integration testing of the application is performed on development fabric, it is then hosted on Azure staging environment and application functionality is tested again. Manual system test cases are created against high level design of the application (Use cases). Web tests created using Microsoft Visual Studio web test features to test the functionality of the application on development environment during integration test can be reused. The web tests created on development environment can be utilized by changing the application URL in test cases to the hosted application URL. The path of application will be modified using context parameters and tests can be executed for application hosted in staging deployment region of Azure account. Since hosted environment provides restricted privilege access to users, system metrics need to be accessed through the Azure diagnostics feature. Proper code instrumentation will be required for trouble shooting of hosted application. Developers will have to write tracing and logging code in the application for writing out information through Azure logging APIs for application monitoring and analyzing defects (if any). Similar to investigating on-premise, applications developers have access to the event viewer, system logs, IIS logs etc., to analyze application issues. However in Azure such facilities are not directly accessible and hence Azure diagnostics API (Microsoft.WindowsAzure.Diagnostics) would have to be relied upon. Users can also connect to an instance of their application deployed on Azure using remote desktop, monitor and trouble shoot the running instance. The diagram below shows a comparative view of people, environment and tools required for system testing an on-premise application vis-à-vis that of an Azure application. 14 Infosys White Paper

15 Figure 10: Comparative View for system testing Performance Testing Performance testing is conducted to confirm how the system would perform under certain workload. Application behavior is tested in terms of various performance aspects under different workload conditions. This includes scalability, reliability, availability and response time of the application, ability to handle load, spikes and throughput. The diagram below lists the steps followed in performance testing of an Azure application. Figure 11: Steps in performance testing In Azure applications, performance testing plays a vital role. Since the infrastructure is abstracted and a standard platform is offered, it becomes essential to gauge the performance of application deployed on the Azure platform. The number of compute instances and VM size will also affect the application performance. Since customers have to pay for number of Infosys White Paper 15

16 specified sized compute instances running at a time, the optimum use of Azure compute instances becomes important for cost-effective utilization of the Azure owned resources. Performance evaluation of application will help derive the factors which can aid in optimizing these resources with respect to cost and performance measures. Azure supports dynamic scalability which makes it easy to test application performance at varying scales which is made available on-demand. Procuring the compute instances and releasing it based on the load is a manual process. But once requested for resources it will be available in no time. It can be automated using some third party applications which continuously monitors the load of deployed applications and does the resource management. Performance testing is performed on hosted Azure instance because application performance should be tested in an environment closely similar to the production environment. Based on the performance requirements, test plan is prepared, performance scripts developed and the approach to execute performance test scripts and capture performance metrics is decided. Performance metrics are captured with performance test scripts execution and analyzed. Since Azure platform is a managed environment, capturing of performance metrics in not easy. Performance parameters are captured using Azure diagnostics with support of diagnostic monitor. The diagram below shows a comparative view of people, environment and tools required for performance testing an on-premise application vis-à-vis that of an Azure application. Figure 12: Comparative View for performance testing Though performance testing is usually performed after system test, the performance goal should be well defined right from the application architecture, design and build. 16 Infosys White Paper

17 Security Testing Security testing involves testing the security at two levels Platform Security Testing To test the platform where application is hosted, is not easily vulnerable to security attacks. Application Security Testing To test that application is not easily vulnerable to security attacks. The Azure platform offered by Microsoft is a managed and standardized environment which is protected by the Azure fabric controllers, network security and personnel level checks and controls making them highly secure. As per Microsoft 6, Windows Azure operates in the Microsoft Global Foundation Services (GFS) infrastructure, portions of which are ISO27001-certified. ISO27001 is recognized worldwide as one of the premiere international information security management standards. Also Azure being offered in a shared model, Microsoft today does not allow performing any platform security tests so as to avoid it affecting other tenants hosted on the same host. However the same cannot be said about the application hosted on Azure. Hence as a part of this phase the focus should primarily be on application security testing. Application security testing is to ensure that application provides the right data and operations to the right person. The confidentiality of data should not be compromised with the functionalities of application. Whatever functionalities are added to the application, should allow secure access to data and operations. Exposing the applications over internet involves more concerns of security since any flaws within the application can make the application vulnerable to security attacks. This makes security testing essential for Azure applications. Figure 13: Accessibility for on-premise and Azure applications Following are the security aspects to verify in an application security test Confidentiality: Protect against the disclosure of information to parties other than the intended recipient Integrity: Ensure received information is correct Authentication: Ensuring received information is from intended known user Authorization: Allow access to intended users Availability: Availability of application in case of denial of service attacks and distributed denial of service attacks Non-repudiation: measures should be taken before the concerned parties deny about an action occurred, a message sent by sender, or received by receiver, etc. Infosys White Paper 17

18 Figure 14: Steps in security testing In traditional security testing almost all security measures can be implemented through firewall. Once the application is deployed, all requests are passed through the firewall and application maintenance team can restrict the application server from users accessing the application. Using this application, the deploying team has control over the users accessing the application. Azure is a common platform for deploying the application. The user can access the application hosted on cloud ubiquitously. There are firewalls installed on Azure platform but it is managed by Microsoft. The application developer and deploying team has the least access to infrastructure and they cannot control the firewalls behavior. Application security testing must be conducted on hosted Azure instance to make sure that the application in production will work as intended. The processes of testing an on-premise and Azure application are quite similar but since the application and data do not remain within an organization s control, application security testing becomes a major concern. The diagram below shows a comparative view of people, environment and tools required for security testing an on-premise application vis-à-vis an Azure application. 18 Infosys White Paper Figure 15: Comparative View for security testing

19 Compliance & Regulatory Testing Compliance & Regulatory testing is conducted to ensure that an application does not violate any organization, industry or government regulations that might lead to legal and possible financial implications. Compliance and regulatory check becomes imperative for the applications hosted on cloud since an application hosted on the public cloud is operated and managed by a third party. The cloud service providers should provide the required compliance such as information about data location, security audits. Hence compliance and regulatory testing is required to ensure that the systems and processes of the cloud service provider have to be compliant with that as is expected by the domain in which the application would operate in. The system must have supportive documentation and/or certification compliance to be handed over to the regulatory agencies (if applicable). Figure 16: Steps in compliance and regulatory testing Below is the list of common compliance requirements of IT systems Sarbanes-Oxley Act (SOX) 3 - The Sarbanes Oxley Act of 2002 sets new or enhanced standards for all U.S. public company boards, management and public accounting firms. The legislation affects both the financial side of corporations and the IT departments. Section 404 of SOX requires management and the external auditor to report on the adequacy of the company's internal control over financial reporting (ICFR). Section 802(a) of the SOX states, whoever knowingly alters, destroys, mutilates, conceals, covers up, falsifies, or makes a false entry in any record, document, or tangible object with the intent to impede, obstruct, or influence the investigation or proper administration of any matter will be panelized. Similar to SOX act in US other countries also have similar act Japan J-SOX Canada Bill 198 Germany - German Corporate Governance Code Australia - Corporate Law Economic Reform Program Act 2004 (CLERP-9) France - Financial Security Law of France Italy - L262/2005 South Africa King Report Infosys White Paper 19

20 Health Insurance Portability and Accountability Act (HIPAA) 4 - Title II of HIPAA, known as the Administrative Simplification (AS) provisions, requires the establishment of national standards for electronic health care transactions. This is intended to help people keep their information private and secure. ISO/IEC ISO/IEC is an Information Security Management System (ISMS) standard which requires that management follows the required measures to keep the information secure in case of security threats and vulnerabilities. Payment Card Industry Data Security Standard (PCI DSS) 6 is a worldwide information security standard defined by the Payment Card Industry Security Standards Council. The standard was created to help organizations that process card payments prevent credit card fraud through increased controls around data and its exposure to compromise. Most of the compliances require that data security and privacy must be maintained and to ensure this they need to conduct security audits in a timely manner. The application must follow the required regulatory compliances which client needs to follow as per their legislation. The developers would need a test to ensure that it follows required regulatory compliances. The diagram below shows a comparative view of people, environment and tools required for regulatory compliance testing an on-premise application vis-à-vis that of an Azure application. Figure 17: Comparative View for compliance and regulatory testing User Acceptance Testing User Acceptance testing is similar to system testing but this is performed by the application end-user before accepting the system. Acceptance testing for Azure applications must be performed on the application deployed on the hosted Azure instance. The user acceptance test environment should be closely similar to the actual production environment. 20 Infosys White Paper

21 Figure 18: Steps in user acceptance testing User acceptance testing approach is quite similar for an on-premise application and Azure application. The diagram below shows a comparative view of people, environment and tools required for user acceptance testing an on-premise application vis-à-vis that of an Azure application. Figure 19: Comparative View for user acceptance testing Once all the defects are fixed and system is tested, it is deployed on the production deployment region on Azure for users. Infosys White Paper 21

22 Benefits realized after application validation with Azure The platform provided by Microsoft Azure offers the capabilities which ease testing applications on Azure. Following are the benefits derived Testing environment setup made easy using Microsoft Azure one can easily set up the kind of environment required to test an application. The application can be easily configured to use specified number of compute or storage instances on Azure as well as the specified VM size. Apart from this, various test environments would not require any additional efforts such as system hardening, resource access lock down, etc., to ensure that the platform has to be made ready to be hosting applications. This will also eliminate administrative overhead of managing multiple test environments. Abstraction for infrastructure As compared to traditional on-premise applications where setting up test environment would mean procuring server class hardware and installing software licenses, Azure provides a platform where a user can directly host applications and not worry about setting up the infrastructure. Infrastructure level details are abstracted and are managed by Microsoft. Since acquiring test and production environments would be easy and time to market of the applicationwould be significantly faster. Standard environment for all test environments on Azure Microsoft Azure offers a standard platform, therefore all test environments setup on Azure will be identical. In traditional testing approach this is a common issue that application works fine in testing but fails in production. One major and common reason is that testing and production environments are not same. In Azure the staging and production environments are exactly same. Hardware configuration and software installed are consistent throughout and it helps avoiding this kind of issues. Effective test environment management The Azure platform is well managed and has robust mechanism for ensuring security, recovery and high availability in place out-of-the-box. Providing these features in the traditional set up will require considerable planning, effort and specialized skills. But with Azure the QoS criteria mentioned and demanded by any enterprise application is delivered implicitly as a part of the Azure services offerings. Dynamic scalability While developing or testing an application if user wants to scale the application, this will require setting up new servers along with possible modifications in the application. On Azure scaling is just a matter of modifying number of compute and storage instances required by application. Control over Operating System Updates and Upgrades Customers can control which operating system updates to apply to their hosted Azure instances. With this customers can ensure that any patches or updates that break their application are not rolled out or the required changes are made in application beforehand. Cost effective When setting up environment for a traditional testing approach, acquiring hardware, installing software, maintaining infrastructure will add the same cost as setting up actual production environment. Whereas in Azure charges are based on utilization, thus saving a lot of time and manpower and resulting in a cost effective solution. 22 Infosys White Paper

23 Challenges of testing applications on Azure Despite the fact that Microsoft Azure platform has got potential for hosting applications without worrying about infrastructure level details, it has certain limitations also. With the abstraction comes the lack of control. Microsoft provides restricted access privileges to users on the Azure platform. Some of these limitations cause difficulties in performing application tests on Azure. These challenges are described as follows Organization Policies Are Not Implicit For Hosted Applications In a traditional approach, all incoming and outgoing network traffic will have to pass the organization firewall and certain security measures and policies are implicit with this. Organizations apply many security policies through firewalls like the kind of response data than can be sent outside or the type of incoming requests. When an application is hosted on Azure, it is no longer in the confinement of the organization, so enforcing the organization policies through firewalls is not possible. Testers will have to perform proper testing to ensure these otherwise implicit parameters are validated and function as expected. Challenge Firewall restriction Policy enforcement Workaround/ Supportive Tools/ Recommendation Though Microsoft Azure provides a highly secure environment, any other security aspect such as imposing site access restrictions, secured transfer of confidential enterprise data, etc., will have to be taken care of by the application and testers will need to test it. Organization policies can t be enforced implicitly since applications run out of the boundaries of an organization s IT system. Application will have to take care of and the testers will need to test it. Development Challenges Table 3: Challenges in enforcing organization policies and their workarounds For an on-premise application, analyzing a bug, debugging and trouble shooting is comparatively easy with support of numerous Microsoft tools, open source tools and third party tools available today. Since Microsoft Azure is a new platform; this kind of support is currently not available. SQL server provides a proficient way to access database, run profiler, check indexing, database tuning, checking query execution plan etc. SQL Azure doesn t support that level of features today. Azure as of today has very limited tools available that can support testing Azure applications. Generation of test data, capturing results, capturing performance matrices, analyzing test results, etc., are very easy and convenient in on-premise applications using various tools. Unavailability of such tools hampers the overall productivity and amount of efforts by developers and testers. Infosys White Paper 23

24 Challenge Less support of debugging and troubleshooting Workaround/ Supportive Tools/ Recommendation Windows Azure diagnostics feature can be used for code instrumentation of Azure applications. Developers can add additional functionality in the application to view system event logs, IIS logs and performance monitor logs using Windows Azure Diagnostics feature. Limited tools available for security testing Limited tools available for performance testing Database access and analysis Test data creation Interop APIs Platform level security is in the control of Microsoft. Application level security measures such as role and membership based access will have to be taken care of by the application developers. Windows Azure diagnostics feature along with Diagnostic Monitor (typeperf utility) of Windows Azure can be leveraged to collect performance parameters of a hosted application. SQL Azure doesn t support full fledged features of SQL server. e.g. profiler, query execution plan, etc. Apart from SQL 2008 R2, Microsoft provides an online database management tool with project name Houston for DB operations on SQL Azure. Test data can be created manually or programmatically for Azure storage and SQL Azure ; inbuilt tools are not available. Microsoft SQL Azure Data Sync tool can be used to replicate database from SQL server to SQL Azure. The applications using interop APIs are not supported on Azure since these require APIs to be registered. This kind of scenario cannot be supported on Azure. Such applications will have to look at hybrid approaches (mix of onpremise and Azure deployment) to build applications on Azure. Table 4: Development challenges and their workarounds Troubleshooting as a Tester Debugging and trouble shooting an Azure application in the development environment works the same way as that of a non-azure application. One can attach debugger to the executing code and debug it in Visual Studio. Or one can use Trace.Write() to get trace related information, If an application hosted on Azure platform has to be debugged, one can use below methods 1. Use IntelliTrace feature of Visual Studio 2010 a. Enable IntelliTrace for roles in Azure application b. Publish application on Azure subscription c. After deployment, open server explorer and expand add current deployment d. Open IntelliTrace in Windows Azure Compute for deployed application e. View IntelliTrace logs and call stack f. Disable IntelliTrace once debugging is done and before deploying the Azure application in production. 2. Use Windows Azure Diagnostics to capture logs 24 Infosys White Paper

25 a. Write traces in application or modify application configuration (.cscfg) to capture performance counters, Windows event logs, IIS logs, application crash dumps, infrastructure logs etc. b. Logs will be written into blob and table storage c. Use tools such as Azure Storage Explorer, Windows Azure Service Management CmdLets, Windows Azure MMC, etc. to view logs 3. Use Remote Desktop Connectivity a. Before publishing the application, configure remote desktop connections and enable connections for all roles b. Provide certificate, user name and password for connection c. Go to Azure management portal, enable remote access and connect to a role d. Download.rdp file and open it. Provide user name, password and connect to the role with remote access. Internet availability is the basic requirement for developing and testing an application on Azure. If Internet is down, testers won t be able to perform cloud testing. Hence, it is mandatory to have internet connectivity all the time to carry out testing on the cloud. There are several services of Microsoft Azure such as Windows Azure Platform for hosted application and Azure storage (Blob, Queue, and Table), AppFabric service bus, Windows Identity Framework, Access Control Service, SQL Azure etc., which an application may be using as building blocks. If an Azure application is using any of the above mentioned services then the connectivity to those services and availability of the services needs to be checked before initiating tests. Say in enterprises, owing to security threats access to several external sites are controlled and more often blocked. They have a locked down environment where many of the ports are also disabled. If an application uses AppFabric service bus, then, ensuring that required ports to connect to the service are open is necessary. Apart from this the requests going to and responses coming from application hosted on Azure will pass through the organization firewall. Testers will have to take care of this aspect also. Corresponding ports will have to be opened and firewall policies might have to be amended for sending requests and receiving responses from an organization. Challenge Internet connectivity Connectivity to Microsoft Azure services Workaround/ Supportive Tools/ Recommendation The tester needs to ensure the proper internet connectivity for smooth testing operations. Connectivity to the Azure blocks e.g. Azure compute, Azure storage (blob, queue, table), SQL Azure, AppFabric service bus, Access control service etc. used by application needs to be ensured. It may require opening some of the ports and few modifications in the firewall policies. Table 5: Challenges in connecting to Azure services from an organization and their workaround Infosys White Paper 25

26 Summary Azure cloud computing offerings in the PaaS space offers developers a platform to not only host and run applications but also a ready-to-use test bed which can help projects reduce the overheads with setting up any world class test facility which has been discussed in this paper and also further explained later in this section. On the application validation front, Azure based projects will benefit in terms of reduced time, effort and cost involved in setting up the various test environments. However there are challenges on the other end, more so since the technology is new and the space still lacks tools and utilities for testing, debugging and troubleshooting and which we have highlighted in this paper along with strategies to mitigate them. The figure shown below summarizes the pros and cons of testing an application on Azure which developers need to be aware of. Figure 20: Pros and cons of application testing on Azure Further to support our views on Azure capabilities of accelerating the testing lifecycle, the graph drawn below show a relative comparison of total cost required for testing an on-premise application vis-à-vis to testing an Azure application. The graph is illustrative and may differ for different projects. 26 Infosys White Paper

27 Figure 21: Comparison of total cost in testing an on-premise vis-à-vis Azure application The graph above shows the capital and operational expenditure incurred in the test life cycle of an application deployed onpremise as well as on Azure. During unit testing and integration testing the costs will be almost identical since both these stages are performed on-premise with similar infrastructure and software requirements. Moving on to the later stages of any testing lifecycle, as the need for the test environments become more and more specialized, the capital and operational expenses will increase. This is primarily owing to the high infrastructure and software requirements of these environments followed by specialized skills required to run and maintain it. Whereas in Azure, since these factors are abstracted by the platform, only operational cost would incur which would be more in line to the demands of the test scenarios. Operational expenses will occur only for the duration an application is deployed on Azure. And as we can see, no charges would accrue the minute the application instance is deleted which would be as soon as the tests have been completed. This pay-as-you-use model is a very useful option available for applications deployed on Azure and needs special attention to be paid by the developers to avoid be charged while no tests are being run. Additionally, since no extra efforts are required for setting up test environments, the testing life cycle span will reduce and an application can go into production early, thus reducing the overall time to market. These operational benefits are a big driver for enterprises that are looking at eliminating their non-productive investments in IT. Test beds being one such area, as discussed in this paper, show a huge potential with developing applications on Azure to help enterprises achieve this goal. Infosys White Paper 27

28 Reference Description [1] Wikipedia URL [2] Windows Azure Security Overview Azure /whitepapers/papers/default.aspx [3] SOX Compliance [4] HIPPA [5] ISO/IEC [6] PCI Data Security Standard MSDN Testing in MSDN Unit testing Web test Validation rule SQL Azure Firewall Azure Storage Explorer OS versioning on Windows Azure Microsoft SQL Azure Data Sync Azure Application Troubleshooting Remote Desktop on Azure application Azure.diagnostics.aspx support.com/2009/12/sql-azure -firewall-tutorial/ storageexplorer.codeplex.com/ Azure /archive/2010/01/11/operating -systemversioning-in-windows - Azure.aspx Azure /sql Azure /datasync/ Infosys White Paper

29 Authors Sidharth Subhash Ghag is a Senior Technology Architect with the Microsoft Technology Center (MTC) in Infosys. With over twelve years of industry experience, he currently leads solutions in Microsoft Technologies in the area of Cloud Computing. In the past he has also worked in the areas of SOA and service-enabling mainframe systems. He has been instrumental in helping Infosys clients with the service orientation of their legacy mainframe systems. Currently he helps customers adopt Cloud computing within their Enterprise. He has authored papers on Cloud computing and service-enabling mainframe systems. Sidharth blogs at Divya Sharma ([email protected]), Technology Analyst with Microsoft Technology Centre (MTC), Infosys and has more than 4 years of experience of developing and designing solutions in Microsoft Technologies such as Windows applications, ASP.net, web services, WCF services etc. Her current work includes exploring Windows Azure platform and implementing applications using technology stack of Azure and deploying those on Azure platform. Divya blogs at Trupti Sarang ([email protected]), Senior Systems Engineer with Microsoft Technology Center (MTC) in Infosys. She has nearly 3 years of industry experience on Microsoft.NET. She currently works on implementing and deploying projects on the Windows Azure platform. Acknowledgement Authors would like to acknowledge Vijayanathan Naganathan, Senior Technology Architect, Independent Validation Services (IVS), Infosys Technologies Ltd. and Bhavin Jayantilal Raichura, Senior Technology Architect, Manufacturing IBU, Infosys Technologies Ltd. for their help in reviewing this paper.