Team Foundation Server / Visual Studio / Testmanager 2012 and Ranorex Documentation Sample Virtual Maschine (VM: TFS2012_U2_Ranorex) Martin Sedlmeier S3CC GmbH Mai 2013 Page 1
Table of Contents Overview... 3 Overview Infrastructure... 5 Start the VM:... 6 Planning... 10 Implementation:... 15 SCM & CI... 18 Integration (Part 1 and Part 2)... 22 Associate... 31 Troubleshooting... 39 Page 2
Overview This document describes a Virtual Machine (VM) which contains a sample integration of Ranorex into the Application Lifecycle Management (ALM) - Process with Team Foundation Server (TFS), Visual Studio (VS) and Test Manager (MTM). The VM TFS2012_U2_Ranorex extends a VM provided by Brian Keller (Microsoft Principal Technical Evangelist for ALM). Many thanks to Brian for this VM (downloaded from http://blogs.msdn.com/b/briankel/). Figure 1: Overview Integration Planning: The story starts with the Planning of the User Stories (PBIs) for a Sprint. Part of this PBI is a tiny Test Case. After manual Testing via Testmanager this Test Case is planned for UI-Automation, should be used in our Build Definition of our Application (CI) and the (automated) Test Case should be part of some Regression Tests (later on). Page 3
The UI-Test (Automated with Ranorex) should start inside Visual Studio (Developer), inside a Build Definition (e.g. Nightly Build) and inside Microsoft Test Manager (Tester). Implementation: The manual Testrun for this Test Case could be finished successfully. Now, we are able to automate this Test Case via Ranorex. SCM & CI: The automated Test runs pretty well in Ranorex, so it is time to add the (Ranorex-) Solution to Source Control in TFS. Additionally we create a Build Definition for this (Ranorex-) Solution and copy the result to a shared Drop-Folder. Integration (Part 1): As next step we integrate the UI-Test into the (Unit)-Testing Framework. This allows us to start the automated UI-Test via TestExplorer in Visual Studio. Integration (Part 2): Now, we are able to integrate this automated UI-Test into the nightly build of our Application. Associate: Finally we associate the automated UI-Test with the Test Case. We created a tiny Lab Environment, so we are able to start the automated UI-Test from Testmanager. Page 4
Overview Infrastructure All Components are installed on a single VM. For sure, this shouldn t be done in real life, this VM is only for demonstrate reasons. All Services are running under the Account Administrator. There is a Build- and Lab- Environment configured (based on localhost). All the work (e.g. automate UI-Tests, add to Source-Control) would be done under the Account./brian (Tribute on Brian Keller;-) ). The VM is in a VMWARE-Format and runs quite well on a Win 8 Notebook. The VM is configured with 4 GB Memory (should be enough) and 2 Processors. You need VMWare-Player (free) for starting the VM locally. All the Passwords (Accounts) are set to P2ssw0rd (Upper P digit 2- lower ssw digit 0 (zero) lower rd). The software is provided with Trail-License Keys (10 days left). It is highly recommended, that this keys will be replaced with valid MSDN-Licenses. Please take a look to this document: http://download.microsoft.com/download/a/9/2/a9253b14-5f23-4bc8-9c7e- F5199DB5F831/Working%20with%20the%20Visual%20Studio%202012%20Update%202%20RTM%20ALM%20Virtual%20Machine.docx You will find some important information about the correct handling of the VM. It describes the Hyper-V Version, but you will find some hints about time-settings in the VM: The System Time in the VM is set to June 2012 (set by a tiny startup-task). After using the VM, it is highly recommended, that you change the value in the File C:\util\SetDateAndTime.bat to the current System Time in the VM (+ 5 min) before you shutdown the VM. The System Time will be set to this Value at the next startup of the VM. Page 5
Start the VM: There are some steps to do to bring the VM (and Services) up and running: Note: please leave the accounts always in this State (e.g. logged on). Description Screenshot Copy VM to a local Folder (e.g. C:\_MyVMs) Start the VM (DoubleClick on VM-Config-File C:\_MyVMs\TFS2012_U2_Ranore x \TFS2012_Update2.vmx) Note: you have to install VMWare Workstation or VMWare Player Because of the configured Lab- Environment after starting the VM the user Administrator is automatically logged in. The Test Agent is started and Online. If the Status is set to disconnected, please refer to the section Troubleshooting at the end of this document. Start the Build Service: via Team Foundation Administrator Console Build Configuration Start Page 6
The Build Service runs as a interactive process : you have to provide the Account-Password (P2ssw0rd) The Build-Service is starting: Minimize the Windows Important: otherwise the UI-Tests will not run Close the TFS Admin Console: File Exit Page 7
Leave the Account Administrator as logged on and switch to user Brian: Menu VM send Ctrl+Alt+Del switch User Menu VM send Ctrl+Alt+Del DoubleClick Account Brian Keller Type in Password (P2ssw0rd) Open a RDP-Session to the Account Administrator on localhost: DoubleClick RDP-Icon on the Desktop of the Account Brian, Type in the Password (P2ssw0rd) for Account Administrator Page 8
Restore down the RDP-Window, but do not minimize it. Important: otherwise the UI-Tests will not run Page 9
Planning Back to our Story: in this section we will go to the Story Backlog, take a look to our PBI and the defined Tasks and Test Cases. We will see the Definition of a Test Plan for Release 1 and a query based Test Suite. We will run our Test Case manually and we take a look to the Test Results of our Test runs. After all, we change the Automation Status on our Test Case from not automated to planned. Description Screenshot Under the Account Brian: Open Browser IE and go to the TFS FF Web Access view Backlog In the Sprint Backlog of the Sprint 3 you will find the PBI Customer can create a new Service Ticket with some Tasks. That means: - the Implementation is planned - the manual Test is planned - the automate Test is planned - the integration is planned.., and trackable in the Daily scrum Meeting Page 10
The PBI contains a linked Test Case. In this State of the Lifecycle of the PBI the Automation State is set to Not Automated. After the Implementation of the PBI (Task implement UI) is done the manual Tests could start Open MTM (Testmanager) Plan Contents Release 1 Sprint 3 (query based) You will find our Test Case. Open Query behind this Test Suite ( edit Query) Page 11
The Query behind the Test Suite is defined with the Iteration Path and the State of the Test Cases. Under Test you will find our Test Case 3 times, because we defined 3 configurations for it (Browser). Start a manual Testrun via Run Run with options Note: there is a Build associate with the current Testrun Check Box Run all the tests manually for a manual Testrun (we will come back for automated Testrun) Page 12
Start the manual Test Run (without Option Create action recording) and save the Result ( Save and Close) You will find all Test Results under View results See all the Test Results under Section Result History. DoubleClick on a specific Test Result Page 13
will open a detail view. After the manual Tests could be finished successfully the Automation Status could be changed to Planned and the Status of the Task Test the implemented UI could be set to Done. Page 14
Implementation: In this Section we will automate the Testrun via Ranorex Test Suite. Description Screenshot Brian will find his Task for automate the Test Case. Note: The Task contains 3 Links (with LinkType): Parent: the PBI to this Task Predecessor: Task for manual Testing (s. Planning) Related: the Test Case which should be automated The Test Steps in the Test Case describes the Test Scenario. Page 15
Open the Ranorex Studio and go to the Test Suite FabrikamFiberUITesting Note: We run the Tests with 3 different Browsers, but we use always the same Ranorex-Recording Modules. (The single Setups for the Browser are disabled, because we will start this Testrun outside of the Ranorex- Studio. At this point we are able to run the Testautomation via Ranorex Studio Run Test Suite Page 16
This works fine so Brian finished his work quite well (and could set the Status of his Task to Done ) Page 17
SCM & CI At this Point we have an automated Test Case (with a Ranorex-Solution). Now we will add this Solution to the TFS-Source Control (SCM) and create a Build-Definition for this Ranorex-Solution (CI). Description Screenshot Brian will find his Task for adding the Ranorex Solution to the TFS-Source Control. Open the Visual Studio (VS), connect to the local TFS VSALM and the Team Project FabrikamFiber in the Collection FabrikamFiberCollection Page 18
Open the Source Control Explorer and take a look to the Ranorex-Solution. You could work with this Solution in Visual Studio or in the Ranorex Solution Page 19
In the Team Explorer under the Section Builds you will find the Build Definition Ranorex Solution (Main) The Build Definition is using a customized Build Template called DefaultTemplate11.1forRanore x This Template builds the Ranorex Solution and copies the output (= Ranorex Binaries) to a shared Drop Location Page 20
The first Part of the Integration is done, Brian leaves the Task-Status to In Progress and set Remaining Work to 3 (hours). Page 21
Integration (Part 1 and Part 2) The next Step in our scenario is the Integration of the Ranorex Solution into the (Unit)-Testing Framework. This means, we would like to start the UI-Tests inside of Visual Studio and not only via Ranorex Studio (Part 1). After this Integration we will be able to add this UI-Test to our Build Definition Fabrikam (Main) with UI-Testing of our Application FabrikamFiber (Part 2). Description Screenshot Brian is still working on this Task. Under the Solution FabrikamFiber.CallCenter you will find a new Test Project. Page 22
The Integration is really simple: just create a TestClass with some Test Methods (as always). The Integration is done via Ranorex TestSuiteRunner. Note: This TestClass is using the Ranorex-Binaries located in the Drop Location of the Ranorex Build Definition. Open the Test Explorer Page 23
and run the Tests You will find the Result under Test Explorer Passed Tests Output report.rxzlog (Ranorex Log File) Note: the TestMethod *_All starts the UI-Tests with the Browser IE and FireFox, TestMethod *_All_WithChrome starts IE, FireFox and Chrome Page 24
Now, we can add this Tests to our new Build Definition (in this Case for the Application-Solution) Fabrikam (Main) with UI.Testing We are using the Default Template. Under Process automated Tests 1. Test Source we define our TestClass (Test Sources Spec) and TestMethod (Test Case Filter) Page 25
Queue a new Build Note: Be sure, that the Build Service is started, the RDP-Session is open and not minimized. Queue a new Build Page 26
As you can see in the RDP-Window the UI- Tests are running under the Build Service (Account administrator) You will find the Test Results of this Build under the Build-Log (Team Explorer Builds my Builds DoubleClick on the finished Build) section test run click on Test entry opens the Test Result- Window Right Mouse Click View Test Results Details Page 27
You will find the Ranorex-Log in the Section Collected Files (on the end of the Log). If the test goes to the Status Failed, you could quite simple create a Bug Workitem. Page 28
The Test Result (including the Ranorex- Log) will be automatically attached to the new Bug. Note: Please set the Iteration Path to the current Release and you will find this Bug in your Product Backlog (via Web Access) Page 29
You could queue a Build also in the Web Access. The Task is still not finished, so Brian let the Status on In Progress and changed the Remaining Work to 1 (hour) Page 30
Associate Finally we have to combine our new TestMethod (implemented via Testing Framework) and the original Test Case. This allows the Tester to run this automated UI-Test within Microsoft Testmanager (MTM). Description Screenshot For associating a TestMethod with a Test Case, we have to start Visual Studio and open our tiny Test Case. There is a Query available: (Team Explorer Work Items Queries Shared Queries Current Sprint DoubleClick on Query Test Case Open the Test Case, go to the Tab Associated Automation and take a look to the set Values. Note: The Automation Status changed to Automated Page 31
Open the Testmanager and you will find our Test Case in the Test Plan Release 1 under the Test Suite Regression Testing (Testing Center Plan Content) Note: This is a Query Based Test Suite. Click on Edit query and take a look to the Query behind the Test Suite. Page 32
Run the Test (Testing Center Test Run Tests Release 1 Regression Testing mark our Test Case Run with options We would like to run this test automatically (do not check option Run all the tests manually leave the Environment to LabForRanorex and start the Test run Note: The selected Environment is on the same VM, in Real Life this should be on different Maschines. Page 33
As you can see in the RDP-Window the UI- Tests will start under the Account Administrator. After the Test Run is finished you will find the Test Result under the Section Attachments. DoubleClick the attached *.trx-file Page 34
will open the Test Results inside Visual Studio. View Test Results Details will open a detail View of the Test Results. Under the section Collected Files you will find the Ranorex- Log. Page 35
You could create a bug directly under the Section Tests Create bug) As you can see under the Tabs Test Cases and Links some additional Infos would be automatically linked to the Bug (open it with DoubleClick). Page 36
You will find this bug in the Product Backlog. You could find a Summary about all of the Test Runs in Testing Center Test Analyze Run. Define the view (Manual or Automated run) Page 37
Our tiny Test Case is still manual runnable. Check the option Run all the Tests manually (Testing Center Test run Tests Release 1 Regression Testing Run with Options Finally all the work is done, so Brian could change the State of his Task to Done. Page 38
Troubleshooting Testagent will not start If the Test Agent does not start (State = disconnected), please go to the Test Agent Configuration Tool (via Start search programs and files type in Test Agent) and set the Password for the User of the Test Agent. Ranorex will not start If Ranorex will not start (Errormessage like : Ranorex License already in use ), please restart the VM (but don t forget to set the new time in c:\util\setdateandtime.bat). Page 39