Applitools Eyes Web Application Guide 1
2 Table of Contents Overview... 3 How to start using Applitools Eyes... 3 Analyzing your first test through Applitools Eyes Web Application... 4 Exploring the home page and navigating the list of tests.... 4 Navigating the main Test Window... 4 Reviewing a step-by-step version of your test... 5 Identifying errors in a certain window and choosing comparison strictness... 6 Comparing the current result with the baseline... 7 Adding new windows to the baseline / updating the baseline... 8 Adding ignorable regions... 9 Floating UI Elements... 11 Adding new defect in your defect tracking system from Applitools Eyes tests... 12 FAQ... 13 Where Can I sign up and get the free version of Applitools Eyes?... 13 How much is an Applitools Eyes License?... 13 What payment options are available?... 13 Where can I find the Applitools Eyes End User Agreement?... 13 Which test automation tools/platforms are supported with Applitools Eyes?... 13
3 Overview Applitools Eyes is a cloud service that automatically validates all the visual aspects of any Web, Mobile and Native app. It allows you to: Automatically validate the correctness of the UI layout, content and appearance of your app. Test your app on any browser, device, OS and programing language, on any screen resolution and form-factor. Keep using your existing test automation tool/infrastructure and add Applitools Eyes capabilities on top of it. How to start using Applitools Eyes In order to start using Applitools Eyes and enjoy all its benefits you will need to follow 5 simple steps: 1. Register online for a free account - Registration is a quick and easy process and can be performed on applitools.com. All we need are a few basic details and you ll be ready to go. 2. Login to Applitools Eyes with your account - As soon as you register and receive your confirmation email, you can use your details (username and password) to login 3. Download and install the Applitools Eyes SDK - Detailed instructions on how to install and use the SDK are available in the Applitools Eyes Quick Start Guide. 4. Create and execute your first Applitools Eyes test Detailed instructions on how to install and use the SDK are available in the Applitools Eyes Quick Start Guide. 5. Analyze your test result through Applitools Eyes web application (see instruction below).
4 Analyzing your first test through Applitools Eyes Web Application Exploring the home page and navigating the list of tests. The home screen of Applitools Eyes displays an overview of all batches and tests that have been performed. The tests are aggregated in batches, and for each test, you can see the test name, test result (based on the color of the text as explained below), test execution time/date, the browser/device that the test was run on, operating system and the screen resolution. By default, all tests of the same batch will be collapsed and when clicking on the batch name, all tests under this batch will be expanded. Batches and tests will be shown in descending order, starting with the test performed most recently. Tests can also be filtered based on the status (failed, successful and running) Color Codes: Green successful test - the result of the test matches the baseline version. Red failed test that contain one or more identified conflicts. Orange The test was aborted or wasn't completed. Grey failed test where all identified issues were resolved. Blue test still running/in progress. Upon selecting a specific test, one can choose to view the detailed test results. Please note that at this point, Eyes will also specify If the selected test is a New Test. If the failed test has since been resolved. Navigating the main Test Window Upon choosing a specific test to review, the user will be directed to the test editor window. The test editor is divided into three main segments: A screenshot of the test step result from the application under test. The Analysis Toolbar. The Navigation Toolbar
5 Test steps with different result from the expected result (the baseline version) are marked as failures (note the screens marked red in the navigation window). The Baseline version contains the expected results for each of the windows that are displayed within a test. The total number of conflicts also appears in red - (1 conflict). Following customer feedback, we ve added a useful new feature that creates and accepts a baseline automatically whenever you run a test for the first time (this means the first run of test within a specific execution environment (OS, browser or screen resolution)). This feature eliminates the need to manually approve an initial baseline and fully automates the process. Now, when you run a test for the first time, it automatically creates a baseline for you (no need to manually accept each window and save it via the Applitools Eyes Test Manager). Next time you run this test on the same platform, it will automatically be compared against the created baseline. Users that might want to stick to the original setup of manually approving their baselines can do by following this quick guide on our website. Reviewing a step-by-step version of your test Using the Navigation Toolbar at the bottom of the test screen, it is possible to view a screen-by-screen run through of the chosen test. Applitools Eyes don t actually store a movie, but present a stop-motion version of the test. This method allows us to store test results forever without exhausting our server capacity. Using the Play button on the far left will start the video. During the run through, the user can inspect both the screen itself and the relevant use of mouse and/or keyboard.
6 The inner Left/Right arrows enable the user to navigate between the different windows of a test. The outer Left/Right arrows enable the user to jump from error to error. Identifying errors in a certain window and choosing comparison strictness When a specific window/test step include an error (marked in red in the test timeline), all errors in this window are marked in purple, but should you fail to see them it is possible to highlight them. Using the Highlight Differences (a flashlight icon) option, the user can view a short burst of circles highlighting all the conflicts of the current test step. In order to simplify the process of analyzing failure in a test, users can change the strictness of Applitools Eyes comparison. Users can choose between: Exact - Only windows that are exact, pixel based, replicas of each other will be considered a success. Strict The default mode where a smart comparison is performed between the actual result and the baseline, taking into consideration both layout and content changes while using Applitools Eyes smart image analysis tools to ignore minor changes that might fail a test when using pixel based comparison. Content Applitools Eyes will only identify content differences while ignoring layout based differences. Layout Applitools Eyes will only identify layout differences while ignoring content differences. Hide Applitools Eyes will ignore all differences for this specific screen and will not fail a test based on them. Currently, the different strictness levels will only be applied to your current review (and won t be saved as the default comparison level for every run of that test from that point on).
7 In future releases, you will be able to select the strictness level for different regions in each window and set this as the baseline for future tests (e.g. ignore only content changes in a region of the screen that is constantly changing like date field). Comparing the current result with the baseline Using the view controls, the Actual window and Expected window (i.e the Baseline version) can be shown individually or side by side, allowing you to see the exact differences between the actual result and the baseline. In essence, there are three ways to view the windows: Side by Side Just the new window Just the baseline window.
8 Using these controls, the user can toggle back and forth between the baseline and current versions. This allows the user to view both version "on top" of each other and dynamically assess the errors. The Zoom controls allow the user to view the image several different ways: Original Size Best-Fit for the current window Custom - using the zoom in/out controls. Adding new windows to the baseline / updating the baseline Upon reviewing each window of a failed test step, the user can choose between two options: In case this failure is a result of a defect in the, you should select Keep baseline to keep the existing baseline image, the user can raise a defect in the defect tracking system and automatically link the defect to the Applitools Eyes test result. The test will keep failing in consequent runs until the required changes are made to fix the identified defect.
9 In case this failure is a result of a legitimate change in the app under test which changed the expected behavior of the app and the user would like to accept the new window as the baseline, the user should select Accept new to accept the result as the new baseline, so from now on this will become the expected result for this window in future tests. Note that in case a similar change occurred in another execution environment (e.g. different mobile device or different browser), the change will automatically propagate to this environment as well, and you will not be required to approve this change in each environment separately. In case the failure is a result of a dynamic region in the application that is constantly changing, but should not fail the test (e.g. a date field or a frame with changing advertisement), Applitools Eyes allows you to define ignore region/s that the test will ignore in future runs, so that changes only in this region of a window will not fail the test. Adding ignorable regions Additional, more advanced, analysis of a specific window can also be performed: Users can mark a specific area of the screen that Eyes will not assess. For example, areas that don t have one expected result can be ignored and therefore will not cause failed tests. This option can be critical when working with a dynamic app or website. There may be specific regions in some windows that are supposed to change every few seconds or minutes like advertisements, news, etc. and the content of these sections is not necessarily relevant to your visual test. Using the ignore region button will allow you to mark a specific region to ignore and in consequent tests, the entire window will be validated while ignoring visual differences in the specific ignore region. The ignore region can be easily expanded/edited/deleted in a similar way to editing a text box in MS Word. The image below illustrates a window with ignore regions (the light blue squares represent ignore regions).
10
11 Floating UI Elements One of the newest elements of Applitools Eyes implemented due to popular demand. Many customers require an easy way to perform functional test scenarios using Applitools Eyes without the need for additional coding. The floating region feature allows the user to highlight a specific region of the page that while being validated, will essentially be allowed to have a dynamic location the user can define a larger area within which they allow the targeted region to appear. Upon execution of the test, the targeted region will be located (and verified) within the larger area, while all other content within the same allowed area will be ignored. To define a floating region, first click the Floating Region icon (see image below) and mark the region that you want to validate. After creating the region, then edit the boundaries where this region is allowed to appear (or "float"): To get you going, here are a few examples where using the Floating Region option could come in handy: Running a test to validate credit card transactions. If a test expects to find a specific generated transaction within a table, but all the other transactions are random and/or the order of the transactions might be different every time the test is run. Running a test to validate search results. If a test is only trying to make sure that certain search results appear following the search for fixed term, but the results may appear in a different position on every test run (see image below).
12 Running a test to validate an area that might move based on other factors. If a page contains random data that can cause our target area to move (not drastically, just slightly upwards or to the left for instance). Adding new defect in your defect tracking system from Applitools Eyes tests Using the report a bug button in the toolbar above the test screen, the user can also choose to create & send a relevant bug report using any defect tracking infrastructure (should one exist).
13 FAQ Where Can I sign up and get the free version of Applitools Eyes? On our website www.applitools.com. Check the topic How to Start Using Applitools Eyes for some more relevant info. How much is an Applitools Eyes License? There are four different licenses available (as well as the free version), each allowing different levels of freedom and capacity of use. The four options are Indie, Indie-Pro, Team and Large team (a custom license). You can find out more about the prices on our pricing page at Applitools.com What payment options are available? We support online payments through credit cards and Paypal. If you would like to use a different payment method, please contact us at info@applitools.com. Monthly and Annual payment terms are available. Where can I find the Applitools Eyes End User Agreement? You can find it in our website under Terms of use Which test automation tools/platforms are supported with Applitools Eyes? Currently Applitools Eyes include built-in SDKs for Selenium (Java,.Net, Python and Ruby), Appium, Microsoft Coded UI and HP UFT. There are also SDKs for.net, Java and Python that can receive images of your application from any test automation solutions. If you would like to use Applitools Eyes with any other test automation tool or with your internal automation infrastructure, please contact support@applitools.com and we will provide you the required information to integrate your existing automation infrastructure to Applitools Eyes.