Mobile App Monitoring. Release Notes. Release 7.0

Transcription

1 Mobile App Monitoring Release 7.0

2 Mobile App Monitoring 7.0 Mobile App Monitoring 7.0 February 2014 Copyright Notice Copyright Keynote Systems, Inc. All rights reserved. THE INFORMATION CONTAINED IN THIS DOCUMENT IS PROVIDED AS IS WITHOUT ANY EXPRESS REPRESENTATIONS OF WARRANTIES. IN ADDITION, KEYNOTE DISCLAIMS ALL IMPLIED REPRESENTATIONS AND WARRANTIES, INCLUDING ANY WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT OF THIRD PARTY INTELLECTURAL PROPERTY RIGHTS. All text and figures included in this publication are the exclusive property of Keynote and may not be copied, reproduced, or used in any way without the express permission in writing of Keynote. Information in this document is subject to change without notice and does not represent a commitment on the part of Keynote. Keynote may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents except as expressly provided in any written license agreement from Keynote. The trademarks or registered trademarks of Keynote Systems, Inc. in the United States and other countries include Keynote, DataPulse, CustomerScope, Keynote Customer Experience Rankings, Perspective, Keynote Red Alert, Keynote WebEffective, The Internet Performance Authority, MyKeynote, SIGOS, SITE, keynote The Mobile & Internet Performance Authority, Keynote FlexUse, Keynote DeviceAnywhere, DeviceAnywhere, Keynote DemoAnywhere, Keynote MonitorAnywhere. All related trademarks, trade names, logos, characters, design and trade dress are trademarks or registered trademarks of Keynote Systems, Inc. in the United States and other countries and may not be used without written permission. All other trademarks are the property of their respective owners. Please forward any comments or suggestions regarding this document to Keynote Support. Keynote Systems, Inc. 777 Mariners Island Blvd. San Mateo, CA ii Keynote Confidential

3 Mobile App Monitoring 7.0 Contents 1 Overview Main Features Known Issues... 7 Keynote Confidential iii

4 Mobile App Monitoring Overview Keynote announces the launch of Mobile App Monitoring, an enterprise-class monitoring service for mobile devices. Mobile App Monitoring 7.0 is a generation up from Mobile Device Perspective 6.0 and presents users with the combination of scripting in Mobile App Monitoring Studio with results in MyKeynote. Script provisioning takes place in the Keynote Service Center. With Mobile App Monitoring, production support teams can easily write/record monitor scripts and schedule them at any frequency on real, live devices. In MyKeynote, you can define custom thresholds for acceptable performance and the alerts that are triggered as a result. 2 Main Features Scripting Interface and Architecture Mobile App Monitoring uses the same visual scripting interface as Keynote s DAE. Users create actions, states, and test cases in organizational containers called projects in Studio. Actions are the basic unit of scripting and can be unpartitioned or implemented separately per device in order to account for differences in interfaces. Timers call out sections of scripts to be measured. You can set performance criteria that trigger alarms for these script sections. Test cases are comprised of calls to actions and states, and a scheduled test case is a measurement that you can graph and analyze in MyKeynote. A project defines reusability of scripts, e.g., you can reuse an action in multiple test cases contained within a project. The figure below shows the Mobile App Monitoring visual scripting environment with an expanded project directory, an action script, command toolbar, and live device. Figure 1 Mobile App Monitoring Visual Scripting Environment Active tab Live device Command toolbar Script canvas Command properties 4 Keynote Confidential

5 Mobile App Monitoring 7.0 Device Interaction in Studio Studio enables you to control real, live devices remotely over the Internet/network to work with them. You can work with devices in the Test Center as well as Monitoring views. Users have access to all device features, including the keyboard, touchscreen, ringer, speaker, battery, power, data cable, and of course, applications. Features in device console that appears to the right of an acquired device enable you to send SMS interrupts to the device, upload applications, access debugging tools and log files, and collaborate with team members by sharing the device screen or device interaction snapshots and video. Figure 2 Acquired Device in Test Center View Sort/filter controls Device list Device console Device log Live device MyKeynote MyKeynote is a powerful web-based tool for visualization, analysis, and reporting the results of Mobile Applicaton Monitoring scheduled measurements. You can also configure alarms from here. Log in to MyKeynote at or click MyKeynote in the Links view of Studio. Figure 3 MyKeynote Landing Page Keynote Confidential 5

6 Mobile App Monitoring 7.0 Opt to view Mobile App measurements so you can select one or more of them to graph. Figure 4 Measurements in MyKeynote Keynote Service Center (KSC) Users with administrative rights can schedule scripts on real devices in the Keynote Service Center. You can also manage your account from here. Figure 5 Keynote Service Center Landing Page You need to enter an agreement ID against which the monitoring measurement is to be provisioned. Figure 6 Entering an Agreement ID in KSC 6 Keynote Confidential

7 Mobile App Monitoring Known Issues Multi-Device Test Cases Multi-device test cases are not supported in this release of Mobile App Monitoring. However, there is support for MO/MT tests that allow you to measure the time taken for a single monitoring transaction between two devices, e.g., sending and receiving an SMS. NOTE The content of the transaction, i.e., the content of the SMS, is controlled by Keynote and includes a queue ID used to measure transaction time. Using Timers Creating timers and then applying them to portions of your script using the Timer command enables you to mark portions of your script to be tracked and measured. You can set up performance thresholds for these script sections and trigger alarms based on the thresholds. Here are some notes on using timers: When creating a timer, set the Timer Type to Script Page. Do not change the default Failure Threshold the performance threshold for setting off alarms is specified in MyKeynote. Timer type Default threshold When inserting a Timer command in your script, use either the Start setting to specify a start point and the Stop setting to specify an end point; do not use the Fail setting. Keynote Confidential 7

8 Mobile App Monitoring 7.0 You must insert a start as well as a stop timer command to measure accurately. If you insert a start timer just above a Wait Event command, every branch of Wait Event (excluding the Timeout branch) must have a stop timer. Alternatively, you can insert a start timer just above Wait Event and a stop timer in the Tie placeholder below the Wait Event. Exporting and Importing Projects If you export and then delete a project from Studio, you cannot re-import the project under the same name. Be sure to use a unique name that has not already been used in your system when importing or creating projects in Studio. In order to export and import projects effectively, the best practice is to create smaller projects with limited image-based commands (e.g., in Wait Event or Find and Touch). Exporting/importing larger projects with lots of images generates errors. Deleting a Dependent JAR If you try to delete a dependent JAR file from project properties, you encounter an error. However, if you re-launch Studio, you should be able to remove the dependency with no problems. Project Sub-Folders Once created, sub-folders of the actions, states, and test cases folders of a project cannot be deleted. Pasting an Action with Dependencies You cannot use the Paste with Dependencies option to copy and paste an action that calls another action from a dependent project. Error Category and Types Users can only add error types to the DAP error category right-click your project > Properties > Error Types. Create and save one error type at a time; if you add multiple error types at a time, only the first one is saved. The name of the DAP error category name is incorrectly displayed as DeviceAnywhere in commands. However, you can choose and insert the correct error types from the DAP category. Effects of Loading Error Types in Commands You can use the error types from the DAP error category for reporting when scripting in Studio. As a result of loading these error types: Switching from the Timeout tab to other tabs in the Find and Touch command can take time. There can be a time lag when dragging some commands, e.g., Send Keys, Find and Touch, onto the script canvas. Looping Through Data Set Values When you specify a command sequence to loop through the values of a data set, only the first value of the data set gets repeatedly utilized or inserted into the device. For example, if your script loops 5 times to insert 5 values from a data set into a device field, only the first value gets inserted for each of the 5 loop iterations. Discarding Changes in Loop Command When you make edits in the Loop command, you cannot undo them using the Discard button. 8 Keynote Confidential

9 Mobile App Monitoring 7.0 Continue with script Option in Wait Event If you choose the Continue with script option in Wait Event to continue even when the command fails or times out, script results incorrectly show the command as successful. The script is also incorrectly shown as having succeeded. Flickering Device Image in Commands Device screens captured in image-based commands can flicker if the device resolution is 1080 x 1920 or higher. To minimize this issue, ensure that your system meets the minimum requirements for Studio and has a large monitor. You can also maximize screen real estate so that scroll bars do not appear in commands by collapsing the program sidebar and other unnecessary panes when scripting. Image Verification Settings in Commands and States The Pixel tolerance slider is incorrectly set to 100% in states for image-based reference points. The slider is set to the correct value of 95% in commands such as Wait Event, Find and Touch, and Navigate To. To view the slides, select Show Options Editor in a command or Advanced Options in a state. Auto-Updating Object Layout When navigating from one application screen to another in Object commands, the object layout captured is not automatically updated. Check and uncheck Auto-Update Layout twice as a workaround. You can also close and reopen the command to refresh it correctly. Calling an Object-Based State from Navigate To Calling an object-based reference point in a state from Navigate To does not work. The object referenced is not correctly identified during script runs. Swipe in Object Commands Swiping to identify and select an object in Object commands does not work in Mobile App Monitoring scripts. Use Send Keys instead to record and implement a swipe. Java States Implementing Java states does not work in Mobile App Monitoring. While you can create a Java state (right-click states folder > New Java State), you cannot build the project successfully thereafter (rightclick project > Build Project). Script Results If you click Upload Result in the script result window after running a script in Studio, you will encounter an error. Studio run results cannot be uploaded to MyKeynote. KSC and MyKeynote from Studio Clicking MyKeynote or Keynote Service Center in the Links view of Studio does not automatically log you into these portals; you must enter your credentials to log in. Exporting Video Exporting video (right-click device > Export Frames > Video from history) of your device session to the file system does not work. NOTE The Toggle Recording command, which enables you to capture device video during a script run, is not currently available in Mobile App Monitoring. Keynote Confidential 9

10 Mobile App Monitoring 7.0 Powering Off a Device Even after you power off a device in Studio (e.g., by holding down the Power button or by disconnecting the battery using the Hardware tab of the device console), the device screen does not go black and the last visible screen is still displayed. However, if you click on the device screen, it does not respond. Acquiring Devices on Macintosh On Studio for Macintosh, if you acquire more than three devices, the device name and number does not appear on the device tab. Acquire three or less devices to avoid this problem. Timestamps for Data Points in Scatter Plots When you opt to view a scatter plot for a measurement in MyKeynote, you can hover over individual data points to see some details such as timestamp of the measurement start time. The timestamp is matched by the placement of the data point along the x-axis (time intervals) of the scatter plot. Figure 7 Hovering Over a Data Point in MyKeynote Scatter Plot If you click on a data point to drill down and view more details, however, the timestamp shown is not accurate. Refer to the details that appear when you hover over a data point for the correct timestamp. Figure 8 Incorrect Timestamp in Drill-Down Details Incorrect timestamp Signal Bars in Data Point Details When you click a data point in a scatter plot to view details, the Signal Bars item reads 0 of <Max Signal>. To display device signal strength information in this field, you need to create a project with an action that captures this information from the device. 1 Create a project called Signal Strength. 10 Keynote Confidential

11 Mobile App Monitoring Declare a global numeric variable called signalstrength with a default value of 0. 3 Create an action that verifies signal strength and then sets the value of the signalstrength variable accordingly. 4 Set up the Signal Strength project as a dependent of your measurement project. 5 Then in your measurement test case, call the action measuring signal strength. NOTE Be sure to publish both your main project and the dependent project prior to provisioning measurements. In the example action below, the Wait Event command uses image verification to check the number of signal bars on a device. There is a branch for each possible outcome, and the value of the signalstrength variable is set accordingly. If the Wait Event command times out because it is unable to verify any of the signal strength images, e.g., if there is no signal, the value of signalstrength remains at 0. The value of signalstrength captured during a measurement displayed in the Signal Bars field in MyKeynote. Figure 9 Action to Capture Signal Strength Branches for each possible outcome 1 bar to 5 bars Image-based verification of number of bars of signal strength Figure 10 Variable Capturing Signal Strength Branch for 5 signal bars Variable value set to 5 Keynote Confidential 11

12 Mobile App Monitoring 7.0 Dashboard Does Not Display Data for Time Periods Less than 12 Hours The Mobile App Monitoring Dashboard in MyKeynote (Dashboards > Mobile App Monitoring) does not display availability, error, or performance data for measurements if you choose time periods of less than 12 hours. This erroneously suggests that the measurements are not active. 12 Keynote Confidential