Mobile App Monitoring. Release Notes. Release 8.0

Transcription

1 Mobile App Monitoring Release 8.0

2 Mobile App Monitoring 8.0 Mobile App Monitoring 8.0 July 2014 Copyright Notice Copyright Keynote Systems, Inc. All rights reserved. THE INFORMATION CONTAINED IN THIS DOCUMENT IS PROVIDED AS IS WITHOUT ANY EXPRESS REPRESENTATIONS OF WARRANTIES. IN ADDITION, KEYNOTE DISCLAIMS ALL IMPLIED REPRESENTATIONS AND WARRANTIES, INCLUDING ANY WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT OF THIRD PARTY INTELLECTURAL PROPERTY RIGHTS. All text and figures included in this publication are the exclusive property of Keynote and may not be copied, reproduced, or used in any way without the express permission in writing of Keynote. Information in this document is subject to change without notice and does not represent a commitment on the part of Keynote. Keynote may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents except as expressly provided in any written license agreement from Keynote. The trademarks or registered trademarks of Keynote Systems, Inc. in the United States and other countries include Keynote, DataPulse, CustomerScope, Keynote Customer Experience Rankings, Perspective, Keynote Red Alert, Keynote WebEffective, The Internet Performance Authority, MyKeynote, SIGOS, SITE, keynote The Mobile & Internet Performance Authority, Keynote FlexUse, Keynote DeviceAnywhere, DeviceAnywhere, Keynote DemoAnywhere, Keynote MonitorAnywhere. All related trademarks, trade names, logos, characters, design and trade dress are trademarks or registered trademarks of Keynote Systems, Inc. in the United States and other countries and may not be used without written permission. All other trademarks are the property of their respective owners. Please forward any comments or suggestions regarding this document to Keynote Support. Keynote Systems, Inc. 777 Mariners Island Blvd. San Mateo, CA ii Keynote Confidential

3 Mobile App Monitoring 8.0 Contents 1 Overview Main Features Updates Resolved Issues Known Issues Keynote Confidential iii

4 Mobile App Monitoring Overview In the 8.0 release of Mobile App Monitoring, users are able to trace application network activity, say in order to improve network security, as part of direct device interaction or in the context of a measurement script. Another scripting enhancement is the ability to view object layout in a tree in Object commands, making it easier to select an object to work with. Users can suspend and resume monitoring on a device while interacting with it in Studio, and view script execution on a device and an execution log in Studio. For project and script management, users now have the ability to create errors specific to their customer account in the MAM User Errors category. 2 Main Features Application Network Activity Users can capture network activity information of a specified application in a PCAP file in the course of direct device interaction in Studio or as part of a script run. In the device console, select Connect device to local computer > App Activity. Here you can specify the maximum PCAP file size, duration of capture, and storage location. Click Start to choose the application to trace from the Application List (the application process name is automatically displayed). When you stop the trace, you see a dialog box to name and change location for the PCAP file. Figure 2-1 App Activity Tab 4 Keynote Confidential

5 Mobile App Monitoring 8.0 To capture network packet information in a PCAP file as part of a monitor script, use a pair of Trace commands to indicate start and stop points for the trace. Although you can use this command in actions as well as test cases, it is recommended that you capture network trace in test cases. To set up your script for the Trace command: 1 Initiate a network trace on the device in interactive mode so you can identify the process name of the application you wish to trace in your script. Then stop the trace. 2 Close all applications on the device. 3 Insert a Trace command to Start the trace before launching the application. 4 To specify the application to trace, create a variable in the Trace command and use the application process name as its value, e.g., for Chrome, the process name is com.android.chrome. 5 Stop the trace before closing the application. You can only capture a single trace (i.e., one Trace command each to start and stop a trace) in any measurement. If your measurement (test case) contains network trace commands, results for script runs in MyKeynote display information from captured PCAP files. Select your measurement in MyKeynote and generate a scatter plot. Select any data point to view details: In addition to the Test Details tab, you will now see the Application Activity and Network And Device Details tabs. The Application Activity tab shows the total number of bytes and elapsed time during the network trace. It shows each TCP network connection with the destination IP address on the left and a graphical Keynote Confidential 5

6 Mobile App Monitoring 8.0 representation of the packets sent and received. Hover over each network connection graphic to see details. Figure 2-2 Network Connection Information Figure 2-3 Application Activity Tab in MyKeynote The Network And Device Details tab shows average network and device statistics captured during the trace. Figure 2-4 Network & Device Details Tab in MyKeynote 6 Keynote Confidential

7 Mobile App Monitoring 8.0 You can download the PCAP file from the Actions drop-down list at the top-right corner. Viewing Monitor Execution in Studio Any user in a customer account can now log in to Studio and acquire a device to watch monitor execution on it live. CAUTION If you interact with a device as a monitor run is being executed, you will interfere with script results and can cause script failure. As you view monitor execution on the device, you can also see a log of device activity: In the device console, select the Tools icon > Sharing tab > Execution log. Figure 2-5 Execution Log Pausing Monitor Execution in Studio When you acquire a device in Studio, it is recommended that you pause any monitor runs before interacting with or scripting on the device. In the device console, select Connect device to local computer > Monitoring > Suspend to halt monitor script runs (or Resume to continue monitoring). Monitoring is automatically resumed when you release a device. Object Tree As with the DOM viewer to view element structure in Web commands, users now have access to an object tree in Object commands. Keynote Confidential 7

8 Mobile App Monitoring 8.0 Figure 2-6 Control to View Object Tree Figure 2-7 Object Tree 8 Keynote Confidential

9 Mobile App Monitoring Updates User Errors and Global Errors Categories Users could previous only add error types to the DAP error category. They can now do so in the MAM User Errors category, which is visible only to users in a particular customer ID. The Global Errors category is available to all customer accounts. You cannot create errors in this category. NOTE All user-created errors from the DAP category in MAM 7.0 have been migrated to the Global Errors category in MAM 8.0. This should not impact running any scripts. It is recommended that going forward, scripts use errors created in the User Errors category and move away legacy user-created errors in the Global Error category. Simplified Timer Creation and Declaration Timer creation in Studio has been simplified so that users no longer need choose a timer type or failure threshold. Additionally, there are only two radio buttons in the Timer command, one for choosing a named timer created earlier, and another to mark the endpoints of a timed sequence. Use Start to mark the beginning of the script segment you wish to time and Stop to mark its end. You can now auto-generate the declaration of the timers used in a measurement (test case). Rightclick your test case > Generate Script Timers > select the correct test case device. The test case parameter, DapScriptPageList, containing the timer list is automatically created. NOTE You can automatically generate the timer list for a test case containing timers, which in turn calls actions and/or other test cases containing timers. Keynote Confidential 9

10 Mobile App Monitoring Resolved Issues Exporting and Importing Projects Two issues with exporting projects have been resolved: After exporting and then deleting a project from Studio, you can now re-import the project under the same name. Exporting larger projects with many images no longer generates errors. Exporting Video Exporting video (right-click device > Export Frames > Video from history) of your device session to the file system now works. Deleting a Dependent JAR You can now delete a dependent JAR file from project properties without encountering an error. Deleting Project Sub-Folders User-created sub-folders of the actions, states, and test cases folders of a project can now be deleted. Pasting an Action with Dependencies You can now use the Paste with Dependencies option to copy and paste an action that calls another action from a dependent project. Error Category and Types Issues with the DAP error category have been resolved. Furthermore, error categories have been realigned to include global errors and user-defined errors (see User Errors and Global Errors Categories). An error with creating and saving multiple error types at a time has been resolved. The name of the DAP error category name was incorrectly displayed as DeviceAnywhere in commands. This is now resolved with the realignment of error categories in MAM projects. Looping Through Data Set Values An issue with inserting data set values where only the first value in the data set was utilized has now been resolved. Discarding Changes in Loop Command When you make edits in the Loop command, you can now undo them using the Discard button. Continue with script Option in Wait Event If you choose the Continue with script option in Wait Event to continue even when the command fails or times out, script results incorrectly showed the command as successful. The script was also incorrectly shown as having succeeded. Results now show the command and script as unsuccessful. Image Verification Settings in Commands and States A discrepancy in the default value of the Pixel tolerance slider in image verification settings has been resolved. The slider is set to 95% in both commands as well as states. To view the slider, select Show Options Editor in a command or Advanced Options in a state. 10 Keynote Confidential

11 Mobile App Monitoring 8.0 Updating Object Layout In place of the Auto-Update Layout button in object commands, users now have a camera icon to update object layout when switching from one device screen to another. Figure 4-1 Updating Layout in Object Commands Click to update layout Object layout in command Calling an Object-Based State from Navigate To You can now call an object-based reference point in a state from the Navigate To command. Java States You can now create a Java state (right-click states folder > New Java State) and build the project successfully thereafter (right-click project > Build Project). Script Results The Upload Result button in the script result window is no longer available after you run a script in Studio, as these results cannot be uploaded to MyKeynote. Acquiring Devices on Macintosh On Studio for Macintosh, if you acquired more than three devices, the device name and number did not appear on the device tab. This has now been fixed and you can acquire any number of devices and view correct information on the device tabs. Timestamps for Data Points in Scatter Plots A discrepancy between the timestamp of a script run when you hover over a data point and when you drill down into its details has now been resolved. Keynote Confidential 11

12 Mobile App Monitoring 8.0 Dashboard Does Not Display Data for Time Periods Less than 12 Hours The Mobile App Monitoring Dashboard in MyKeynote (Dashboards > Mobile App Monitoring) did not display availability, error, or performance data for measurements for time periods less than 12 hours, erroneously suggesting that the measurements were not active. This issue has been fixed. 5 Known Issues Error Message When Attempting Interactive Trace Without Suspending Monitoring You must suspend monitoring on a device before attempting an interactive trace on any of its applications. If you attempt a trace without suspending monitoring, you will be able to start the trace, but on clicking the Stop button, you see the error Stop Trace Failed. However the trace is stopped and no data is captured in the PCAP file; you will not be able to save the PCAP file. Releasing a Device Without Resuming Monitoring Activity If you attempt to shut Studio down with a device still acquired, you are given the option to release the device or to retain it for the next Studio session. If you suspend monitoring on a device and then retain it for a subsequent session, monitoring is automatically resumed when you log back in. You will need to suspend monitoring on the device again. Application Upload Uploading applications onto Studio devices using the device console or the Load Application command is not supported. Execution Log Limits Studio crashes if you view the Execution log (in the device console s Sharing tab) for more extended periods of time. To view live script execution activity on the device for a period of more than 2 hours, use the Mobile App Monitoring Dashboard in MyKeynote instead. Date and Time in Execution Log The date and time are not displayed in the Studio execution log if the preceding scheduled run ended in failure. Error Types and Categories If you do not choose any error types in your script, the first error in the Global Errors category is triggered when the script fails. If you insert an error in a command and then delete the error type from the project, the first userdefined error is displayed in the command and in Studio script results when the script fails. Be sure to use a different error type in your script to avoid this. If you insert an error in a command and then delete the error type from the project, Studio does not warn you that you are using a deleted error; MyKeynote displays Unknown Error when the script fails. Be sure to use a different error type in your script to avoid this. Studio Results for Timers When you run a script with a timed sequence in Studio, the results incorrectly show that the offset of the Stop Timer command from the start of the script is 0 seconds. 12 Keynote Confidential

13 Mobile App Monitoring 8.0 Find and Touch Command Failure When the Find and Touch command fails during a script run in Studio, script results do not display the Description field. This field should display the error type triggered as a result of command failure. Discarding Edits in Set Variable Command When you click Discard in the Set Variable command, the variable selected is cleared but its value persists. However, the value is also correctly cleared when you check in the script. Advanced Device Filter Searching the device list using the advanced filter (filtering by device features supported in Studio such as carrier network, sending SMS interrupts to device, etc.) is not supported Device IP Address in MyKeynote When you drill down to view details of a data point in a measurement containing a network trace, there is a discrepancy in the device IP address displayed. The Application Activity tab sometimes displays a 127.x.x.x address, while the Network And Device Details tab displays the correct IP address. CPU Utilization Display The CPU Utilization statistic on the Network And Device Details tab of MyKeynote sometimes erroneously shows a negative percentage for Android devices. Canceling Project Export or Import Project export/import is completed even if you click the Cancel button during the process. Studio Performance Studio has a delayed response to certain tasks such as creating a project, opening project properties, and implementing an action on a specific device. Slow performance is also observed when using commands, for example: When switching from the Timeout tab to other tabs in the Find and Touch command When dragging some commands, e.g., Send Keys, Find and Touch, onto the script canvas Multi-Device Test Cases Multi-device test cases are not supported in this release of Mobile App Monitoring. However, there is support for MO/MT tests that allow you to measure the time taken for a single monitoring transaction between two devices, e.g., sending and receiving an SMS. NOTE The content of the transaction, i.e., the content of the SMS, is controlled by Keynote and includes a queue ID used to measure transaction time. Flickering Device Image in Commands Device screens captured in image-based commands can flicker if the device resolution is 1080 x 1920 or higher. To minimize this issue, ensure that your system meets the minimum requirements for Studio and has a large monitor. You can also maximize screen real estate so that scroll bars do not appear in commands by collapsing the program sidebar and other unnecessary panes when scripting. Swipe in Object Commands Swiping to identify and select an object in Object commands does not work in Mobile App Monitoring scripts. Use Send Keys instead to record and implement a swipe. Keynote Confidential 13

14 Mobile App Monitoring 8.0 KSC and MyKeynote from Studio Clicking MyKeynote or Keynote Service Center in the Links view of Studio does not automatically log you into these portals; you must enter your credentials to log in. Toggle Recording Command The Toggle Recording command, which enables you to capture device video during a script run, is not currently available in Mobile App Monitoring. Powering Off a Device Even after you power off a device in Studio (e.g., by holding down the Power button or by disconnecting the battery using the Hardware tab of the device console), the device screen does not go black and the last visible screen is still displayed. However, if you click on the device screen, it does not respond. Signal Bars in Data Point Details When you click a data point in a scatter plot to view details, the Signal Bars item reads 0 of <Max Signal>. To display device signal strength information in this field, you need to create a project with an action that captures this information from the device. 1 Create a project called Signal Strength. 2 Declare a global numeric variable called signalstrength with a default value of 0. 3 Create an action that verifies signal strength and then sets the value of the signalstrength variable accordingly. 4 Set up the Signal Strength project as a dependent of your measurement project. 5 Then in your measurement test case, call the action measuring signal strength. NOTE Be sure to publish both your main project and the dependent project prior to provisioning measurements. In the example action below, the Wait Event command uses image verification to check the number of signal bars on a device. There is a branch for each possible outcome, and the value of the signalstrength variable is set accordingly. If the Wait Event command times out because it is unable to verify any of the signal strength images, e.g., if there is no signal, the value of signalstrength remains at 0. The value of signalstrength captured during a measurement displayed in the Signal Bars field in MyKeynote. 14 Keynote Confidential

15 Mobile App Monitoring 8.0 Figure 5-1 Action to Capture Signal Strength Branches for each possible outcome 1 bar to 5 bars Image-based verification of number of bars of signal strength Figure 5-2 Variable Capturing Signal Strength Branch for 5 signal bars Variable value set to 5 Keynote Confidential 15