Submitting UITests at the Command Line Overview This guide discusses how to submit your UITests tests to Xamarin Test Cloud using the command line. This scenario is appropriate for continuous integration servers or builds that performed at the command line using a script. It will also discuss strategies for signing your Android APK s with custom keystores. Requirements It is assumed that UITests have been written and are running in your local environment, and that you have the test-cloud.exe application installed via NuGet. It is assumed that a Test Cloud project has been created, your account has been added to the team for that project, and you have the Team API Key. Please see the Organizations & Teams document for more Details. Familiarity with the command line is required. Internet access to the Test Cloud web servers is required. Firewalls should be configured to allow traffic to the Test Cloud servers at the following IP addresses: 195.249.159.238 195.249.159.239 The Test Cloud servers require port 80 and 443. Android applications must request the INTERNET permission in AndroidManifest.XML. Submitting UITests
Tests are submitted to the using test-cloud.exe command line application, which is distributed as a part of the Xamarin.UITest NuGet package. This is a multi-purpose utility that will submit both Android and ios tests, and it works on both OS X and Windows. The following table describes some of the more important command line parameters for test-cloud.exe, which may be found in the packages directory as shown in the screenshot below. It is not a complete list of all parameters. Parameter --assembly-dir <PATH> --devices <DEVICE ID> --user <EMAIL> --nunit-xml --sign-info --dsym --category <NUNIT- CATEGORY> --fixture <NUNIT- FIXTURE> --test-chunk --fixture-chunk Description The path to the folder holding the test assemblies. The device ID that was provided in the Test Cloud Upload dialog. The e-mail address of the team member submitting the tests. The filename to which test results are exported, formatted as NUnit results XML. Optional. Android only. Supply a signing information file that will be used to sign the Test Server APK. See the section below for more details. Optional. ios only. Will upload the dsym files along with the application and tests. This allows for more detail in the log files. Optional. NUnit category to run. (Can be used multiple times) NUnit fixture / namespace to run. (Can be used multiple times) Run tests in parallel by method Run tests in parallel by fixture It is important that all commands be run from the solution directory. As an example, the following screenshot of the CreditCardValidation folder on Github identifies the solution project for the CreditCardValidation-UITest project:
It is important that all commands be run from the root of the solution directory. As an example, the following screenshot of the CreditCardValidation folder on Github identifies the solution project for the CreditCardValidation-UITest project: Note: Xamarin Test Cloud will terminate test runs that take longer than 180 minutes. Submitting ios Tests The following snippet is the simplest command line that could be used to submit an ios application and tests to Test Cloud: test-cloud.exe submit <PATH-TO-IPA> <TEAM API KEY> --devices=
<DEVICES> --assembly-dir=<path-to-test-assembly-dir> --user=<email> Submitting Android Tests The following snippet is the simplest command line that could be used to submit an Android application and tests to Test Cloud: test-cloud.exe submit <PATH-TO-APK> <TEAM API KEY> --devices= <DEVICES> --assembly-dir=<path-to-test-assembly-dir> --user=<email> Only release builds of an Android application should be submitted to Test Cloud. test-cloud.exe will sign the application APK and the test server APK with the same keystore before uploading to Test Cloud. It is possible to control this signing step as described in the next section. Signing Android APK's test-cloud.exe will use one of three algorithms to sign the APKs prior to submission: 1. Use Build-in Keystore. This is the default behaviour. test-cloud.exe will sign the application and the test server APK using it s own keystore. 2. Provide your own Keystore. The second option is to provide your own keystore (along with credentials) that test-cloud.exe will use to sign the application and test server APK. The following snippet shows the additional command line parameters: test-cloud.exe submit <PATH-TO-APK> <TEAM API KEY> keystore <PATH-TO-KEYSTORE> <KEYSTORE-PASSWORD> <KEYALIAS> <KEYALIAS- PASSWORD> This option allows for complete control over what key will be used to sign the application, but it presents a potential security risk because the application keystore must be included in source code and the password for the keystore will be in clear text in build scripts. 1. Use a signing information file. This option involves using test-cloud.exe to extract a few files from a signed test server APK, which are then stored in a small zip file, known
as a signing information file. The main reason for using a signing information file is so that it can be added to source code control. This ensures that the application and test server are consistently signed with the same key information. The following snippet shows how to create the signing information file: test-cloud.exe gen-sign-info <PATH-TO-SIGNED_APK> <PATH_TO_KEYSTORE> <KEYSTORE-PASSWORD> <KEYALIAS> <KEYALIAS-PASSWORD> --dir <DIRECTORY- TO-SAVE-SIGN-INFO> When run, test-cloud.exe will create a file called testserver.si in the directory specified by the --dir parameter. Note: The signing information file must be recreated each time the Xamarin.UITest package is updated. Summary This guide discussed how to use test-cloud.exe to upload Xamarin.UITests and applications to Xamarin Test Cloud for testing. It also discusses several options for how to sign Android APKs prior to submitting test runs to Test Cloud.