Xamarin Test Cloud – All you have to know: part1

Xamarin Test Cloud – All you have to know: part1


Short introduction

Mobile development is becoming more and more popular nowadays. It is not only about creating apps by small companies or startups. It is also connected with enterprise. Many different devices are in use especially if you refer to Android.

How you can check that your application will look and behave appropriate on every device?

With Xamarin Test Cloud you can test mobile applications and see reports at the end.

In this article I would like to introduce you to Xamarin Test Cloud and show what are the possible options.

What do I need to start?

1) Xamarin Test Cloud account (you can use free tiral version)

2) Xamarin Test Recorder (either for Windows or MAC)


Let’s start

Firstly I would like to write about basics about which you have to know before you start testing your mobile apps.

Xamarin Test Cloud is a cloud-based service that provides an automatable way to do UI Acceptance Testing of mobile apps. It provides hundreds of devices. As always there are tests behind written to check different application functionalities.

It is not mystery that manual tests are expensive and sometimes there are too many functionalities to do it manually. Xamarin Test Cloud provides automated UI tests. Below I will describe two possible frameworks available to write tests:


Xamarin UITest

Framework based on the popular NUnit testing library which enables developers mto write tests in C#. It is especially recommended for developers who wrote their automated tests before. To read more about NUnit library please refer to this link.



Framework which enables developers to write their tests in Ruby using Cucumber tool. This kind of tests are very close to Behavior Driven Development methodology. Big advantage of using this framework is that tests can be written in the language of the business by anyone following grammatical rules imposed by Cucumber tool.


Decision about which framework should you use is up to you – both of them will test proper behavior of the application.

Remember that boths are based on the Calabash underneath.

In the below picture you can see simple architecture and flow of Xamarin Test Cloud testing with listed actions:


1) Developer creates app package with tests

2) App package is uploaded to the Xamarin Test Cloud

3) Xamarin Test Cloud installs application on test devices

4) Once every test is completed Xamarin Test Cloud prepares report about the result of tests


Continious Integration

As you saw above developers can submit app packages with tests manually – but with reference to bigger apps Continious Integration can be used and Xamarin Test Cloud supports it:


1) Developer commits changes to the repository

2) Build server gets latest commited version and created app package

3) Application package with tests is uploaded to the Xamarin Test Cloud

4) Xamarin Test Cloud is testing app on different devices

5) Report with result is sent to developer


Test Cloud Frameworks – how does it work?

Frameworks mentioned above (UITest or Calabash) are to test the mobile app functionality but they do not interact with the app directly. There is one more component required to run these tests – Xamarin Test Cloud Agent.

Xamarin Test Cloud Agent is responsible for running tests on the devices – it is separate application installed by Xamarin Test Cloud alongside your application.


Calabash and UITest frameworks use a client-server architecture:

1) Test Cloud Server Application – lightweight server app runs on the device which listen for clients to initate selected test. It communicates over HTTP via JSON.

2) Test Cloud Agent – responsible for providing information for the server about which test should be now executed.


Xamarin Test Cloud Agent on Android


Test Cloud Server Application runs as a separate application which has permision to cooperate with the actual application for tests. Such scenario is possible because Test Cloud signs both apps with the same key.

You do not have to add any additional references in your code to make it running.


Xamarin Test Cloud Agent on iOS


On the iOS Test Cloud looks different – Test Cloud Server Application is built into the tested application.

To achieve that you have to add NuGet package: Xamarin Test Cloud Agent.

Agent should be included only in Debug mode. When you prepare your app for release it should be removed by Xamarin.iOS Linker.


Run tests in Xamarin Test Cloud

Now you know what is the structure of the testing framework and how does it work. When you upload your application to Test Cloud it host the tests and running them on different devices (user dooes not have to run tests manually form the command prompt):



Calabash or UITest – which framework should I choose?

There is no one good answer for that – you will achieve what you have planned with both of them.

Below diagram provided by Xamarin should help you choose the right way:


If you have your application already written in C# (Xamarin) you propably should choose UITest framework because you are familiar with the language. What’s more you can run all tests from the IDE (Visual Studio or Xamarin Studio). UITest is based on the popular NUnit framework so there is also a big support from the community.

For the native applications (Java, Swift, Obj-C or hybrid apps) Calabash framework will be good choice. You can create cross-platform tests in any text editor (Ruby IDE too) without any special tools.


What kind of tests do we have in mobile development?


Lets start from the bottom.

1) Unit Tests – this is the type of tests which is the most costly. The purpose of it is to test a small unit of the application (often this is a method) isolating it and verifying if its working properly.

2) Components Tests – this is the type of tests which verifying if components (like classes or objects) are working properly. For example testing the whole functionality of “Car” class is called component test.

3) Integration Tests – the purpose of this type of test is to check proper integration between classes – for instance if class “Car” behaves properly with “Tire” class.

4) Acceptance Tests – these tests are carried out to validate the functionality of the application. They just check if the app functionality meets the expectations related to the functionality agreed at the beginning of development.

5) UI Tests – this is very often done by QA (Quality Assurance) team manually. They check if user interface is implemented correctly (by clicking buttons, testing different flows). This is something what can be done with Xamarin Test Cloud.

Last type of test is called “Regression test” and it is not included in the picture but I will explain. This type of test is being used to check if our current code works with the previous versions (for instance if newly written class works properly with others).


Sum up

In this article I tried to collect all the information which is important about Xamarin Test Cloud platform. Now you know what are possible tests frameworks and how test are performed on the Android and iOS.

Now you know what kind of tests are available with reference to mobile development.

In the next article I will try to show how to write tests and run them with Xamarin Test Cloud to create automated UI Tests.