December 23, 2019

How to run an automation test script from Azure as part of CI-CD

  December 23, 2019
There is no DevOps without the word ‘Automation’.

When considering a Azure pipeline, automation is present in almost all its phases like continuous integration, continuous testing, continuous deployment and in application performance monitoring.

From Azure we can control our automation scripts to run from ADO test plans and get results back in our test cases.

Attaching automated scripts along with CI build pipelines will provide the option to implement shift left approach in testing. Automated smoke test suite and regression suite triggers ensures the quality delivery of the application to the users.

Tools like SonarQube, Checkmarx etc helps us to attain code quality and standards in an automated way.

Here let's have look on how we can execute an automated UI test from Azure.

Get test Artifacts

First thing we need to have is the test artifact which is having all the test scripts.

There are two ways we can execute automated tests in Azure.

  • Using a build pipeline

We can add both build and tests to a single pipeline. If we are using this, we will be creating the artifact and using that in the same pipeline as below.

Here we have added the task Download build artifacts as part of the second agent job.
Artifact can be downloaded by mentioning the name of the artifact used in Publish artifact task.

So here all the build and test execution will happens in the same pipeline, if we have any unit test cases or component level test cases we can add those here and run along with the build itself.

Enabling continues integration in the build pipeline will trigger the pipeline to execute whenever a commit happens in the mapped repository.

This will ensure minimum quality of application/code while building and this can be considered as shift left approach in testing.
  • Using a Release
We can add tests in Azure release also.Usually we will be getting a full product once after a release, If you are following any CI-CD process. So we can add tests covering end to end flow of the application as part of the release. This is the place where we usually add Smoke and regression suites to run.

For adding tests in release we need to use the artifact created in the build pipeline. So we will be mentioning the pipeline and artifacts details in Artifact section of release.

Just like in the build, we can schedule or Auto trigger the release whenever a new release is available.

We can setup timer also for release trigger.

Adding stages to the release

We can add each environments as each stages in a release and separate tasks can be done for each stages. We can add our automated test for each of these stages as VsTest tasks.

These tasks will be slimier as the second set of jobs that we mentioned in Build pipeline based test execution. Since we have a separate artifact section,here we don't need to mention the download artifact task.

These agent jobs will be referring the mapped artifact by default.
If we have more than one artifact used in the release , we can select which needs to be used for the agent job.

VsTest and Visual Studio Test Platform Installer

  • Visual Studio Test Platform Installer

This task will help us to install all the necessary files needed execute a test in the agent machine. We need to use this task first in agent machine to make it VsTest test execution ready.(Azure agent should be ready in the agent machine)

If we are using Visual Studio Test Platform Installer in an agent we need to mention that in the VsTest task also in the execution options section.

Just select Installed by Tools Installer under Test platform version field.

  • Visual studio Test

VsTest task can be used to run any tests created using different testing frameworks. It supports tool like Selenium, Appium,Protractor , Coded UI etc and Unit test frameworks like MsTest, NUnit etc.

Output and configuration of each of these tools will be different, so Azure provides different configuration options inside VsTest task.

Running MsTest tests in Azure

There are three main options to select tests in azure, They are :
  • Test assemblies
  • Test plan
  • Test run

Test Assemblies

We can use the tests using assemblies option to select all the tests inside our dll file. If you are using MsTest, this will find all the methods with TestMethod annotation with it.
We don't need any test cases from azure test plan to be associated with test scripts to use this method.

We have to mention the Test files as the solution dll and VsTest task itself will fetch all the tests in it.
Now using test filter criteria we can selectively run the test scripts and results will be shown in the test run of the release.

Test Plan

Test plan option will help us to execute the test scripts based on associated test automation mapped in test plan.

Refer here for more details :

Once the mapping is completed, we need to select the Test plan, test suite and test configuration inside VsTest task.

When this task is executed, it will search for all the test cases with associated automation in the given test plan combination .Test results will be shown in Execution history of test case as well as in test runs.
If the test scripts associated previously was not available in the repo, this task will throw an error while execution.

Test Run

The third option we have is using the test run, This is almost same as the Test plan but in here we will be able to run a test case directly from test plan other that attaching it to a particular release.

While using this task Run ID will be taken dynamically and that test will be executed using the release created. We can create a release using this option and set all the configurations.

Run test case using Release

Once the release is set, we can use this release in Azure test plans to run a single test or a group of selected tests.
One precondition for this will be , the test case should have a associated automation.

To execute the automation we need to select the test/tests from test suite and click on three dots.

Then select Run>Run with options.

This will provide us with the option to select mode of run.

Here we need to provide the following details :
  • Select test type and runner 
Select it as Automated tests using release stage since we are using the release with test run option in it
  • Select a build
Select the latest build of your project.

  • Release Pipeline
Here we need to select the name of release pipeline we created with Test Run option

Then select the stage and Click on OK

This will start with the execution of selected test cases in the agent mentioned in release.

To know what happens when a test run is started refer here :

If any associated test is not found or if there is problem with the solution, we will get a failure as below.

Identifying tests:Found 0 automated test(s)

If the version of MsTest is wrong or if we select a release which is not having release with Test run option, we may get below error.

Validating stage:The selected stage does not have the right version or settings of the Visual Studio Test task to run tests.

Selecting default release for a test plan

We also have a option in Azure to set a default release for the test plan from test plan settings.


Thanks for reading How to run an automation test script from Azure as part of CI-CD

« Prev Post

1 comment:

Bookmark this website for more workarounds and issue fixes.

Verify Zip file contents without extracting using C# + Selenium

While doing automation testing, we may get a scenario where we need to download a zip file and to validate the contents inside it. One way t...