A guide to Parallel Test Execution with Cypress Dashboard & Jenkins

Every project team in an organization prefers their products to be flawless or contain the least possible minor defects, and therefore there is a heavy emphasis on the importance of testing thoroughly. Automation testing comes in handy when there is a need to provide assistance to any QA team by subsequently reducing test efforts. Many teams like to have their end-to-end regression suites automated and running beforehand for this purpose. Automation does indeed help in the aforementioned scenario, but if there are hundreds of test cases in the automation test suite then the test run can take up quite a long time. Typically, organizations’ management like to have test results provided as soon as possible, especially when the team is in need of immediate assistance in testing.

This is where parallel testing jumps in. As the name suggests, parallel test execution refers to having your tests running in parallel rather than sequentially. This means that rather than executing one by one, your tests will run side by side, and hence reduce the overall test time. There are many benefits of parallel testing. It reduces time, saves team effort, and aids in the development of successfully working software.

This guide will cover how to implement parallel execution with Cypress. Cypress provides the functionality of Parallelization since it can take a long time to run all the tests serially. By running them parallelly, the team can save time and money. This can be done by having your test project running with Continuous Integration since true test parallelization can be carried out through CI/CD by setting up multiple machines to run in a CI environment. According to their official documentation, Cypress itself does not recommend the usage of the parallel parameter for a single machine as it can create load on your system if the CPU is not powerful or high-end enough to handle multiple concurrent tests at the same time, and it is for this reason that the concept of running your tests in multiple containers is introduced and supported. However, if you have a single powerful machine hosting the CICD environment and have the capacity and resources to support memory-consuming tasks, then parallelization is possible on a single machine as well.

Using Cypress, Parallel Execution in its true sense, is only possible with Continuous Integration. It also requires your project to be set up with Cypress Dashboard, in order to record tests. This is necessary because before running in parallel, your tests need to run at least once with Cypress Dashboard. The dashboard service records the average time a test takes to run, and that time duration eventually supports splitting the tests to multiple machines or instances. Each spec file in your project is assigned to the available machine or instance according to Cypress’ balance strategy. Additionally, the run order can hence not be guaranteed each time when parallelized.

This guide will cover how to set up your Cypress project to the Dashboard and start running tests in parallel on a single machine.

Connect to Cypress Dashboard

The first step is to connect to Dashboard. To do so, launch Cypress Runner and select the Runs tab, then click Connect to Dashboard and set up the project.

Once the project is set up, it will generate a project ID and record key. These two values are used to uniquely identify the Cypress project. The project ID needs to be stored in the cypress.json file and the record key can be used while running the tests, to start recording with Cypress Dashboard.

Record tests run with Cypress Dashboard

The record key basically means that your project is allowed to record tests to the Dashboard Service, which is a requirement for parallel execution. It is the key that enables you to write and create runs. The tests can be run using the following command:

cypress run --record --key <record key>

Once this is executed locally or through CI/CD, the test run will be recorded and the results will be displayed on Cypress Dashboard.

Now that the project is successfully set up and a test run has been completed, you can move to the parallel execution of tests. You can either set up multiple machines or utilize the single machine your CI provider is set up on.

In order to allow the tests to run in parallel, the parallel flag needs to be used along with the record flag and key.

cypress run --record --key <record key> --parallel

You can store this command in the package.json file

The Process for Parallel Execution

• The CI machine contacts the Dashboard service to specify which spec file to run.
• The machine receives the spec file to run by Cypress.
• Cypress calculates the duration to test each file (these were recorded in the initial run)
• Based on the estimations, Cypress distributes the spec file one by one to each available instance in a way that minimizes the overall time of the test run.
• The tests are run on each instance as per availability until all the tests have been executed.
• Upon completion, there may be a run completion delay or waiting period, in case any more relevant post-execution work remains.

To put it simply, in order to have multiple instances, you need to specify the number of stages and run the above-mentioned command in each stage. Cypress will launch the number of instances respectively and the ‘ — parallel’ key will ensure that the tests are split into the instances. This can only be done through CI and not locally.

Parallel execution of Cypress Tests through Jenkins

One of the simplest and most straightforward ways to perform parallel execution is to use Jenkins File. A Jenkins File is a text file containing the definition of a Jenkins Pipeline. It is usually stored in the root directory of the repository. Usually, pipelines consist of stages and steps. This file follows Groovy syntax.

Parallel execution can easily be performed using a Declarative Pipeline, which contains additional parameters, such as parallel stage and agent. For Cypress, you also need to have the NodeJS plugin installed in Jenkins, and the name specified for its installation needs to be provided under tools in the Jenkins File.

Defining a Declarative Pipeline for Parallel Execution

• Create a file in the root directory, and name it “Jenkinsfile”
• Start with a simple pipeline section.
• Under the pipeline, specify the agent. You can set it to ‘any’ since we are using a single machine, which is the only one available. Without the agent directive, the pipeline would be considered invalid.
• Specify the tools. In this case, you need NodeJS so mention the name set for this plugin.
• Specify the stages directive. This tells what the “work tasks” of the pipeline consist of.
• Under the stages, mention the stage with a name. The stage refers to one of those tasks. A stage can have several sub-stages as well.
• Under stage, state the steps. These are the steps to perform the tasks. The steps should contain the batch or shell commands needed for execution.
• For parallel execution, you must create a parallel section under stage. This in turn can have multiple stages and steps under it, as shown in the example below. Define stages under the parallel section as many times as you require the number of instances, and mention the command for parallel execution under them. Make sure that the ‘ — parallel’ flag is passed, otherwise the entire test suite will be executed on each instance.
• You can use ‘echo’ to print to the console, and ‘sh’ for shell commands, which we will be using to run tests.
• At the end, you can have a stage to generate the reports. One thing to note here, is that if one stage (excluding the ones in the parallel stage) fails, the next sequential stage may not be executed due to the failure of the previous. For example, if even a single test fails, the last stage to generate the final report will not be run due to the previous failure. To avoid this, you can state a post section to handle such scenarios and define the command as per the condition. For commands that must always run, you can state them under the always condition.

This will result in a basic structure for a declarative pipeline. There are many more options that you can add as per your need, such as environment and when conditions.

Once that’s done, save your file and push your changes. In the end, you will have a pipeline that looks like this:

Set up Jenkins job

Launch Jenkins and Add New Item and select Pipeline.

Firstly, click on the Pipeline tab. Then, under Definition, select Pipeline script from SCM, and add your Git repository URL and credentials.

Also specify the branch name and path to Jenkinsfile.

Click on Apply and then Save.

To run the tests, click on Build now. After running your tests you will be able to see the average time taken during each stage as well.

As per the commands provided in this example, Cypress will divide the test suite into 3 stages and the tests will be divided one by one into these three instances as per the availability.

This way you can perform parallel execution of Cypress Tests using Cypress Dashboard and Jenkins on your machine.

While there are many advantages to parallel execution with Cypress, there are some limitations to this feature as well.

Firstly, Cypress Dashboard is a paid service. The free one also provides parallel execution, but the number of test runs per month is limited to 500 only. This means that if you have a total of 100 tests in your project, you will be able to run the entire suite only 5 times as Cypress Dashboard equates one test case running to one test run. Therefore, if you need your tests to run on a daily or hourly basis, you will need to opt for a paid version that would provide you with the number of test runs per need.

Secondly, you might run into resource and capacity limitations depending on the performance of your system. If you have a powerful machine that can handle heavy workloads, launching multiple instances would not be a problem. However, if your machine has limited resources then you might get some unexpected and strange failures in the test run, which are otherwise passing, or your system might lag sufficiently and the test run may lag or get blocked. So, in order to overcome these challenges, it is better to have a separate designated system for your Jenkins or have it hosted on the cloud and these concerns will hence be overcome.

Cypress Dashboard has made it much easier to parallelize tests, as not only does it provide online access to your recorded test results, but also provides rich analytics and diagnostics for your test results.