Automation: Triggering automated tests

Most teams who integrate their automated tests with TestRail trigger and execute their automated tests outside of TestRail (e.g. as part of a continuous integration system) and use TestRail's API to submit the test results. However, some teams prefer to implement a system to directly trigger test results from TestRail and this document explains how to do exactly this using a combination of a UI script and custom server-side scripts.

A UI Script is used to add a new button to TestRail's user interface and a server-side script launches the automated tests. The example executes tests on the test run level but this can be changed to support test plans or just individual tests.

Please note that this document assumes basic knowledge of PHP as well as web development (HTML, CSS and JS). Understanding and extending this example requires experience with all of the above and this document cannot substitute for this.

The source code for this example can be found on GitHub. Please choose the version that matches your TestRail version:

Trigger test run example (TestRail 4.x)
The GitHub repository with the UI and server-side script for TestRail 4.x

Trigger test run example (TestRail 3.x)
The GitHub repository with the UI and server-side script for TestRail 3.x

Workflow

The following diagram illustrates the basic workflow of the integration:

The UI script adds a new button to the toolbar for test runs. When clicked it sends an HTTP request to the server-side script which in turn triggers the automated tests. The script or tests then report back the test results via TestRail's API.

UI script

You can add the UI script (trigger.ui) in TestRail's administration area under Administration > Customizations > Add UI Script. Simply copy and paste the whole UI script from the GitHub repository to the Configuration text box. After adding the script you should see a new Start Tests button in the toolbar for test runs:

Trigger script

The trigger script (trigger.php) needs to be placed into TestRail's installation directory on the web server next to TestRail's index.php file. If you plan to use a different location you also need to change the URL in the UI script accordingly.

The next step is to configure the following settings for TestRail's API:

define('TESTRAIL_API_ADDRESS', 'http://<server>/testrail');
define('TESTRAIL_API_USER', 'user@example.com');
define('TESTRAIL_API_PASSWORD', '***');

There's a lot of boiler plate code in the script, e.g. to handle the script arguments and submit test results back to TestRail's API. The interesting bit is the following function:

function execute_test($run_id, $case_id, $test_id)
{
    ..
}

This function is expected to trigger automated tests for a given test run and case (test) and should return a valid TestRail status ID. The example of this function simply returns a random status ID but real implementations could trigger tests by calling external command line tools, an external API or something else.

Security considerations

The example server-side script doesn't include any provisions to authenticate script calls. So before using this script on a production server - especially on a server that is accessible over the Internet - make sure that you actually authenticate any script accesses. The easiest way to do this would be to enable HTTP authentication for the script in your web server and change the JavaScript code to include the login credentials in all HTTP calls.