Getting started with TestRail

TestRail is a web-based test case management tool. It is used by testers, developers and team leads to manage, track and organize software testing efforts. TestRail allows team members to enter test cases, organize test suites, execute test runs and track their results, all from a modern and easy to use web interface. Teams benefit from four main advantages when adopting TestRail:

  • Manage test cases, suites and test runs
    The traditional way of managing and documenting test cases on paper, in wikis, bug tracking tools or general document management systems quickly reaches its limits. TestRail enables you to create, manage and organize test cases and suites within an optimized user interface and application structure.
  • Boosting testing productivity
    TestRail's todo lists, filters and email notifications help coordinate test runs and increase the productivity and responsibility of testers. The goal is to make sure that every team member knows his tasks at all times and that team leads can assign new tasks to testers depending on their workload.
  • Real-time insights into your testing progress
    To make important project decisions, it is critical to have access to detailed information about the testing progress and test results. Questions such as “How many percent of a test run have we successfully completed?”, “How are we progressing on the upcoming milestone?” or “Which tests have been executed in the past 24 hours?” can only be answered efficiently by using a comprehensive test case management software such as TestRail, and TestRail makes these details readily available.
  • Organize and track software testing efforts
    The organization and coordination of multiple parallel tests, be it for different projects or releases, is often complex and time-consuming. To not lose track of your testing efforts, TestRail helps you manage important software-specific data and structures such as project milestones and makes it easy to integrate with bug tracking tools.

The dashboard

After logging in to TestRail, the first page you usually see is the dashboard. The dashboard provides you with an overview of available projects, recent project activities and your todos. When you navigate to a project (by clicking on a project title), you switch to the project view and land on the project's overview page, showing project details such as test suites, active test runs, project activity and so on. Whenever you need to switch to another project, just return to the dashboard by clicking the link in the upper left corner.

Test cases and suites

A test case in TestRail consists of a description of the test's prerequisites, a list of test steps and the expected result. A test case can ideally be verified by a single tester in a short period of time and confirms a specific functionality, documents a task or verifies a project artifact.

In TestRail, test cases are organized into test suites. A test suite is a collection of test cases and is often created for specific project modules or areas. How you organize your test suites depends on the size of your project. If you have a lot of test cases for a project module, it is recommended to further break down your test suites and create suites for specific features or functionality. Ideally, a test suite consists of 50-1000 test cases.

To further organize test cases, test cases of a suite can be grouped into so-called sections. Sections are used to organize test cases into groups to make it easier for testers to identify related cases.

For example, if you use multiple test cases to verify a specific application dialog or web page, these test cases should be grouped in a section. Depending on how complex your project is and how fine-grained your test cases are, a section should ideally consist of 10-50 test cases.

Test runs and tests

To run a test and enter test results for the cases you added, you start a test run for a particular test suite. While a test suite is just like a plan that specifies how an application is tested, a test run is an actual test you are conducting.

For most projects you will most likely start multiple test runs for a particular test suite over time. For example, if you are releasing multiple versions of a software program, you might want to conduct a test run for each new release. Likewise, you can have multiple test runs for a particular test suite active at the same time. This can make sense if you want to execute a particular test suite for multiple configurations (such as different operating systems). You can then start a test run for each different configuration you want to test against.

When you start a new test run (e.g. by clicking the Run a Test button in the toolbar of a test suite), you can choose to either include all cases of a test suite or select the cases manually. If you include all cases in a test run, new cases that you add to the test suite are also automatically added to the run.

A run consists of individual tests for each case that you add. By default, each test has one of five different statuses that are signaled in TestRail by specific colors. You can add a test result and change the status of a test either by clicking the Add Result button on the test page, or by changing it directly on the run page.

The following test statuses are available by default:

  • Untested
    By default, new tests have the status Untested. Once a test result has been added to a test, it can never receive the Untested status again.
  • Passed
    A test is marked as Passed when a tester verified the test steps and the expected results.
  • Failed
    A tester marks a test as Failed if one of the specified test steps resulted in an error or if the expected result differs from the actual test result.
  • Retest
    If a developer or tester determines that a test should be tested again, she marks the test as Retest. E.g., if a test originally failed and the developer fixed the problem, the developer can mark it to be retested.
  • Blocked
    The Blocked status is used to signal that a test cannot be executed currently because of some external dependency (such as a bug that needs to be fixed before being able to complete to test). It is often used in combination with the Retest status.

Once a test run has been completed, you can close it from the run's edit page. Tests of a closed run cannot be edited or changed, making it ideal for archiving test runs. Additionally, if you change any of the associated cases' attributes (such as the expected result), the change won't be applied to the closed and archived run.

Projects and milestones

Projects are the main organizational unit in TestRail. It is generally recommended to create a TestRail project for every real software project you want to manage within TestRail. All other data such as test suites, runs, test results, milestones etc. are directly associated with a specific project. See the following illustration to see how the different entities relate to each other:

You can also add your project milestones to TestRail. A milestone can be any project milestone you deem to be important, such as a planned public software release, an internal test version or a new beta release for an important customer. Once you have added your milestones to TestRail, you can assign test runs to specific milestones.

Assigning test runs to milestones has the advantage that you can easily track the milestones' test results and progress separately. Especially if you are working on multiple milestones in parallel or if you have many test runs active at the same time, managing milestones within TestRail is a great help.

Test plans and configurations

When you need to manage multiple test runs and configurations for a single project, TestRail's test plans come in handy. A test plan allows you to start multiple test runs at once, either if you have many test suites or if you want to test against multiple configurations (where a configuration can be anything you need to test your project against, such as different operating systems or web browsers).

To create a test plan, simply click the Add Test Plan button from the Test Runs & Results page. Now just select one or more test suites from the sidebar to add them to the plan. Just like you do with a single test run, you can configure all properties of the test runs such as the name, the cases you want to include and so on.

You often have to test a specific test run against multiple configurations such as operating systems or web browser versions. To make it easier to create a separate test run for each configuration combination, you can specify and select all configurations for your test runs within a plan. To do this, simply click the Select Configurations link of a run and select or add your configurations.

Todos and notifications

Tests can be assigned to team members. You can either assign tests when a run is created or do so afterwards from the run or test pages. Once a test has been assigned to a user, the test appears on the user's todo list. Every user has its own todo list for each project she's working on, while the Dashboard provides a quick way to see the user's todos for all projects. The todo page can also be used to view the todos of other team members or the entire team at once, providing an easy way for team leads to assign tests to team members with the least workload.

Another relevant feature are email notifications. Email notifications help test owners to learn about test changes made by other users. For example, if a test has been assigned to a user (i.e. the test owner) and another user adds a comment or result to the test, the test owner is notified by email. It is also possible to subscribe to tests of other users or even entire runs by clicking the small email icons on the relevant pages.

Tracking progress and activity

Besides making it easier to manage test suites and enter test results, providing an easy way for all team members to track the test progress is one of TestRail's most useful features. You can view the test results and testing activity of runs, milestones and entire projects on the individual resource pages.

You can choose between the status, activity and progress view on the test run and milestone pages from the sidebar, while you can switch between the project history and test results on the project's overview page. Additional reporting features can be found on the Reports tab.

Learn more and try out TestRail

You've reached the end of this Getting Started guide. If you haven't already, the easiest way to try out TestRail is to create a trial on our servers and this just takes two minutes to set up:

Try out TestRail

Please do not hesitate to contact us if you have any questions that aren't answered in this guide. Also feel free to use our support forum in this case.