A customer who develops software approached us asking what tool we would recommend for their testing team.

Since the customer already used Jira for business requirements and development, we focused on add-ons for Atlassian.

The customer needed a tool for a minor team (30 testers) that could do manual tests, but the emphasis was on automated tests and easy integration with CI systems. From the portfolio of available tools, we chose Xray because of its user-friendly GUI, its many integrations with testing frameworks, and its wide range of settings.

Xray – brief overview

Xray is a Jira plugin designed for test team management, test plans, and test results logging. Xray is built on the fact that whatever is possible and makes sense to record as a JIRA task has a task type created. This makes it possible to have custom process settings for the functions, which can be an advantage, especially in test design and test planning. In its basic setup, Xray works with the following set of functions:

• Test Set – a package of tests that are related
• Test – separate test
• Pre-Condition – a condition that must be met before the test can be executed
• Test Plan – a plan that defines a set of tests to be executed for a specific version
• Test Execution – specific execution of tests and recording of results

In addition to these types of tasks, Xray can be used to work with the requirements whose implementation we want to test. Coverage maps can be created by linking tasks of type Requirement to Tests and Tests to tasks of type Bug, allowing us to see which Bug/Test impacts which requirement.

The Xray itself contains different perspectives on test execution. The creator of Xray, Xpand IT, also supplies the Xporter plugin, which provides the ability to report tests with the ability to export reports to .xlsx, .docx, .pdf, .docm, .dotm, .rtf, .odt, .svg, .png formats. If you use the EazyBI plugin (blog HERE) with Jira, Xray adds predefined dimensions and measured values to this plugin.

Getting started with Xray

Before creating tests in Xray, we need to set up Xray. Since Xray can also work with linking to requirements and tasks with errors, Xpand IT (the creator of Xray) recommends choosing from the following project settings:

1. All-in-one project—In this mode, system requirements, including implementation tasks, tests, and bugs, are in one project.
2. Separate implementation project from test project—This mode puts implementation-related things, such as requirements, implementation tasks, and bugs, into the “Implementation project” and test-related tasks, such as tests, conditions, test suites, test plans, and test execution, into the “Test project.”
3. Specialized project for tests only – in this mode, there are three projects: implementation project (requirements, implementation tasks), project with tests (Tests, Test sets and conditions) and project for testing (Test plans, test execution, bugs)
4. Complete decoupling – compared to a “Specialized test-only project”, this mode also decouples the bug logging to an external project.
5. Requirements in an isolated system—This setting allows you to connect tests to requirements registered in other systems, such as Enterprise Architect.

We can proceed to the test design phase once Xray and the test project have been selected and configured.

Test Design

Xray allows you to record two types of tests: manual and automatic. These tests can be merged into test sets, or since version 3.0 is a test repository, they can be sorted into folders. Tests or test suites can be linked to requirements, where the test coverage status can be displayed directly on the requirement page. Some may also appreciate the ability to add input conditions for tests and reuse them for other tests.

Manual tests consist of steps describing the step, the input data, and the expected result. Steps can be rearranged, deleted, and attachments added. Another advantage is the use of wiki markup formatting for the description of the test and the description of the steps, data, and desired outcomes.

Xray also allows you to record automated tests. The developer is working on Cucumber tests, a category with direct support for scripting in Jira with the backing for highlighting Gherkin language keywords. These tests can later be exported to Cucumber feature files that can be run on the test environment.

Other automated tests can be created in Jira, provided we know the test identifier in a particular language or in a more straightforward way – by initially importing the test results into Jira. Xray will then link each subsequent import of the results of a given test to an existing test.

Test planning

Planning is done by creating test plans linked to the tests or test suites to be tested in that plan. Since version 3.0, the planning view can be used in a test repository. In this view, tests can be added from the repository and the test plan modified.

Conducting tests

To start testing in Xray, we must create a Test Execution task. If we have planned the testing via test plans, we can create such a task directly from the test plan. The references to the tests included in the plan will be automatically copied to the Test Execution task. If we have not planned or need to do testing outside the plan, we can create this task directly and add the required tests.

For manual tests, the actual execution of the tests takes place on a particular screen, which is displayed when you click on the run test button from the test execution task. This screen displays information about the currently executing test, including a description, any input conditions, and the test steps. The tester can report errors directly from this screen or continue with the following test directly from this screen after the test has been successfully executed.

The results of automated tests can be imported directly into the Test Execution task. Tests that failed to pair with existing tests are automatically created and assigned to that Test Execution.

Report of results

Xray includes several gadgets in Jira to track testing progress. Still, it also includes testing-related reports such as the Test Coverage Map, Test Coverage over Time, Test Plan Report, Test Execution Report, and Test Run Report.

What we can help you with

In the beginning, the Xray needs to be set up. To do this, the following steps are necessary:

1. Gathering and analysing testing requirements—what projects will be tested, how often, what the release cycles are, what the structure of the test team is, etc.
2. identification of approaches to testing – each type of project has its specifics, and therefore, there are usually multiple types of approaches
3. unification of testing approaches – for the sake of simplifying Xray administration, it is pretty essential to reduce the types of approaches to a minimum number in this step

Based on these steps, we can set up Jira and Xray to meet the client’s requirements as closely as possible.

We help clients integrate with CI systems where reporting automated test results directly to Jira after the build is completed on CI can be achieved.

Maroš Strmenský
Atlassian Architect / Xray Specialist

If you need assistance from experts with implementing or configuring Atlassian products or need advice on how to use them most effectively in your company, please do not hesitate to contact us.

Our Atlassian Solutions