SoTest User Documentation¶

The latest version of this documentation is hosted on docs.sotest.io.

The purpose of these documents is to describe to users how to interact with SoTest in software development project teams.

Overview¶

Testing a software product on all supported hardware platforms is typically not done during development. Instead, QA teams inspect the product for possible regression before every release. This mode of operation leads to late discovery of regressions. Development teams then need to correct these regressions under time pressure during the release phase. Bugs that are discovered late are usually more costly to fix as the process to correction at this point involves multiple departments.

SoTest is an infrastructure product developed by Cyberus Technology GmbH that automates software tests on baremetal hardware. It aims to make testing on hardware as simple as running unit tests. Software product development teams can use it to automate hardware tests in order to integrate them into a fast synchronous end-to-end testing development workflow.

Workflow¶

The following diagram describes the typical workflow in projects where a SoTest deployment performs product testing on hardware:

Typical Developer Workflow with SoTest

This diagram also points out the 2 typical ways to interact with SoTest. Either, users can

manually schedule Test Runs or
let their CI schedule Test Runs automatically.

Concepts¶

In order to organize hardware tests in a useful abstract notion, SoTest introduces a few simple concepts. These become the typical vocabulary that developers use when referring to things around automated hardware testing.

Hardware Platform¶

A hardware platform is a specific configuration of hardware that is made available for testing by a SoTest deployment.

Hardware platforms can be different models of laptops, servers, desktop computers with specific hardware, such as monitors, docking stations or other peripherals, installed.

Multiple instances per hardware platform may exist under the same hardware platform name. SoTest treats these as identical machines to speed up test execution.

Boot Item¶

A boot item describes how to boot one test in a hardware-platform agnostic way.

Each boot item contains, among others, a path to a bootable binary (including its command line parameters), optional lists of additional binaries to load or additional files required for running the test, and a list of machines to execute the test on.

A boot item that has been executed on a specific hardware platform is saved in a database as a successful or failed test. A Web UI visualizes the test results.

Test Run¶

A Test Run is a list of boot items that are each to be executed on one or more hardware platforms.

The SoTest Web UI lists all existing Test Runs in a pageable and searchable overview, called the Test Run Index. Test Runs are listed with their ID, Priority, Creation Date, User, Name and result information.

Example Test Run Index

An overview over the result of a Test Run is displayed in the Test Run Results page as a matrix of boot items. Every row represents a boot item, while the column shows for which hardware platforms it is configured to execute.

Example Test Run with Results

(The small (s) appendix to some of the boot items in the screenshot indicate that some test cases have been skipped on these hardware platforms.)

A new Test Run can be created using the SoTest REST API. The test workloads are then scheduled to run on the selected hardware platforms.

Architecture¶

In order to provide hardware testing service for multiple hardware platforms, a deployed SoTest system is structured roughly like this:

SoTest Architecture Overview

SoTest Database¶

The SoTest Database stores information about all current and past hardware Test Runs along with their results and makes them accessible via the Web UI that can be browsed by humans, but also via the REST API.

SoTest Cluster¶

Hardware platforms are managed in clusters. Each cluster consists of one SoTest Controller which maintains its own private network in which it controls many individual hardware platforms. Projects can have multiple SoTest clusters distributed on different sites that all report to the same SoTest database.

The Controller performs remote power cycling on a hardware platform, then provides it with all needed binaries at boot time, and finally records the relevant test output. After interpreting the test output, it posts back the test results to the SoTest database.