Skip to main content

Concepts

Let's imagine we are building a software to check whether workers at a construction site are wearing hard hats. Our software takes an image as input and uses an object detection algorithm to identify people and helmets within the image.

Our software could be complex. We could break it down to smaller components and write unit tests for each component. We could also write integration tests that feed a few images to our workflow code under test and check the coordinates of identified objects within an image. We would need a large number of test inputs to gain confidence that our software is production-ready and can handle real-world inputs. This may be a problem:

  • Describing the expected behavior of our software for each test input is difficult.
  • Future improvements to our software may require adjusting the expected values which is cumbersome.

Touca aims to solve these problems by taking a different approach. Instead of asserting the output of each test against hard-coded expected values, Touca captures the actual behavior and performance of our software for each test case and compares them against a previous trusted version.

Psst!

This is a great time to sign up on Touca. You will need your API Key and URL to complete this hands-on tutorial. See here for instructions.

In this tutorial, we will use the Touca CLI to see how Touca works in action. In the next tutorial, we will learn how to use Touca SDKs to write tests and capture data programmatically.

pip install touca

Touca CLI has various commands for different operations. Let's use the login command to log into the Touca server and set our API credentials.

touca login

Touca CLI is also extensible to help us add plugins that save us from writing code for common use-cases. Let's install a "demo" plugin and run it to download sample data for version v1.0 of our hard helmets example and make them available in the tutorial directory of our current working directory.

touca plugin add plugins://demo
touca demo download v1.0

To test v1.0, we have run our code under test with 10 different images and have captured the output of our software, including a JSON file that lists the detected objects in each image and a JPG file visualizing those detected objects.

Since this is the first working version of our software, we can spend some time inspecting the results and verifying their correctness. If we were using unit testing, we would start writing assertions against the JSON output. With Touca, we can capture the output test results and establish them as baseline.

touca check --suite=helmets tutorial/v1.0

Unlike snapshot testing, Touca stores the captured test results on a remote server. The server automatically compares the results for newly submitted versions against our baseline version and reports the differences in near real-time. This approach removes the need for storing snapshot files in version control and enables comparing any two versions of your software and visualizing their differences. And since building software is often a team effort, Touca gives your team controlled access to the state of your software workflows and how they evolve over time.

In our hard helmets example, now that we established a baseline, we can re-run touca check every time we make changes to our software to compare subsequent versions against the baseline.

touca demo download v1.1
touca check --suite=helmets tutorial/v1.1

Let's navigate to the Touca server to see how v1.1 of our hard helmets example behaves compared to v1.0 and decide whether those differences are improvements or unintended regressions.

For the above test case, it looks like v1.1 performs object detections more accurately. We can inspect the differences in other test cases to decide if v1.1 is indeed working better. If we choose to ship v1.1, we can promote this version as the new baseline so that subsequent versions are compared against it.

touca check is a convenient way to track changes in the output of our software without the need to write code. It works similar to diff except you don't need to have the two sides of comparison available at the same time. But touca check is limited to capturing the output of a process such as generated files and standard streams. Many complex system components that need regression testing rarely produce any external output. Touca provides SDKs for different programming languages that allow capturing values of variables and runtime of functions from our code.