Let's imagine we are building a software to check whether workers at a construction site are wearing hard hats. Our software takes an image as input and uses an object detection algorithm to identify people and helmets within the image.
Our software could be complex. We could break it down to smaller components and write unit tests for each component. We could also write integration tests that feed a few images to our workflow code under test and check the coordinates of identified objects within an image. We would need a large number of test inputs to gain confidence that our software is production-ready and can handle real-world inputs. This may be a problem:
- Describing the expected behavior of our software for each test input is difficult.
- Future improvements to our software may require adjusting the expected values which is cumbersome.
Touca aims to solve these problems by taking a different approach. Instead of asserting the output of each test against hard-coded expected values, Touca captures the actual behavior and performance of our software for each test case and compares them against a previous trusted version.
In this tutorial, we will use the Touca CLI to see how Touca works in action. In the next tutorial, we will learn how to use Touca SDKs to write tests and capture data programmatically.
pip install touca
brew install touca
Touca CLI has various commands for different operations. Let's use the
login command to log into the Touca server and set our API credentials.
Touca CLI is also extensible to help us add plugins that save us from writing
code for common use-cases. Let's install a "demo" plugin and run it to download
sample data for version
v1.0 of our
hard helmets example and make them
available in the
tutorial directory of our current working directory.
touca plugin add plugins://demo
touca demo download v1.0
v1.0, we have run our code under test with 10 different images and
have captured the output of our software, including a JSON file that lists the
detected objects in each image and a JPG file visualizing those detected
Since this is the first working version of our software, we can spend some time inspecting the results and verifying their correctness. If we were using unit testing, we would start writing assertions against the JSON output. With Touca, we can capture the output test results and establish them as baseline.
touca check --suite=helmets tutorial/v1.0
Unlike snapshot testing, Touca stores the captured test results on a remote server. The server automatically compares the results for newly submitted versions against our baseline version and reports the differences in near real-time. This approach removes the need for storing snapshot files in version control and enables comparing any two versions of your software and visualizing their differences. And since building software is often a team effort, Touca gives your team controlled access to the state of your software workflows and how they evolve over time.
hard helmets example, now that we established a baseline, we can re-run
touca check every time we make changes to our software to compare subsequent
versions against the baseline.
touca demo download v1.1
touca check --suite=helmets tutorial/v1.1
Let's navigate to the Touca server to see how
v1.1 of our
example behaves compared to
v1.0 and decide whether those differences are
improvements or unintended regressions.
For the above test case, it looks like
v1.1 performs object detections more
accurately. We can inspect the differences in other test cases to decide if
v1.1 is indeed working better. If we choose to ship
v1.1, we can promote
this version as the new baseline so that subsequent versions are compared
touca check is a convenient way to track changes in the output of our software
without the need to write code. It works similar to
diff except you don't need
to have the two sides of comparison available at the same time. But
touca check is limited to capturing the output of a process such as generated
files and standard streams. Many complex system components that need regression
testing rarely produce any external output. Touca provides SDKs for different
programming languages that allow capturing values of variables and runtime of
functions from our code.