Skip to main content

Writing Tests

Let's assume we'd like to test a software that takes the username of a student and returns basic information about them including their name, date of birth, and GPA.

def find_student(username: str) -> Student:
# ...

Where Student has the following properties:

@dataclass
class Student:
username: str
fullname: str
dob: datetime.date
gpa: float

Here's a Touca test we can write for our code under test:

import touca
from students import find_student

@touca.workflow
def students_test(username: str):
with touca.scoped_timer("find_student"):
student = find_student(username)
touca.assume("username", student.username)
touca.check("fullname", student.fullname)
touca.check("birth_date", student.dob)
touca.check("gpa", student.gpa)

With the following general pattern:

import touca

@touca.workflow
def name_of_suite(testcase: str):
# your code goes here

The code we insert as our workflow under test generally performs the following operations:

  1. Map a given testcase name to its corresponding input.
  2. Call the code under test with that input.
  3. Describe the behavior and performance of the code under test.

You can define any number of test workflows and run one or all of them as part of the same test. The test runner will execute the code under test with the test cases for each workflow and submits the test results to the corresponding suite. You can use command line option --filter to limit the test run to any given workflow.

@touca.workflow(testcases=["alice", "bob", "charlie"])
def students_test(username: str):
student = code_under_test.find_student(username)
touca.check("gpa", student.gpa)

@touca.workflow(testcases=["banana", "orange"])
def fruits_test(name: str):
fruit = code_under_test.find_color(name)
touca.check("color", fruit.color)

Describing the Behavior

For any given username, we can call our code under test and capture the important properties of its output that we expect to remain the same in future versions of our software.

We can start small and capture the entire returned object as a Touca result:

touca.check("student", student)

What if we decided to add a field to the return value of the function that reported whether the profile was fetched from the cache? Since this information may change every time we run our tests, we can choose to capture different fields as separate entities.

touca.assume("username", student.username)
touca.check("fullname", student.fullname)
touca.check("birth_date", student.dob)
touca.check("gpa", student.gpa)

This approach allows Touca to report differences in a more helpful format, providing analytics for different fields. If we changed the implementation to always capitalize student names, we could better visualize the differences to make sure that only the value associated with key fullname changes across our test cases.

Note that we used Touca function assume to track the username. Touca does not visualize the values captured as assumption unless they are different.

We can capture the value of any number of variables, including the ones that are not exposed by the interface of our code under test. In our example, let us imagine that our software calculates GPA of students based on their courses.

If we are just relying on the output of our function, it may be difficult to trace a reported difference in GPA to its root cause. Assuming that the courses enrolled by a student are not expected to change, we can track them without redesigning our API:

def calculate_gpa(courses: List[Course]):
touca.check("courses", courses)
return sum(k.grade for k in courses) / len(courses) if courses else 0

Touca data capturing functions remain no-op in production environments. They are only activated when running in the context of a Touca test workflow.

Describing the Performance

Just as we can capture values of variables to describe the behavior of different parts of our software, we can capture the runtime of different functions to describe their performance. Touca can notify us when future changes to our implementation result in significant changes in the measured runtime values.

touca.start_timer("find_student")
student = find_student(username)
touca.stop_timer("find_student")

The two functions start_timer and stop_timer provide fine-grained control for runtime measurement. If they feel too verbose, we can opt to use scoped_timer instead:

with touca.scoped_timer("find_student"):
student = find_student(username)

It is also possible to add measurements obtained by other performance benchmarking tools.

touca.add_metric("external_source", 1500)

In addition to these data capturing functions, the test framework automatically tracks the wall-clock runtime of every test case and reports it to the Touca server.

Like other data capturing functions, we can use Touca performance logging functions in production code, to track runtime of internal functions for different test cases. The functions introduced above remain no-op in production environments.

Running the test

We can run our test from the command line:

touca config set api-key=<TOUCA_API_KEY>
touca config set api-url=<TOUCA_API_URL>
touca test --testcase alice bob charlie

Touca SDK captures the Student object with all its properties and submits that information to the Touca server. We can check this output on the web app but we can also ask the SDK to generate a JSON result file for us:

touca test --save-as-json

You can use --help to learn about available command line options.

Notice that we are not specifying the list of test cases anymore. When they are not explicitly provided, the SDK fetches this list from the Touca server.