Skip to main content

Setting Test Cases

Programmatic Testcase Declaration

In addition to passing testcases as command-line arguments or letting the SDK fetch and reuse the set of test cases already submitted to the baseline version, sometimes you may want to specify your test cases as code or infer them during the test runtime, based on the set of local files or datasets in a given directory, or rows of a given column in a CSV file, or a nested array of JSON response of a particular API endpoint, etc.

In Python, you could use the testcases parameter of the @touca.workflow decorator which accepts either a list, a generator, or a generator function.

@touca.workflow(testcases=["alice", "bob", "charlie"])
def students_test(username: str):
student = code_under_test.find_student(username)
touca.check("gpa", student.gpa)

Here's the same snippet if we were to use a generator function.

def find_testcases():
for username in ['alice', 'bob', 'charlie']:
yield username

def students_test(username: str):
student = code_under_test.find_student(username)
touca.check("gpa", student.gpa)

Best Practices

Identifying the code under test is the first step to developing any test tool. We recommend that you choose your code under test such that it constitutes a pure function (in its mathematic sense) whose return value remains the same for the same argument, and its evaluation has no side effects. This way, for any implementation of our code, any given input will yield a specific output. If a subsequent implementation yields a different output for the same input, we can argue that the changes in the implementation have introduced regressions.

The type and definition of the input to our code under test are arbitrary and restricted only by the requirements of our workflow. But Touca tests always take a short, unique, and URL-friendly string as test case. As an example, a possible test case for our Touca test may be the name of a locally stored file or directory in which the input for the code under test is stored. It is up to us to load the actual input to our code under test based on the filename provided as test case.

The effectiveness of a Touca test depends, in part, on the variety of its test cases. Ideally, we should have enough test cases to cover all execution branches of our code under test. This may be difficult to achieve in real-world applications where the code under test may be arbitrarily complicated. As a more practical alternative, we recommend that the test cases be chosen such that they represent the range of typical inputs given to our software in production environments.