Your First Touca Test

You've made it this far. Great! 👍🏼

We assume you have followed our Setup Your Account tutorial to create an account on Touca. In this document, we will show you how to write and run a simple regression test to submit your first test results to the Touca server.

This is a hands-on tutorial. It's only fun if you follow along. 👨🏻‍💻

Code Under Test

Let us imagine we are building a profile database software that retrieves personal information of students based on their username.

Python

def find_student(username: str) -> Student:

C++

Student find_student(const std::string& username);

TypeScript

async function find_student(username: string): Promise<Student>;

Java

public static Student findStudent(final String username);

Where type Student could be defined as follows:

Python

@dataclass
class Student:
    username: str
    fullname: str
    dob: datetime.date
    gpa: float

C++

struct Student {
  std::string username;
  std::string fullname;
  Date dob;
  float gpa;
};

TypeScript

interface Student {
  username: string;
  fullname: string;
  dob: Date;
  gpa: number;
}

Java

import java.time.LocalDate;

public final class Student {
    public String username;
    public String fullname;
    public LocalDate dob;
    public double gpa;
}

Clone the Touca repository to a directory of your choice.

git clone git@github.com:trytouca/trytouca.git

Navigate to the examples directory for your preferred programming language. Each example serves as a standalone hands-on tutorial.

Let's focus on the Main API example examples/<lang>/02_<lang>_main_api which includes two modules students and students_test. The students module represents our code under test: the production code for our profile database software. Our code under test can have any complexity. It may call various nested functions, connect to database, and scrape the web to return information about a student given their username.

Check out the students module for a possible "current" implementation:

Python

python/02_python_main_api/students.py

def find_student(username: str) -> Student:
    sleep(0.2)
    data = next((k for k in students if k[0] == username), None)
    if not data:
        raise ValueError(f"no student found for username: ${username}")
    return Student(data[0], data[1], data[2], calculate_gpa(data[3]))

C++

cpp/02_cpp_main_api/students.cpp

Student find_student(const std::string& username) {
  std::this_thread::sleep_for(std::chrono::milliseconds(200));
  if (!students.count(username)) {
    throw std::invalid_argument("no student found for username: " + username);
  }
  const auto& student = students.at(username);
  return {
    student.username,
    student.fullname,
    student.dob,
    calculate_gpa(student.courses)
  };
}

TypeScript

js/02_node_main_api/students.ts

export async function find_student(username: string): Promise<Student> {
  await new Promise((v) => setTimeout(v, 200));
  const data = students.find((v) => v.username === username);
  if (!data) {
    throw new Error(`no student found for username: ${username}`);
  }
  const { courses, ...student } = data;
  return { ...student, gpa: calculate_gpa(courses) };
}

Java

java/02_java_main_api/Students.java

public static Student findStudent(final String username) {
  Thread.sleep(200);
  for (Student student : Students.students) {
    if (student.username.equals(username)) {
      return new Student(data.username, data.fullname, data.dob,
          calculateGPA(data.courses));
    }
  }
  throw new NoSuchElementException(
      String.format("No student found for username: %s", username));
}

Writing a Touca Test

With Touca, we call our workflow under test with various inputs and try to describe the behavior and performance of our implementation by capturing values of variables and runtime of functions as results and metrics. While this is similar to unit testing, there are fundamental differences:

• Instead of hard-coding inputs to our code under test, we pass them via the testcase parameter to our Touca test workflow.
• Instead of hard-coding expected outputs for each test case, we use Touca data capturing functions to record the actual values of important variables.
• Instead of being bound to checking the output value of our code under test, we can track value of any variable and runtime of any function in our code under test.

These differences in approach stem from a difference in objective. Unlike unit testing, our goal is not to verify that our code behaves correctly. We want to check that it behaves and performs as well as before. This way, we can start changing our implementation without causing regressions in our overall software.

Here is a possible implementation for our first Touca test code:

Python

python/02_python_main_api/students_test.py

import touca
from students import find_student

@touca.Workflow
def students_test(username: str):
    student = find_student(username)
    touca.assume("username", student.username)
    touca.check("fullname", student.fullname)
    touca.check("birth_date", student.dob)
    touca.check("gpa", student.gpa)

if __name__ == "__main__":
    touca.run()

C++

cpp/02_cpp_main_api/students_test.cpp

#include "students.hpp"
#include "students_types.hpp"
#include "touca/touca.hpp"

int main(int argc, char* argv[]) {
  touca::workflow("find_student", [](const std::string& username) {
    const auto& student = find_student(username);
    touca::assume("username", student.username);
    touca::check("fullname", student.fullname);
    touca::check("birth_date", student.dob);
    touca::check("gpa", student.gpa);
  });
  return touca::run(argc, argv);
}

TypeScript

js/02_node_main_api/students_test.ts

import { touca } from "@touca/node";
import { find_student } from "./students";

touca.workflow("students_test", async (username: string) => {
  const student = await find_student(username);
  touca.assume("username", student.username);
  touca.check("fullname", student.fullname);
  touca.check("birth_date", student.dob);
  touca.check("gpa", student.gpa);
});

touca.run();

Java

java/02_java_main_api/StudentsTest.java

import io.touca.Touca;

public final class StudentsTest {

  @Touca.Workflow
  public void findStudent(final String username) {
    Student student = Students.findStudent(username);
    Touca.assume("username", student.username);
    Touca.check("fullname", student.fullname);
    Touca.check("birth_date", student.dob);
    Touca.check("gpa", student.gpa);
  }

  public static void main(String[] args) {
    Touca.run(StudentsTest.class, args);
  }

}

Notice the absence of hard-coded inputs and expected outputs. Each Touca workflow, takes a short, unique, and URL-friendly testcase name, maps that to a corresponding input and passes that input to our code under test. In the above code snippet, once we receive the output of our find_student workflow, we use check to track various characteristics of that output. Touca notifies us if these characteristics change in a future version of our find_student workflow.

We can track any number of variables in each Touca test workflow. More importantly, we can track important variables that might not necessarily be exposed through the interface of our code under test. In our example, our software computes the GPA of a student based on their courses and using an internal function calculate_gpa. With Touca, we can check this function for regression by tracking both the calculated GPA and the list courses, without creating a separate test workflow.

Python

python/02_python_main_api/students.py

def calculate_gpa(courses: List[Course]):
    touca.check("courses", courses)
    return sum(k.grade for k in courses) / len(courses) if courses else 0

C++

cpp/02_cpp_main_api/students.cpp

float calculate_gpa(const std::vector<Course>& courses) {
  touca::check("courses", courses);
  const auto& sum = std::accumulate(courses.begin(), courses.end(), 0.0f,
    [](const float sum, const Course& course) {
      return sum + course.grade;
    });
  return courses.empty() ? 0.0f : sum / courses.size();
}

TypeScript

js/02_node_main_api/students.ts

function calculate_gpa(courses: Course[]): number {
  touca.check("courses", courses);
  return courses.length
    ? courses.reduce((sum, v) => sum + v.grade, 0) / courses.length
    : 0.0;
}

Java

java/02_java_main_api/Students.java

private static double calculateGPA(final Course[] courses) {
    Touca.check("courses", courses);
    double sum = Arrays.asList(courses).stream().mapToDouble(item -> item.grade).sum();
    return courses.length == 0 ? sum / courses.length : 0.0;
}

Notice that we are using Touca check inside our production code. Touca data capturing functions are no-op in the production environment. When they are executed by Touca workflow in a test environment, they start capturing values and associating them with the active test case.

Lastly, Touca helps us track changes in the performance of different parts of our code, for any number of test cases. While there are various patterns and facilities for capturing performance benchmarks, the most basic functions are start_timer and stop_timer for measuring runtime of a given piece of code, as shown below.

Python

python/02_python_main_api/students_test.py

import touca
from students import find_student

@touca.Workflow
def students_test(username: str):
    with touca.scoped_timer("find_student"):
        student = find_student(username)
    touca.assume("username", student.username)
    touca.check("fullname", student.fullname)
    touca.check("birth_date", student.dob)
    touca.check("gpa", student.gpa)
    touca.add_metric("external_source", 1500)

C++

cpp/02_cpp_main_api/students_test.cpp

#include "students.hpp"
#include "students_types.hpp"
#include "touca/touca.hpp"

int main(int argc, char* argv[]) {
  touca::workflow("find_student", [](const std::string& username) {
    touca::start_timer("find_student");
    const auto& student = find_student(username);
    touca::stop_timer("find_student");
    touca::assume("username", student.username);
    touca::check("fullname", student.fullname);
    touca::check("birth_date", student.dob);
    touca::check("gpa", student.gpa);
    touca::add_metric("external_source", 1500);
  });
  return touca::run(argc, argv);
}

TypeScript

js/02_node_main_api/students_test.ts

import { touca } from "@touca/node";
import { find_student } from "./students";

touca.workflow("students_test", async (username: string) => {
  touca.start_timer("find_student");
  const student = await find_student(username);
  touca.stop_timer("find_student");
  touca.assume("username", student.username);
  touca.check("fullname", student.fullname);
  touca.check("birth_date", student.dob);
  touca.check("gpa", student.gpa);
  touca.add_metric("external_source", 1500);
});

touca.run();

Java

java/02_java_main_api/StudentsTest.ts

import io.touca.Touca;

public final class StudentsTest {

  @Touca.Workflow
  public void findStudent(final String username) {
    Touca.startTimer("find_student");
    Student student = Students.findStudent(username);
    Touca.stopTimer("find_student");
    Touca.assume("username", student.username);
    Touca.check("fullname", student.fullname);
    Touca.check("birth_date", student.dob);
    Touca.check("gpa", student.gpa);
    Touca.addMetric("external_source", 1500);
  }

  public static void main(String[] args) {
    Touca.run(StudentsTest.class, args);
  }

}

There is so much more that we can cover, but for now, let us accept the above code snippet as the first version of our Touca test code and proceed with running this test.

Running a Touca Test

Let us now use one of Touca SDKs to write a test that could help us detect future changes in the overall behavior or performance of our profile database software.

Python

Navigate to directory python/02_python_main_api in the examples repository and create a virtual environment using Python v3.6 or newer.

python -m venv .env
source .env/bin/activate

Install Touca SDK as a third-party dependency:

pip install touca

C++

Navigate to directory cpp/02_cpp_main_api in the examples repository and run build.sh or build.bat depending on your platform using CMake 3.14 or newer. This command produces executables in a ./local/dist/bin directory.

./build.sh

This example uses CMake's FetchContent to pull Touca SDK as a dependency. See our SDK documentation for instructions to use Conan, instead.

TypeScript

Navigate to directory js/02_node_main_api in the examples repository and use either of npm or yarn to build examples using Node v12 or newer.

npm install
npm run build

If you must use older versions of node, at least consider upgrading npm: npm i -g npm@latest

Java

Navigate to directory java/02_java_main_api in the examples repository and use gradle to build examples using Java 8 or newer.

./gradlew build

We can run Touca test from the command line, passing the following information as command line arguments.

• API Key: to authenticate with the Touca server
• API URL: to specify where test results should be submitted to
• Revision: to specify the version of our code under test
• Testcases: to specify what inputs should be given to our workflow under test

We can find API Key and API URL on the Touca server. We can use any string value for Revision. More importantly, we can pass any number of test cases to the code under test, without ever changing our test logic.

Python

touca config set api-key=<TOUCA_API_KEY>
touca config set api-url=<TOUCA_API_URL>
touca test --revision v1.0 --testcase alice bob charlie

C++

export TOUCA_API_KEY=<TOUCA_API_KEY>
export TOUCA_API_URL=<TOUCA_API_URL>
./local/dist/bin/example_cpp_main_api --revision v1.0 --testcase alice,bob,charlie

TypeScript

export TOUCA_API_KEY=<TOUCA_API_KEY>
export TOUCA_API_URL=<TOUCA_API_URL>
node 02_node_main_api/dist/students_test.js --revision v1.0 --testcase alice bob charlie

Java

export TOUCA_API_KEY=<TOUCA_API_KEY>
export TOUCA_API_URL=<TOUCA_API_URL>
gradle runExampleMain --args='--revision v1.0 --testcase alice bob charlie'

In real-world scenarios, we may have too many test cases to specify as command line arguments. We can write our test cases to a file and pass the path to that file using the --testcase-file option. Alternatively, we can add our test cases directly to the Touca server. When test cases are not provided via --testcase or --testcase-file options, Touca SDKs attempt to retrieve them from the Touca server.

The above command produces the following output.

Touca Test Framework
Suite: students_test/v1.0

 1.  PASS   alice      (127 ms)
 2.  PASS   bob        (123 ms)
 3.  PASS   charlie    (159 ms)

Tests:      3 passed, 3 total
Time:       0.57 s

✨   Ran all test suites.

At this point, we should see the results of our test on the Touca server. This is a big milestone. Congratulations! 🎉

Notice that this version is shown with a star icon to indicate that it is the baseline version of our Suite. Touca will compare subsequent versions of our software against the test results submitted for this version.

In the next section, we will see how to use Touca to understand the differences between different versions of our software, investigate their root cause, communicate our findings with our team members, and update the baseline version.