Core API

Previously, we covered the high-level API of our Node.js SDK and learned how to test a find_student software using the Touca test framework:

import { touca } from "@touca/node";
import { find_student } from "./students";

touca.workflow("students_test", async (username: string) => {
  const student = await find_student(username);
  // insert code here to describe the behavior
  // and performance of the workflow under test
});

touca.run();

Functions touca.workflow and touca.run are the entry-points to the Touca test framework. In addition to running our workflow under test with different test cases, the test framework provides facilities that include reporting progress, handling errors, parsing command line arguments, and many more. We intentionally designed this API to abstract away these common features to let developers focus on their workflow under test.

Touca SDK for Node.js provides a separate lower-level Client API that offers more flexibility and control over how tests are executed and how their results are handled. This API is most useful when integrating Touca with other existing test frameworks.

import { touca } from "@touca/node";
import { find_student } from "./students";

(async () => {
  await touca.configure();
  for (const username of await touca.get_testcases()) {
    touca.declare_testcase(username);

    const student = await find_student(username);
    // insert code here to describe the behavior
    // and performance of the workflow under test

    await touca.post();
    await touca.save_json(`touca_${username}.json`);
    await touca.save_binary(`touca_${username}.bin`);
    touca.forget_testcase(username);
  }
  await touca.seal();
})();

The above code uses the low-level Touca Client API to perform the same operations as the Touca test framework, without handling errors, reporting progress, and handling command line arguments. In this section, we will review the functions used in this code and explain what they do.

Configuring the Client

Touca Client requires a one-time call of function configure. This configuration effectively activates all other Touca functions for capturing data and submission of results. Therefore, this function must be called from our test tool, and not from our code under test. This design enables us to leave the calls to Touca data capturing functions in our production code without having to worry about their performance impact.

The configure function can take various configuration parameters including the Touca API Key and API URL. You can also specify an external JSON configuration file via the file option. Check out Node.js SDK reference API documentation for the full list of acceptable configuration parameters and their impact.

await touca.configure({
  api_key: "<TOUCA_API_KEY>",
  api_url: "<TOUCA_API_URL>",
  revision: "<TOUCA_TEST_VERSION>"
});

Touca API Key should be treated as a secret. We advise against hard-coding this parameter.

The three common parameters, API Key, API URL, and version of the code under test can also be set as environment variables TOUCA_API_KEY, TOUCA_API_URL, and TOUCA_TEST_VERSION. Environment variables always override the parameters passed to the configure function.

All of the configuration parameters passed to configure are optional. When api_key and api_url are missing, the client is configured in the offline mode. It can still capture data and store them to files but it will not submit them to the Touca server.

You can always force the client to run in offline mode by passing the offline parameter to the configure function.

Preparing Test Cases

for (const username of await touca.get_testcases()) {
  // insert the code to run for each test case
}

The test framework expects test cases to be specified via the Touca server UI or via command line arguments. With the Client API, you can obtain the list of test cases from any source and pass them, one by one, to your code under test using a simple for loop.

You can still use the function get_testcases to obtain the list of test cases from the Touca server, as our high-level API does. This function should be called when the client is configured to run in offline mode.

Declaring Test Cases

Once the client is configured, you can call declare_testcase once for each test case to indicate that all subsequent captured data and performance benchmarks belong to the specified test case, until a different test case is declared.

for (const username of await touca.get_testcases()) {
  touca.declare_testcase(username);
  // now we can start calling our code under test
  // and describing its behavior and performance
}

With Touca, we consider test cases as a set of unique names that identify different inputs to our code under test. These inputs can be anything as long as they are expected to produce the same behavior every time our code is executed.

Similar to touca.configure, we should only call touca.declare_testcase from our test tool, and not from our code under test.

Capturing Test Results

In the previous document, we reviewed the main Touca functions for describing behavior and performance of our code under test, by capturing values of important variables and runtime of interesting functions. In this section, we dive a little deeper to explain how Touca tracks values of variables and performance benchmarks.

Preserving Data Types

Touca data capturing functions such as touca.check, preserve the types of all captured data so that the Touca server can compare them in their original type.

touca.check("username", student.username);
touca.check("fullname", student.fullname);
touca.check("birth_date", student.dob);
touca.check("gpa", student.gpa);

In the example above, touca.check stores value of properties username and fullname as string while properties dob and gpa are stored as Date and number respectively. The server visualizes possible differences in these values based on their types.

The SDK is designed to handle iterables and custom objects by serializing their elements and properties. This makes it possible for us to add object student as a single entity, if we so choose.

Customizing Data Serialization

While Touca data capturing functions automatically support objects and custom types, it is possible to override the serialization logic for any given non-primitive data type.

Consider the following definition for a custom class Course.

export class Course {
  constructor(public readonly name: string, public readonly grade: number) {}
}

By default, the SDK serializes objects of this class using by serializing all of its public properties. This behavior results in object Course('math', 3.9) to be serialized as {name: 'math', grade: '3.9}. We can use touca.add_serializer to override this default behavior. The following code results in the same object to be serialized as ['math', 3.9]:

touca.add_serializer(Course.name, (x: Course) => [x.name, x.grade]);
for (const course of student.courses) {
  touca.add_array_element("courses", course);
  touca.add_hit_count("number of courses");
}

While our serializer changed the way Course data is serialized and visualized, it still preserved both properties of this object. If we like to exclude the property name during serialization and limit the comparison to grade, we could use (x: Course) => x.grade instead.

It is sufficient to register each serializer once per lifetime of the test application.

Submitting Test Results

Once we execute our code under test for each test case and describe its behavior and performance, we can submit them to the Touca server.

await touca.post();

The server stores the captured data, compares them against the submitted data for pervious versions of our code, visualizes any differences, and reports them in real-time.

It is possible to call touca.post multiple times during the runtime of our test tool. Test cases already submitted to the Touca server whose results have not changed, will not be resubmitted. It is also possible to add new results for an already submitted test case. Any subsequent call to touca::post will resubmit the modified test cases.

We generally recommend that touca.post be called every time the code under test is executed for a given test case. This practice ensures real-time feedback about the test results, as they are being executed.

Storing Test Results

We can choose to store captured test results and performance benchmarks for one or more of our declared test cases on the local filesystem for further processing or later submission to the Touca server.

await touca.save_binary(`touca_${username}.bin`);
await touca.save_json(`touca_${username}.json`);

We can store captured data in JSON or binary format using touca.save_json or touca.save_binary respectively. While JSON files are preferable for quick inspections, only binary files may be posted to the Touca server at a later time.

Forgetting Test Cases

If you are submitted thousands of test cases for each version of your workflow and capture significant amount of information for each test case, you can use touca.forget_testcase to release all the captured information from process memory, when you are done with a given test case.

await touca.forget_testcase();

Sealing Test Results

When all the test cases are executed for a given version of our code under test, we have the option to seal the version to let the server know that no further test result is expected to be submitted for it. This allows the server to send the final comparison result report to interested users, as soon as it is available.

await touca.seal();

Sealing the version is optional. The Touca server automatically performs this operation once a certain amount of time has passed since the last test case was submitted.