touca/touca.hpp is the only header file of Touca SDK for C++ that you need to include in your regression test code. It provides all the functions necessary to configure the core library, capture results and submit them to the Touca server.

This section documents the API exposed by this header file.

Test Runner

Touca SDK has a high-level API that makes it easy to write tests and to run them using our built-in test runner.

The test runner abstracts away many of the common features expected of a test tool, such as parsing and processing configuration options, logging, error handling, managing test results on the filesystem and submitting them to the Touca server. An example implementation invokes touca::run() after registering one or more test workflows via touca::workflow().

int main(int argc, char* argv[]) {
  touca::workflow("is_prime", [](const std::string& testcase) {
    const auto number = std::stoul(testcase);
    touca::check("output", is_prime(number));
  });
  return touca::run(argc, argv);
}

int touca::run(int argc, char *argv[])

Serves as the main entrypoint to the built-in test runner.

Runs your test workflows one by one, with the appropriate configuration options, and to handle all captured test results by submitting them to the Touca server or writing them into the local filesystem.

This function works similar to the application’s main() function and is expected to be called within the main() function. This design gives you the possibility to perform custom actions before and after invoking the test runner.

See also

touca::workflow() that should be called at least once prior to calling this function.

Parameters:

argc – number of arguments provided to the application
argv – list of arguments provided to the application

Returns:

exit status that could be returned by the test application’s main() function

Adding Test Workflows

void touca::workflow(const std::string &name, const std::function<void(const std::string&)> workflow_callback, const std::function<void(WorkflowOptions&)> options_callback = nullptr)

Registers a test workflow to be run with one or more test cases by the Touca test runner.

While we make no assumption about the input to your code under test, the input parameter to function workflow_callback is always the identifier for your test case with type const std::string&. It is up to the test workflow to perform the mapping from the identifier to the actual test input.

See also

touca::WorkflowOptions for a list of supported options.

Parameters:

name – name of this test workflow to be used as the test suite slug
workflow_callback – function that calls your code under test once for each test case.
options_callback – optional function that helps you set certain configuration options for this particular workflow.

struct WorkflowOptions

Configuration options that can be set for individual test workflows when calling the high-level API function touca::workflow().

Setting these parameters is optional. The test runner has built-in mechanism to attempt to find the appropriate value for each option based on the overall configuration options of the overall test.

Subclassed by touca::Workflow

Public Members

std::string suite: Name of the suite to be used that overrides the name of the workflow specified as the first parameter to touca::workflow().

std::string version: Version of the code under test. When this parameter is not set, and is not otherwise specified when running the test, the test runner queries the Touca server to find the most recent submitted version for this suite and uses a minor increment of that version.

std::vector<std::string> testcases: List of test cases to be given one by one to the test workflow. When this parameter is not set, and is not otherwise specified when running the test, the test runner fetches and reuses the list of submitted test cases for the baseline version of this suite.

Configuring the Test Runner

void touca::configure_runner(const std::function<void(RunnerOptions&)> runner_options_callback)

High-level function that lets you customize the behavior of the built-in test runner at runtime, before running the test workflows.

Calling this function is optional. In most cases, it is easier to configure the test runner via command-line arguments, environment variables, or by using a configuration profile.

Any customization using this function will be applied before performing other methods of configuration (such as parsing command-line arguments).

int main(int argc, char* argv[]) {
  touca::configure_workflow([](touca::RunnerOptions& x) {
    x.save_binary = true;
  });
  touca::workflow("is_prime", [](const std::string& testcase) {
    const auto number = std::stoul(testcase);
    touca::check("output", is_prime(number));
  });
  return touca::run(argc, argv);
}

See also

touca::RunnerOptions for a list of supported options.

Parameters:: runner_options_workflow – function that helps you customize the test runner behavior.

struct RunnerOptions : public touca::ClientOptions 

Configuration options supported by the built-in test runner.

Public Members

bool save_binary = false: Store all the data points captured for each test case into a local file in binary format. Touca binary archives can later be inspected using the Touca CLI and submitted to a Touca server instance.

bool save_json = false: Store all the data points captured for each test case into a local file in JSON format. Unlike Touca binary archives, these JSON files are only helpful for manual inspection of the captured test results and are not supported by the Touca server.

bool overwrite_results = false: Overwrite the locally generated test results for a given testcase if the results directory already exists.

bool no_color = false: Do not use ANSI colors when reporting the test progress in the standard output.

bool redirect_output = true: Capture the standard output and standard error of the code under test and redirect them to a local file.

bool skip_logs = false: Indicates whether to generate a copy of the standard output of the test into a Console.log file.

std::string config_file: Relative or full path to a configuration file to be loaded and applied at runtime.

std::string output_directory: Relative or full path to the directory in which Touca test results are written, when the runner is configured to write them into the local filesystem.

std::string log_level = "info": Level of detail to use when publishing log events to the external loggers.

std::vector<std::string> testcases: Set of testcases to feed one by one to all the registered workflows. When not provided, the test runner uses the set of testcases configured for each workflow. If that set is empty, the test runner attempts to retrieve and reuse the set of testcases submitted for the baseline version of each workflow.

std::string workflow_filter: Limits the test to running the specified workflow as opposed to all the registered workflows.

std::vector<Workflow> workflows: The set of all registered workflows.

bool submit_async = false: Submits test results asynchronously if set.

void touca::add_sink(std::unique_ptr<Sink> sink, const Sink::Level level = Sink::Level::Info)

Registers a sink to subscribe to the test runner log events.

Parameters:

sink – sink instance to be called when a log event is published
level – minimum level of detail to subscribe to

Core Library

Touca C++ SDK provides a separate lower-level Client API that offers more flexibility and control over how tests are executed and how their results are handled. This API is most useful when integrating Touca with other existing test frameworks.

Configuring the Library

void touca::configure(const std::function<void(ClientOptions&)> options = nullptr)

Attempts to configure the core Touca library.

Must be called before declaring testcases or adding results to the client.

touca::configure([](ClientOptions& x){
    x.api_key = "03dda763-62ea-436f-8395-f45296e56e4b"
    x.api_url = "https://api.touca.io"
    x.team = "acme",
    x.students = "students",
    x.version = "v1.0",
});

Client is considered configured if it can capture test results and store them locally on the filesystem. To do so, configuration options team, suite, and version shall be provided, directly or indirectly, together in a single call, or separately in a sequence of calls, in order for the client to be considered as configured. These options may be specified, in part or in full, as components of the configuration parameter api-url.

In addition to the configuration parameters above, the parameters api-url and api-key shall be provided for the client to be able to submit captured test results to the server. As an example, the same configuration options above could have been provided via the following code snippet.

touca::configure([](ClientOptions& x){
    x.api_key = "03dda763-62ea-436f-8395-f45296e56e4b"
    x.api_url = "https://api.touca.io/@/acme/student/v1.0"
});

If the API URL is missing and API Key is set, the API URL will be set to https://api.touca.io.

Since the function is designed to never throw and to allow users to configure the client through a sequence of separate calls, we recommend that you use touca::is_configured() to check if the client is configured and learn about any potential error using touca::configuration_error().

if (!touca::is_configured()) {
    std::cerr << touca::configuration_error() << std::endl;
}

See also

touca::ClientOptions for a list of supported configuration parameters

Parameters:: options – a callback function for setting configuration parameters

struct ClientOptions

Configuration options supported by the low-level Core API library.

Use the touca::configure function for setting these options programmatically. When using the test runner, you can set any subset of these options without hard-coding the values using a variety of methods such as command-line arguments, environment variables, JSON-formatted configuration file, Touca CLI configuration profiles, etc. See touca::RunnerOptions to learn more.

Subclassed by touca::RunnerOptions

Public Members

std::string api_key

API Key issued by the Touca server that identifies who is submitting the test results.

Since the value should be treated as a secret, we strongly recommend that you do not hard-code this option and pass it via other methods such as setting the environment variable TOUCA_API_KEY (ideal for the CI environment) or using the Touca CLI to set this option in your configuration profile to be automatically loaded at runtime (ideal for local development).

std::string api_url

URL to Touca server API

Defaults to when api_key is specified. If you are self-hosting the Touca server, we encourage using the Touca CLI to set this option in your configuration profile to be automatically loaded at runtime.

std::string team

Slug of your team on the Touca server

Since it is unlikely for your team slug to change, we encourage using the Touca CLI to set this option in your configuration profile to be automatically loaded at runtime.

std::string suite

Name of the test suite to submit test results to

When using the test runner, value of the first parameter to touca::workflow is used by default.

std::string version

Version of your code under test

Since this version is expected to change, we encourage setting option via the environment variable TOUCA_TEST_VERSION or passing it as a command-line option.

When using the test runner, you may also skip setting this option to let the test runner query the Touca server for the most recent version of your suite and use a minor version increment.

bool offline = false

Disables all communications with the Touca server

Determines whether client should connect with the Touca server during the configuration. Will be set to false when neither api_url nor api_key are set.

bool concurrency = true

Isolates the testcase scope to calling thread

Determines whether the scope of test case declaration is bound to the thread performing the declaration, or covers all other threads. Defaults to true.

If set to true, when a thread calls touca::declare_testcase, all other threads also have their most recent test case changed to the newly declared test case and any subsequent call to data capturing functions such as touca::check will affect the newly declared test case.

bool touca::is_configured()

Checks if the client is configured to perform basic operations including capturing data points and storing them locally on the filesystem. Note that when api_key and api_url are missing, the client is considered as configured even though it cannot submit the test results to the server.

See also

touca::configuration_error() for extracting the most recent reason when touca::configure() is called but the client is not configured.

Returns:: true if the client is configured to perform basic operations.

std::string touca::configuration_error()

Provides the most recent error, if any, encountered during client configuration.

See also

touca::is_configured() for checking if the client is configured before attempting to access and handle any configuration error.

Returns:: short description of the most recent configuration error

void touca::add_logger(const std::shared_ptr<touca::logger> logger)

Registers a custom logger that is notified when an event of potential interest takes place.

This function enables users to register their own logger derived from touca::logger and listen for log events generated by the client library. Log events include warnings and errors if client library is misused or fails to perform an instructed action. Adding a logger is optional.

Parameters:: logger – a custom logger that is notified of log events generated by the core library.

Declaring Testcases

void touca::declare_testcase(const std::string &name)

Declares the name of the testcase to which all subsequent results will be submitted until a new testcase is declared.

Unless configuration options concurrency is set to false, when a thread calls declare_testcase all other threads also have their most recent testcase changed to the newly declared one.

Parameters:: name – name of the testcase to be declared

void touca::forget_testcase(const std::string &name)

Removes all logged information associated with a given testcase.

Removes from memory, all information that is logged for the previously declared testcase, for all the threads, regardless of whether configuration option concurrency is set.

This function does not remove the test results results from the server, in case they are already submitted. It clears all information about that testcase from the client library such that switching back to an already declared or already submitted testcase would behave similar to when that testcase was first declared.

Calling this function is useful in long-running regression tests, after submission of the testcase to the server, if memory consumed by the client library is a concern or if there is a risk that a future testcase with a similar name may be executed.

Parameters:: name – name of the testcase to be removed from memory

Capturing Test Results

template<typename Char, typename Value> void touca::check(Char &&key, const Value &value)

Captures the value of a given variable as a test result for the declared testcase and associates it with the specified key.

Primary data capturing function for adding test results for the declared testcase.

Template Parameters:

Char – type of string to be associated with the value stored as a result. Expected to be convertible to std::basic_string<char>.
Value – original type of value value to be stored as a result in association with the given key key.

Parameters:

key – name to be associated with the logged test result.
value – value to be logged as a test result.

template<typename Char, typename Value> void touca::assume(Char &&key, const Value &value)

Logs a given value as an assumption for the declared testcase and associates it with the specified key.

Assumptions are a special category of data points that are hardly ever expected to change for a given test case between different versions of the workflow.

Assumptions are treated differently by the Touca server: The server specially highlights assumptions if they are different between two test versions and removes them from user focus if they remain unchanged. Therefore, assumptions are particularly helpful for verifying assumptions about input data and their properties.

See also

touca::check() as the primary data capturing function.

Template Parameters:

Char – type of string to be associated with the value stored as an assumption. Expected to be convertible to std::basic_string<char>.
Value – original type of value value to be stored as an assumption in association with given key key.

Parameters:

key – name to be associated with the logged test result.
value – value to be logged as an assumption

template<typename Char, typename Value> void touca::add_array_element(Char &&key, const Value &value)

Adds a given data point as an element of an ordered set of data points that are stored as one single test result entity for the testcase which is associated with the specified key.

Could be considered as a helper utility function. This method is particularly helpful to log a list of elements as they are found:

for (const auto number : numbers) {
    if (isPrime(number)) {
        touca::add_array_element("prime numbers", number);
        touca::add_hit_count("number of primes");
    }
}

This pattern can be considered as a syntactic sugar for the following alternative:

std::vector<unsigned> primes;
for (const auto number : numbers) {
    if (isPrime(number)) {
        primes.emplace_back(number);
    }
}
if (!primes.empty()) {
    touca::check("prime numbers", primes);
    touca::check("number of primes", primes.size());
}

The items added to the list are not required to be of the same type. The following code is acceptable:

touca::add_array_element("elements", 42);
touca::add_array_element("elements", "forty three");

See also

touca::check() as the primary data capturing function.

Since: v1.1

Template Parameters:

Char – type of string to be associated with the value stored as an element. Expected to be convertible to std::basic_string<char>.
Value – original type of value value to be stored as an element of an array associated with given key key.

Parameters:

key – name to be associated with the logged test result.
value – element to be appended to the array

Throws:

touca::detail::runtime_error – if the specified key is already associated with a test result whose type is not a derivative of touca::array.

void touca::add_hit_count(const std::string &key)

Increments the value of the data point associated with key key. Creates the data point with the initial value of 1 if it does not exist.

May be considered as a helper utility function. This method is particularly helpful to track variables whose values are determined in loops with indeterminate execution cycles:

for (const auto number : numbers) {
    if (isPrime(number)) {
        add_array_element("prime numbers", number);
        add_hit_count("number of primes");
    }
}

This pattern can be considered as a syntactic sugar for the following alternative:

std::vector<unsigned> primes;
for (const auto number : numbers) {
    if (isPrime(number)) {
        primes.emplace_back(number);
    }
}
if (!primes.empty()) {
    touca::check("prime numbers", primes);
    touca::check("number of primes", primes.size());
}

See also

touca::check() as the primary data capturing function.

Since: v1.1

Parameters:: key – name to be associated with the logged test result.
Throws:: touca::detail::runtime_error – if the specified key is already associated with a test result which was not an integer.

Capturing Metrics

void touca::add_metric(const std::string &key, const unsigned duration)

Adds a performance measurement collected with the help of this library as a performance benchmark (metric).

Since: v1.2.0

Parameters:

key – name to be associated with the performance metric
duration – duration in number of milliseconds

void touca::start_timer(const std::string &key)

Starts performance measurement for a given metric.

Records the time of invocation of this function, associates it with the given key and awaits a future call to stop_timer with the same key to log the duration as a performance metric.

Since: v1.1

Parameters:: key – name to be associated with the performance metric

void touca::stop_timer(const std::string &key)

Stops performance measurement for a given metric.

Logs a performance metric whose value is the duration between this call and a previous call to start_timer with the same key.

Since: v1.1

Parameters:: key – name to be associated with the performance metric

class scoped_timer: a simple class that helps clients log the duration between its instantiation and destruction as a performance metric.

TOUCA_SCOPED_TIMER: convenience macro for logging performance of a function as a performance metric.

Saving Test Results

void touca::save_binary(const std::string &path, const std::vector<std::string> &testcases = {}, const bool overwrite = true)

Stores the test results in binary format in a file with the specified path.

Stores the test results assigned to given set of testcases in a file with the specified path in binary format. We do not recommend as a general practice for regression test tools to locally store their test results. This feature may be helpful for special cases such as when test tools have to be run in environments that have no access to the Touca server (e.g. running with no network access).

Parameters:

path – path to file in which test results should be stored
testcases – set of names of testcases whose results should be stored. if given set is empty, all test cases will be stored in the specified file.
overwrite – determines whether to overwrite any file that exists in the specified path. Defaults to true.

void touca::save_json(const std::string &path, const std::vector<std::string> &testcases = {}, const bool overwrite = true)

Stores test results in JSON format in a file with the specified path.

Stores test results assigned to given set of testcases in a file with the specified path in JSON format.

Parameters:

path – path to file in which test results should be stored
testcases – set of names of testcases whose results should be stored to disk. if given set is empty, all testcases will be stored in the specified file.
overwrite – determines whether to overwrite any file that exists in the specified path. Defaults to true.

Submitting Test Results

Post::Status touca::post(const Post::Options &options = {})

Submits all test results recorded so far to Touca server.

Posts all test results of all testcases declared by this client to Touca server in flatbuffers format. Should only be called after the client is configured.

It is possible to call touca::post() multiple times during runtime of the test tool. Test cases already submitted to the server whose test results have not changed, will not be resubmitted. It is also possible to add test results to a testcase after it is submitted to the server. Any subsequent call to touca::post() will resubmit the modified testcase.

Throws:: touca::detail::runtime_error – if client is not configured or that it is configured to operate without communicating with the server.
Returns:: enum indicating the status of submitted test results.

Sealing a Version

void touca::seal()

Notifies Touca server that all test cases were executed and no further test result is expected to be submitted.

Expected to be called by the test tool once all test cases are executed and all test results are posted.

Sealing the version is optional. The Touca server automatically performs this operation once a certain amount of time has passed since the last test case was submitted. This duration is configurable from the “Settings” tab in “Suite” Page.

Since: v1.3

Throws:: touca::detail::runtime_error – if client is not configured or that it is configured to operate without communicating with the server.

Extending Touca Type System

template<typename T, typename> struct serializer

Non-specialized template declaration of conversion logic for handling objects of custom types by the Touca SDK for C++.

Allows users developing regression tools to provide explicit full specialization of this class that makes it convenient to pass objects of their non-trivial type directly to Touca API functions that accept test results.

The following example illustrates a specialization of serializer for a custom type Date.

struct Date {
  const unsigned short year;
  const unsigned short month;
  const unsigned short day;
};

template <>
struct touca::serializer<Date> {
  data_point serialize(const Date& value) {
    return object("Date")
      .add("year", value.year)
      .add("month", value.month)
      .add("day", value.day);
  }
};

Once declared, this specialization allows user to directly pass objects of type Date to Touca server API that accepts test results.

Date date { 1961, 8, 4 };
touca::check("birthday", date);

Noteworthy, that declaring a conversion logic for Date, enables objects of this type to be used as smaller pieces of even more complex types:

struct Person {
  const std::string name;
  const Date birthday;
};

template <>
struct touca::serializer<Person> {
  data_point serialize(const Person& value) {
    return object("Person")
      .add("name", val.name)
      .add("birthday", val.birthday);
  }
};

Person person { "alex", { 1961, 8, 4 } };
touca::check("person", person);

Template Parameters:: T – type whose handling logic is to be implemented in serialize member function.