Core API
Previously, we covered the high-level API of our C++ SDK
and learned how to test a find_student
software using the Touca test
framework:
#include "students.hpp"
#include "students_types.hpp"
#include "touca/touca.hpp"
int main(int argc, char* argv[]) {
touca::workflow("find_student", [](const std::string& username) {
const auto& student = find_student(username);
// insert code here to describe the behavior
// and performance of the workflow under test
});
return touca::run(argc, argv);
}
Functions touca::workflow
and touca::run
are the entry-points to the Touca
test framework. In addition to running our workflow under test with different
test cases, the test framework provides facilities that include reporting
progress, handling errors, parsing command line arguments, and many more. We
intentionally designed this API to abstract away these common features to let
developers focus on their workflow under test.
Touca SDK for C++ provides a separate lower-level Client API that offers more flexibility and control over how tests are executed and how their results are handled. This API is most useful when integrating Touca with other existing test frameworks.
#include "touca/touca.hpp"
#include "students_test.hpp"
int main() {
touca::configure({
{ "api-key", "<TOUCA_API_KEY>" },
{ "api-url", "<TOUCA_API_URL>" },
{ "version", "<TOUCA_TEST_VERSION>" }
});
for (const username of touca::get_testcases()) {
touca::declare_testcase(username);
const auto& student = find_student(username);
// insert code here to describe the behavior
// and performance of the workflow under test
touca::post();
touca::save_binary("touca_" + username + ".bin");
touca::save_json("touca_" + username + ".json");
touca::forget_testcase(username);
}
touca::seal();
}
The above code uses the low-level Touca Client API to perform the same operations as the Touca test framework, without handling errors, reporting progress, and handling command line arguments. In this section, we will review the functions used in this code and explain what they do.
Configuring the Client
Touca Client requires a one-time call of function configure
. This
configuration effectively activates all other Touca functions for capturing data
and submission of results. Therefore, this function must be called from our test
tool, and not from our code under test. This design enables us to leave the
calls to Touca data capturing functions in our production code without having to
worry about their performance impact.
The configure
function can take various configuration parameters including the
Touca API Key and API URL. You can also specify an external JSON configuration
file via the file
option. Check out C++ SDK reference API documentation for
the full list of acceptable configuration parameters and their impact.
touca::configure({
{ "api-key": "<TOUCA_API_KEY>" },
{ "api-url": "<TOUCA_API_URL>" },
{ "revision": "<TOUCA_TEST_VERSION>" },
});
Touca API Key should be treated as a secret. We advise against hard-coding this parameter.
The three common parameters, API Key, API URL, and version of the code under
test can also be set as environment variables TOUCA_API_KEY
, TOUCA_API_URL
,
and TOUCA_TEST_VERSION
. Environment variables always override the parameters
passed to the configure
function.
All of the configuration parameters passed to configure
are optional. When
api_key
and api_url
are missing, the client is configured in the offline
mode. It can still capture data and store them to files but it will not submit
them to the Touca server.
You can always force the client to run in offline mode by passing the offline
parameter to the configure
function.
Preparing Test Cases
for (const username of touca::get_testcases()) {
// insert the code to run for each test case
}
The test framework expects test cases to be specified via the Touca server UI or via command line arguments. With the Client API, you can obtain the list of test cases from any source and pass them, one by one, to your code under test using a simple for loop.
You can still use the function get_testcases
to obtain the list of test cases
from the Touca server, as our high-level API does. This function should be
called when the client is configured to run in offline mode.
Declaring Test Cases
Once the client is configured, you can call declare_testcase
once for each
test case to indicate that all subsequent captured data and performance
benchmarks belong to the specified test case, until a different test case is
declared. We can change this behavior for multi-threaded software workflows.
for (const username of touca::get_testcases()) {
touca::declare_testcase(username);
// now we can start calling our code under test
// and describing its behavior and performance
}
With Touca, we consider test cases as a set of unique names that identify different inputs to our code under test. These inputs can be anything as long as they are expected to produce the same behavior every time our code is executed.
Similar to touca::configure
, we should only call touca::declare_testcase
from our test tool, and not from our code under test.
Capturing Test Results
In the previous document, we reviewed the main Touca functions for describing behavior and performance of our code under test, by capturing values of important variables and runtime of interesting functions. In this section, we dive a little deeper to explain how Touca tracks values of variables and performance benchmarks.
Preserving Data Types
Touca data capturing functions such as touca::check
, preserve the types of all
captured data so that the Touca server can compare them in their original type.
touca::check("username", student.username);
touca::check("fullname", student.fullname);
touca::check("gpa", student.gpa);
In the example above, touca::check
stores value of properties username
and
fullname
as std::string
while property gpa
is stored as float
. The
server visualizes possible differences in these values based on their types.
Customizing Data Serialization
Touca C++ SDK has built-in support for many commonly-used types of the standard library and can be extended to support custom data types.
Consider the following definition for a user-defined type Date
.
struct Date {
unsigned short year;
unsigned short month;
unsigned short day;
};
Using the natively supported types, we can add a value of type Date
as three
separate test results that each cover individual member variables. But this
practice is cumbersome and impractical for real-world complex data types. To
solve this, we can extend Touca type system to support type Date
using partial
template specialization.
#include "touca/touca.hpp"
template <>
struct touca::serializer<Date> {
data_point serialize(const Date& value) {
return object("Date")
.add("year", value.year)
.add("month", value.month)
.add("day", value.day);
}
};
Once the client library learns how to handle a custom type, it automatically
supports handling it as sub-component of other types. As an example, with the
above-mentioned partial template specialization for type Date
, we can start
adding test results of type std::vector<Date>
or std::map<string, Date>
.
touca::check("birth_date", student.dob);
Consult with the Reference API documentation for more information and examples for supporting custom types.
Submitting Test Results
Once we execute our code under test for each test case and describe its behavior and performance, we can submit them to the Touca server.
touca::post();
The server stores the captured data, compares them against the submitted data for pervious versions of our code, visualizes any differences, and reports them in real-time.
It is possible to call touca::post
multiple times during the runtime of our
test tool. Test cases already submitted to the Touca server whose results have
not changed, will not be resubmitted. It is also possible to add new results for
an already submitted test case. Any subsequent call to touca::post
will
resubmit the modified test cases.
We generally recommend that touca::post
be called every time the code under
test is executed for a given test case. This practice ensures real-time feedback
about the test results, as they are being executed.
Storing Test Results
We can choose to store captured test results and performance benchmarks for one or more of our declared test cases on the local filesystem for further processing or later submission to the Touca server.
touca::save_binary("touca_" + username + ".bin");
touca::save_json("touca_" + username + ".json");
We can store captured data in JSON or binary format using touca::save_json
or
touca::save_binary
respectively. While JSON files are preferable for quick
inspections, only binary files may be posted to the Touca server at a later
time.
Forgetting Test Cases
If you are submitted thousands of test cases for each version of your workflow
and capture significant amount of information for each test case, you can use
touca::forget_testcase
to release all the captured information from process
memory, when you are done with a given test case.
touca::forget_testcase();
Sealing Test Results
When all the test cases are executed for a given version of our code under test, we have the option to seal the version to let the server know that no further test result is expected to be submitted for it. This allows the server to send the final comparison result report to interested users, as soon as it is available.
touca::seal();
Sealing the version is optional. The Touca server automatically performs this operation once a certain amount of time has passed since the last test case was submitted.