The test driver must correctly generate the files specified by the --log-file and --trs-file option (even when the tested program fails or crashes).
The .log file should ideally contain all the output produced by the tested program, plus optionally other information that might facilitate debugging or analysis of bug reports. Apart from that, its format is basically free.
The .trs file is used to register some metadata through the use
of custom reStructuredText fields. This metadata is expected to be
employed in various ways by the parallel test harness; for example, to
count the test results when printing the testsuite summary, or to decide
which tests to rerun upon make recheck. Unrecognized metadata
in a .trs file is currently ignored by the harness, but this might
change in the future. The list of currently recognized metadata follows.
:test-result: ¶The test driver must use this field to register the results of each
test case run by a test script file. Several :test-result: fields
can be present in the same .trs file; this is done in order to
support test protocols that allow a single test script to run more test
cases.
The only recognized test results are currently PASS, XFAIL,
SKIP, FAIL, XPASS and ERROR. These results,
when declared with :test-result:, can be optionally followed by
text holding the name and/or a brief description of the corresponding
test; the harness will ignore such extra text when generating
test-suite.log and preparing the testsuite summary.
:recheck:If this field is present and defined to no, then the corresponding
test script will not be run upon a make recheck. What
happens when two or more :recheck: fields are present in the same
.trs file is undefined behavior.
:copy-in-global-log:If this field is present and defined to no, then the content
of the .log file will not be copied into the global
test-suite.log. We allow to forsake such copying because, while
it can be useful in debugging and analysis of bug report, it can also be
just a waste of space in normal situations, e.g., when a test script is
successful. What happens when two or more :copy-in-global-log:
fields are present in the same .trs file is undefined behavior.
:test-global-result:This is used to declare the "global result" of the script. Currently,
the value of this field is needed only to be reported (more or less
verbatim) in the generated global log file $(TEST_SUITE_LOG),
so it’s quite free-form. For example, a test script which runs 10 test
cases, 6 of which pass and 4 of which are skipped, could reasonably have
a PASS/SKIP value for this field, while a test script which runs
19 successful tests and one failed test could have an ALMOST
PASSED value. What happens when two or more :test-global-result:
fields are present in the same .trs file is undefined behavior.
Let’s see a small example. Assume a .trs file contains the following lines:
:test-result: PASS server starts :global-log-copy: no :test-result: PASS HTTP/1.1 request :test-result: FAIL HTTP/1.0 request :recheck: yes :test-result: SKIP HTTPS request (TLS library wasn't available) :test-result: PASS server stops
Then the corresponding test script will be rerun by make check,
will contribute with five test results to the testsuite summary
(three of these tests being successful, one failed, and one skipped), and
the content of the corresponding .log file will not be
copied into the global log file test-suite.log.