$Id: README 6942 2016-01-15 18:21:28Z gavin $
Copyright 2010, 2011, 2012, 2013, 2014, 2015, 2016
Free Software Foundation, Inc.
Copying and distribution of this README file, with or without
modification, are permitted in any medium without royalty provided the
copyright notice and this notice are preserved.
Files anywhere within the texinfo/tp/tests/ subdirectory which have no
other copyright notice are hereby placed in the public domain.
This directory contains tests of the texi2any program.
To run this testsuite, sed, awk, diff (with -u) are used.
For LaTeX2HTML tests (not run by default), mktemp
understanding the -d option is also required.
Some tests may also require a case insensitive filesystem.
Running the testsuite
This testsuite can be run one of the following ways:
* Using the makefile rules: "make check" (in this directory) for the
base tests, "make long-checks" for the long tests, or "make all-checks"
for both. These use the scripts under the test_scripts directory.
* Using ./parser_tests.sh with the directories one want to run the tests
in given as arguments, for example "./parser_tests.sh formatting".
Checking test results
The test results are in the out_parser directories; the reference
results are in res_parser directories. There can be these pairs.
- out_parser/ and res_parser/ for html output in texi2html style;
- out_parser_html/ and res_parser_html/ for html output in texi2any style;
- out_parser_info/ and res_parser_info/ for info results;
and diffs are put in the diffs/ subdir.
For example, to investigate a failure in the
test_scripts/layout_formaatting_docbook.sh test, compare the contents of the
files in the layout_res_parser/formatting_docbook and
layout_out_parser/formatting_docbook directories, e.g.
Also look in the layout/diffs directory.
If the test results are as expected, copy the output files onto the
reference files. Otherwise, if they need more investigation, see "Tests
specification" below, to find out what input file, etc. was used for the
test. Sometimes you need to use "makeinfo -F" to force an output file
to be created.
When running tests via parser_tests.sh or run_parser_all.sh, then
a test failure causes
to be printed. If there is a diff with the references, then
is printed. If there are no reference to compare with, then
no res: a_test_directory
If a whole directory check run using './parser_tests.sh' had no failure,
the directory name is printed followed by a fail; it is followed
by a ok if there was a failure.
The test results are in a directory below the out directories. This
directory name can be considered as the test name. Let's call it a_test
for the remainder of the explanation. The stderr output is in
out_parser/a_test/a_test.2, stdout output is in
out_parser/a_test/a_test.1. All the commands called are output in
Regenerating test results
Giving -copy as the first argument to ../run_parser_all.sh or
parser_tests.sh (or long_tests.sh or tex_html_tests.sh, which need
environment variables LONG_TESTS resp. TEX_HTML_TESTS, or ALL_TESTS, to
be set to "yes", but otherwise operate similarly) will cause the
reference results to be regenerated. For example, to update the
reference results for one particular test:
$ ./parser_tests.sh -copy formatting
The copy-* make targets update all the references for various large sets
of tests via this method.
To update a group of long tests:
$ LONG_TESTS=yes ./parser_tests.sh -copy nested_formats
Exception: in the many_input_files subdirectory, the *parser*.sh scripts
can't be used for updating. Instead, the Makefile's copy-tests and
copy-tex-html targets do it directly.
TeX4ht and LaTeX2HTML tests
The tex4ht and latex2HTML related tests are not run automatically, as
some test results involving latex2HTML or tex4ht depend on the setup and
version of these tools.
These tests may be run by
For the tests that use latex2html, to avoid the test failing if there is
a dot in the cwd, mktemp is used to create a temporary directory, and
the directory is passed through on the command line.
Test runs are driven by the contents of the tests-parser.txt file in
each subdirectory. This is a line-oriented file. A # starts a
comment, except that the very first line of the file can be, e.g.,
# formats :html
which says which formats to generate.
Otherwise, each non-empty non-comment line describes a test. The first
word on the line is the test name which corresponds also with the
resulting directory (what we called a_test above). It is followed by
the source manual name. The source manual name has to have the .texi
extension. Optionally, additional arguments can be given on the rest of
So, for example, the line
specifies that for the test a_test, the file manual.texi is processed
and results are put are put in the a_test directory. The line:
a_test_split_chapter manual.texi --split chapter
specifies that the results of the processing of manual.texi with additional
command line arguments --split chapter will be in the a_test_split_chapter
Using "texi" as the test name is special. In that case the directory name
is constructed by appending _ followed by the manual name without the .texi
extension to "texi". And the result is not processed normally, but instead
macros are expanded and regions that are not processed removed and the
result is put in a file with the same name as the manual.
results in the following: in the directory out_parser/texi_manual the file
manual.texi will have macros expanded and ignored regions removed.
A common .texi file should be in the top-level directory (there is such an
example with coverage_macro.texi).
Init files are searched for in ../t/init/.
There are three subdirectories containing tests, each of which is
organized like this one: htmlxref, htmlxref-only_mono and htmlxref-only_split.
These groups of tests are in their own subdirectorie so they can use
their own htmlxref.cnf init file (which is copied to the build dir in
tests/Makefile.am). In htmlxref-only_split, using a .texinfo directory
Creating a new test
- choose the subdir here for the new test. Each subdir only generates
output in one or two formats (according to the `formats' line in
its tests-parser.txt), so if you want to test a particular output
format, use the right place.
- add the line to subdir/tests-parser.txt as described above.
- most probably, create the needed subdir/*.texi file
- cd subdir; ../run_parser_all.sh newtest
This will create subdir/out_parser*/newtest/ with the test results
(as explained above), as well as stdout in newtest.1 and stderr in newtest.2.
Look carefully to be sure they are as they should be.
For XML output, perhaps check validity using the commands in tp/TODO.
When things look good, use the -copy option to create the
subdir/res_parser*/newtest/ subdirs and copy the output files there.
The test run will be logged in subdir/test_log/newtest.log,
including the exact invocation of texi2any, which can alter anything.
For even more, use sh -vx to show exactly what run_parser_all is doing.
When satisfied, add the .texi in subdir/Makefile.am.
Of course commit all the files and try a make check (or long-checks,
as needed) afterward to verify it gets run, and passes.
See tp/t/README for info about those tests.