pcc/ standards and regression testing

Please document on this page about methods for standards testing, compiler validation, and regression testing.

OpenBSD regress

OpenBSD's CVS contains code for regression tests for pcc at /usr/src/regress/usr.bin/pcc/.

TODO: please document how to use OpenBSD regress

See bsd.regress.mk(5). You can simply do "make regress".

GCC Testsuite

GCC Testsuite can also be used. See here for details.

mcpp's Validation Suite for Standard C Conformance of Preprocessing

The BSD-licensed mcpp (Matsui cpp) at http://mcpp.sourceforge.net/ provides the "Validation Suite for Standard C Conformance of Preprocessing".

There are three directories containing tests: test-t, test-l, test-c. In test-t most of the files end with .t suffix. These should be preprocessed (not compiled) one by one. First look at the content of a file. A comment inside tells you which feature is tested and what should be the result. Then run the file through preprocessor and inspect the output. Skip the files which test C++ preprocessor features (the files with .cc suffix and some .t files too). I'm not aware of any way how to automate tests in test-t directory.

TODO: explore tests in test-l directory.

test-c directory contains C source files, which should be preprocessed by the tested preprocessor, compiled by any compiler and then executed. List of available tests is written in n_i_.lst file. Each test when executed prints "started" and then another line "success" if the test is passed successfully. Output of each test is stored in a temporary file. Those temp files are later analyzed by cpp_test tool, which considers the test as successfull, if the coresponding temp file contains two lines "started" and "success". So cpp_test tool automates testing, but only for tests in test-c directory (only for compiled tests). To compile cpp_test tool simply enter the tool directory and type:

cc cpp_test.c -o cpp_test

Then go to the test-c directory and run the tests in automated fashion (here we assume that we test PCC compiler):

../tool/cpp_test PCC "/usr/local/bin/pcc -D__GNUC__ -o %s %s.c" "rm %s" < n_i_.lst

First argument is a name of tested compiler (can be whatever you want), second argument is a command used to compile a test program, third is for cleaning up compiled binaries and on standard input is expected a list of files (tests) which should be compiled.

NOTE: The __GNUC__ define is a hack, which I used on NetBSD, to avoid function renaming error in defs.h. Is there a more clean way how to solve this? I was told that this issue was fixed in NetBSD-current.

The summary is available in file PCC.sum, where on the last line is written number of tests which failed.

Another example. To test gcc compiler type:

../tool/cpp_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n_i_.lst

Automated testing gives you just an overview of how good/bad your compiler is. To debug the problems, you have to run preprocessor on test file manually.

The very long ./doc/cpp-test.html document provides details how to run cpp_test tool.

The ./doc/mcpp-summary.pdf file describes the results of the tests versus several cpp implementations.

TODO:

Pkgsrc

Pkgsrc package build system can be used with pcc compiler if you want a way to quickly test "pcc" for a lot of "third-party" software (including doing entire bulk builds attempting thousands of packages):

PKGSRC_COMPILER=pcc

Also set PCCBASE to where your bin/pcc is at (/usr/pkg is the default).

Note that NetBSD's pkgsrc package build system is portable and is used as the primary package build system for DragonFly and some distributions of Linux as well as NetBSD. It also has support for several operating systems.

pkgsrc also has support for selecting a custom compiler suite, including: pcc, Tru64 ccc, Intel icc, SGI IRIS, HPUX compiler, SGI's MIPSpro 64 and 32 compilers, Sun WorkShip/Forte/Sun ONE Studio, GCC, and IBM's XL compiler. It also has support for psuedo compilers: ccache, distcc and f2c.

Also cross-build support for modular X.org and its dependencies in pkgsrc is also being worked on.

(So pkgsrc with "pcc" seems like a good way to do lots of tests.)

LCC test suite

http://drh.svnrepository.com/svn/lcc/trunk/tst/

Compile foo.c and pipe foo.0 to it. And compare the output with foo.1bk.

Details are in the "Building the Compiler and Accessories" section in the Installing lcc Installation guide.

Commercial Compiler Testing

Click here for commercial services that provide compiler validation and testing.

The NIST F77 Testsuite

To test the F77 implementation the NIST Testsuite is available from http://troutmask.apl.washington.edu/~kargl/nist.tgz.

Using other programs testsuites

There are quite a bunch of other programs that provide testsuites, and running them can often reveal deeply hidden bugs in the compiler.
Among the programs often used to test pcc are: