Back to wiki

RCS History

/ TOP / standards_and_regression_testing.mdwn

Revision 1.19


Please document on this page about methods for standards testing, compiler validation,
and regression testing.

### OpenBSD regress

OpenBSD's CVS contains code for regression tests for pcc at [/usr/src/regress/usr.bin/pcc/](http://www.openbsd.org/cgi-bin/cvsweb/src/regress/usr.bin/pcc/).

TODO: please document how to use OpenBSD regress

See [bsd.regress.mk(5)](http://www.openbsd.org/cgi-bin/man.cgi?query=bsd.regress.mk&sektion=5&manpath=OpenBSD+Current).
You can simply do "make regress".

### GCC Testsuite

[[GCC_Testsuite]] can also be used. See [[here|GCC_Testsuite]] for details.

### mcpp's Validation Suite for Standard C Conformance of Preprocessing

The BSD-licensed mcpp (Matsui cpp) at <http://mcpp.sourceforge.net/>
provides the "Validation Suite for Standard C Conformance of Preprocessing".

There are three directories containing tests: test-t, test-l, test-c.
In test-t most of the files end with .t suffix. These should be preprocessed (not
compiled) one by one. First look at the content of a file. A comment
inside tells you which feature is tested and what should be the result.
Then run the file through preprocessor and inspect the output. Skip
the files which test C++ preprocessor features (the files with .cc suffix
and some .t files too). I'm not aware of any way how to automate tests
in test-t directory.

 *TODO: explore tests in test-l directory*.

test-c directory contains C source files, which should be preprocessed
by the tested preprocessor, compiled by any compiler and then executed.
List of available tests is written in n\_i\_.lst file. Each test when
executed prints "started" and then another line "success" if the test
is passed successfully. Output of each test is stored in a temporary file.
Those temp files are later analyzed by cpp\_test tool, which considers
the test as successfull, if the coresponding temp file contains two lines
"started" and "success". So cpp\_test tool automates testing, but **only** for
tests in test-c directory (only for compiled tests). To compile cpp\_test
tool simply enter the tool directory and type:

    cc cpp_test.c -o cpp_test

Then go to the test-c directory and run the tests in automated fashion
(here we assume that we test PCC compiler):

    ../tool/cpp_test PCC "/usr/local/bin/pcc -D__GNUC__ -o %s %s.c" "rm %s" < n_i_.lst

First argument is a name of tested compiler (can be whatever you want),
second argument is a command used to compile a test program, third is for cleaning
up compiled binaries and on standard input is expected a list of files (tests)
which should be compiled.

 *NOTE: The \_\_GNUC\_\_ define is a hack, which I used on NetBSD, to avoid
function renaming error in defs.h. Is there a more clean way how to
solve this? I was told that this issue was fixed in NetBSD-current.*

The summary is available in file PCC.sum, where on the last line is
written number of tests which failed.

Another example. To test gcc compiler type:

    ../tool/cpp_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n_i_.lst

Automated testing gives you just an overview of how good/bad
your compiler is. To debug the problems, you have to run preprocessor
on test file manually.

The very long ./doc/cpp-test.html document provides details how to
run cpp\_test tool.

The ./doc/mcpp-summary.pdf file describes the results of the tests versus 
several cpp implementations.

TODO:

* test this and document here and send results back to the mcpp project to be included in their results (but I would fix the obvious bugs before doing that).
* Document the format of *.sum summary file.
      
### Pkgsrc

Pkgsrc package build system can be used with pcc compiler if you want a way to quickly test "pcc" for a lot of "third-party" software (including doing entire bulk builds attempting thousands of packages):

    PKGSRC_COMPILER=pcc

Also set `PCCBASE` to where your bin/pcc is at (/usr/pkg is the default).

Note that NetBSD's pkgsrc package build system is portable 
and is used as the primary package build system for DragonFly and some 
distributions of Linux as well as NetBSD. It also has support for several 
operating systems.

pkgsrc also has support for selecting a custom compiler suite, including: pcc, Tru64 ccc, 
Intel icc, SGI IRIS, HPUX compiler, SGI's MIPSpro 64 and 32 compilers, Sun 
WorkShip/Forte/Sun ONE Studio, GCC, and IBM's XL compiler. It also has support 
for psuedo compilers: ccache, distcc and f2c.

Also cross-build support for modular X.org and its dependencies in pkgsrc is also being worked on.

(So pkgsrc with "pcc" seems like a good way to do lots of tests.)

### LCC test suite

<http://drh.svnrepository.com/svn/lcc/trunk/tst/>

Compile foo.c and pipe foo.0 to it. And compare the output with [foo.1bk](http://drh.svnrepository.com/svn/lcc/trunk/x86/linux/tst/).

Details are in the "Building the Compiler and Accessories" section in the [Installing lcc](http://drh.svnrepository.com/svn/lcc/trunk/doc/install.html) Installation guide.


### Commercial Compiler Testing

Click [[here|Commercial_Validation]] for commercial services that provide compiler validation and testing.

### The NIST F77 Testsuite

To test the F77 implementation the NIST Testsuite is available from [http://troutmask.apl.washington.edu/~kargl/nist.tgz](http://troutmask.apl.washington.edu/~kargl/nist.tgz).

Powered by rcshistory.cgi 0.3