Back to wiki

RCS History

/ TOP / standards_and_regression_testing.mdwn

Revision 1.15


Please document on this page about methods for standards testing, compiler validation,
and regression testing.

### OpenBSD regress

OpenBSD's CVS contains code for regression tests for pcc at [/usr/src/regress/usr.bin/pcc/](http://www.openbsd.org/cgi-bin/cvsweb/src/regress/usr.bin/pcc/).

TODO: please document how to use OpenBSD regress

See [bsd.regress.mk(5)](http://www.openbsd.org/cgi-bin/man.cgi?query=bsd.regress.mk&sektion=5&manpath=OpenBSD+Current).
You can simply do "make regress".

### GCC Testsuite

[[GCC_Testsuite]] can also be used. See [[here|GCC_Testsuite]] for details.

### mcpp's Validation Suite for Standard C Conformance of Preprocessing

The BSD-licensed mcpp (Matsui cpp) at <http://mcpp.sourceforge.net/>
provides the "Validation Suite for Standard C Conformance of Preprocessing".

There are three directories containing tests: test-t, test-l, test-c.
In test-t are files with *.t suffix. These should be preprocessed (not
compiled) one by one. First look at the content of a file. A comment
inside tells you which feature is tested and what should be the result.
Then run the file through preprocessor and inspect the output. Skip
the files which test C++ preprocessor features. There's currently no way
how to automate execution of these tests.

test-c directory contains C source files, which should be compiled
by the tested compiler and then executed. At the beginning the test
prints "started" and upon successfull completion is printed "success".
List of available tests is written in n\_i\_.lst file. This process
can be automated by cpp\_test tool in tool directory. To compile this
tool simply enter the tool directory and type:

 $ cc cpp\_test.c -o cpp\_test

Then go to the test-c directory and run the tests in automated fashion
(here we assume that we test PCC compiler):

 $ ../tool/cpp\_test PCC "/usr/local/bin/pcc -D\_\_GNUC\_\_ -o %s %s.c" "rm %s" < n\_i\_.lst

First argument is a name of tested compiler (can be whatever you want),
second argument is a command to compile a test, third is how to delete
the compiled test and on standard input is expected a list of files (tests)
which should be compiled.

NOTE: The \_\_GNUC\_\_ define is a hack, which I used on NetBSD, to avoid
function renaming error in defs.h. Is there a more clean way how to
solve this?

The summary is available in file PCC.sum, where on the last line is
written number of tests which failed.

Another example. To test gcc compiler type:

 $ ../tool/cpp\_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n\_i\_.lst

Automated testing gives you just an overview of how good/bad
your compiler is. To debug the problems, you have to run preprocessor
on test file manually.

The very long ./doc/cpp-test.html document provides 
details how to run cpp\_test tool.

The ./doc/mcpp-summary.pdf file describes the results of the tests versus 
several cpp implementations.

TODO: test this and document here and send results back to the mcpp project to be included in their results (but I would fix the obvious bugs before doing that).

### Pkgsrc

Pkgsrc package build system can be used with pcc compiler if you want a way to quickly test "pcc" for a lot of "third-party" software (including doing entire bulk builds attempting thousands of packages):

    PKGSRC_COMPILER=pcc

Also set `PCCBASE` to where your bin/pcc is at (/usr/pkg is the default).

Note that NetBSD's pkgsrc package build system is portable 
and is used as the primary package build system for DragonFly and some 
distributions of Linux as well as NetBSD. It also has support for several 
operating systems.

pkgsrc also has support for selecting a custom compiler suite, including: pcc, Tru64 ccc, 
Intel icc, SGI IRIS, HPUX compiler, SGI's MIPSpro 64 and 32 compilers, Sun 
WorkShip/Forte/Sun ONE Studio, GCC, and IBM's XL compiler. It also has support 
for psuedo compilers: ccache, distcc and f2c.

Also cross-build support for modular X.org and its dependencies in pkgsrc is also being worked on.

(So pkgsrc with "pcc" seems like a good way to do lots of tests.)

### LCC test suite

<http://drh.svnrepository.com/svn/lcc/trunk/tst/>

Compile foo.c and pipe foo.0 to it. And compare the output with [foo.1bk](http://drh.svnrepository.com/svn/lcc/trunk/x86/linux/tst/).

Details are in the "Building the Compiler and Accessories" section in the [Installing lcc](http://drh.svnrepository.com/svn/lcc/trunk/doc/install.html) Installation guide.


### Commercial Compiler Testing

Click [[here|Commercial_Validation]] for commercial services that provide compiler validation and testing.

Powered by rcshistory.cgi 0.3