Back to wiki
Please document on this page about methods for standards testing, compiler validation, and regression testing. ### OpenBSD regress OpenBSD's CVS contains code for regression tests for pcc at [/usr/src/regress/usr.bin/pcc/](http://www.openbsd.org/cgi-bin/cvsweb/src/regress/usr.bin/pcc/). TODO: please document how to use OpenBSD regress See [bsd.regress.mk(5)](http://www.openbsd.org/cgi-bin/man.cgi?query=bsd.regress.mk&sektion=5&manpath=OpenBSD+Current). You can simply do "make regress". ### GCC Testsuite [[GCC_Testsuite]] can also be used. See [[here|GCC_Testsuite]] for details. ### mcpp's Validation Suite for Standard C Conformance of Preprocessing The BSD-licensed mcpp (Matsui cpp) at <http://mcpp.sourceforge.net/> provides the "Validation Suite for Standard C Conformance of Preprocessing". There are three directories containing tests: test-t, test-l, test-c. In test-t are files with *.t suffix. These should be preprocessed (not compiled) one by one. First look at the content of a file. A comment inside tells you which feature is tested and what should be the result. Then run the file through preprocessor and inspect the output. Skip the files which test C++ preprocessor features. There's currently no way how to automate execution of these tests. test-c directory contains C source files, which should be compiled by the tested compiler and then executed. At the beginning the test prints "started" and upon successfull completion is printed "success". List of available tests is written in n\_i\_.lst file. This process can be automated by cpp\_test tool in tool directory. To compile this tool simply enter the tool directory and type: $ cc cpp\_test.c -o cpp\_test Then go to the test-c directory and run the tests in automated fashion (here we assume that we test PCC compiler): $ ../tool/cpp\_test PCC "/usr/local/bin/pcc -D\_\_GNUC\_\_ -o %s %s.c" "rm %s" < n\_i\_.lst First argument is a name of tested compiler (can be whatever you want), second argument is a command to compile a test, third is how to delete the compiled test and on standard input is expected a list of files (tests) which should be compiled. NOTE: The \_\_GNUC\_\_ define is a hack, which I used on NetBSD, to avoid function renaming error in defs.h. Is there a more clean way how to solve this? The summary is available in file PCC.sum, where on the last line is written number of tests which failed. Another example. To test gcc compiler type: $ ../tool/cpp\_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n\_i\_.lst Automated testing gives you just an overview of how good/bad your compiler is. To debug the problems, you have to run preprocessor on test file manually. The very long ./doc/cpp-test.html document provides details how to run cpp\_test tool. The ./doc/mcpp-summary.pdf file describes the results of the tests versus several cpp implementations. TODO: test this and document here and send results back to the mcpp project to be included in their results (but I would fix the obvious bugs before doing that). ### Pkgsrc Pkgsrc package build system can be used with pcc compiler if you want a way to quickly test "pcc" for a lot of "third-party" software (including doing entire bulk builds attempting thousands of packages): PKGSRC_COMPILER=pcc Also set `PCCBASE` to where your bin/pcc is at (/usr/pkg is the default). Note that NetBSD's pkgsrc package build system is portable and is used as the primary package build system for DragonFly and some distributions of Linux as well as NetBSD. It also has support for several operating systems. pkgsrc also has support for selecting a custom compiler suite, including: pcc, Tru64 ccc, Intel icc, SGI IRIS, HPUX compiler, SGI's MIPSpro 64 and 32 compilers, Sun WorkShip/Forte/Sun ONE Studio, GCC, and IBM's XL compiler. It also has support for psuedo compilers: ccache, distcc and f2c. Also cross-build support for modular X.org and its dependencies in pkgsrc is also being worked on. (So pkgsrc with "pcc" seems like a good way to do lots of tests.) ### LCC test suite <http://drh.svnrepository.com/svn/lcc/trunk/tst/> Compile foo.c and pipe foo.0 to it. And compare the output with [foo.1bk](http://drh.svnrepository.com/svn/lcc/trunk/x86/linux/tst/). Details are in the "Building the Compiler and Accessories" section in the [Installing lcc](http://drh.svnrepository.com/svn/lcc/trunk/doc/install.html) Installation guide. ### Commercial Compiler Testing Click [[here|Commercial_Validation]] for commercial services that provide compiler validation and testing.
Powered by rcshistory.cgi 0.3