Back to wiki
Please document on this page about methods for standards testing, compiler validation, and regression testing. ### OpenBSD regress OpenBSD's CVS contains code for regression tests for pcc at [/usr/src/regress/usr.bin/pcc/](http://www.openbsd.org/cgi-bin/cvsweb/src/regress/usr.bin/pcc/). TODO: please document how to use OpenBSD regress See [bsd.regress.mk(5)](http://www.openbsd.org/cgi-bin/man.cgi?query=bsd.regress.mk&sektion=5&manpath=OpenBSD+Current). You can simply do "make regress". ### GCC Testsuite [[GCC_Testsuite]] can also be used. See [[here|GCC_Testsuite]] for details. ### mcpp's Validation Suite for Standard C Conformance of Preprocessing The BSD-licensed mcpp (Matsui cpp) at <http://mcpp.sourceforge.net/> provides the "Validation Suite for Standard C Conformance of Preprocessing". There are three directories containing tests: test-t, test-l, test-c. In test-t most of the files end with .t suffix. These should be preprocessed (not compiled) one by one. First look at the content of a file. A comment inside tells you which feature is tested and what should be the result. Then run the file through preprocessor and inspect the output. Skip the files which test C++ preprocessor features (the files with .cc suffix and some .t files too). I'm not aware of any way how to automate tests in test-t directory. *TODO: explore tests in test-l directory*. test-c directory contains C source files, which should be preprocessed by the tested preprocessor, compiled by any compiler and then executed. List of available tests is written in n\_i\_.lst file. Each test when executed prints "started" and then another line "success" if the test is passed successfully. Output of each test is stored in a temporary file. Those temp files are later analyzed by cpp\_test tool, which considers the test as successfull, if the coresponding temp file contains two lines "started" and "success". So cpp\_test tool automates testing, but **only** for tests in test-c directory (only for compiled tests). To compile cpp\_test tool simply enter the tool directory and type: cc cpp_test.c -o cpp_test Then go to the test-c directory and run the tests in automated fashion (here we assume that we test PCC compiler): ../tool/cpp_test PCC "/usr/local/bin/pcc -D__GNUC__ -o %s %s.c" "rm %s" < n_i_.lst First argument is a name of tested compiler (can be whatever you want), second argument is a command used to compile a test program, third is for cleaning up compiled binaries and on standard input is expected a list of files (tests) which should be compiled. *NOTE: The \_\_GNUC\_\_ define is a hack, which I used on NetBSD, to avoid function renaming error in defs.h. Is there a more clean way how to solve this? I was told that this issue was fixed in NetBSD-current.* The summary is available in file PCC.sum, where on the last line is written number of tests which failed. Another example. To test gcc compiler type: ../tool/cpp_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n_i_.lst Automated testing gives you just an overview of how good/bad your compiler is. To debug the problems, you have to run preprocessor on test file manually. The very long ./doc/cpp-test.html document provides details how to run cpp\_test tool. The ./doc/mcpp-summary.pdf file describes the results of the tests versus several cpp implementations. TODO: * test this and document here and send results back to the mcpp project to be included in their results (but I would fix the obvious bugs before doing that). * Document the format of *.sum summary file. ### Pkgsrc Pkgsrc package build system can be used with pcc compiler if you want a way to quickly test "pcc" for a lot of "third-party" software (including doing entire bulk builds attempting thousands of packages): PKGSRC_COMPILER=pcc Also set `PCCBASE` to where your bin/pcc is at (/usr/pkg is the default). Note that NetBSD's pkgsrc package build system is portable and is used as the primary package build system for DragonFly and some distributions of Linux as well as NetBSD. It also has support for several operating systems. pkgsrc also has support for selecting a custom compiler suite, including: pcc, Tru64 ccc, Intel icc, SGI IRIS, HPUX compiler, SGI's MIPSpro 64 and 32 compilers, Sun WorkShip/Forte/Sun ONE Studio, GCC, and IBM's XL compiler. It also has support for psuedo compilers: ccache, distcc and f2c. Also cross-build support for modular X.org and its dependencies in pkgsrc is also being worked on. (So pkgsrc with "pcc" seems like a good way to do lots of tests.) ### LCC test suite <http://drh.svnrepository.com/svn/lcc/trunk/tst/> Compile foo.c and pipe foo.0 to it. And compare the output with [foo.1bk](http://drh.svnrepository.com/svn/lcc/trunk/x86/linux/tst/). Details are in the "Building the Compiler and Accessories" section in the [Installing lcc](http://drh.svnrepository.com/svn/lcc/trunk/doc/install.html) Installation guide. ### Commercial Compiler Testing Click [[here|Commercial_Validation]] for commercial services that provide compiler validation and testing.
Powered by rcshistory.cgi 0.3