--- standards_and_regression_testing.mdwn 2007/10/05 22:02:09 1.14 +++ standards_and_regression_testing.mdwn 2007/10/26 00:06:19 1.15 @@ -19,14 +19,55 @@ The BSD-licensed mcpp (Matsui cpp) at <http://mcpp.sourceforge.net/> provides the "Validation Suite for Standard C Conformance of Preprocessing". -It comes with cpp\_test tool that can be used to test 265 tests and -provides scores. The very long ./doc/cpp-test.html document provides +There are three directories containing tests: test-t, test-l, test-c. +In test-t are files with *.t suffix. These should be preprocessed (not +compiled) one by one. First look at the content of a file. A comment +inside tells you which feature is tested and what should be the result. +Then run the file through preprocessor and inspect the output. Skip +the files which test C++ preprocessor features. There's currently no way +how to automate execution of these tests. + +test-c directory contains C source files, which should be compiled +by the tested compiler and then executed. At the beginning the test +prints "started" and upon successfull completion is printed "success". +List of available tests is written in n\_i\_.lst file. This process +can be automated by cpp\_test tool in tool directory. To compile this +tool simply enter the tool directory and type: + + $ cc cpp\_test.c -o cpp\_test + +Then go to the test-c directory and run the tests in automated fashion +(here we assume that we test PCC compiler): + + $ ../tool/cpp\_test PCC "/usr/local/bin/pcc -D\_\_GNUC\_\_ -o %s %s.c" "rm %s" < n\_i\_.lst + +First argument is a name of tested compiler (can be whatever you want), +second argument is a command to compile a test, third is how to delete +the compiled test and on standard input is expected a list of files (tests) +which should be compiled. + +NOTE: The \_\_GNUC\_\_ define is a hack, which I used on NetBSD, to avoid +function renaming error in defs.h. Is there a more clean way how to +solve this? + +The summary is available in file PCC.sum, where on the last line is +written number of tests which failed. + +Another example. To test gcc compiler type: + + $ ../tool/cpp\_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n\_i\_.lst + +Automated testing gives you just an overview of how good/bad +your compiler is. To debug the problems, you have to run preprocessor +on test file manually. + +The very long ./doc/cpp-test.html document provides details how to run cpp\_test tool. The ./doc/mcpp-summary.pdf file describes the results of the tests versus several cpp implementations. -TODO: test this and document here and send results back to the mcpp project to be included in their results. +TODO: test this and document here and send results back to the mcpp project to be included in their results (but I would fix the obvious bugs before doing that). ### Pkgsrc
Powered by rcshistory.cgi 0.3