Back to wiki

RCS History

/ TOP / standards_and_regression_testing.mdwn

Differences for revision 1.15 from 1.14

--- /standards_and_regression_testing.mdwn	2007/10/05 22:02:09	1.14
+++ /standards_and_regression_testing.mdwn	2007/10/26 00:06:19	1.15
@@ -19,14 +19,55 @@
 The BSD-licensed mcpp (Matsui cpp) at <>
 provides the "Validation Suite for Standard C Conformance of Preprocessing".
-It comes with cpp\_test tool that can be used to test 265 tests and 
-provides scores. The very long ./doc/cpp-test.html document provides 
+There are three directories containing tests: test-t, test-l, test-c.
+In test-t are files with *.t suffix. These should be preprocessed (not
+compiled) one by one. First look at the content of a file. A comment
+inside tells you which feature is tested and what should be the result.
+Then run the file through preprocessor and inspect the output. Skip
+the files which test C++ preprocessor features. There's currently no way
+how to automate execution of these tests.
+test-c directory contains C source files, which should be compiled
+by the tested compiler and then executed. At the beginning the test
+prints "started" and upon successfull completion is printed "success".
+List of available tests is written in n\_i\_.lst file. This process
+can be automated by cpp\_test tool in tool directory. To compile this
+tool simply enter the tool directory and type:
+ $ cc cpp\_test.c -o cpp\_test
+Then go to the test-c directory and run the tests in automated fashion
+(here we assume that we test PCC compiler):
+ $ ../tool/cpp\_test PCC "/usr/local/bin/pcc -D\_\_GNUC\_\_ -o %s %s.c" "rm %s" < n\_i\_.lst
+First argument is a name of tested compiler (can be whatever you want),
+second argument is a command to compile a test, third is how to delete
+the compiled test and on standard input is expected a list of files (tests)
+which should be compiled.
+NOTE: The \_\_GNUC\_\_ define is a hack, which I used on NetBSD, to avoid
+function renaming error in defs.h. Is there a more clean way how to
+solve this?
+The summary is available in file PCC.sum, where on the last line is
+written number of tests which failed.
+Another example. To test gcc compiler type:
+ $ ../tool/cpp\_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n\_i\_.lst
+Automated testing gives you just an overview of how good/bad
+your compiler is. To debug the problems, you have to run preprocessor
+on test file manually.
+The very long ./doc/cpp-test.html document provides 
 details how to run cpp\_test tool.
 The ./doc/mcpp-summary.pdf file describes the results of the tests versus 
 several cpp implementations.
-TODO: test this and document here and send results back to the mcpp project to be included in their results.
+TODO: test this and document here and send results back to the mcpp project to be included in their results (but I would fix the obvious bugs before doing that).
 ### Pkgsrc

Powered by rcshistory.cgi 0.3