Back to wiki

RCS History

/ TOP / standards_and_regression_testing.mdwn

Differences for revision 1.16 from 1.15


--- /standards_and_regression_testing.mdwn	2007/10/26 00:06:19	1.15
+++ /standards_and_regression_testing.mdwn	2007/10/26 08:36:50	1.16
@@ -20,55 +20,65 @@
 provides the "Validation Suite for Standard C Conformance of Preprocessing".
 
 There are three directories containing tests: test-t, test-l, test-c.
-In test-t are files with *.t suffix. These should be preprocessed (not
+In test-t most of the files end with .t suffix. These should be preprocessed (not
 compiled) one by one. First look at the content of a file. A comment
 inside tells you which feature is tested and what should be the result.
 Then run the file through preprocessor and inspect the output. Skip
-the files which test C++ preprocessor features. There's currently no way
-how to automate execution of these tests.
-
-test-c directory contains C source files, which should be compiled
-by the tested compiler and then executed. At the beginning the test
-prints "started" and upon successfull completion is printed "success".
-List of available tests is written in n\_i\_.lst file. This process
-can be automated by cpp\_test tool in tool directory. To compile this
+the files which test C++ preprocessor features (the files with .cc suffix
+and some .t files too). I'm not aware of any way how to automate tests
+in test-t directory.
+
+ *TODO: explore tests in test-l directory*.
+
+test-c directory contains C source files, which should be preprocessed
+by the tested preprocessor, compiled by any compiler and then executed.
+List of available tests is written in n\_i\_.lst file. Each test when
+executed prints "started" and then another line "success" if the test
+is passed successfully. Output of each test is stored in a temporary file.
+Those temp files are later analyzed by cpp\_test tool, which considers
+the test as successfull, if the coresponding temp file contains two lines
+"started" and "success". So cpp\_test tool automates testing, but **only** for
+tests in test-c directory (only for compiled tests). To compile cpp\_test
 tool simply enter the tool directory and type:
 
- $ cc cpp\_test.c -o cpp\_test
+    cc cpp_test.c -o cpp_test
 
 Then go to the test-c directory and run the tests in automated fashion
 (here we assume that we test PCC compiler):
 
- $ ../tool/cpp\_test PCC "/usr/local/bin/pcc -D\_\_GNUC\_\_ -o %s %s.c" "rm %s" < n\_i\_.lst
+    ../tool/cpp_test PCC "/usr/local/bin/pcc -D__GNUC__ -o %s %s.c" "rm %s" < n_i_.lst
 
 First argument is a name of tested compiler (can be whatever you want),
-second argument is a command to compile a test, third is how to delete
-the compiled test and on standard input is expected a list of files (tests)
+second argument is a command used to compile a test program, third is for cleaning
+up compiled binaries and on standard input is expected a list of files (tests)
 which should be compiled.
 
-NOTE: The \_\_GNUC\_\_ define is a hack, which I used on NetBSD, to avoid
+ *NOTE: The \_\_GNUC\_\_ define is a hack, which I used on NetBSD, to avoid
 function renaming error in defs.h. Is there a more clean way how to
-solve this?
+solve this? I was told that this issue was fixed in NetBSD-current.*
 
 The summary is available in file PCC.sum, where on the last line is
 written number of tests which failed.
 
 Another example. To test gcc compiler type:
 
- $ ../tool/cpp\_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n\_i\_.lst
+    ../tool/cpp_test GCC "gcc -std=iso9899:199409 -o%s %s.c" "rm %s" < n_i_.lst
 
 Automated testing gives you just an overview of how good/bad
 your compiler is. To debug the problems, you have to run preprocessor
 on test file manually.
 
-The very long ./doc/cpp-test.html document provides 
-details how to run cpp\_test tool.
+The very long ./doc/cpp-test.html document provides details how to
+run cpp\_test tool.
 
 The ./doc/mcpp-summary.pdf file describes the results of the tests versus 
 several cpp implementations.
 
-TODO: test this and document here and send results back to the mcpp project to be included in their results (but I would fix the obvious bugs before doing that).
+TODO:
 
+* test this and document here and send results back to the mcpp project to be included in their results (but I would fix the obvious bugs before doing that).
+* Document the format of *.sum summary file.
+      
 ### Pkgsrc
 
 Pkgsrc package build system can be used with pcc compiler if you want a way to quickly test "pcc" for a lot of "third-party" software (including doing entire bulk builds attempting thousands of packages):

Powered by rcshistory.cgi 0.3