diff -r f28ad4577863 -r 38ea5c71f04f doc/loupe-test.txt --- a/doc/loupe-test.txt Mon Jan 30 09:11:07 2012 +0000 +++ b/doc/loupe-test.txt Tue Feb 14 22:17:03 2012 +0000 @@ -142,7 +142,64 @@ │ Line 3 column 29 - Query possible scanno arid │ └───────────────────────────────────────────────────────────────┘ -Caveats -------- +False-positives +--------------- -There is no support yet for testcases which are expected to fail. +Most of the time, the input can be tweaked so that all warnings bookloupe +reports represent real errors in the text. Sometimes, however, this either +cannot be done and still test what we need to. In these cases we need a +means to describe these false-positives (warnings that do not describe +a real error). This is important so that a later version of bookloupe can +be improved to not issue the false-positive warning and still pass the +test. In order to do this, we need to describe the warnings in a more +structures manner, like this: + + ┌───────────────────────────────────────────────────────────────┐ + │**************** OPTIONS **************** │ + │-s │ + │**************** INPUT **************** │ + │'In a moment,' Peter replied,' I'm just coming.' │ + │ │ + │'Underneath the girls' scarves. │ + │ │ + │**************** WARNINGS **************** │ + │ │ + │ │ + │ │ + │ Wrongspaced singlequotes? │ + │ │ + │ │ + │ │ + │ Mismatched singlequotes? │ + │ │ + │ │ + │ │ + │ │ + │ Mismatched singlequotes? │ + │ │ + │ │ + └───────────────────────────────────────────────────────────────┘ + +Here, we use the "WARNINGS" tag instead of "EXPECTED" to denote that we +wish to use structured warnings and the list of warnings is enclosed in +an ... node. + +Each warning, or potential warnings is then described using either an +"error" node (for warnings that represent real errors in the text), a +"false-positive" node (for warnings that do not represent real errors), +or a "false-negative" node (for warnings that should be issued, but that +are not yet detected by bookloupe). + +Within each warning node, there are then one or more "at" nodes which +list the acceptable locations for the warning to be reported at (the +first listed should be the preferred location) and exactly one "text" +node which must match the text of the warning issued. + +A testcase will pass if all the warnings marked as errors were issued and +if no warnings were issued that are not listed in one form or another. +If the testcase passes with an expected failure (ie., issues a warning +for a false positive or does not issue a warning for a false negative), +then the test is counted as passed, but a note will be printed describing +this, eg.: + +sample: PASS (with 1 of 1 false positives and 1 of 1 false negatives)