ali@5: bookloupe test framework
ali@5: ========================
ali@0:
ali@0: Running existing testcases
ali@0: --------------------------
ali@0:
ali@5: The test harness (the program that runs a test) is called loupe-test. The
ali@5: various testcases are stored in multiple text files, typically with a .tst
ali@5: extension.
ali@0:
ali@5: To run a testcase when all of bookloupe, loupe-test and the testcase file are
ali@0: in the current directory simply do something like:
ali@0:
ali@5: % loupe-test missing-space.tst
ali@0:
ali@0: from a command prompt. Under MS-Windows, this is called a command window and
ali@0: the prompt will normally look slightly different, eg.,
ali@0:
ali@5: C:\DP> loupe-test missing-space.tst
ali@0:
ali@0: To run all the tests in the current directory, do something like this:
ali@0:
ali@5: % loupe-test *.tst
ali@0:
ali@5: If bookloupe is not in the current directory or you want to run the testsuite
ali@5: against gutcheck (the program that bookloupe is based on), then you can set an
ali@5: environment variable (BOOKLOUPE) to point at it. For example, on MS-Windows
ali@5: you might do:
ali@0:
ali@5: C:\DP> set BOOKLOUPE=C:\GUTCHECK\GUTCHECK.EXE
ali@5: C:\DP> loupe-test *.tst
ali@0:
ali@0: Writing your own testcases
ali@0: --------------------------
ali@0:
ali@0: Writing a new testcase is pretty painless. Most testcases follow this simple
ali@0: pattern:
ali@0:
ali@0: ┌──────────────────────────────────────────┐
ali@0: │**************** INPUT **************** │
ali@0: │"Look!John, over there!" │
ali@0: │**************** EXPECTED ****************│
ali@0: │ │
ali@0: │"Look!John, over there!" │
ali@0: │ Line 1 column 6 - Missing space? │
ali@0: └──────────────────────────────────────────┘
ali@0:
ali@0: The sixteen asterisks in this example form what is known as the "flag". This
ali@0: flag must come before and after all tags (eg., INPUT and EXPECTED). In the
ali@5: unlikely event that you need sixteen asterisks at the start of a line of text,
ali@0: then simply choose a different flag and use it throughout the file (flags
ali@0: can be any sequence of ASCII characters except control codes and space).
ali@0:
ali@5: Note that the header that bookloupe and gutcheck normally output is not
ali@5: included in the expected output. This avoids problems with not knowing
ali@5: beforehand the name of the file that bookloupe/gutcheck will be asked to
ali@5: look at (and saves typing!). bookloupe (and gutcheck) prints a blank line
ali@5: before each warning. These are not part of the header and so do need to
ali@5: be included.
ali@0:
ali@5: To test that bookloupe produces no output, you still need to include
ali@0: an EXPECTED tag, just with no text following it. If there is no EXPECTED
ali@5: tag, then loupe-test will consider that no expectation exists and won't
ali@5: check the output at all.
ali@0:
ali@10: Non-ASCII testcases
ali@10: -------------------
ali@10:
ali@10: The testcase definitions (the .tst files) are always written in UTF-8
ali@10: which is a superset of ASCII. Since gutcheck does not understand UTF-8
ali@10: this causes a problem when it is desired to include characters that
ali@10: are not in ASCII in a testcase. To solve this problem it is possible
ali@10: to specify an encoding to use for the test. It is very important to
ali@10: undertand that this specifies the encoding that loupe-test will use to
ali@10: talk to bookloupe/gutcheck and _not_ the encoding of the .tst file
ali@10: (which remains UTF-8). gutcheck understands Latin-1 (at least to a
ali@10: limited extent), the canonical name for which is ISO-8859-1:
ali@10:
ali@10: ┌─────────────────────────────────────────────────────────────────┐
ali@10: │**************** ENCODING **************** │
ali@10: │ISO-8859-1 │
ali@10: │**************** INPUT **************** │
ali@10: │"Hello," he said, "I wanted to bave a tête-à-tête with you." │
ali@10: │**************** EXPECTED **************** │
ali@10: │ │
ali@10: │"Hello," he said, "I wanted to bave a tête-à-tête with you." │
ali@10: │ Line 1 column 31 - Query word bave - not reporting duplicates│
ali@10: └─────────────────────────────────────────────────────────────────┘
ali@10:
ali@10: Embedded linefeeds
ali@10: ------------------
ali@10:
ali@10: One of the tests that bookloupe/gutcheck need to do is check that all
ali@10: lines are ended with CR NL (as required by PG) rather than the UNIX
ali@10: standard NL. loupe-test deliberately ignores the line endings in testcase
ali@10: definition files and always uses CR NL. Thus there is needed a means
ali@10: to embed a linefeed (aka newline) character into the input to be sent
ali@10: to bookloupe/gutcheck to test that it correctly identified the problem.
ali@10: loupe-test recognises the unicode symbol for linefeed (U+240A): ␊ which
ali@10: can be used for this purpose instead of a normal newline.
ali@10:
ali@10: Passing command line options
ali@10: ----------------------------
ali@10:
ali@10: Some of bookloupe's functionality is only available using command line
ali@10: options. loupe-test provides a means of specifying these with the
ali@10: OPTIONS tag:
ali@10:
ali@10: ┌──────────────────────────────────────────┐
ali@10: │**************** OPTIONS **************** │
ali@10: │-m │
ali@10: │-d │
ali@10: │**************** INPUT **************** │
ali@10: │“He went thataway!” │
ali@10: │**************** EXPECTED ****************│
ali@10: └──────────────────────────────────────────┘
ali@10:
ali@10: Extra input files
ali@10: -----------------
ali@10:
ali@10: Under certain circumstances, bookloupe reads other input files than just
ali@10: the ebook. These can be specified in the testcase definition file by
ali@10: adding the name of the file to the INPUT tag:
ali@10:
ali@10: ┌───────────────────────────────────────────────────────────────┐
ali@10: │**************** OPTIONS **************** │
ali@10: │-u │
ali@10: │**************** INPUT(gutcheck.typ) **************** │
ali@10: │arid │
ali@10: │**************** INPUT **************** │
ali@10: │I am the very model of a modern Major-General, │
ali@10: │I've information vegetable, animal, and mineral, │
ali@10: │I know the kings of England, arid I quote the fights historical│
ali@10: │From Marathon to Waterloo, in order categorical; │
ali@10: │I'm very well acquainted, too, with matters mathematical, │
ali@10: │I understand equations, both the simple and quadratical, │
ali@10: │About binomial theorem I'm teeming with a lot o' news-- │
ali@10: │With many cheerful facts about the square of the hypotenuse. │
ali@10: │**************** EXPECTED **************** │
ali@10: │ │
ali@10: │I know the kings of England, arid I quote the fights historical│
ali@10: │ Line 3 column 29 - Query possible scanno arid │
ali@10: └───────────────────────────────────────────────────────────────┘
ali@10:
ali@19: False-positives
ali@19: ---------------
ali@10:
ali@19: Most of the time, the input can be tweaked so that all warnings bookloupe
ali@19: reports represent real errors in the text. Sometimes, however, this either
ali@19: cannot be done and still test what we need to. In these cases we need a
ali@19: means to describe these false-positives (warnings that do not describe
ali@19: a real error). This is important so that a later version of bookloupe can
ali@19: be improved to not issue the false-positive warning and still pass the
ali@19: test. In order to do this, we need to describe the warnings in a more
ali@19: structures manner, like this:
ali@19:
ali@19: ┌───────────────────────────────────────────────────────────────┐
ali@19: │**************** OPTIONS **************** │
ali@19: │-s │
ali@19: │**************** INPUT **************** │
ali@19: │'In a moment,' Peter replied,' I'm just coming.' │
ali@19: │ │
ali@19: │'Underneath the girls' scarves. │
ali@19: │ │
ali@19: │**************** WARNINGS **************** │
ali@19: │ │
ali@19: │ │
ali@19: │ │
ali@19: │ Wrongspaced singlequotes? │
ali@19: │ │
ali@19: │ │
ali@19: │ │
ali@19: │ Mismatched singlequotes? │
ali@19: │ │
ali@19: │ │
ali@19: │ │
ali@19: │ │
ali@19: │ Mismatched singlequotes? │
ali@19: │ │
ali@19: │ │
ali@19: └───────────────────────────────────────────────────────────────┘
ali@19:
ali@19: Here, we use the "WARNINGS" tag instead of "EXPECTED" to denote that we
ali@19: wish to use structured warnings and the list of warnings is enclosed in
ali@19: an ... node.
ali@19:
ali@19: Each warning, or potential warnings is then described using either an
ali@19: "error" node (for warnings that represent real errors in the text), a
ali@19: "false-positive" node (for warnings that do not represent real errors),
ali@19: or a "false-negative" node (for warnings that should be issued, but that
ali@19: are not yet detected by bookloupe).
ali@19:
ali@19: Within each warning node, there are then one or more "at" nodes which
ali@19: list the acceptable locations for the warning to be reported at (the
ali@19: first listed should be the preferred location) and exactly one "text"
ali@19: node which must match the text of the warning issued.
ali@19:
ali@19: A testcase will pass if all the warnings marked as errors were issued and
ali@19: if no warnings were issued that are not listed in one form or another.
ali@19: If the testcase passes with an expected failure (ie., issues a warning
ali@19: for a false positive or does not issue a warning for a false negative),
ali@19: then the test is counted as passed, but a note will be printed describing
ali@19: this, eg.:
ali@19:
ali@19: sample: PASS (with 1 of 1 false positives and 1 of 1 false negatives)