ali@5: bookloupe test framework
ali@5: ========================
ali@0:
ali@0: Running existing testcases
ali@0: --------------------------
ali@0:
ali@5: The test harness (the program that runs a test) is called loupe-test. The
ali@5: various testcases are stored in multiple text files, typically with a .tst
ali@5: extension.
ali@0:
ali@5: To run a testcase when all of bookloupe, loupe-test and the testcase file are
ali@0: in the current directory simply do something like:
ali@0:
ali@5: % loupe-test missing-space.tst
ali@0:
ali@0: from a command prompt. Under MS-Windows, this is called a command window and
ali@0: the prompt will normally look slightly different, eg.,
ali@0:
ali@5: C:\DP> loupe-test missing-space.tst
ali@0:
ali@0: To run all the tests in the current directory, do something like this:
ali@0:
ali@5: % loupe-test *.tst
ali@0:
ali@5: If bookloupe is not in the current directory or you want to run the testsuite
ali@5: against gutcheck (the program that bookloupe is based on), then you can set an
ali@5: environment variable (BOOKLOUPE) to point at it. For example, on MS-Windows
ali@5: you might do:
ali@0:
ali@5: C:\DP> set BOOKLOUPE=C:\GUTCHECK\GUTCHECK.EXE
ali@5: C:\DP> loupe-test *.tst
ali@0:
ali@105: When a testcase fails, loupe-test shows the output of bookloupe (or gutcheck)
ali@105: up until the point where it deviates from the expected result and displays a
ali@105: carat (^) to point to the exact column where the deviation occurred. Sometimes
ali@105: it can still be difficult to work out what is happening and so loupe-test also
ali@105: supports a -o option which will simply print bookloupe's output without comment
ali@105: or checking.
ali@105:
ali@0: Writing your own testcases
ali@0: --------------------------
ali@0:
ali@0: Writing a new testcase is pretty painless. Most testcases follow this simple
ali@0: pattern:
ali@0:
ali@0: ┌──────────────────────────────────────────┐
ali@0: │**************** INPUT **************** │
ali@0: │"Look!John, over there!" │
ali@0: │**************** EXPECTED ****************│
ali@0: │ │
ali@0: │"Look!John, over there!" │
ali@0: │ Line 1 column 6 - Missing space? │
ali@0: └──────────────────────────────────────────┘
ali@0:
ali@0: The sixteen asterisks in this example form what is known as the "flag". This
ali@0: flag must come before and after all tags (eg., INPUT and EXPECTED). In the
ali@5: unlikely event that you need sixteen asterisks at the start of a line of text,
ali@0: then simply choose a different flag and use it throughout the file (flags
ali@0: can be any sequence of ASCII characters except control codes and space).
ali@0:
ali@5: Note that the header that bookloupe and gutcheck normally output is not
ali@5: included in the expected output. This avoids problems with not knowing
ali@5: beforehand the name of the file that bookloupe/gutcheck will be asked to
ali@5: look at (and saves typing!). bookloupe (and gutcheck) prints a blank line
ali@5: before each warning. These are not part of the header and so do need to
ali@5: be included.
ali@0:
ali@5: To test that bookloupe produces no output, you still need to include
ali@0: an EXPECTED tag, just with no text following it. If there is no EXPECTED
ali@5: tag, then loupe-test will consider that no expectation exists and won't
ali@5: check the output at all.
ali@0:
ali@10: Non-ASCII testcases
ali@10: -------------------
ali@10:
ali@10: The testcase definitions (the .tst files) are always written in UTF-8
ali@10: which is a superset of ASCII. Since gutcheck does not understand UTF-8
ali@10: this causes a problem when it is desired to include characters that
ali@10: are not in ASCII in a testcase. To solve this problem it is possible
ali@10: to specify an encoding to use for the test. It is very important to
ali@10: undertand that this specifies the encoding that loupe-test will use to
ali@10: talk to bookloupe/gutcheck and _not_ the encoding of the .tst file
ali@10: (which remains UTF-8). gutcheck understands Latin-1 (at least to a
ali@10: limited extent), the canonical name for which is ISO-8859-1:
ali@10:
ali@10: ┌─────────────────────────────────────────────────────────────────┐
ali@10: │**************** ENCODING **************** │
ali@10: │ISO-8859-1 │
ali@10: │**************** INPUT **************** │
ali@10: │"Hello," he said, "I wanted to bave a tête-à-tête with you." │
ali@10: │**************** EXPECTED **************** │
ali@10: │ │
ali@10: │"Hello," he said, "I wanted to bave a tête-à-tête with you." │
ali@10: │ Line 1 column 31 - Query word bave - not reporting duplicates│
ali@10: └─────────────────────────────────────────────────────────────────┘
ali@10:
ali@10: Embedded linefeeds
ali@10: ------------------
ali@10:
ali@10: One of the tests that bookloupe/gutcheck need to do is check that all
ali@101: lines are ended with CR LF (as required by PG) rather than the UNIX
ali@101: standard LF. loupe-test deliberately ignores the line endings in testcase
ali@101: definition files and uses the expected CR LF. Thus there is needed a means
ali@10: to embed a linefeed (aka newline) character into the input to be sent
ali@10: to bookloupe/gutcheck to test that it correctly identified the problem.
ali@10: loupe-test recognises the unicode symbol for linefeed (U+240A): ␊ which
ali@10: can be used for this purpose instead of a normal newline.
ali@10:
ali@101: UNIX-style newlines
ali@101: -------------------
ali@101:
ali@101: To make life easier for users on UNIX and similar platforms, bookloupe
ali@101: recognises the case of all lines terminated with UNIX-style newlines.
ali@101: It notes this in the summary but does not issue any warnings. We thus
ali@101: need some way to test this case which we do by the NEWLINES tag:
ali@101:
ali@101: ┌──────────────────────────────────────────────────────────────────────────┐
ali@101: │**************** NEWLINES **************** │
ali@101: │LF │
ali@101: │**************** INPUT **************** │
ali@101: │Katherine was assailed by a sudden doubt. Had she mailed that letter? Yes,│
ali@101: │she was certain of that. She had run out to the mail box at ten o'clock │
ali@101: │at night especially to mail it. What had gone wrong? Why wasn't there │
ali@101: │someone to meet her? │
ali@101: └──────────────────────────────────────────────────────────────────────────┘
ali@101:
ali@101: The possible options are CRLF for DOS-style newlines (the default) and
ali@101: LF for UNIX-style newlines.
ali@101:
ali@10: Passing command line options
ali@10: ----------------------------
ali@10:
ali@10: Some of bookloupe's functionality is only available using command line
ali@10: options. loupe-test provides a means of specifying these with the
ali@10: OPTIONS tag:
ali@10:
ali@10: ┌──────────────────────────────────────────┐
ali@10: │**************** OPTIONS **************** │
ali@10: │-m │
ali@10: │-d │
ali@10: │**************** INPUT **************** │
ali@10: │“He went thataway!” │
ali@10: │**************** EXPECTED ****************│
ali@10: └──────────────────────────────────────────┘
ali@10:
ali@10: Extra input files
ali@10: -----------------
ali@10:
ali@10: Under certain circumstances, bookloupe reads other input files than just
ali@10: the ebook. These can be specified in the testcase definition file by
ali@10: adding the name of the file to the INPUT tag:
ali@10:
ali@10: ┌───────────────────────────────────────────────────────────────┐
ali@10: │**************** OPTIONS **************** │
ali@10: │-u │
ali@10: │**************** INPUT(gutcheck.typ) **************** │
ali@10: │arid │
ali@10: │**************** INPUT **************** │
ali@10: │I am the very model of a modern Major-General, │
ali@10: │I've information vegetable, animal, and mineral, │
ali@10: │I know the kings of England, arid I quote the fights historical│
ali@10: │From Marathon to Waterloo, in order categorical; │
ali@10: │I'm very well acquainted, too, with matters mathematical, │
ali@10: │I understand equations, both the simple and quadratical, │
ali@10: │About binomial theorem I'm teeming with a lot o' news-- │
ali@10: │With many cheerful facts about the square of the hypotenuse. │
ali@10: │**************** EXPECTED **************** │
ali@10: │ │
ali@10: │I know the kings of England, arid I quote the fights historical│
ali@10: │ Line 3 column 29 - Query possible scanno arid │
ali@10: └───────────────────────────────────────────────────────────────┘
ali@10:
ali@105: Non standard output
ali@105: -------------------
ali@105:
ali@105: Bookloupe normally follows a standard pattern when printing warnings which
ali@105: loupe-test knows how to interpret. Occasionally this is not suitable and
ali@105: the testcase needs to specify exactly what should be printed. This can
ali@105: be done by adding a literal stdout to the EXPECTED tag:
ali@105:
ali@105: ┌───────────────────────────────────────────────────────────────┐
ali@105: │**************** OPTIONS **************** │
ali@105: │--dump-config │
ali@105: │**************** EXPECTED(stdout) **************** │
ali@105: │# Trivial configuration │
ali@105: │ │
ali@105: │[options] │
ali@105: │dp=true │
ali@105: └───────────────────────────────────────────────────────────────┘
ali@105:
ali@19: False-positives
ali@19: ---------------
ali@10:
ali@19: Most of the time, the input can be tweaked so that all warnings bookloupe
ali@19: reports represent real errors in the text. Sometimes, however, this either
ali@19: cannot be done and still test what we need to. In these cases we need a
ali@19: means to describe these false-positives (warnings that do not describe
ali@19: a real error). This is important so that a later version of bookloupe can
ali@19: be improved to not issue the false-positive warning and still pass the
ali@19: test. In order to do this, we need to describe the warnings in a more
ali@19: structures manner, like this:
ali@19:
ali@19: ┌───────────────────────────────────────────────────────────────┐
ali@19: │**************** OPTIONS **************** │
ali@19: │-s │
ali@19: │**************** INPUT **************** │
ali@19: │'In a moment,' Peter replied,' I'm just coming.' │
ali@19: │ │
ali@19: │'Underneath the girls' scarves. │
ali@19: │ │
ali@19: │**************** WARNINGS **************** │
ali@19: │ │
ali@19: │ │
ali@19: │ │
ali@19: │ Wrongspaced singlequotes? │
ali@19: │ │
ali@19: │ │
ali@19: │ │
ali@19: │ Mismatched singlequotes? │
ali@19: │ │
ali@19: │ │
ali@19: │ │
ali@19: │ │
ali@19: │ Mismatched singlequotes? │
ali@19: │ │
ali@19: │ │
ali@19: └───────────────────────────────────────────────────────────────┘
ali@19:
ali@19: Here, we use the "WARNINGS" tag instead of "EXPECTED" to denote that we
ali@19: wish to use structured warnings and the list of warnings is enclosed in
ali@19: an ... node.
ali@19:
ali@19: Each warning, or potential warnings is then described using either an
ali@19: "error" node (for warnings that represent real errors in the text), a
ali@19: "false-positive" node (for warnings that do not represent real errors),
ali@19: or a "false-negative" node (for warnings that should be issued, but that
ali@19: are not yet detected by bookloupe).
ali@19:
ali@19: Within each warning node, there are then one or more "at" nodes which
ali@19: list the acceptable locations for the warning to be reported at (the
ali@19: first listed should be the preferred location) and exactly one "text"
ali@19: node which must match the text of the warning issued.
ali@19:
ali@19: A testcase will pass if all the warnings marked as errors were issued and
ali@19: if no warnings were issued that are not listed in one form or another.
ali@19: If the testcase passes with an expected failure (ie., issues a warning
ali@19: for a false positive or does not issue a warning for a false negative),
ali@19: then the test is counted as passed, but a note will be printed describing
ali@19: this, eg.:
ali@19:
ali@19: sample: PASS (with 1 of 1 false positives and 1 of 1 false negatives)
ali@101:
ali@101: The summary
ali@101: -----------
ali@101:
ali@101: As part of the header (the first section of output), bookloupe may display
ali@101: a number of summary lines. These are characterized by a leading ASCII
ali@101: long arrow (-->) and generally say something about the ebook as a whole
ali@101: rather than individual lines. Where it is desired to test for the presence
ali@101: of a summary line, a "summary" node can be included within the "expected"
ali@101: node of a testcase using structured warnings. The "summary" node can contain
ali@101: one or more "text" nodes which indicate the text of lines that must be
ali@101: present in the summary section in order for the test to pass. No account is
ali@101: taken of the order of such lines and other summary lines may also be present.