Bookloupe documentation bookloupe: lists possible common formatting errors in a Project Gutenberg candidate file. Bookloupe is based on gutcheck, written by Jim Tinsley. It is a command line program and can be used under Microsoft Windows, Mac or Unix. For Windows-only people, there is an appendix at the end with brief instructions for running it. Current version: 2.1 This software is Copyright Jim Tinsley 2000-2005 and J. Ali Harlow 2012 onwards. Bookloupe comes wih ABSOLUTELY NO WARRANTY. For details, read the file COPYING. This is Free Software; you may redistribute it under certain conditions (GPL). See http://www.juiblex.co.uk/pgdp/bookloupe/ for the latest version. Recent changes in behaviour Each new version of bookloupe brings bug fixes and improvements. Sometimes the behaviour is also changed in ways that might be unexpected: Odd characters The check for "odd" characters (tab, tilde, carat, forward slash and asterisks) is disabled in bookloupe 2.0 when the character set is switched from ASCII/ISO-8859-1 to UNICODE (ie., when the "There are a lot of foreign letters here." message is printed). As of bookloupe 2.1 these tests operate independently of the character set selected. Users may notice this change most especially in the case of the DP-specific /* ... */ markup. Bookloupe 2.0 often did not warn when this markup was encountered even when the --dp switch was not given. Bookloupe 2.1 will warn about this markup unless dp-specific mode is switched on, paranoid mode is switched off or the ebook contains more than 10 lines containing asterisks. In the last case --> 11 lines in this file contain asterisks. Not reporting them. will be printed. Usage is: bookloupe [OPTION...] filename Options: -d, --dp ignores some DP-specific markup -e, --no-echo switches off Echoing of lines -s, --squote checks Single quotes --typo checks Typos -p, --qpara sets strict quotes checking for Paragraphs --no-paranoid switches OFF typo checking and extra checks -l, --no-line-end turns off Line-end checks -o, --overview produces an Overview only -y, --stdout sets error messages to stdout -h, --header echos the header fields -m, --markup ignore some common HTML markup -u, --usertypo warns about words in a user-defined typo file -v, --verbose forces individual reporting of minor problems -w, --web special mode for web uploads (for future use) --charset=NAME the set of characters valid for this ebook --dump-config dump the current configuration There are also inverted options available which are useful when it is desired to override an option set in the configuration file: --no-dp, --echo, --no-squote, --no-typo, --no-qpara, --paranoid, --line-end, --no-overview, --no-stdout, --no-header, --no-markup, --no-usertypo --no-verbose. Note: there is no --no-web since --web simply selects a set of options. Finally there are a couple of options that toggle the state of options rather than setting or unsetting them: -t (for typo) and -x (for typo and paranoid). These are mainly intended for compatability with gutcheck. Running bookloupe without any parameters will display a brief help message. Sample usage: bookloupe warpeace.txt More detail: Configuration file Bookloupe will look for a file named bookloupe.ini to read as a configuration file. Options set in a configuration file can be overridden from the command line as required. The following directories are searched in order: 1) The current working directory. When run from the command line, this is the directory you ran it from. When run from guiguts it will normally be the directory that contains the guiguts program. 2) The directory containing the bookloupe program. 3) The user's configuration directory. Under MS-Windows this is normally CSIDL_LOCAL_APPDATA which is typically set to C:\Documents and Settings\\Local Settings\Application Data. On other platforms this is normally $XDG_CONFIG_HOME which, if not set defaults to $HOME/.config The directories to search can also be changed using the $BOOKLOUPE_CONFIG_PATH environment variable which is a colon separated (semi-colon separated under MS-Windows) list of directories. The configuration file is a key file. This is very similar to, but not identical to a typical ini file as found under MS-Windows. Key files consist of a number of groups which start with the group name enclosed in square brackets on a line by itself. Bookloupe recognises just one group, "options". Then below the group name there follows the keys and their values for that group, one per line in the format key=value. Most of bookloupe's options are flags (ie., either on or off). For these keys, the value must be either "true" or "false". The file may also contain comment lines which begin with the # symbol. The names of the keys follow the long option names. A sample configuration file is provided (in sample.ini). The file will need to be copied to bookloupe.ini before bookloupe will read it. You can also use the --dump-config option to write a configuration file for you. For example, if you typically want to run bookloupe with the --dp and --squote options, then you might do: $ bookloupe --dp --squote --dump-config > configuration.ini $ ren configuration.ini bookloupe.ini (Don't be tempted to merge these two steps or bookloupe will see an empty configuration file and complain.) This same idea can also be used to modify an existing configuration. Character encoding Bookloupe will handle e-texts encoded in UTF-8 (preferred), ISO-8859-1 (also known as Latin-1), or WINDOWS-1252 (also known, incorrectly, as ansi). The output will be in the same encoding as the input e-text. Character set (--charset) Character encodings have an implicit set of characters that can be encoded and thus define a set of characters that can be present in the text. However sometimes it is desirable that not all characters that can be encoded should be present in a text. The set of characters that should be present is known as the character set. The default setting for the character set (called auto) does the same as gutcheck for Windows-1252 encoded texts for compatability: If the file is predominately ASCII then the set of legal characters is ASCII and warnings are issued whenever non-ASCII characters are encountered. The message will either warn of non-ASCII or non-ISO-8859-1 characters as appropriate. If the file contains a significant number of non-ASCII characters then a message is printed as follows: --> There are a lot of foreign letters here. Not reporting them. and the character set is widened to include all possible characters. For UTF-8 encoded texts, auto selects UNICODE. Most character sets are simply defined in bookloupe as the set of all characters that can be encoded in the encoding of the same name. UNICODE is an exception and includes only the characters assigned in the relevant Unicode standard but excluding the Private Use Area characters. Note that the relevant Unicode standard is given by the version of glib in use rather than by any code in bookloupe and thus can vary from system to system. PG texts however are likely to be using characters assigned in very early Unicode standards, thus mitigating this issue. Echoing lines (--no-echo to switch off) You may find it convenient, when reviewing Bookloupe's suggestions, to see the line that Bookloupe is questioning. That way, you can often see at a glance whether it is a real error that needs to be fixed, or a false positive that should be in the text, but Bookloupe's limited programming doesn't understand. By default, bookloupe echoes these lines, but if you don't want to see the lines referred to, --no-echo will switch it OFF. Quotes (--squote and --qpara switches) Bookloupe always looks for unbalanced doublequotes in a paragraph. It is a common convention for writers not to close quotes in a paragraph if the next paragraph opens with quotes and is a continuation by the same speaker. Bookloupe therefore does not normally report unclosed quotes if the next paragraph begins with a quote. If you need to see all unclosed quotes, even where the next paragraph begins with a quote, you should use the -p switch. Singlequotes (', `, ‘ and ’) are a problem, since the same character can be used for an apostrophe. I'm not sure that it is possible to get 100% accuracy on singlequotes checking, particularly since dialect, quite common in PG texts, upsets the normal rules so badly. Consider the sentence: 'Tis often said that a man's a man for a' that. As humans, we recognize that both apostrophes are used for contractions rather than quotes, but it isn't easy to get a program to recognize that. Since bookloupe makes too many mistakes when trying to match singlequotes, it doesn't look for unbalanced singlequotes unless you specify the --squote switch. Consider these sentences, which illustrate the main cases: 'Tis often said that a fool and his money are soon parted. 'Becky's goin' home,' said Tom. The dogs' tails wagged in unison. Those 'pack dogs' of yours look more like wolves. Typos (--typo switch) It's not bookoupe's job to be a spelling checker, but it does check for a list of common typos and OCR errors if you use the --typo switch. (The -t and -x switchs also toggle typo checking.) It also checks for character combinations, especially involving h and b, which are often confused by OCR, that rarely or never occur. For example, it queries "tbe" in a word. Now, "the" often occurs, but "tbe" is very rare (heartbeat, hotbed), so I'm playing the odds - a few false positives for many errors found. Similarly with "ii", which is a very common OCR error. Bookloupe suppresses multiple reporting of the first 40 "typos" found. This is to remove the annoyance of seeing something like "FN" (footnote) or "LK" (initials) flagged as a typo 147 times in a text. Line-end checking (--no-line-end switch to disable) All PG texts should have a Carriage Return (CR - character 13) and a Line Feed (LF - character 10) at end of each line, regardless of what O/S you made them on. DOS/Windows, Unix and Mac have different conventions, but the final text should always use a CR/LF pair as its line terminator. By default, bookloupe verifies that every line does have the correct terminator, but if you're on a work-in-progress in Linux, you might want to convert the line-ends as a final step, and not want to see thousands of errors every time you run bookloupe before that final step, so you can turn off this checking with the --no-line-end switch. Paranoid mode (--no-paranoid switch to disable: Trust No One :-) --no-paranoid switches OFF some extra checks like standalone 1 and 0 queries. Overview mode (--overview switch) This mode just gives a count of queries found instead of a detailed list. Header quote (--header switch) If you use the --header switch, bookloupe will also display the Title, Author, Release and Edition fields from the PG header. This is useful mostly for the automated checks we do on recently-posted texts. Errors to stdout (--stdout switch) If you're just running bookloupe normally, you can ignore this. It's only there for programs that provide a front end to bookloupe. It makes error messages appear within the output of bookloupe so that the front end knows whether bookloupe ran OK. Verbose reporting (--verbose switch) Normally, if bookloupe sees lots of long lines, short lines, spaced dashes, non-ASCII characters or dot-commas ".," it assumes these are features of the text, counts and summarizes them at the top of its report, but does not list them individually. If the verbose switch is on, bookloupe will list them all. Markup interpretation (--markup switch) Normally, bookloupe flags anything it suspects of being HTML markup as a possible error. When you use the --markup switch, however, it matches anything that looks like markup against a short list of common HTML tags and entities. If the markup is in that list, it either ignores the markup, in the case of a tag, or "interprets" the markup as its nearest ASCII equivalent, in the case of an entity. So, for example, using this switch, bookloupe will "see" “He went thataway!” as "He went thataway!" and report accordingly. This switch does not, not, NOT check the validity of HTML; it exists so that you can run bookloupe on most HTML texts for PG, and get sane results. It does not support all tags. It does not support all entities. When it sees a tag or entity it does not recognize, it will query it as HTML just as if you hadn't specified the --markup switch. Bookloupe will automatically switch on markup interpretation if it sees a lot of tags that appear to be markup, so mostly, you won't have to specify this. User-defined typos (--usertypo switch) If you have a file named bookloupe.typ or gutcheck.typ either in your current working directory or in the directory from which you explicitly invoked bookoupe, but not necessarily on your path, and if you specify the --usertypo switch, bookloupe will query any word specified in that file. The file is simple: one word, in lower case, per line. Be careful not to put multiple words onto a line, or leave any rubbish other than the word on the line. You should have received a sample file bookloupe.typ with this package. The file may be encoded in UTF-8 (preferred), ISO-8859-1 (also known as Latin-1), or WINDOWS-1252 (also known, incorrectly, as ansi). Ignore DP markup (--dp switch) Distributed Proofreaders (http://www.pgdp.net) has for some time been the main source of PG texts, and proofers there use special conventions. This switch understands those conventions, so that people can use bookloupe on files in process that still haven't had the special conventions removed yet. The special conventions supported are page-separators and "", "", "/*", "*/", "/#", "#/", "/$", "$/". Dump the current configuration (--dump-config switch) The --dump-config switch can be used to dump the current configuration. This is a combination of the internal defaults, the configuration file (if any) and the command line options. If a configuration file is present, any comments found in that file will be preserved in the dumped configuration. If there is no configuration file, then a default set of comments to go with the internal default configuration is generated. You will probably only run bookloupe on a text once or maybe twice, just prior to uploading; it usually finds a few formatting problems; it also usually finds queries that aren't problems at all - it often questions Tables of Contents for having short lines, for example. These are called "false positives," and need a human to decide on them. The text should be standard prose, and already close to PG normal format (plain text, about 70 characters per line with blank lines between paragraphs). Bookloupe merely draws your attention to things that might be errors. It is NOT a substitute for human judgement. Formatting choices like short lines may be for a reason that this program can't understand. Even the most careful human proofing can leave errors behind in a text, and there are several automated checks you can do to help find them. Of these, spellchecking (with _very_ careful human judgement) is the most important and most useful. Bookloupe does perform some basic typo-checking if you ask it to, but its focus is on formatting errors specific to PG texts— mismatched quotes, non-ASCII characters, bad spacing, bad line length, HTML tags perhaps left from a conversion, unbalanced brackets. Suggestions for additional checks would be appreciated and duly considered, but no guarantees that they will be implemented. How does Jim Tinsley use gutcheck? Practically everyone I give gutcheck to asks me how _I_ use it. Well, when I get a text for posting, say filename.txt, I run gutcheck -o filename.txt That gives me a quick idea what I'm dealing with. It'll tell me what kind of problems gutcheck sees, and give me an idea of how much more work needs to be done on the text. Keep in mind that gutcheck doesn't do anything like a full spellcheck, but when I see a text that has a lot of problems, I assume that it probably needs a spellcheck too. Having got a feel for the ballpark, I run gutcheck filename.txt > jj where jj is my personal, all-purpose filename for temporary data that doesn't need to be kept. Then I open filename.txt and jj in a split-screen view in my editor, and work down the text, fixing whatever needs fixing, and skipping whatever doesn't. If your editor doesn't split-screen, you can get much the same effect by opening your original file in your normal editor, and jj (or your equivalent name) in something like Notepad, keeping both in view at the same time. Twice a day, an automatic process looks at all recently-posted texts, and emails Michael, me, and sometimes other people with their gutcheck summaries. Explanations of common bookloupe messages: --> 74 lines in this file have white space at end PG texts shouldn't have extra white space added at end of line. Don't worry too much about this; they're not doing any harm, and they'll be removed during posting anyway. --> 348 lines in this file are short. Not reporting short lines. --> 84 lines in this file are long. Not reporting long lines. --> 8 lines in this file are VERY long! If there are a lot of long or short lines, bookloupe won't list them individually. The short lines version of this message is commonly seen when gutchecking poetry and some plays, where the normal line length is shorter than the standard for prose. A "VERY long" line is one over 80 characters. You normally shouldn't have any of these, but sometimes you may have to render a table that must be that long, or some special preformatted quotation that can't be broken. --> There are 75 spaced dashes and em-dashes in this file. Not reporting them. The PG standard for an emdash--like these--is two minus signs with no spaces before or after them. However, some older texts used spaced dashes - like these -- and if there are very many such spaced dashes in the file, bookoupe just draws your attention to it and doesn't list them individually. Line 3020 - Non-ASCII character 233 Standard PG texts should use only ASCII characters with values up to 127; however, non-English, accented characters can be represented according to several different non-ASCII encoding schemes, using values over 127. If you have a plain English text with a few accented characters in words like cafe or tete-a-tete, you might replace the accented characters with their unaccented versions. The English pound sign is another commonly-seen non-ASCII character. If you have enough non-ASCII characters in your text that you feel removing them would degrade your text, you should probably consider doing a UTF-8 text. Line 1207 - Non-ISO-8859 character 156 Even in "8-bit" texts, there are distinctions between code sets. The ISO-8859 family of 8-bit code sets is the most commonly used in PG, and these sets do not define values in the range 128 through 159 as printable characters. It's quite common for someone on a Windows or Mac machine to use a non-ISO character inadvertently, so this message warns that the character is not only not ASCII, but also outside the ISO-8859 range. Line 46 - Tab character? Some editors and WPs will put in Tab characters (character 9) to indicate indented text. You should not use these in a PG text, because you can't be sure how they will appear on a reader's screen. Find the Tab, and replace it with the appropriate number of spaces. Line 1327 - Tilde character? The tilde character (~) might be legitimately used, but it's the character commonly used by OCR software to indicate a place where it couldn't make out the letter, so bookloupe flags it. Line 1347 - Asterisk? Asterisks are reported only in paranoid mode (see -x). Like tildes, they are often used to indicate errors, but they are also legitimately used as line delimiters and footnote markers. Line 1451 - Long line 129 PG texts should have lines shorter than 76. There may be occasions where you decide that you really have to go out to 79 characters, but the sample above says that line 1451 is 129 characters long— probably two lines run together. Line 1590 - Short line? PG texts should have lines longer than 54 characters. However, there are special cases like poetry and tables of contents where the lines _should_ be shorter. So treat bookloupe warnings about short lines carefully. Sometimes it's a genuine formatting problem; sometimes the line really needs to be short. Hint: bookloupe will not flag lines as short if they are indented —if they start with a space. I like to start inserted stanzas and other such items indented with a couple of spaces so that they stand out from the main text anyway. Line 1804 - Begins with punctuation? Lines should normally not begin with commas, periods and so on. An exception is ellipses . . . which can happen at start of line. Line 1850 - Spaced em-dash? The PG standard for an em-dash--like these--is two minus signs with no spaces before or after them. Bookloupe flags non-PG em-dashes - like this one. Normally, you will replace it with a PG-standard em-dash. Line 1904 - Query he/be error? Bookloupe makes a very minor effort to look for that scourge of all proofreaders, "be" replacing "he" or vice-versa, and draws your attention to it when it thinks it has found one. Line 2017 - Query digit in a1most The digit 1 is commonly OCRed for the letter l, the digit 0 for the letter O, and so on. When bookloupe sees a mix of digits and letters, it warns you. It may generate a false positive for something like 7am. Line 2083 - Query standalone 0 In paranoid mode (see -x) only, bookloupe warns about the digit 0 and the number 1 standing alone as a word. This can happen if the OCR misreads the words O or I. Line 2115 - Query word whetber If you have switched typo-checking on, bookloupe looks for potential typos, especially common h/b errors. It's not infallible; it sometimes queries legit words, but it's always worth taking a look. Line 2190 column 14 - Missing space? Omitting a space is a very common error,especially coming from OCRed text,and can be hard for a human to spot. The commas in the previous sentence illustrate the kind of thing I mean. Line 2240 column 48 - Spaced punctuation? The flip side of the "missing space" error , here , is when extra spaces are added before punctuation . Some old texts appear to add extra spaces around punctuation consistently, but this was a typographical convention rather than the author's intent, and the extra "spaces" should be removed when preparing a PG text. Line 2301 column 19 - Unspaced quotes? Another common spacing problem occurs in a phrase like "You wait there,"he said. Line 2385 column 27 - Wrongspaced quotes? Bookloupe checks whether a quote seems to be a start or end quote, and queries those that appear to be misplaced. This does give rise to false positives when quotes are nested, for example: "And how," she asked, "will your "friends" help you now?" but these false positives are worth it because of the many cases that this test catches, notably those like: "And how, "she said," will your friends help you now?" Sometimes a "wrongspaced quotes" query will arise because an earlier quote in the paragraph was omitted, so if the place specified seems to be OK, look back to see whether there's a problem in the preceding lines. Line 2400 - HTML Tag? 

    Some PG texts have been converted from HTML, and not all of the
    HTML tags have been removed.



    Line 2402 - HTML symbol? &emdash;

    Similarly, special HTML symbol characters can survive into PG
    texts. Can occasionally produce amusing false positives like
    . . . Marwick & Co were well known for it;



    Line 2540 - Mismatched quotes

    Another bookloupe mainstay—unclosed doublequotes in a paragraph.
    See the discussion of quotes in the switches section near the
    start of this file.

    Since the mismatch doesn't occur on any one line, bookloupe quotes
    the line number of the first blank line following the paragraph,
    since this is the point where it reconciles the count of quotes.
    However, if bookloupe is echoing lines, that is, you haven't used
    the -e switch, it will show the _first_ line of the paragraph,
    to help you find the place without using line numbers. The
    offending paragraph is therefore between the quoted line and
    the line number given.



    Line 2587 - Mismatched single quotes

    Only checked with the -s switch, since checking single quotes is
    not a very reliable process. Otherwise, the same logic as for
    doublequotes applies.



    Line 2877 - Mismatched round brackets?

    Also curly and square brackets. Texts with a lot of brackets, like
    plays with bracketed stage instructions, may have mismatches.


    Line 3150 - No CR?
    Line 3204 - Two successive CRs?
    Line 3281 position 75 - CR without LF?

    These are the invalid line-end warnings. See the discussion of
    line-end checking in the switches section near the start of this
    file. If you see these, and your editor doesn't show anything
    wrong, you should probably try deleting the characters just before
    and after the line end, and the line-end itself, then retyping the
    characters and the line-end.


    Line 2940 - Paragraph starts with lower-case

    A common error in an e-text is for an extra blank line

    to be put in, like the blank line above, and this often
    shows up as a new paragraph beginning with lower case.
    Sometimes the blank line is deliberate, as when a
    quotation is inserted in a speech. Use your judgement.


    Line 2987 - Extra period?

    An extra period. is a. common problem in OCRed text. and usually
    arises when a speck of dust on the page is mistaken for a period.
    or. as occasionally happens. when a comma loses its tail.


    Line 3012 column 12 - Double punctuation?

    Double punctuation., like that,, is a common typo and
    scanno. Some books have much legit double punctuation,
    like etc., etc., but it's worth checking anyway.



            *       *       *        *

For Windows-only users who are unfamiliar with DOS:

    If you're a Windows-only user, you need to save
    bookloupe.exe into the folder (directory) where the
    text file you want to check is. Let's say your
    text file is in C:\gut, then you should save
    bookloupe.exe into C:\gut.

    Now get to a console. You can do this by
    selecting the "Command Prompt" or "MS-DOS Prompt"
    option that will be somewhere on your
    Start/Programs menu.

    Now get into the C:\gut directory.
    You can do this using the cd (change directory)
    command, like this:
        cd \gut
    and your prompt will change to
        C:\gut>
    so you know you're in the right place.

    Now type
        bookloupe yourfile.txt
    and you'll see bookloupe's report

    By default, bookloupe prints its queries to screen.
    If you want to create a file of them, to edit
    against the text, you can use the greater-than
    sign (>) to tell it to output the report to a
    file. For example, if you want its report in a
    file called queries.lst, you could type

        bookloupe yourfile.txt > queries.lst

    The queries.lst file will then contain the listing
    of possible formatting errors, and you can
    edit it alongside your text.

    Whatever you do, DON'T make the filename after
    the greater-than sign the name of a file already
    on your disk that you want to keep, because
    the greater-than sign will cause bookloupe to
    replace any existing file of that name.

    So, for example, if you have two Tolstoy files
    that you want to check, called WARPEACE.TXT and
    ANNAK.TXT, make sure that neither of these names
    is ever used following the greater-than sign.
    To check these correctly, you might do:

    bookloupe warpeace.txt > war.lst

    and

    bookloupe annak.txt > annak.lst

    separately. Then you can look at war.lst and annak.lst
    to see the bookloupe reports.

For Windows-only users who want to use bookloupe from guiguts:

    1) If you haven't already done so, download bookloupe-win32-xxx.zip
    from http://www.juiblex.co.uk/pgdp/bookloupe/

    2) Extract the files into a suitable folder, e.g. C:\DP\bookloupe

    3) Start Guiguts

    4) Choose Preferences | File Paths | Set File Paths..

    5) Click the "Locate Gutcheck..." button

    6) Browse to the folder where you extracted bookloupe

    7) Double-click bookloupe.exe

    Now, whenever you do "Gutcheck" in Guiguts, it will run bookloupe
    instead. Since the output will look very like gutcheck output, you
    may want to check that it is actually bookloupe that is running. To do
    this, look at the black command line message window, which will say:

    "bookloupe: Check and report on an e-text".

    To return to using gutcheck for any reason, repeat steps 4 and 5
    above, and then,

    6b) Browse back to the gutcheck folder, which is in a "tools"
    folder inside the main Guiguts folder. It will be something like
    "C:\DP\guiguts-win\tools\gutcheck", depending on where you installed
    Guiguts originally.

    7b) Double-click gutcheck.exe

    Now doing "Gutcheck" in Guiguts will run gutcheck itself, and the
    message in the black window should read:

    "gutcheck: Check and report on an e-text".