diff -r 000000000000 -r 218904410231 doc/gutcheck.txt
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/doc/gutcheck.txt Fri Jan 27 00:28:11 2012 +0000
@@ -0,0 +1,742 @@
+
+
+ Gutcheck documentation
+
+
+gutcheck: lists possible common formatting errors in a Project
+Gutenberg candidate file. It is a command line program and can be used
+under Win32 or Unix (gutcheck.c should compile anywhere; if it doesn't,
+tell me). For Windows-only people, there is an appendix at the end
+with brief instructions for running it.
+
+
+Current version: 0.99. Users of 0.98 see end of file for changes.
+
+You should also have received the licence file COPYING, a README file,
+gutcheck.c, the source code, and gutcheck.exe, a DOS executable, with
+this file.
+
+This software is Copyright Jim Tinsley 2000-2005.
+
+Gutcheck comes wih ABSOLUTELY NO WARRANTY. For details, read the file COPYING.
+This is Free Software; you may redistribute it under certain conditions (GPL).
+
+See http://gutcheck.sourceforge.net for the latest version.
+
+
+Usage is: gutcheck [-setopxlywm] filename
+ where:
+ -s checks Single quotes
+ -e switches off Echoing of lines
+ -t checks Typos
+ -o produces an Overview only
+ -p sets strict quotes checking for Paragraphs
+ -x (paranoid) switches OFF typo checking and extra checks
+ -l turns off Line-end checks
+ -y sets error messages to stdout
+ -w is a special mode for web uploads (for future use)
+ -v (verbose) forces individual reporting of minor problems
+ -m interprets Markup of some common HTML tags and entities
+ -u warns about words in a user-defined typo file gutcheck.typ
+ -d ignores some DP-specific markup
+
+Running gutcheck without any parameters will display a brief help message.
+
+Sample usage:
+
+ gutcheck warpeace.txt
+
+
+More detail:
+
+ Echoing lines (-e to switch off)
+
+ You may find it convenient, when reviewing Gutcheck's
+ suggestions, to see the line that Gutcheck is questioning.
+ That way, you can often see at a glance whether it is
+ a real error that needs to be fixed, or a false positive
+ that should be in the text, but Gutcheck's limited
+ programming doesn't understand.
+
+ By default, gutcheck echoes these lines, but if you don't
+ want to see the lines referred to, -e will switch it OFF.
+
+
+ Quotes (-s and -p switches)
+
+ Gutcheck always looks for unbalanced doublequotes in a
+ paragraph. It is a common convention for writers not to
+ close quotes in a paragraph if the next paragraph opens
+ with quotes and is a continuation by the same speaker.
+
+ Gutcheck therefore does not normally report unclosed quotes
+ if the next paragraph begins with a quote. If you need
+ to see all unclosed quotes, even where the next paragraph
+ begins with a quote, you should use the -p switch.
+
+ Singlequotes (') are a problem, since the same character
+ is used for an apostrophe. I'm not sure that it is
+ possible to get 100% accuracy on singlequotes checking,
+ particularly since dialect, quite common in PG texts,
+ upsets the normal rules so badly. Consider the sentence:
+ 'Tis often said that a man's a man for a' that.
+ As humans, we recognize that both apostrophes are used
+ for contractions rather than quotes, but it isn't easy
+ to get a program to recognize that.
+
+ Since Gutcheck makes too many mistakes when trying to match
+ singlequotes, it doesn't look for unbalanced singlequotes
+ unless you specify the -s switch.
+
+ Consider these sentences, which illustrate the main cases:
+
+ 'Tis often said that a fool and his money are soon parted.
+
+ 'Becky's goin' home,' said Tom.
+
+ The dogs' tails wagged in unison.
+
+ Those 'pack dogs' of yours look more like wolves.
+
+
+
+ Typos (-t switch)
+
+ It's not Gutcheck's job to be a spelling checker, but it
+ does check for a list of common typos and OCR errors if you
+ use the -t switch. (The -x switch also turns typo checking on.)
+
+ It also checks for character combinations, especially involving
+ h and b, which are often confused by OCR, that rarely or never
+ occur. For example, it queries "tbe" in a word. Now, "the" often
+ occurs, but "tbe" is very rare (heartbeat, hotbed), so I'm
+ playing the odds - a few false positives for many errors found.
+ Similarly with "ii", which is a very common OCR error.
+
+ Gutcheck suppresses multiple reporting of the first 40 "typos"
+ found. This is to remove the annoyance of seeing something like
+ "FN" (footnote) or "LK" (initials) flagged as a typo 147 times
+ in a text.
+
+
+ Line-end checking (-l switch to disable)
+
+ All PG texts should have a Carriage Return (CR - character 13)
+ and a Line Feed (LF - character 10) at end of each line,
+ regardless of what O/S you made them on. DOS/Windows, Unix
+ and Mac have different conventions, but the final text should
+ always use a CR/LF pair as its line terminator.
+
+ By default, Gutcheck verifies that every line does have
+ the correct terminator, but if you're on a work-in-progress
+ in Linux, you might want to convert the line-ends as a final
+ step, and not want to see thousands of errors every time you
+ run Gutcheck before that final step, so you can turn off
+ this checking with the -l switch.
+
+
+ Paranoid mode (-x switch to disable: Trust No One :-)
+
+ -x switches OFF typo-checking, the -t flag, automatically
+ and some extra checks like standalone 1 and 0 queries.
+
+
+ Overview mode (-o switch)
+
+ This mode just gives a count of queries found
+ instead of a detailed list.
+
+
+ Header quote (-h switch)
+
+ If you use the -h switch, gutcheck will also display
+ the Title, Author, Release and Edition fields from the
+ PG header. This is useful mostly for the automated
+ checks we do on recently-posted texts.
+
+
+ Errors to stdout (-y switch)
+
+ If you're just running gutcheck normally, you can ignore
+ this. It's only there for programs that provide a front
+ end to gutcheck. It makes error messages appear within
+ the output of gutcheck so that the front end knows whether
+ gutcheck ran OK.
+
+
+ Verbose reporting (-v switch)
+
+ Normally, if gutcheck sees lots of long lines, short lines,
+ spaced dashes, non-ASCII characters or dot-commas ".," it
+ assumes these are features of the text, counts and summarizes
+ them at the top of its report, but does not list them
+ individually. If the -v switch is on, gutcheck will list them all.
+
+
+ Markup interpretation (-m switch)
+
+ Normally, gutcheck flags anything it suspects of being HTML
+ markup as a possible error. When you use the -m switch,
+ however, it matches anything that looks like markup against
+ a short list of common HTML tags and entities. If the markup
+ is in that list, it either ignores the markup, in the case
+ of a tag, or "interprets" the markup as its nearest ASCII
+ equivalent, in the case of an entity. So, for example, using
+ this switch, gutcheck will "see"
+
+ “He went thataway!”
+
+ as
+
+ "He went thataway!"
+
+ and report accordingly.
+
+ This switch does not, not, NOT check the validity of HTML;
+ it exists so that you can run gutcheck on most HTML texts
+ for PG, and get sane results. It does not support all tags.
+ It does not support all entities. When it sees a tag or entity
+ it does not recognize, it will query it as HTML just as if
+ you hadn't specified the -m switch.
+
+ Gutcheck 0.99 will automatically switch on markup interpretation
+ if it sees a lot of tags that appear to be markup, so mostly, you
+ won't have to specify this.
+
+ User-defined typos (-u switch)
+
+ If you have a file named gutcheck.typ either in your current
+ working directory or in the directory from which you explicitly
+ invoked gutcheck, but not necessarily on your path, and if you
+ specify the -u switch, gutcheck will query any word specified
+ in that file. The file is simple: one word, in lower case, per
+ line. 999 lines are allowed for. Be careful not to put multiple
+ words onto a line, or leave any rubbish other than the word on
+ the line. You should have received a sample file gutcheck.typ
+ with this package.
+
+ Ignore DP markup (-d switch)
+
+ Distributed Proofreaders (http://www.pgdp.net) is currently
+ (2005) the main source of PG texts, and proofers there use
+ special conventions. This switch understands those conventions,
+ so that people can use gutcheck on files in process that still
+ haven't had the special conventions removed yet. The special
+ conventions supported in 0.99 are page-separators and
+ "", "", "/*", "*/", "/#", "#/", "/$", "$/".
+
+
+You will probably only run gutcheck on a text once or maybe twice,
+just prior to uploading; it usually finds a few formatting problems;
+it also usually finds queries that aren't problems at all - it often
+questions Tables of Contents for having short lines, for example.
+These are called "false positives", and need a human to decide on
+them.
+
+The text should be standard prose, and already close to PG normal
+format (plain text, about 70 characters per line with blank lines
+between paragraphs).
+
+Gutcheck merely draws your attention to things that might be errors.
+It is NOT a substitute for human judgement. Formatting choices like
+short lines may be for a reason that this program can't understand.
+
+Even the most careful human proofing can leave errors behind in a
+text, and there are several automated checks you can do to help find
+them. Of these, spellchecking (with _very_ careful human judgement) is
+the most important and most useful.
+
+Gutcheck does perform some basic typo-checking if you ask it to,
+but its focus is on formatting errors specific to PG texts -
+mismatched quotes, non-ASCII characters, bad spacing, bad line
+length, HTML tags perhaps left from a conversion, unbalanced
+brackets.
+
+Suggestions for additional checks would be appreciated and duly
+considered, but no guarantees that they will be implemented.
+
+
+
+
+ How do _I_ use it?
+
+Practically everyone I give gutcheck to asks me how _I_ use it.
+Well, when I get a text for posting, say filename.txt, I run
+
+ gutcheck -o filename.txt
+
+That gives me a quick idea what I'm dealing with. It'll tell
+me what kind of problems gutcheck sees, and give me an idea
+of how much more work needs to be done on the text. Keep in
+mind that gutcheck doesn't do anything like a full spellcheck,
+but when I see a text that has a lot of problems, I assume that
+it probably needs a spellcheck too.
+
+Having got a feel for the ballpark, I run
+
+ gutcheck filename.txt > jj
+
+where jj is my personal, all-purpose filename for temporary data
+that doesn't need to be kept. Then I open filename.txt and jj in
+a split-screen view in my editor, and work down the text, fixing
+whatever needs fixing, and skipping whatever doesn't. If your
+editor doesn't split-screen, you can get much the same effect by
+opening your original file in your normal editor, and jj (or your
+equivalent name) in something like Notepad, keeping both in view
+at the same time.
+
+Twice a day, an automatic process looks at all recently-posted
+texts, and emails Michael, me, and sometimes other people with
+their gutcheck summaries.
+
+
+
+ Future development of gutcheck
+
+Gutcheck has gone about as far as it can, given its current
+structure. In order to add better singlequotes checking,
+sentence checking, better he/be checking and other good stuff
+that I'd like to see, I'll have to rewrite it from a different
+angle - looking at the syntax instead of the lines. And I'll
+probably get around to that sooner or later.
+
+Meantime, I'm just trying to get this version stabilized, so
+please report any bugs you find. When it is stable, I'll run
+up a Windows port for those timid souls who can't look a
+command line in the eye. :-)
+
+And I've started work on gutspell, a companion to gutcheck
+which will concentrate on spelling problems. PG spelling
+problems are unusual, since the range of texts we cover is
+so wide, and I'll be taking a somewhat unorthodox approach
+to writing this spelling-checker _specifically_ for texts
+containing a lot of dialect and uncommon words that have
+probably already been spell-checked against a standard
+modern dictionary.
+
+
+
+
+Explanations of common gutcheck messages:
+
+ --> 74 lines in this file have white space at end
+
+ PG texts shouldn't have extra white space added at end of line.
+ Don't worry too much about this; they're not doing any harm,
+ and they'll be removed during posting anyway.
+
+
+ --> 348 lines in this file are short. Not reporting short lines.
+ --> 84 lines in this file are long. Not reporting long lines.
+ --> 8 lines in this file are VERY long!
+
+ If there are a lot of long or short lines, Gutcheck won't list
+ them individually. The short lines version of this message
+ is commonly seen when gutchecking poetry and some plays, where
+ the normal line length is shorter than the standard for prose.
+ A "VERY long" line is one over 80 characters. You normally
+ shouldn't have any of these, but sometimes you may have to render
+ a table that must be that long, or some special preformatted
+ quotation that can't be broken.
+
+
+ --> There are 75 spaced dashes and em-dashes in this file. Not reporting them.
+
+ The PG standard for an emdash--like these--is two minus signs
+ with no spaces before or after them. However, some older texts
+ used spaced dashes - like these -- and if there are very many
+ such spaced dashes in the file, gutcheck just draws your
+ attention to it and doesn't list them individually.
+
+
+
+ Line 3020 - Non-ASCII character 233
+
+ Standard PG texts should use only ASCII characters with values
+ up to 127; however, non-English, accented characters can be
+ represented according to several different non-ASCII encoding
+ schemes, using values over 127. If you have a plain English text
+ with a few accented characters in words like cafe or tete-a-tete,
+ you should replace the accented characters with their unaccented
+ versions. The English pound sign is another commonly-seen
+ non-ASCII character. If you have enough non-ASCII characters in
+ your text that you feel removing them would degrade your text
+ unacceptably, you should probably consider doing an 8-bit text
+ as well as a plain-ASCII version.
+
+
+
+ Line 1207 - Non-ISO-8859 character 156
+
+ Even in "8-bit" texts, there are distinctions between code sets.
+ The ISO-8859 family of 8-bit code sets is the most commonly used
+ in PG, and these sets do not define values in the range 128 through
+ 159 as printable characters. It's quite common for someone on a
+ Windows or Mac machine to use a non-ISO character inadvertently,
+ so this message warns that the character is not only not ASCII,
+ but also outside the ISO-8859 range.
+
+
+
+ Line 46 - Tab character?
+
+ Some editors and WPs will put in Tab characters (character 9) to
+ indicate indented text. You should not use these in a PG text,
+ because you can't be sure how they will appear on a reader's
+ screen. Find the Tab, and replace it with the appropriate number
+ of spaces.
+
+
+ Line 1327 - Tilde character?
+
+ The tilde character (~) might be legitimately used, but it's the
+ character commonly used by OCR software to indicate a place where
+ it couldn't make out the letter, so gutcheck flags it.
+
+
+
+ Line 1347 - Asterisk?
+
+ Asterisks are reported only in paranoid mode (see -x).
+ Like tildes, they are often used to indicate errors, but they are
+ also legitimately used as line delimiters and footnote markers.
+
+
+
+ Line 1451 - Long line 129
+
+ PG texts should have lines shorter than 76. There may be occasions
+ where you decide that you really have to go out to 79 characters,
+ but the sample above says that line 1451 is 129 characters long -
+ probably two lines run together.
+
+
+
+ Line 1590 - Short line?
+
+ PG texts should have lines longer than 54 characters. However,
+ there are special cases like poetry and tables of contents where
+ the lines _should_ be shorter. So treat Gutcheck warnings about
+ short lines carefully. Sometimes it's a genuine formatting
+ problem; sometimes the line really needs to be short.
+
+ Hint: gutcheck will not flag lines as short if they are indented
+ - if they start with a space. I like to start inserted stanzas
+ and other such items indented with a couple of spaces so that
+ they stand out from the main text anyway.
+
+
+
+ Line 1804 - Begins with punctuation?
+
+ Lines should normally not begin with commas, periods and so on.
+ An exception is ellipses . . . which can happen at start of line.
+
+
+
+ Line 1850 - Spaced em-dash?
+
+ The PG standard for an em-dash--like these--is two minus signs
+ with no spaces before or after them. Gutcheck flags non-PG
+ em-dashes - like this one. Normally, you will replace it with a
+ PG-standard em-dash.
+
+
+
+ Line 1904 - Query he/be error?
+
+ Gutcheck makes a very minor effort to look for that scourge of all
+ proofreaders, "be" replacing "he" or vice-versa, and draws your
+ attention to it when it thinks it has found one.
+
+
+
+ Line 2017 - Query digit in a1most
+
+ The digit 1 is commonly OCRed for the letter l, the digit 0 for
+ the letter O, and so on. When gutcheck sees a mix of digits and
+ letters, it warns you. It may generate a false positive for
+ something like 7am.
+
+
+
+ Line 2083 - Query standalone 0
+
+ In paranoid mode (see -x) only, gutcheck warns about the digit 0
+ and the number 1 standing alone as a word. This can happen if the
+ OCR misreads the words O or I.
+
+
+
+ Line 2115 - Query word whetber
+
+ If you have switched typo-checking on, gutcheck looks for
+ potential typos, especially common h/b errors. It's not
+ infallible; it sometimes queries legit words, but it's
+ always worth taking a look.
+
+
+
+ Line 2190 column 14 - Missing space?
+
+ Omitting a space is a very common error,especially coming from
+ OCRed text,and can be hard for a human to spot. The commas in
+ the previous sentence illustrate the kind of thing I mean.
+
+
+
+ Line 2240 column 48 - Spaced punctuation?
+
+ The flip side of the "missing space" error , here , is when extra
+ spaces are added before punctuation . Some old texts appear to add
+ extra spaces around punctuation consistently, but this was a
+ typographical convention rather than the author's intent, and the
+ extra "spaces" should be removed when preparing a PG text.
+
+
+
+ Line 2301 column 19 - Unspaced quotes?
+
+ Another common spacing problem occurs in a phrase like "You wait
+ there,"he said.
+
+
+
+ Line 2385 column 27 - Wrongspaced quotes?
+
+ As of version 0.98, gutcheck adds extra checks on whether a quote
+ seems to be a start or end quote, and queries those that appear to
+ be misplaced. This does give rise to false positives when quotes are
+ nested, for example:
+
+ "And how," she asked, "will your "friends" help you now?"
+
+ but these false positives are worth it because of the many cases
+ that this test catches, notably those like:
+
+ "And how, "she said," will your friends help you now?"
+
+ Sometimes a "wrongspaced quotes" query will arise because an earlier
+ quote in the paragraph was omitted, so if the place specified seems
+ to be OK, look back to see whether there's a problem in the preceding
+ lines.
+
+
+
+ Line 2400 - HTML Tag?
+
+ Some PG texts have been converted from HTML, and not all of the
+ HTML tags have been removed.
+
+
+
+ Line 2402 - HTML symbol? &emdash;
+
+ Similarly, special HTML symbol characters can survive into PG
+ texts. Can occasionally produce amusing false positives like
+ . . . Marwick & Co were well known for it;
+
+
+
+ Line 2540 - Mismatched quotes
+
+ Another gutcheck mainstay - unclosed doublequotes in a paragraph.
+ See the discussion of quotes in the switches section near the
+ start of this file.
+
+ Since the mismatch doesn't occur on any one line, gutcheck quotes
+ the line number of the first blank line following the paragraph,
+ since this is the point where it reconciles the count of quotes.
+ However, if gutcheck is echoing lines, that is, you haven't used
+ the -e switch, it will show the _first_ line of the paragraph,
+ to help you find the place without using line numbers. The
+ offending paragraph is therefore between the quoted line and
+ the line number given.
+
+
+
+ Line 2587 - Mismatched single quotes
+
+ Only checked with the -s switch, since checking single quotes is
+ not a very reliable process. Otherwise, the same logic as for
+ doublequotes applies.
+
+
+
+ Line 2877 - Mismatched round brackets?
+
+ Also curly and square brackets. Texts with a lot of brackets, like
+ plays with bracketed stage instructions, may have mismatches.
+
+
+ Line 3150 - No CR?
+ Line 3204 - Two successive CRs?
+ Line 3281 position 75 - CR without LF?
+
+ These are the invalid line-end warnings. See the discussion of
+ line-end checking in the switches section near the start of this
+ file. If you see these, and your editor doesn't show anything
+ wrong, you should probably try deleting the characters just before
+ and after the line end, and the line-end itself, then retyping the
+ characters and the line-end.
+
+
+ Line 2940 - Paragraph starts with lower-case
+
+ A common error in an e-text is for an extra blank line
+
+ to be put in, like the blank line above, and this often
+ shows up as a new paragraph beginning with lower case.
+ Sometimes the blank line is deliberate, as when a
+ quotation is inserted in a speech. Use your judgement.
+
+
+ Line 2987 - Extra period?
+
+ An extra period. is a. common problem in OCRed text. and usually
+ arises when a speck of dust on the page is mistaken for a period.
+ or. as occasionally happens. when a comma loses its tail.
+
+
+ Line 3012 column 12 - Double punctuation?
+
+ Double punctuation., like that,, is a common typo and
+ scanno. Some books have much legit double punctuation,
+ like etc., etc., but it's worth checking anyway.
+
+
+
+ * * * *
+
+For Windows-only users who are unfamiliar with DOS:
+
+ If you're a Windows-only user, you need to save
+ gutcheck.exe into the folder (directory) where the
+ text file you want to check is. Let's say your
+ text file is in C:\GUT, then you should save
+ GUTCHECK.EXE into C:\GUT.
+
+ Now get to a DOS prompt. You can do this by
+ selecting the "Command Prompt" or "MS-DOS Prompt"
+ option that will be somewhere on your
+ Start/Programs menu.
+
+ Now get into the C:\GUT directory.
+ You can do this using the CD (change directory)
+ command, like this:
+ CD \GUT
+ and your prompt will change to
+ C:\GUT>
+ so you know you're in the right place.
+
+ Now type
+ gutcheck yourfile.txt
+ and you'll see gutcheck's report
+
+ By default, gutcheck prints its queries to screen.
+ If you want to create a file of them, to edit
+ against the text, you can use the greater-than
+ sign (>) to tell it to output the report to a
+ file. For example, if you want its report in a
+ file called QUERIES.LST, you could type
+
+ gutcheck yourfile.txt > queries.lst
+
+ The queries.lst file will then contain the listing
+ of possible formatting errors, and you can
+ edit it alongside your text.
+
+ Whatever you do, DON'T make the filename after
+ the greater-than sign the name of a file already
+ on your disk that you want to keep, because
+ the greater-than sign will cause gutcheck to
+ replace any existing file of that name.
+
+ So, for example, if you have two Tolstoy files
+ that you want to check, called WARPEACE.TXT and
+ ANNAK.TXT, make sure that neither of these names
+ is ever used following the greater-than sign.
+ To check these correctly, you might do:
+
+ gutcheck warpeace.txt >war.lst
+
+ and
+
+ gutcheck annak.txt > annak.lst
+
+ separately. Then you can look at war.lst and annak.lst
+ to see the gutcheck reports.
+
+ * * * *
+
+
+For existing 0.98 users upgrading to 0.99:
+
+ If you run on old 16-bit DOS or Windows 3.x, I'm afraid
+ you're out of luck. I'm not saying it _can't_ be compiled
+ to run on 16-bit, but the executable with the package is
+ for Win32 only. *nix users won't notice the change at all.
+
+
+ There are two new switches: -u and -d.
+ See above for full rundown.
+
+
+Here's a list of the new errors:
+
+ Line 1456 - Carat character?
+
+ I^ve found a few.
+
+
+ Line 1821 - Forward slash?
+
+ Common error for italicized "I", or so /'ve found.
+
+
+ Line 2139 - Query missing paragraph break?
+
+ "Come here, son." "Do I _have_ to go, dad?"
+ Like that. False positives in some texts. Sorry 'bout that,
+ but these are often errors.
+
+
+ Line 2200 - Query had/bad error?
+
+ Clear enough. Doesn't catch as many as I'd like it to,
+ but rarely gives false alarms.
+
+
+ Line 2268 - Query punctuation after the?
+
+ Some words, like "the", very rarely have punctuation
+ following them. Others, like "Mrs", usually have a
+ period, but never a comma. Occasional false positives.
+
+
+ Line 2380 - Query possible scanno arid
+
+ It found one of your user-defined typos when you
+ used the -u switch.
+
+
+ Line 2511 - Capital "S"?
+
+ Surprisingly common specific case, like: Jane'S
+
+
+ Line 3469 - endquote missing punctuation?
+
+ OK. This one can really cause a lot of false positives
+ in some books, but it switches itself off if it finds
+ more than 20 in a text, unless you force it to list them
+ all with the -v switch.
+ "Hey, dad" Johnny said, "can we go now?"
+ is a common punctuation-missing error.
+
+
+ Line 4266 - Mismatched underscores?
+
+ Like mismatched anything else!
+
+