cmdTokenizerobject
Command tokenizer for US English. Other language modules should provide their own tokenizers to allow for differences in punctuation and other lexical elements.
cmdTokenizer : Tokenizer
Superclass Tree (in declaration order)
cmdTokenizer
Tokenizer
` object`
Summary of Properties
patAlphaDashAlpha patPunct patSpelledTens patSpelledUnits rules_
Summary of Methods
acceptAbbrTok buildOrigText tokCvtAbbr tokCvtApostropheS tokCvtPluralApostrophe tokCvtSpelledNumber
Inherited from Tokenizer :
deleteRule deleteRuleAt insertRule insertRuleAt tokCvtLower tokCvtSkip tokenize
Properties
patAlphaDashAlpha
add the part after the hyphen
patPunct
no description available
patSpelledTens
some pre-compiled regular expressions
patSpelledUnits
no description available
rules_OVERRIDDEN
no description available
Methods
acceptAbbrTok (txt)
Check to see if we want to accept an abbreviated token - this is a token that ends in a period, which we use for abbreviated words like “Mr.” or “Ave.” We’ll accept the token only if it appears as given - including the period - in the dictionary. Note that we ignore truncated matches, since the only way we’ll accept a period in a word token is as the last character; there is thus no way that a token ending in a period could be a truncation of any longer valid token.
buildOrigText (toks)
Given a list of token strings, rebuild the original input string. We can’t recover the exact input string, because the tokenization process throws away whitespace information, but we can at least come up with something that will display cleanly and produce the same results when run through the tokenizer.
tokCvtAbbr (txt, typ, toks)
Process an abbreviated token.
When we find an abbreviation, we’ll enter it with the abbreviated word minus the trailing period, plus the period as a separate token. We’ll mark the period as an “abbreviation period” so that grammar rules will be able to consider treating it as an abbreviation – but since it’s also a regular period, grammar rules that treat periods as regular punctuation will also be able to try to match the result. This will ensure that we try it both ways - as abbreviation and as a word with punctuation - and pick the one that gives us the best result.
tokCvtApostropheS (txt, typ, toks)
Handle an apostrophe-s word. We’ll return this as two separate tokens: one for the word preceding the apostrophe-s, and one for the apostrophe-s itself.
tokCvtPluralApostrophe (txt, typ, toks)
Handle a plural apostrophe word (“the smiths’ house”). We’ll return this as two tokens: one for the plural word, and one for the apostrophe.
tokCvtSpelledNumber (txt, typ, toks)
Handle a spelled-out hyphenated number from 21 to 99. We’ll return this as three separate tokens: a word for the tens name, a word for the hyphen, and a word for the units name.
TADS 3 Library Manual
Generated on 5/16/2013 from TADS version 3.1.3