enchant.tokenize: String tokenization functions for PyEnchant¶
An important task in spellchecking is breaking up large bodies of text into their constituent words, each of which is then checked for correctness. This package provides Python functions to split strings into words according to the rules of a particular language.
Each tokenization function accepts a string as its only positional argument, and returns an iterator that yields tuples of the following form, one for each word found:
The meanings of these fields should be clear: word is the word
that was found and pos is the position within the text at which
the word began (zero indexed, of course). The function will work
on any string-like object that supports array-slicing; in particular
character-array objects from the
array module may be used.
The iterator also provides the attribute
offset which gives the current
position of the tokenizer inside the string being split, and the method
set_offset() for manually adjusting this position. This can be used for
example if the string’s contents have changed during the tokenization
To obtain an appropriate tokenization function for the language
identified by tag, use the function
tknzr = get_tokenizer("en_US") for (word,pos) in tknzr("text to be tokenized goes here") do_something(word)
This library is designed to be easily extendible by third-party
authors. To register a tokenization function for the language
tag, implement it as the function tokenize within the
module enchant.tokenize.<tag>. The function
will automatically detect it. Note that the underscore must be
used as the tag component separator in this case, in order to
form a valid python module name. (e.g. “en_US” rather than “en-US”)
Currently, a tokenizer has only been implemented for the English language. Based on the author’s limited experience, this should be at least partially suitable for other languages.
This module also provides various implementations of Chunkers and Filters. These classes are designed to make it easy to work with text in a variety of common formats, by detecting and excluding parts of the text that don’t need to be checked.
Chunker is a class designed to break a body of text into large chunks
of checkable content; for example the
HTMLChunker class extracts the
text content from all HTML tags but excludes the tags themselves.
Filter is a class designed to skip individual words during the checking
process; for example the
URLFilter class skips over any words that
have the format of a URL.
For example, to spellcheck an HTML document it is necessary to split the text into chunks based on HTML tags, and to filter out common word forms such as URLs and WikiWords. This would look something like the following:
tknzr = get_tokenizer("en_US",(HTMLChunker,),(URLFilter,WikiWordFilter))) text = "<html><body>the url is http://example.com</body></html>" for (word,pos) in tknzer(text): ...check each word and react accordingly...
- class enchant.tokenize.Chunker(text: str)¶
Base class for text chunking functions.
A chunker is designed to chunk text into large blocks of tokens. It has the same interface as a tokenizer but is for a different purpose.
- class enchant.tokenize.EmailFilter(tokenizer: Type[tokenize] | Filter)¶
Filter skipping over email addresses. This filter skips any words matching the following regular expression:
That is, any words that resemble email addresses.
- class enchant.tokenize.Filter(tokenizer: Type[tokenize] | Filter)¶
Base class for token filtering functions.
A filter is designed to wrap a tokenizer (or another
Filter) and do two things:
skip over tokens
split tokens into sub-tokens
Subclasses have two basic options for customising their behaviour. The method
_skip()may be overridden to return True for words that should be skipped, and False otherwise. The method
_split()may be overridden as tokenization function that will be applied to further tokenize any words that aren’t skipped.
- class enchant.tokenize.HTMLChunker(text: str)¶
Chunker for breaking up HTML documents into chunks of checkable text.
The operation of this chunker is very simple - anything between a “<” and a “>” will be ignored. Later versions may improve the algorithm slightly.
- class enchant.tokenize.HashtagFilter(tokenizer: Type[tokenize] | Filter)¶
Filter skipping over #hashtag. This filter skips any words matching the following regular expression:
That is, any words that are #hashtag.
- class enchant.tokenize.MentionFilter(tokenizer: Type[tokenize] | Filter)¶
Filter skipping over @mention. This filter skips any words matching the following regular expression:
That is, any words that are @mention.
- class enchant.tokenize.URLFilter(tokenizer: Type[tokenize] | Filter)¶
Filter skipping over URLs. This filter skips any words matching the following regular expression:
That is, any words that are URLs.
- class enchant.tokenize.WikiWordFilter(tokenizer: Type[tokenize] | Filter)¶
Filter skipping over WikiWords. This filter skips any words matching the following regular expression:
That is, any words that are WikiWords.
- class enchant.tokenize.basic_tokenize(text: str)¶
Tokenizer class that performs very basic word-finding.
This tokenizer does the most basic thing that could work - it splits text into words based on whitespace boundaries, and removes basic punctuation symbols from the start and end of each word.
- class enchant.tokenize.empty_tokenize¶
Tokenizer class that yields no elements.
- enchant.tokenize.get_tokenizer(tag: str | None = None, chunkers: Iterable[Type[Chunker] | Type[Filter]] | None = None, filters: Iterable[Type[Filter]] | None = None) tokenize ¶
Locate an appropriate tokenizer by language tag.
This requires importing the function tokenize from an appropriate module. Modules tried are named after the language tag, tried in the following order:
the entire tag (e.g. “en_AU.py”)
the base country code of the tag (e.g. “en.py”)
If the language tag is None, a default tokenizer (actually the English one) is returned. It’s unicode aware and should work OK for most latin-derived languages.
If a suitable function cannot be found, raises
If given and not None, chunkers and filters must be lists of chunker classes and filter classes respectively. These will be applied to the tokenizer during creation.
- class enchant.tokenize.tokenize(text: str)¶
Base class for all tokenizer objects.
Each tokenizer must be an iterator and provide the
offsetattribute as described in the documentation for this module.
While tokenizers are in fact classes, they should be treated like functions, and so are named using lower_case rather than the CamelCase more traditional of class names.
- class enchant.tokenize.unit_tokenize(text: str)¶
Tokenizer class that yields the text as a single token.
- enchant.tokenize.wrap_tokenizer(tk1: Type[tokenize] | Filter, tk2: Type[tokenize] | Filter) Filter ¶
Wrap one tokenizer inside another.
This function takes two tokenizer functions tk1 and tk2, and returns a new tokenizer function that passes the output of tk1 through tk2 before yielding it to the calling code.