The built-in tokenizers split text into words, n-grams, or dates. When none
of them fits your data, write your own and place it in the strategy formula
with custom_tokens():
name ~ normalize_text() + custom_tokens(my_tokenizer).
Arguments
- text
A character vector to tokenize.
- tokenizer
A plain function, or an object with a
tokenize()method.
Details
The tokenizer can be a plain function that takes a character vector and
returns a list of token vectors. It can also be a fitted tokenizer object
with a tokenize() method, which is how a tokenizer with learned state,
such as a trained subword vocabulary, enters a strategy.
Whichever form you pass, it must return a list with one element per input
string, each element a character vector of that record's tokens
(character(0) for a record with none). custom_tokens() checks this and
stops with a clear error if the result does not fit.
See also
tokenize(), the method a fitted tokenizer provides;
word_tokens() for the standard whitespace tokenizer.
Other token generators:
generate_ngrams(),
numeric_tokens(),
subword_tokens(),
word_tokens()
