When you already know which tokens mean the same thing (a curated synonym
list, brand-name variants, a code-to-label table), use_dictionary() rewrites
each token to its group label so the variants collapse to one token and match.
Use it when the mapping is known in advance; when you instead want joinery to
discover near-duplicates from the data, use fuzzy_tokens().
Arguments
- text
A character vector of tokens to look up.
- dict
A data.table::data.table with a
tokenscolumn and atoken_groupcolumn. Rows whosetokensvalue matches an input token supply that token's group label.
Value
A list of character vectors, one per input element, holding the
matched group labels (empty when the token is not in dict).
Details
Tokens absent from the dictionary return no group, so chain this after a token generator and keep a sharper field alongside it.
See also
fuzzy_tokens() to discover groups from the data instead.
Other token transformers:
drop_numeric_tokens(),
drop_short_tokens(),
extract_initials(),
filter_stopwords(),
fuzzy_tokens(),
token_shapes()
Examples
dict <- data.table::data.table(
tokens = c("example", "sample"),
token_group = c("example/sample", "example/sample")
)
use_dictionary("example", dict)
#> [[1]]
#> [1] "example/sample"
#>
use_dictionary("nonexistent", dict)
#> [[1]]
#> character(0)
#>