Word tokens fail quietly on compound words and misspellings:
"Maschinenbaugesellschaft" and "Maschinenbau GmbH" share no word, so
they never meet in a word-token match. Subword tokens fix this by learning,
from your own data, a vocabulary of frequent word pieces and splitting every
record into those pieces. Two spellings of the same name then share most of
their pieces even when they share no whole word.
find_subwords() is the learning step. Run it once on the text of a column
(after the same cleaning you use in your strategy), then place the fitted
model in the strategy formula with subword_tokens():
sw <- find_subwords(normalize_text(register$name))
strat <- search_strategy(
name ~ normalize_text() + subword_tokens(sw)
)The fitted model is self-contained: you can save it with saveRDS() and
reuse it in a later session, and the same model always splits text the same
way.
Usage
find_subwords(
x,
vocab_size = 500L,
algorithm = c("bpe", "unigram"),
keep_boundary = FALSE,
sample_n = NULL,
columns = NULL
)Arguments
- x
The training text: a character vector, or a table (data frame, tibble, data.table, DuckDB table) together with
columns.- vocab_size
Number of subword pieces to learn. Default
500.- algorithm
"bpe"(default) or"unigram", the two learning algorithms SentencePiece offers. Start with"bpe"; try"unigram"if its splits look better on your data.- keep_boundary
Keep the word-boundary marker on each piece? The default
FALSEstrips it, so the same piece matches whether it starts a word or not; that is usually what record linkage wants.- sample_n
Optional cap on the number of rows used for training. On a corpus of millions of rows a uniform sample of a few hundred thousand fits an equivalent vocabulary in a fraction of the time.
- columns
For table input: character vector naming the text column(s) to train on.
Value
A fitted Subword_Model, ready for subword_tokens().
Details
Fit the model on the text as the strategy will see it: apply the same
cleaning steps (for example normalize_text()) to the training text that
precede subword_tokens() in the formula. Training on raw text and
applying to cleaned text (or the other way round) degrades the splits.
vocab_size sets how fine the splits are. Small vocabularies split
aggressively into short pieces; large ones keep frequent words whole and
only split rare ones. The default of 500 suits mid-sized name corpora;
large corpora often support 1000 to 8000. On a very small corpus the
fitted vocabulary can come out smaller than requested; the model records
the size actually fitted. rarity_distribution() on the prepared tokens
is the read for judging the result.
Very frequent single-character pieces are normal and mostly harmless: they
carry almost no rarity, so they contribute little to a score. Compose
subword_tokens(sw) + drop_short_tokens(2) to drop them outright.
See also
subword_tokens() to use the model in a strategy;
find_stopwords() for the same fit-then-apply pattern on stopwords.
Examples
# \donttest{
if (requireNamespace("sentencepiece", quietly = TRUE)) {
firms <- rep(c(
"maschinenbau mueller", "tischlerei schmidt",
"holzbau wagner", "metallbau krause"
), 40)
sw <- find_subwords(firms, vocab_size = 60)
sw
subword_tokens("maschinenbaugesellschaft mueller", sw)
}
#> [[1]]
#> [1] "ma" "sch" "inen" "bau" "g" "e" "se" "ll" "sch" "a"
#> [11] "f" "t" "mue" "ller"
#>
# }
