Drop short tokens from token lists

Removes tokens shorter than min_nchar characters from a token column. Where word_tokens()'s own min_nchar filters length at tokenisation, this filters length after a token transform - which is where it matters for the phonetic encoders (as_cologne(), as_soundex(), as_metaphone()) and generate_ngrams(): those produce short codes, and a 1-2 character code maps to a very large equivalence class (low distinctiveness), so it behaves as a false-match magnet. Chain drop_short_tokens() after the encoder to keep only the discriminative codes.

Operates on the list-of-character token vectors produced by earlier steps, mirroring filter_stopwords() / drop_numeric_tokens().

Usage

drop_short_tokens(tokens, min_nchar = 2)

Arguments

tokens

A list of character vectors.

min_nchar

Whole number; tokens with fewer than this many characters are dropped. Default 2.

Value

A list of character vectors with short tokens removed.

See also

filter_stopwords() and drop_numeric_tokens() for the same list-column idea with other drop rules; word_tokens() for the same length cut applied at tokenisation instead.

Other token transformers: drop_numeric_tokens(), extract_initials(), filter_stopwords(), fuzzy_tokens(), token_shapes(), use_dictionary()

Examples

drop_short_tokens(list(c("BAU", "AG", "X")))
#> [[1]]
#> [1] "BAU" "AG" 
#> 
# list(c("BAU", "AG"))  # the 1-char token is dropped at the default min_nchar = 2

# keep only Cologne codes of 4+ digits (drops the collision-prone short class)
drop_short_tokens(as_cologne(list(c("Bülau", "Mertens"))), min_nchar = 4)
#> [[1]]
#> [1] "67268"
#> 
# list("67268")