Filter out stopwords from token lists

Some tokens carry no matching signal but appear everywhere: legal forms like GMBH or LTD, articles, generic words. Because they are common they create many spurious matches and fan out blocks. filter_stopwords() removes named tokens so matching rests on the distinctive ones. The comparison is case-insensitive.

Usage

filter_stopwords(tokens, stopwords)

Arguments

tokens

A list of character vectors, as produced by word_tokens().

stopwords

A character vector of tokens to remove (case-insensitive).

Value

A list of character vectors with the stopwords removed.

Details

It transforms a token column, so it runs after a token generator such as word_tokens().

See also

drop_numeric_tokens() to remove house numbers the same way.

Other token transformers: drop_numeric_tokens(), drop_short_tokens(), extract_initials(), fuzzy_tokens(), token_shapes(), use_dictionary()

Examples

filter_stopwords(list(c("MUELLER", "GMBH")), stopwords = c("gmbh"))
#> [[1]]
#> [1] "MUELLER"
#> 
# list("MUELLER")