Convert tokens to shape signatures

Reduces each token to its letter/digit pattern: every letter becomes "A", every digit "N", anything else "X". The signature ignores the actual characters and keeps only the layout, which is useful for matching on the format of a code or identifier (postal codes, licence plates, product codes) rather than its exact value, or as a coarse blocking key.

Usage

token_shapes(tokens)

Arguments

tokens

A list of character vectors.

Value

A list of character vectors of shape signatures, one signature per input token.

Details

It transforms a token column, so it runs after a token generator such as word_tokens().

See also

Other token transformers: drop_numeric_tokens(), drop_short_tokens(), extract_initials(), filter_stopwords(), fuzzy_tokens(), use_dictionary()

Examples

token_shapes(list(c("MUELLER", "A12B")))
#> [[1]]
#> [1] "AAAAAAA" "ANNA"   
#> 
# list(c("AAAAAAA", "ANNA"))