R/text_core.R
TokenizeWithRules.Rd
A wrapper around `tok` which applies `rules`, then tokenizes, then applies `post_rules`
TokenizeWithRules(tok, rules = NULL, post_rules = NULL)
tokenizer
rules
post_rules
None