R/text_core.R
tokenize1.Rd
Call `TokenizeWithRules` with a single text
tokenize1(text, tok, rules = NULL, post_rules = NULL)
text
tok
rules
post_rules
None