R/text_core.R
BaseTokenizer.Rd
Basic tokenizer that just splits on spaces
BaseTokenizer(split_char = " ")
separator
None