HF_TokenClassBeforeBatchTransform

Handles everything you need to assemble a mini-batch of inputs and targets, as well as decode the dictionary produced

HF_TokenClassBeforeBatchTransform(
  hf_arch,
  hf_tokenizer,
  ignore_token_id = -100,
  max_length = NULL,
  padding = TRUE,
  truncation = TRUE,
  is_split_into_words = TRUE,
  n_tok_inps = 1,
  ...
)

Arguments

hf_arch

architecture

hf_tokenizer

tokenizer

ignore_token_id

ignore token id

max_length

maximum length

padding

padding or not

truncation

truncation or not

is_split_into_words

to split into_words

n_tok_inps

number tok inputs

...

additional arguments

Value

None

Details

as a byproduct of the tokenization process in the `encodes` method.