HF_TokenClassBeforeBatchTransform
Handles everything you need to assemble a mini-batch of inputs and targets, as well as decode the dictionary produced
HF_TokenClassBeforeBatchTransform( hf_arch, hf_tokenizer, ignore_token_id = -100, max_length = NULL, padding = TRUE, truncation = TRUE, is_split_into_words = TRUE, n_tok_inps = 1, ... )
hf_arch |
architecture |
hf_tokenizer |
tokenizer |
ignore_token_id |
ignore token id |
max_length |
maximum length |
padding |
padding or not |
truncation |
truncation or not |
is_split_into_words |
to split into_words |
n_tok_inps |
number tok inputs |
... |
additional arguments |
as a byproduct of the tokenization process in the 'encodes' method.
None
Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.