Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!
Get Started for Free
Documentation
R
fastai

HF_CausalLMBeforeBatchTransform

HF_CausalLMBeforeBatchTransform

Description

Handles everything you need to assemble a mini-batch of inputs and targets, as well as decode the dictionary produced

Usage

HF_CausalLMBeforeBatchTransform(
  hf_arch,
  hf_tokenizer,
  max_length = NULL,
  padding = TRUE,
  truncation = TRUE,
  is_split_into_words = FALSE,
  n_tok_inps = 1,
  ignore_token_id = -100,
  ...
)

Arguments

hf_arch

architecture
hf_tokenizer

tokenizer
max_length

maximum length
padding

padding or not
truncation

truncation or not
is_split_into_words

to split into words
n_tok_inps

number tok inputs
ignore_token_id

ignore token id
...

additional arguments

Details

as a byproduct of the tokenization process in the 'encodes' method.

Value

None

fastai

Interface to 'fastai'

v2.0.7
Apache License 2.0
Authors
Turgut Abdullayev [ctb, cre, cph, aut]
Initial release

We don't support your browser anymore

Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.