fastai: tokenize_folder – R documentation

Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!

tokenize_folder

Tokenize_folder

Description

Tokenize text files in 'path' in parallel using 'n_workers'

Usage

tokenize_folder(
  path,
  extensions = NULL,
  folders = NULL,
  output_dir = NULL,
  skip_if_exists = TRUE,
  output_names = NULL,
  n_workers = 6,
  rules = NULL,
  tok = NULL,
  encoding = "utf8"
)

Arguments

`path`	path
`extensions`	extensions
`folders`	folders
`output_dir`	output_dir
`skip_if_exists`	skip_if_exists
`output_names`	output_names
`n_workers`	number of workers
`rules`	rules
`tok`	tokenizer
`encoding`	encoding

Value

None

fastai

Interface to 'fastai'

v2.0.7

Apache License 2.0

Authors

Turgut Abdullayev [ctb, cre, cph, aut]

Initial release