fastai: tokenize_csv – R documentation

Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!

tokenize_csv

Tokenize_csv

Description

Tokenize texts in the 'text_cols' of the csv 'fname' in parallel using 'n_workers'

Usage

tokenize_csv(
  fname,
  text_cols,
  outname = NULL,
  n_workers = 4,
  rules = NULL,
  mark_fields = NULL,
  tok = NULL,
  header = "infer",
  chunksize = 50000
)

Arguments

`fname`	file name
`text_cols`	text columns
`outname`	outname
`n_workers`	numeber of workers
`rules`	rules
`mark_fields`	mark fields
`tok`	tokenizer
`header`	header
`chunksize`	chunk size

Value

None

fastai

Interface to 'fastai'

v2.0.7

Apache License 2.0

Authors

Turgut Abdullayev [ctb, cre, cph, aut]

Initial release