Become an expert in R — Interactive courses, Cheat Sheets, certificates and more!
Get Started for Free

tokenize_df

Tokenize_df

Description

Tokenize texts in 'df[text_cols]' in parallel using 'n_workers'

Usage

tokenize_df(
  df,
  text_cols,
  n_workers = 6,
  rules = NULL,
  mark_fields = NULL,
  tok = NULL,
  res_col_name = "text"
)

Arguments

df

data frame
text_cols

text columns
n_workers

number of workers
rules

rules
mark_fields

mark_fields
tok

tokenizer
res_col_name

res_col_name

Value

None

fastai

Interface to 'fastai'

v2.0.7
Apache License 2.0
Authors
Turgut Abdullayev [ctb, cre, cph, aut]
Initial release

We don't support your browser anymore

Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.