Remove sentences based on their token lengths or a pattern match
Removes sentences from a corpus or a character vector shorter than a specified length.
corpus_trim(
x,
what = c("sentences", "paragraphs", "documents"),
min_ntoken = 1,
max_ntoken = NULL,
exclude_pattern = NULL
)
char_trim(
x,
what = c("sentences", "paragraphs", "documents"),
min_ntoken = 1,
max_ntoken = NULL,
exclude_pattern = NULL
)x |
corpus or character object whose sentences will be selected. |
what |
units of trimming, |
min_ntoken, max_ntoken |
minimum and maximum lengths in word tokens (excluding punctuation) |
exclude_pattern |
a stringi regular expression whose match (at the sentence level) will be used to exclude sentences |
a corpus or character vector equal in length to the input. If
the input was a corpus, then the all docvars and metadata are preserved.
For documents whose sentences have been removed entirely, a null string
("") will be returned.
txt <- c("PAGE 1. This is a single sentence. Short sentence. Three word sentence.",
"PAGE 2. Very short! Shorter.",
"Very long sentence, with multiple parts, separated by commas. PAGE 3.")
corp <- corpus(txt, docvars = data.frame(serial = 1:3))
corp
# exclude sentences shorter than 3 tokens
corpus_trim(corp, min_ntoken = 3)
# exclude sentences that start with "PAGE <digit(s)>"
corpus_trim(corp, exclude_pattern = "^PAGE \\d+")
# trimming character objects
char_trim(txt, "sentences", min_ntoken = 3)
char_trim(txt, "sentences", exclude_pattern = "sentence\\.")Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.