function tokenizing rtext objects
function tokenizing rtext objects
## S3 method for class 'rtext' text_tokenize( string, regex = NULL, ignore.case = FALSE, fixed = FALSE, perl = FALSE, useBytes = FALSE, non_token = FALSE )
string |
text to be tokenized |
regex |
regex expressing where to cut see (see grep) |
ignore.case |
whether or not reges should be case sensitive (see grep) |
fixed |
whether or not regex should be interpreted as is or as regular expression (see grep) |
perl |
whether or not Perl compatible regex should be used (see grep) |
useBytes |
byte-by-byte matching of regex or character-by-character (see grep) |
non_token |
should information for non-token, i.e. those patterns by which the text was splitted, be returned as well |
Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.