Create Strings as tokens...
|
|
Create a new Strings object as a list of tokens.
Settings
-
Text
-
the text to be tokenized.
-
Separators
-
determines the separator characters. If left empty a space will be used as a separator
Behaviour
Multiple consecutive separators in the text will be treated as one.
Examples
-
Example 1
Create Strings as tokens: "a b c", " "
-
will produce a Strings object with 3 strings in it: "a", "b" and "c".
-
Example 2
Create Strings as tokens: "a b c ", " "
-
will also produce a Strings object with 3 strings in it: "a", "b" and "c".
-
Example 3
Create Strings as tokens: "a,b,c", ","
-
will produce a Strings object with 3 strings in it: "a", "b" and "c".
-
Example 4
Create Strings as tokens: "a, b, c", ","
-
will produce a Strings object with 3 strings in it: "a", " b" and " c".
-
Example 5
Create Strings as tokens: "a, b, c", " ,"
-
will produce a Strings object with 3 strings in it: "a", "b" and "c".
-
Example 6
Create Strings as tokens: "a,,b,c", " ,"
-
will also produce a Strings object with 3 strings in it: "a", "b" and "c".
-
Example 7
Create Strings as tokens: "a, ,b,c", ","
-
will produce a Strings with 4 strings in it: "a"," ", "b" and "c".
-
Example 8
Create Strings as tokens: "a,b,c,", ","
-
will produce a Strings object with 3 strings in it: "a", "b" and "c".
-
Example 9
Create Strings as tokens: "a,b,c, ", ","
-
will produce a Strings object with 4 strings in it: "a", "b", "c" and " ".
-
Example 10
Create Strings as tokens: "A string" + tab$+ "of ..tokens" + newline$+ "and some more tokens", " .," + tab$+ newline$
-
will produce a Strings object with 8 strings in it: "A", "string", "of", "tokens", "and", "some", "more" and "tokens".
© djmw, April 17, 2017