tokenize
tokenize(
text,splitpoints?):Token[]
Defined in: core/prepare/tokenize/index.ts:212
Parameters
Section titled “Parameters”string
splitpoints?
Section titled “splitpoints?”Splitpoint[] = []
Returns
Section titled “Returns”Token[]
tokenize(
text,splitpoints?):Token[]
Defined in: core/prepare/tokenize/index.ts:212
string
Splitpoint[] = []
Token[]