달빛약속 / tokenize

Function: tokenize()

tokenize(text, splitpoints): Token[]

Parameters

• text: string

• splitpoints: Brand<number, "Splitpoint">[] = []

Returns

Token[]

Defined in

core/prepare/tokenize/index.ts:208