Last updated 3 days ago
Was this helpful?
Model
The model to tokenize with.
Text
The text content to be tokenized.
token_ids[]
A list of tokens.
token_ids[].token_id
ID of the token
token_ids[].string_token
Origin text
token_ids[].token_bytes
Converted token values
