Tokenize Text

Tokenize Text

Graiphic · 2025-09-04

Public technical reference documentation from Graiphic. For current SOTA installation and toolkit guides, visit the documentation center.

Description

Tokenize input text and store it in a ptr.

Documentation illustration (illustration unavailable in the archive).

Input parameters

ONNX in : object, tokenizer session.
text : string, the raw text input that will be transformed into a sequence of tokens by the tokenizer.

Output parameters

Documentation illustration (illustration unavailable in the archive). ONNX out : object, tokenizer session.

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).