Generate Streaming Text

Public technical reference documentation from Graiphic. For current SOTA installation and toolkit guides, visit the documentation center.

Description

Runs text generation with the Llama model from a preprocessed sequence of tokens. The model generates the output incrementally and returns tokens one by one as they are produced (streaming mode).

Documentation illustration (illustration unavailable in the archive).

Input parameters

ONNX in : object, llama generator session.

Output parameters

Documentation illustration (illustration unavailable in the archive). ONNX out : object, llama generator session.
output_string : string, the latest generated token, returned as a string.
Documentation illustration (illustration unavailable in the archive). is_done : boolean, indicates whether the text generation process has finished.

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).