TensorRT

Sessions Parameters : cluster

intra_op_num_threads : integer, number of threads used within each operator to parallelize computations. If the value is 0, ONNX Runtime automatically uses the number of physical CPU cores.
inter_op_num_threads : integer, number of threads used between operators, to execute multiple graph nodes in parallel. If set to 0, this parameter is ignored when execution_mode is ORT_SEQUENTIAL. In ORT_PARALLEL mode, 0 means ORT automatically selects a suitable number of threads (usually equal to the number of cores).
execution_mode : enum, controls whether the graph executes nodes one after another or allows parallel execution when possible.ORT_SEQUENTIAL runs nodes in order, ORT_PARALLEL runs them concurrently.
deterministic_compute : boolean, forces deterministic execution, meaning results will always be identical for the same inputs.
graph_optimization_level : enum, defines how much ONNX Runtime optimizes the computation graph before running the model.
optimized_model_file_path : path, file path to save the optimized model after graph analysis.
profiling output dir : path, specifies the directory where ONNX Runtime will save profiling output files. If you set this parameter to a valid (non-empty) path, profiling is automatically enabled. However, if the path is empty, profiling will not be activated.

TRT Parameters : cluster

Device And Stream : cluster,

Device Index : integer, ID of the GPU used for inference (default = 0).
Use Custom Compute Stream? : boolean, if true, uses a user-defined CUDA compute stream instead of the default stream.
Custom Compute Stream (Pointer) : integer, pointer or reference to the user’s CUDA stream to be used during inference.

Parser & Subgraph Options : cluster,

Max Partition Attemps : integer, maximum number of attempts for TensorRT to partition and identify compatible subgraphs within the ONNX model.
Minimum Subgraph Size : integer, minimum number of nodes required in a subgraph before TensorRT can take ownership of it.

Memory & Workspace Management : cluster,

Max Workspace Memory (Bytes) : integer, maximum workspace memory allocated for TensorRT (0 = use maximum available GPU memory).
Share Memory Between Subgraphs : boolean, if true, allows multiple TensorRT subgraphs to reuse the same execution context and memory buffers.
Auxiliary Stream Count : integer, defines how many auxiliary CUDA streams are created per main inference stream.

• • • • -1 = automatic heuristic mode
      • 0 = minimize memory usage
      • >0 = allow multiple auxiliary streams for parallel execution

Precision & Numeric Modes : cluster,

Enable FP16 : boolean, enables half-precision (float16) computation for faster performance on supported GPUs.
Enable BF16 : boolean, enables bfloat16 precision for models trained in BF16 format.
Enable INT8 : boolean, enables INT8 quantization for maximum inference speed.
INT8 Calibration Table Name : path, specifies the name or path of the calibration table used for INT8 mode.
Use Native INT8 Calibration? : boolean, if true, uses the calibration table generated directly by TensorRT.

Engine Build, Cache & Paths : cluster,

Enable Engine Caching? : boolean, saves compiled TensorRT engines to disk to avoid rebuilding on future runs.
Engine Cache Directory : path, path where the TensorRT engine cache will be stored.
Enable Engine Decryption? : boolean, enables support for encrypted engine files.
Decryption Library Path : path, path to the library used to decrypt TensorRT engine files.
Build Engines Sequentially? : boolean, forces TensorRT to build engines one at a time instead of in parallel.
Enable Timing Cache? : boolean, enables TensorRT’s timing cache to accelerate repeated builds.
Timing Cache Directory : path, path where timing cache data will be stored.
Force Timing Cache Use? : boolean, forces the reuse of the timing cache even if the device profile has changed.
Enable Detailed Logs? : boolean, enables detailed logging for each build step and timing stage.
Use Build Heuristics? : boolean, uses heuristic algorithms to reduce engine build time (may affect optimal performance).
Enable Weight Sparsity? : boolean, allows TensorRT to exploit sparsity in model weights for better performance.
Engine Optimization Level (0–5)) : integer, controls the engine builder optimization level.

• • • • 0–2 = fast build, reduced performance
      • 3 = default balance between speed and quality
      • 4–5 = best performance, longer build time

Paths & Plugins : cluster,

Extra Plugin Libraries : array, list (semicolon-separated) of additional plugin library paths to load.
Tactic Source Rules : string, defines which tactic sources TensorRT should include or exclude (e.g. “-CUDNN,+CUBLAS”).
Enabled Preview Features : enum, comma-separated list of experimental TensorRT features to enable (e.g. “ALIASED_PLUGIN_IO_10_03”).

Profile Shapes : cluster,

Minimum Input Shapes : string, specifies the smallest input dimensions used to build the TensorRT engine.
Maximum Input Shapes : string, specifies the “typical” input size that TensorRT will optimize for.
Optimal Input Shapes : string, defines the upper bound for dynamic input dimensions accepted by the engine.

Advanced & DLA / CUDA Graph : cluster,

Enable DLA Acceleration? : boolean, enables inference execution on NVIDIA’s Deep Learning Accelerator (if supported).
DLA Core Index : integer, selects which DLA core to use (0 = default).
Dump TensorRT Subgraphs? : boolean, dumps all TensorRT-compiled subgraphs to disk for debugging.
Force LayerNorm in FP32? : boolean, forces LayerNorm operations to run in full precision (FP32) for numerical stability.
Enable CUDA Graph Execution? : boolean, executes the TensorRT engine using CUDA Graph for improved launch efficiency.