CUDA

Sessions Parameters : cluster

intra_op_num_threads : integer, number of threads used within each operator to parallelize computations. If the value is 0, ONNX Runtime automatically uses the number of physical CPU cores.
inter_op_num_threads : integer, number of threads used between operators, to execute multiple graph nodes in parallel. If set to 0, this parameter is ignored when execution_mode is ORT_SEQUENTIAL. In ORT_PARALLEL mode, 0 means ORT automatically selects a suitable number of threads (usually equal to the number of cores).
execution_mode : enum, controls whether the graph executes nodes one after another or allows parallel execution when possible.ORT_SEQUENTIAL runs nodes in order, ORT_PARALLEL runs them concurrently.
deterministic_compute : boolean, forces deterministic execution, meaning results will always be identical for the same inputs.
graph_optimization_level : enum, defines how much ONNX Runtime optimizes the computation graph before running the model.
optimized_model_file_path : path, file path to save the optimized model after graph analysis.
profiling output dir : path, specifies the directory where ONNX Runtime will save profiling output files. If you set this parameter to a valid (non-empty) path, profiling is automatically enabled. However, if the path is empty, profiling will not be activated.

CUDA Parameters : cluster

Device & Stream : cluster,

Device Index : integer, ID of the CUDA device used for inference (default = 0).
Use Custom Compute Stream? : boolean, if true, the model will run using a user-provided CUDA compute stream.
Custom Compute Stream (Pointer) : integer, pointer or reference to the user CUDA stream.
Copy Using Default Stream? : boolean, if true, host-device transfers are executed in the default CUDA stream.

Memory Management : cluster,

GPU Memory Limit (Bytes) : integer, maximum GPU memory the CUDA EP can allocate (default = unlimited).
Arena Extension Strategy : enum, defines how the GPU memory arena grows : “NextPowerOfTwo” (default) or “SameAsRequested”.
Default Memory Arena Config : integer, optional pointer to a custom OrtArenaCfg; overrides other memory parameters.

CuDNN & Kernel Optimization : cluster,

cuDNN Convolution Algorithm Search : enum, strategy for cuDNN convolution kernel selection : Exhaustive, Heuristic, or Default.
Use Max Workspace for cuDNN? : boolean, if true, cuDNN may use the largest available workspace for best performance.
Pad Conv1D Inputs to NC1D? : boolean, automatically pads 1D convolution inputs to [N,C,1,D] or [N,C,D,1] for compatibility.
Fuse Convolution + Bias? : boolean, enables cuDNN Frontend fusion of convolution and bias layers (JIT-compiled kernels).
Scaled Dot-Product Attention Kernel Option : boolean, selects the implementation of the SDPA kernel (0 = default).
Enable Strict SkipLayerNorm Mode? : boolean, forces LayerNormalization kernel instead of SkipLayerNorm for higher accuracy at lower speed.

Execution & Precision Controls : cluster,

Enable CUDA Graph Execution? : boolean, captures model execution as a CUDA Graph for reduced launch overhead (requires static shapes).
Use EP-Level Unified Stream? : boolean, runs all CUDA operations through one unified execution stream at the provider level.
Prefer NHWC Tensor Layout? : boolean, uses NHWC (channels-last) tensor format where supported for better memory access.
Use TensorFloat-32 (TF32)? : boolean, enables TensorFloat-32 tensor cores (default = true) for faster matrix math with minimal precision loss.

Tunable Operator System : cluster,

Enable Tunable Operators? : boolean, activates runtime operator tuning for optimal kernel selection.
Enable Tuning Mode for Tunable Operators? : boolean, allows benchmarking multiple kernel variants to select the fastest one.
Max Tuning Duration (ms) : integer, maximum duration (in milliseconds) for kernel tuning before fallback to default.