UpdatedAugust 20, 2025

MatMulBnb4

Description

MatMulBnb4 is a MatMul with weight quantized with 4 bits using either FP4 or NF4 data type (https://arxiv.org/pdf/2305.14314.pdf). It does Matrix Multiplication like MatMul (https://github.com/onnx/onnx/blob/main/docs/Operators.md#matmul) with differences: 1. Input B is a 2D constant Matrix. Its input feature count and output feature count are specified by attribute ‘K’ and ‘N’. 2. Input B is quantized with 4 bits with quantization data type specified by attribute ‘quant_type’. It is transposed, flattened and quantized blockwisely with block size specified by attribute ‘block_size’. And block_size is not an arbitrary number and must be a power of 2 and not smaller than 16, like 16, 32, 64, 128,.. 3. Input B’s quantization constants or scales are specified by input ‘absmax’.

Input parameters

specified_outputs_name : array, this parameter lets you manually assign custom names to the output tensors of a node.

Graphs in : cluster, ONNX model architecture.

A (heterogeneous) – T1 : object, the input tensor, not quantized.
B (heterogeneous) – T2 : object, 1-dimensional quantized data for weight.
absmax (heterogeneous) – T1 : object, quantization constants.

Parameters : cluster,

K : integer, size of each input feature.
Default value “0”.
N : integer, size of each output feature.
Default value “0”.
block_size : integer, number of groupsize used for weight quantization. It needs to be a power of 2 and not smaller than 16.
Default value “0”.
quant_type : enum, quantization data type.
Default value “FP4”.
training_mode : boolean, indicate if the ops run in training_mode.
Default value “False”.
transB : boolean, whether B should be transposed on the last two dimensions before doing multiplication.
Default value “True”.
training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

name (optional) : string, name of the node.

Output parameters

Y (heterogeneous) – T1 : object, tensor. The output tensor has the same rank as the input.

Type Constraints

T1 in (tensor(float), tensor(float16), tensor(bfloat16)) : Constrain input and output types to float/half_float/brain_float tensors.

T2 in (tensor(uint8)) : Constrain quantized weight types to uint8.

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).

Deep Learning

Installation guide

Execution providers

General

Architecture

Layers

Nodes

Nodes

Activation

Mono Input

Parameters

Graph Function

Graph

File

Get & Set

Runtime

Create

Academic Training

Inference

Training

Exec

Academic

Input

Advanced

Add Weight

Index

Name

Convert

From ONNX

To ONNX

Format Weight

Get Weight

Metrics

Pre/Post Process

Set Weight

More

Layers parameters

Nodes Parameters

Accelerator

Quick start

Installation guide

Execution providers

General

API

Execution

Open

Graph

File

Exec

Constant

Generator

Full Train Step

Eval Step

Train Step