QAttention

 Graphs in : cluster, ONNX model architecture.

input (heterogeneous) – T1 : object, 3D input tensor with shape (batch_size, sequence_length, input_hidden_size).
weights (heterogeneous) – T2 : object, 2D input tensor with shape (input_hidden_size, 3 * hidden_size), hidden_size = num_heads * head_size.
bias (heterogeneous) – T3 : object, 1D input tensor with shape (3 * hidden_size).
input_scale (heterogeneous) – T3 : object, scale of quantized input tensor. It’s a scalar, which means a per-tensor/layer quantization.
weight_scale (heterogeneous) – T3 : object, scale of weight scale. It’s a scalar or a 1D tensor, which means a per-tensor/per-column quantization.Its size should be 3 * hidden_size if it is per-column quantization.
mask_index (optional, heterogeneous) – T4 : object, attention mask index with shape (batch_size).
input_zero_point (optional, heterogeneous) – T1 : object, zero point of quantized input tensor.It’s a scalar, which means a per-tensor/layer quantization.
weight_zero_point (optional, heterogeneous) – T2 : object, zero point of quantized weight tensor. It’s a scalar or a 1D tensor, which means a per-tensor/per-column quantization.Its size should be 3 * hidden_size if it is per-column quantization.
past (optional, heterogeneous) – T3 : object, past state for key and value with shape (2, batch_size, num_heads, past_sequence_length, head_size).

 Parameters : cluster,

do_rotary : boolean, whether to use rotary position embedding.
Default value “False”.
mask_filter_value : float, the value to be filled in the attention mask.
Default value “-10000”.
num_heads : integer, number of attention heads.
Default value “0”.
past_present_share_buffer : integer, corresponding past and present are same tensor, its shape is (2, batch_size, num_heads, max_sequence_length, head_size).
Default value “0”.
scale : float, custom scale will be used if specified.
Default value “0”.
unidirectional : boolean, whether every token can only attend to previous tokens.
Default value “False”.
 training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
 lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

 name (optional) : string, name of the node.

 Graphs out : cluster, ONNX model architecture.

output (heterogeneous) – T3 : object, 3D output tensor with shape (batch_size, sequence_length, hidden_size).
present (optional, heterogeneous) – T3 : object, present state for key and value with shape (2, batch_size, num_heads, past_sequence_length + sequence_length, head_size).