DecoderMaskedMultiHeadAttention

 Graphs in : cluster, ONNX model architecture.

query – T : object, query with shape (batch_size, 1, hidden_size) or packed QKV with shape (batch_size, 1, 2 * hidden_size + v_hidden_size).
key (optional) – T : object, key with shape (batch_size, 1, hidden_size) for self attention or past_key with shape (batch_size, num_heads, kv_sequence_length, head_size) for cross attention.
value (optional) – T : object, value with shape (batch_size, 1, v_hidden_size) for self attention or past_value with shape (batch_size, num_heads, kv_sequence_length, head_size) for cross attention.
mask_index (optional) – M : object, mask values of shape (batch_size, total_sequence_length) or (batch_size, kv_sequence_length).
relative_position_bias (optional) – T : object, additional add to QxK’ with shape (batch_size or 1, num_heads or 1, sequence_length, total_sequence_length).
past_key (optional) – T : object, past state for key with shape (batch_size, num_heads, past_sequence_length, head_size) for self attentionWhen past_present_share_buffer is set, its shape is (batch_size, num_heads, max_sequence_length, head_size). The keys buffer is re-ordered in such a way that its virtual sub-tensor of shape (batch_size, num_heads, max_sequence_length, head_size) which may be perceived as being of shape (batch_size, num_heads, max_sequence_length, head_size / x, x) is reordered to become (batch_size, num_heads, head_size / x, max_sequence_length, x) where `x = 16 / sizeof(T)`.
past_value (optional) – T : object, past state for value with shape (batch_size, num_heads, past_sequence_length, head_size) for self attentionWhen past_present_share_buffer is set, its shape is (batch_size, num_heads, max_sequence_length, head_size).
past_sequence_length (optional) – M : object, when past_present_share_buffer is used, it is required to specify past_sequence_length (could be 0).Cross Attention doesn’t need this input.
beam_width (optional) – M : object, the beam width that is being used while decoding. If not provided, the beam width will be assumed to be 1.
cache_indirection (optional) – M : object, a buffer of shape [batch_size, beam_width, max_output_length] where an `[i, j, k]` entry specifies which beam the `k`-th token came from for the `j`-th beam for batch `i` in the current iteration.
bias (optional) – T : object, bias tensor with shape (hidden_size + hidden_size + v_hidden_size) from input projection.

 Parameters : cluster,

mask_filter_value : float, the value to be filled in the attention mask.
Default value “-10000”.
num_heads : integer, number of attention heads.
Default value “0”.
output_qk : integer, need output the cross attention MatMul(Q, K).
Default value “0”.
past_present_share_buffer : integer, corresponding past and present are same tensor, its size is (batch_size, num_heads, max_sequence_length, head_size).
Default value “0”.
scale : float, custom scale will be used if specified. Default value is 1/sqrt(head_size).
Default value “0”.
 training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
 lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

 name (optional) : string, name of the node.

 Graphs out : cluster, ONNX model architecture.

output – T : object, 3D output tensor with shape (batch_size, sequence_length, v_hidden_size).
present_key (optional) – T : object, present state for key with shape (batch_size, num_heads, total_sequence_length, head_size). If past_present_share_buffer is set, its shape is (batch_size, num_heads, max_sequence_length, head_size), while effective_seq_length = (past_sequence_length + kv_sequence_length).
present_value_cache (optional) – T : object, present state for value with shape (batch_size, num_heads, total_sequence_length, head_size). If past_present_share_buffer is set, its shape is (batch_size, num_heads, max_sequence_length, head_size), while effective_seq_length = (past_sequence_length + kv_sequence_length).
qk (optional) – QK : object, normalized Q * K, of shape (batch_size, num_heads, 1, total_sequence_length).