DecoderAttention
Description
This DecoderAttention supports self attention and cross attention, key and value cache, and key_padding_mask. The attention mask is not support at the moment. Some boolean parameters are passed by runtime input for generic purpose
Documentation illustration (illustration unavailable in the archive).
Input parameters
specified_outputs_name : array, this parameter lets you manually assign custom names to the output tensors of a node.
Graphs in : cluster, ONNX model architecture.
query – T : object, 3D input tensor with shape (sequence_length, batch_size, hidden_size), hidden_size = num_heads * head_size. key – T : object, 3D input tensor with shape (total_sequence_length, batch_size, hidden_size). q_weight – T : object, 2D input tensor with shape (hidden_size, hidden_size). kv_weight – T : object, 2D input tensor with shape (hidden_size, 2 * hidden_size). bias – T : object, 2D input tensor with shape (hidden_size, 2 * hidden_size). key_padding_mask (optional) – B : object, 2D input tensor with shape (batch_size, total_sequence_length). key_cache (optional) – T : object, input tensor with shape (batch_size, num_heads, sequence_length or total_sequence_length, head_size). value_cache (optional) – T : object, input tensor with shape (batch_size, num_heads, sequence_length or total_sequence_length, head_size). static_kv – B : object, if static_kv = true, cross-attention; else self-attention. use_past – B : object, if use_past = true, use cache; else no cache. has_layer_state – B : object, if has_layer_state = true, layer_state = {} or [a,b]; else layer_state = None. has_key_padding_mask – B : object, has_key_padding_mask or not.
Documentation illustration (illustration unavailable in the archive).
Parameters : cluster,
mask_filter_value : float, the value to be filled in the attention mask.
Default value “-10000”.
Documentation illustration (illustration unavailable in the archive). num_heads : integer, number of attention heads.
Default value “0”.
training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.
name (optional) : string, name of the node.
Documentation illustration (illustration unavailable in the archive).
Output parameters
Graphs out : cluster, ONNX model architecture.
output – T : object, 3D output tensor with shape (sequence_length, batch_size, hidden_size). new_key_cache (optional) – T : object, output tensor with shape (batch_size, num_heads, new sequence_length, head_size). new_value_cache (optional) – T : object, output tensor with shape (batch_size, num_heads, new sequence_length, head_size).
Documentation illustration (illustration unavailable in the archive).
Type Constraints
T in (tensor(float), tensor(float16)) : Constrain input and output types to float and float16 tensors.
B in (tensor(bool)) : Constrain key_padding_mask to bool tensors.
Example
All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).