GatedRelativePositionBias
Description
GatedRelativePositionBias computes attention biases based on relative positions, with a gating mechanism that adapts their influence. It outputs a bias tensor added to the attention scores in Transformer models.
Documentation illustration (illustration unavailable in the archive).
Input parameters
specified_outputs_name : array, this parameter lets you manually assign custom names to the output tensors of a node.
Graphs in : cluster, ONNX model architecture.
query_layer (heterogeneous) – T : object, tensor with shape (batch_size, seq_len, num_heads x head_size) or (token_count, num_heads x head_size). query_bias (heterogeneous) – T : object, 1-d tensor with shape (num_heads x head_size). rel_pos (heterogeneous) – T : object, tensor with shape (1, num_head, seq_len, seq_len). weight (heterogeneous) – T : object, gemm weight for the gated_ur_linear, shape (head_size, D), D is divisible by 2. bias (heterogeneous) – T : object, bias for the gated_ur_linear, shape (D). eco_a (heterogeneous) – T : object, tensor of shape (1, num_heads, 1, 1). token_offset (optional, heterogeneous) – M : object, offset of each token with shape (batch_size, seq_len).
Documentation illustration (illustration unavailable in the archive).
Parameters : cluster,
num_heads : integer, number of attention heads.
Default value “0”.
training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.
name (optional) : string, name of the node.
Documentation illustration (illustration unavailable in the archive).
Output parameters
output (heterogeneous) – T : object, output tensor with shape (batch_size, num_heads, seq_len, seq_len).
Type Constraints
T in (tensor(float), tensor(float16)) : Constrain input and output types to float tensors.
M in (tensor(int32)) : Constrain token_offset to integer types.
Example
All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).