LongformerAttention

LongformerAttention

Description

Longformer Self Attention with a local context and a global context. Tokens attend locally: Each token attends to its W previous tokens and W succeeding tokens with W being the window length. A selected few tokens attend globally to all other tokens.

Documentation illustration (illustration unavailable in the archive).

The attention mask is of shape (batch_size, sequence_length), where sequence_length is a multiple of 2W after padding. Mask value < 0 (like -10000.0) means the token is masked, 0 otherwise.
Global attention flags have value 1 for the tokens attend globally and 0 otherwise.

Input parameters

specified_outputs_name : array, this parameter lets you manually assign custom names to the output tensors of a node.

 Graphs in : cluster, ONNX model architecture.

input (heterogeneous) – T : object, 3D input tensor with shape (batch_size, sequence_length, hidden_size), hidden_size = num_heads * head_size.
weight (heterogeneous) – T : object, 2D input tensor with shape (hidden_size, 3 * hidden_size).
bias (heterogeneous) – T : object, 1D input tensor with shape (3 * hidden_size).
mask (heterogeneous) – T : object, attention mask with shape (batch_size, sequence_length).
global_weight (heterogeneous) – T : object, 2D input tensor with shape (hidden_size, 3 * hidden_size).
global_bias (heterogeneous) – T : object, 1D input tensor with shape (3 * hidden_size).
global (heterogeneous) – G : object, global attention flags with shape (batch_size, sequence_length).

Documentation illustration (illustration unavailable in the archive).

 Parameters : cluster,

num_heads : integer, number of attention heads.
Default value “0”.
window : integer, one sided attention windows length W, or half of total window length.
Default value “0”.
 training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
 lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

 name (optional) : string, name of the node.

Documentation illustration (illustration unavailable in the archive).

Output parameters

output (heterogeneous) – T : object, 3D output tensor with shape (batch_size, sequence_length, hidden_size).

Type Constraints

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).