UpdatedAugust 19, 2025

LongformerAttention

Description

Longformer Self Attention with a local context and a global context. Tokens attend locally: Each token attends to its W previous tokens and W succeeding tokens with W being the window length. A selected few tokens attend globally to all other tokens.

The attention mask is of shape (batch_size, sequence_length), where sequence_length is a multiple of 2W after padding. Mask value < 0 (like -10000.0) means the token is masked, 0 otherwise.
Global attention flags have value 1 for the tokens attend globally and 0 otherwise.

Input parameters

specified_outputs_name : array, this parameter lets you manually assign custom names to the output tensors of a node.

Graphs in : cluster, ONNX model architecture.

input (heterogeneous) – T : object, 3D input tensor with shape (batch_size, sequence_length, hidden_size), hidden_size = num_heads * head_size.
weight (heterogeneous) – T : object, 2D input tensor with shape (hidden_size, 3 * hidden_size).
bias (heterogeneous) – T : object, 1D input tensor with shape (3 * hidden_size).
mask (heterogeneous) – T : object, attention mask with shape (batch_size, sequence_length).
global_weight (heterogeneous) – T : object, 2D input tensor with shape (hidden_size, 3 * hidden_size).
global_bias (heterogeneous) – T : object, 1D input tensor with shape (3 * hidden_size).
global (heterogeneous) – G : object, global attention flags with shape (batch_size, sequence_length).

Parameters : cluster,

num_heads : integer, number of attention heads.
Default value “0”.
window : integer, one sided attention windows length W, or half of total window length.
Default value “0”.
training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

name (optional) : string, name of the node.

Output parameters

output (heterogeneous) – T : object, 3D output tensor with shape (batch_size, sequence_length, hidden_size).

Type Constraints

T in (tensor(float), tensor(float16)) : Constrain input and output types to float tensors.

G in (tensor(int32)) : Constrain to integer types.

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).

SOTA

Installation guide

General

Accelerator Toolkit

Installation guide

Execution providers

General

Execution

CUDA Advanced

Construct Ptr Input Data

Index

Name

Exec

Exec

Mono

Multi

Input

Deep Learning Toolkit

Installation guide

Execution providers

General

Architecture

Layers

Nodes

Nodes

Activation

Mono Input

Parameters

Graph Function

Graph

File

Get & Set

Runtime

Create

Academic Training

Inference

Training

Exec

Academic

Input

Advanced

Add Weight

Index

Name

Convert

From ONNX

To ONNX

Format Weight

Get Weight

More

Layers parameters

Nodes Parameters

Computer Vision Toolkit

Installation guide

General

Tools

Image Manipulation

Image Modification

Files

Pixel Editing

Draw

Grayscale

Color

Region of Interest

Additional Windows

Session

Functions

Filters

Form

Inspection

Operators

Pattern

Treatment

Video Writter

CUDA Toolkit

Installation guide

General

Array

CuBLAS

Elementary