Forum
Login
Register

Welcome to our Support Center
Updated

EmbedLayerNormalization

Description

EmbedLayerNormalization is the fusion of embedding layer in BERT model, with optional mask processing. The embedding layer takes input_ids (word IDs) and segment_ids (sentence IDs) to look up word_embedding, position_embedding, and segment_emedding; the embeddings are added then applied layer normalization using gamma and beta tensors. The last input mask is optional. If mask is provided, mask index (that is position of first 0 in mask, or number of words) will be calculated.

 

Input parameters

 

specified_outputs_name : array, this parameter lets you manually assign custom names to the output tensors of a node.

 Graphs in : cluster, ONNX model architecture.

input_ids (heterogeneous) – T1 : object, 2D words IDs with shape (batch_size, sequence_length).
 segment_ids (optional, heterogeneous) – T1 : object, 2D segment IDs with shape (batch_size, sequence_length).
word_embedding (heterogeneous) – T : object, 2D with shape (,hidden_size).
position_embedding (heterogeneous) – T : object, 2D with shape (, hidden_size).
segment_embedding (optional, heterogeneous) – T : 2D with shape (, hidden_size).
 gamma (heterogeneous) – T : object, 1D gamma tensor for layer normalization with shape (hidden_size).
beta (heterogeneous) – T : object, 1D beta tensor for layer normalization with shape (hidden_size).
mask (optional, heterogeneous) – T1 : object, 2D attention mask with shape (batch_size, sequence_length).
 position_ids (optional, heterogeneous) – T1 : object, 2D position ids with shape (batch_size, sequence_length) or (1, sequence_length).

 Parameters : cluster,

epsilon : float, the epsilon value to use to avoid division by zero.
Default value “1E-5”.
mask_index_type : enum, the mask index tensor type for shape inference (0: None, 1: 1D mask_index).
Default value “None”.
 training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
 lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

 name (optional) : string, name of the node.

Output parameters

 Graphs out : cluster, ONNX model architecture.

output (heterogeneous) – T : object, 3D output tensor with shape (batch_size, sequence_length, hidden_size).
mask_index (optional, heterogeneous) – T1 : object, 1D mask_index tensor with shape (batch_size).
embedding_sum (optional, heterogeneous) – T : object, sum of word_embedding and position_embedding without layer normalization.

Type Constraints

T in (tensor(float), tensor(float16)) : Constrain input and output float tensors types.

T1 in (tensor(int32)) : Constrain input and output integer tensors types.

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).
Table of Contents