Forum
Login
Register

Welcome to our Support Center
UpdatedAugust 19, 2025

LayerNormalization

Description

This is layer normalization defined in ONNX as function. The overall computation can be split into two stages. The first stage is standardization, which makes the normalized elements have zero mean and unit variances.

 

 

The computation required by standardization can be described by the following equations.

Mean = ReduceMean<axes=normalized_axes>(X)
D = Sub(X, Mean)
DD = Mul(D, D)
Var = ReduceMean<axes=normalized_axes>(DD)
VarEps = Add(Var, epsilon)
StdDev = Sqrt(VarEps)
InvStdDev = Reciprocal(StdDev)
Normalized = Mul(D, InvStdDev)

 

whereΒ normalized_axesΒ isΒ [axis,Β ...,Β rankΒ ofΒ XΒ -Β 1]. The variablesΒ VarΒ andΒ StdDevΒ stand for variance and standard deviation, respectively. The second output isΒ MeanΒ and the last one isΒ InvStdDev. Depending onΒ stash_typeΒ attribute, the actual computation must happen in different floating-point precision. For example, ifΒ stash_typeΒ is 1, this operator casts all input variables to 32-bit float, perform the computation, and finally castΒ NormalizedΒ back to the original type ofΒ X. The second stage then scales and shifts the outcome of the first stage using

NormalizedScaled = Mul(Normalized, Scale)
Y = Add(NormalizedScaled, B)

 

The second stage doesn’t depends onΒ stash_type. All equations are inΒ this syntax. The same variable (i.e., input, output, and attribute) uses the same name in the equations above and this operator’s definition. LetΒ dΒ indicate the i-th dimension ofΒ X. IfΒ X’s shape isΒ [d[0],Β ...,Β d[axis-1],Β d[axis],Β ...,Β d[rank-1]], the shape ofΒ MeanΒ andΒ InvStdDevΒ isΒ [d[0],Β ...,Β d[axis-1],Β 1,Β ...,Β 1].Β YΒ andΒ XΒ have the same shape. This operator supports unidirectional broadcasting (tensorsΒ ScaleΒ andΒ BΒ should be unidirectional broadcastable to tensorΒ X); for more details please checkΒ Broadcasting in ONNX.

 

 

 

Input parameters

 

specified_outputs_name :Β array, this parameter lets you manually assign custom names to the output tensors of a node.

Β Graphs in :Β cluster, ONNX model architecture.

X (heterogeneous) – T : object, tensor to be normalized.
 Scale (heterogeneous) – T : object, scale tensor.
B (optional, heterogeneous) – T : object, bias tensor.

Β Parameters :Β cluster,

axis : integer, the first normalization dimension. If rank(X) is r, axis’ allowed range is [-r, r). Negative value means counting dimensions from the back.
Default value β€œ-1”.
epsilon : float, the epsilon value to use to avoid division by zero.
Default value β€œ1e-05”.
stash_typeΒ : integer, type of Mean and InvStdDev. This also specifies stage one’s computation precision.
Default value β€œ1”.
Β training_mode :Β boolean, whether the layer is in training mode (can store data for backward).
Default value β€œFalse”.
Β lda coeff :Β float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value β€œ1”.

Β name (optional) :Β string,Β name of the node.

Output parameters

Β Graphs out :Β cluster, ONNX model architecture.

Y (heterogeneous) – T : object, normalized tensor.
Mean (optional, heterogeneous) – U : object, saved mean used during training to speed up gradient computation.
InvStdDev (optional, heterogeneous) – U : object, saved inverse standard deviation used during training to speed up gradient computation.

Type Constraints

T in (tensor(bfloat16),Β tensor(double),Β tensor(float),Β tensor(float16)) : Constrain input types and output Y type to float tensors.

U in (tensor(bfloat16),Β tensor(float)) : Type of Mean and InvStdDev tensors.

Example

All these exemples are snippets PNG, you can drop these Snippet onto the block diagram and get the depicted code added to your VI (Do not forget to install Deep Learning library to run it).
Table of Contents