RotaryEmbedding

 Graphs in : cluster, ONNX model architecture.

input (heterogeneous) – T : object, 3D tensor with shape (batch_size, sequence_length, hidden_size) or 4D with shape (batch_size, num_heads, sequence_length, head_size).
position_ids (heterogeneous) – M : object, 1D tensor with shape (1) or 2D tensor with shape (batch_size, sequence_length).
cos_cache (heterogeneous) – T : object, 2D tensor with shape (max_sequence_length, head_size / 2) or (max_sequence_length, rotary_embedding_dim / 2).
sin_cache (heterogeneous) – T : object, 2D tensor with shape (max_sequence_length, head_size / 2) or (max_sequence_length, rotary_embedding_dim / 2).

 Parameters : cluster,

interleaved : boolean, indicates whether the input has real and imaginary parts interleaved. If false, meaning the first half of the input consists of real values and the second half consists of imaginary values.
Default value “False”.
is_packed_batching : boolean, ragged batch inputs or not.
Default value “False”.
num_heads : integer, number of attention heads. Must use with rotary_embedding_dim.
Default value “0”.
rotary_embedding_dim : integer, rotary embedding dimension.
Default value “0”.
scale : float, custom scale will be used if specified.
Default value “1”.
 training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
 lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

 name (optional) : string, name of the node.