MoE

 Graphs in : cluster, ONNX model architecture.

input (heterogeneous) – T : object, 2D input tensor with shape (num_tokens, hidden_size) or 3D input tensor with shape (batch_size, sequence_length, hidden_size).
router_probs (heterogeneous) – T : object, 2D input tensor with shape (num_tokens, num_experts).
fc1_experts_weights (heterogeneous) – T : object, 3D input tensor with shape (num_experts, fusion_size * inter_size, hidden_size), where fusion_size is 2 for fused swiglu, and 1 otherwise.
fc1_experts_bias (optional, heterogeneous) – T : object, 2D optional input tensor with shape (num_experts, fusion_size * inter_size).
fc2_experts_weights (heterogeneous) – T : object, 3D input tensor with shape (num_experts, hidden_size, inter_size).
fc2_experts_bias (optional, heterogeneous) – T : object, 2D optional input tensor with shape (num_experts, hidden_size).
fc3_experts_weights (optional, heterogeneous) – T : object, 3D optional input tensor with shape (num_experts, inter_size, hidden_size).
fc3_experts_bias (optional, heterogeneous) – T : object, 2D optional input tensor with shape (num_experts, inter_size).

 Parameters : cluster,

 activation_alpha : float, alpha parameter used in activation function.
Default value “1”.
 activation_beta : float, beta parameter used in activation function.
Default value “0”.
activation_type : enum, activation function to use. Choose from relu, gelu, silu, swiglu and identity.
Default value “relu”.
k : integer, number of top experts to select from expert pool.
Default value “1”.
 normalize_routing_weights : boolean, whether to normalize routing weights.
Default value “False”.
swiglu_fusion : enum, 0: not fused, 1: fused and interleaved. 2: fused and not interleaved.
Default value “Not Fused”.
 swiglu_limit : float, The limit used to clamp in SwiGLU. No clamp when limit is not provided.
Default value “0”.
 use_sparse_mixer : boolean, whether to use sparse mixer.
Default value “False”.
 training? : boolean, whether the layer is in training mode (can store data for backward).
Default value “True”.
 lda coeff : float, defines the coefficient by which the loss derivative will be multiplied before being sent to the previous layer (since during the backward run we go backwards).
Default value “1”.

 name (optional) : string, name of the node.