Activation Layer

Specialization of the Layer for activation function. The activation function \(\varphi\) is applied (either element-wise or layer-wise).

\[\mathbf{z} = \varphi(\mathbf{x})\]

Usage

eLayer/Activation"

Configuration

These are settings required by this module.

Function
  • Usage: e[“Function”] = string

  • Description: Indicates the activation function for the weighted inputs to the current layer.

  • Options:

    • Elementwise/Linear”: Applies a linear transformation element-wise.

    • Elementwise/Tanh”: Applies tanh element-wise.

    • Elementwise/ReLU”: Applies an element-wise rectifier linear unit function.

    • Elementwise/Logistic”: Applies an element-wise logistic (sigmoid) function.

    • Elementwise/Clip”: Clips the input (s) element-wise into the alpha < s < beta range.

    • Elementwise/Log”: Applies the element-wise log function.

    • Elementwise/SoftSign”: Applies an element-wise soft sign function (currently only supported with the Eigen library).

    • Softmax”: Applies the layer-wide softmax operation.

Alpha
Beta
Output Channels
  • Usage: e[“Output Channels”] = unsigned integer

  • Description: Indicates the size of the output vector produced by the layer.

Weight Scaling
  • Usage: e[“Weight Scaling”] = float

  • Description: Factor that is mutliplied by the layers’ weights.

Engine
  • Usage: e[“Engine”] = string

  • Description: Specifies which Neural Network backend engine to use.

  • Options:

    • Korali”: Uses Korali’s lightweight NN support. (CPU Sequential - Does not require installing third party software other than Eigen)

    • OneDNN”: Uses oneDNN as NN support. (CPU Sequential/Parallel - Requires installing oneDNN)

    • CuDNN”: Uses cuDNN as NN support. (GPU - Requires installing cuDNN)

Mode
  • Usage: e[“Mode”] = string

  • Description: Specifies the execution mode of the Neural Network.

  • Options:

    • Training”: Use for training. Stores data during forward propagation and allows backward propagation.

    • Inference”: Use for inference only. Only runs forward propagation. Faster for inference.

Layers
  • Usage: e[“Layers”] = knlohmann::json

  • Description: Complete description of the NN’s layers.

Timestep Count
  • Usage: e[“Timestep Count”] = unsigned integer

  • Description: Provides the sequence length for the input/output data.

Batch Sizes
  • Usage: e[“Batch Sizes”] = List of unsigned integer

  • Description: Specifies the batch sizes.

Default Configuration

These following configuration will be assigned by default. Any settings defined by the user will override the given settings specified in these defaults.

{
"Alpha": 1.0,
"Batch Sizes": [],
"Beta": 0.0,
"Engine": "Korali",
"Input Values": [],
"Output Channels": 0,
"Uniform Generator": {
    "Maximum": 1.0,
    "Minimum": -1.0,
    "Type": "Univariate/Uniform"
    },
"Weight Scaling": 1.0
}