Activation Layer
Specialization of the Layer for activation function. The activation function \(\varphi\) is applied (either element-wise or layer-wise).
Usage
eLayer/Activation"
Configuration
These are settings required by this module.
- Function
Usage: e[“Function”] = string
Description: Indicates the activation function for the weighted inputs to the current layer.
Options:
“Elementwise/Clip”: Clips the input (s) element-wise into the alpha < s < beta range.
“Elementwise/Linear”: Applies a linear transformation element-wise.
“Elementwise/Log”: Applies the element-wise log function.
“Elementwise/Logistic”: Applies an element-wise logistic (sigmoid) function.
“Elementwise/ReLU”: Applies an element-wise rectifier linear unit function.
“Elementwise/SoftReLU”: Applies an element-wise soft rectifier function (currently only supported with OneDNN and the Eigen library).
“Elementwise/SoftSign”: Applies an element-wise soft sign function (currently only supported with the Eigen library).
“Elementwise/Tanh”: Applies tanh element-wise.
“Softmax”: Applies the layer-wide softmax operation.
- Alpha
Usage: e[“Alpha”] = float
Description: First (alpha) argument to the activation function, as detailed in https://oneapi-src.github.io/oneDNN/dev_guide_eltwise.html
- Beta
Usage: e[“Beta”] = float
Description: Second (beta) argument to the activation function, as detailed in https://oneapi-src.github.io/oneDNN/dev_guide_eltwise.html
- Output Channels
Usage: e[“Output Channels”] = unsigned integer
Description: Indicates the size of the output vector produced by the layer.
- Weight Scaling
Usage: e[“Weight Scaling”] = float
Description: Factor that is mutliplied by the layers’ weights.
- Engine
Usage: e[“Engine”] = string
Description: Specifies which Neural Network backend engine to use.
Options:
“Korali”: Uses Korali’s lightweight NN support. (CPU Sequential - Does not require installing third party software other than Eigen)
“OneDNN”: Uses oneDNN as NN support. (CPU Sequential/Parallel - Requires installing oneDNN)
“CuDNN”: Uses cuDNN as NN support. (GPU - Requires installing cuDNN)
- Mode
Usage: e[“Mode”] = string
Description: Specifies the execution mode of the Neural Network.
Options:
“Training”: Use for training. Stores data during forward propagation and allows backward propagation.
“Inference”: Use for inference only. Only runs forward propagation. Faster for inference.
- Layers
Usage: e[“Layers”] = knlohmann::json
Description: Complete description of the NN’s layers.
- Timestep Count
Usage: e[“Timestep Count”] = unsigned integer
Description: Provides the sequence length for the input/output data.
- Batch Sizes
Usage: e[“Batch Sizes”] = List of unsigned integer
Description: Specifies the batch sizes.
Default Configuration
These following configuration will be assigned by default. Any settings defined by the user will override the given settings specified in these defaults.
{ "Alpha": 1.0, "Batch Sizes": [], "Beta": 0.0, "Engine": "Korali", "Input Values": [], "Output Channels": 0, "Uniform Generator": { "Maximum": 1.0, "Minimum": -1.0, "Name": "Neural Network / Uniform Generator", "Type": "Univariate/Uniform" }, "Weight Scaling": 1.0 }