AdaBelief
This implements AdaBelief for gradient based optimisation as published in https://arxiv.org/abs/2010.07468.
Usage
e["Solver"]["Type"] = "Optimizer/AdaBelief"
Results
These are the results produced by this solver:
- Best Gradient(x)
Usage: e[“Results”][“Best Gradient(x)”] = List of real number
Description: Values of dF(x) for the x parameters that produced the best F(x) found so far.
- Best F(x)
Usage: e[“Results”][“Best F(x)”] = real number
Description: Optimal value of F(x) found so far.
- Best Parameters
Usage: e[“Results”][“Best Parameters”] = List of real number
Description: Value for the x parameters that produced the best F(x).
Variable-Specific Settings
These are settings required by this module that are added to each of the experiment’s variables when this module is selected.
- Lower Bound
Usage: e[“Variables”][index][“Lower Bound”] = real number
Description: [Hint] Lower bound for the variable’s value.
- Upper Bound
Usage: e[“Variables”][index][“Upper Bound”] = real number
Description: [Hint] Upper bound for the variable’s value.
- Initial Value
Usage: e[“Variables”][index][“Initial Value”] = real number
Description: [Hint] Initial value at or around which the algorithm shall start looking for an optimum.
- Initial Mean
Usage: e[“Variables”][index][“Initial Mean”] = real number
Description: [Hint] Initial mean for the proposal distribution. This value must be defined between the variable’s Mininum and Maximum settings (by default, this value is given by the center of the variable domain).
- Initial Standard Deviation
Usage: e[“Variables”][index][“Initial Standard Deviation”] = real number
Description: [Hint] Initial standard deviation of the proposal distribution for a variable (by default, this value is given by 30% of the variable domain width).
- Minimum Standard Deviation Update
Usage: e[“Variables”][index][“Minimum Standard Deviation Update”] = real number
Description: [Hint] Lower bound for the standard deviation updates of the proposal distribution for a variable. Korali increases the scaling factor sigma if this value is undershot.
- Values
Usage: e[“Variables”][index][“Values”] = List of real number
Description: [Hint] Locations to evaluate the Objective Function.
Configuration
These are settings required by this module.
- Beta1
Usage: e[“Solver”][“Beta1”] = real number
Description: Smoothing factor for momentum update.
- Beta2
Usage: e[“Solver”][“Beta2”] = real number
Description: Smoothing for gradient update.
- Eta
Usage: e[“Solver”][“Eta”] = real number
Description: Learning Rate (Step Size)
- Epsilon
Usage: e[“Solver”][“Epsilon”] = real number
Description: Term to facilitate numerical stability.
Termination Criteria
These are the customizable criteria that indicates whether the solver should continue or finish execution. Korali will stop when at least one of these conditions are met. The criteria is expressed in C++ since it is compiled and evaluated as seen here in the engine.
- Min Gradient Norm
Usage: e[“Solver”][“Min Gradient Norm”] = real number
Description: Specifies the minimal norm for the gradient of function with respect to Parameters.
Criteria:
(_k->_currentGeneration > 1) && (_gradientNorm <= _minGradientNorm)
- Max Gradient Norm
Usage: e[“Solver”][“Max Gradient Norm”] = real number
Description: Specifies the minimal norm for the gradient of function with respect to Parameters.
Criteria:
(_k->_currentGeneration > 1) && (_gradientNorm >= _maxGradientNorm)
- Max Value
Usage: e[“Solver”][“Max Value”] = real number
Description: Specifies the maximum target fitness to stop maximization.
Criteria:
_k->_currentGeneration > 1 && (+_bestEverValue > _maxValue)
- Min Value Difference Threshold
Usage: e[“Solver”][“Min Value Difference Threshold”] = real number
Description: Specifies the minimum fitness differential between two consecutive generations before stopping execution.
Criteria:
_k->_currentGeneration > 1 && (fabs(_currentBestValue - _previousBestValue) < _minValueDifferenceThreshold)
- Max Infeasible Resamplings
Usage: e[“Solver”][“Max Infeasible Resamplings”] = unsigned integer
Description: Maximum number of resamplings per candidate per generation if sample is outside of Lower and Upper Bound.
Criteria:
(_maxInfeasibleResamplings > 0) && (_infeasibleSampleCount >= _maxInfeasibleResamplings)
- Max Model Evaluations
Usage: e[“Solver”][“Max Model Evaluations”] = unsigned integer
Description: Specifies the maximum allowed evaluations of the computational model.
Criteria:
_maxModelEvaluations <= _modelEvaluationCount
- Max Generations
Usage: e[“Solver”][“Max Generations”] = unsigned integer
Description: Determines how many solver generations to run before stopping execution. Execution can be resumed at a later moment.
Criteria:
_k->_currentGeneration > _maxGenerations
Default Configuration
These following configuration will be assigned by default. Any settings defined by the user will override the given settings specified in these defaults.
{ "Beta1": 0.9, "Beta2": 0.999, "Epsilon": 1e-08, "Eta": 0.001, "Model Evaluation Count": 0, "Termination Criteria": { "Max Generations": 10000000000, "Max Gradient Norm": 1000000000000.0, "Max Infeasible Resamplings": 1000000, "Max Model Evaluations": 1000000000, "Max Value": Infinity, "Min Gradient Norm": 1e-12, "Min Value Difference Threshold": -Infinity }, "Variable Count": 0 }
Variable Defaults
These following configuration will be assigned to each of the experiment variables by default. Any settings defined by the user will override the given settings specified in these defaults.
{ "Initial Mean": NaN, "Initial Standard Deviation": NaN, "Initial Value": NaN, "Lower Bound": -Infinity, "Minimum Standard Deviation Update": 0.0, "Upper Bound": Infinity, "Values": [] }