Optimization¶
Initialize and update model weights during training¶
Optimizers¶
Create an AdaDelta optimizer with respective parameters |
|
Create an AdaGrad optimizer with respective parameters |
|
Create an Adam optimizer with respective parameters |
|
Create an optimizer by name and parameters |
|
Get an updater closure that can take list of weight and gradient and return updated list of weight |
|
Create a Nesterov Accelerated SGD( NAG) optimizer |
|
Create an RMSProp optimizer with respective parameters |
|
Create an SGD optimizer with respective parameters |
Initialization¶
Xavier initializer |
|
Create initialization of argument like arg.array |
|
Internal default value initialization scheme |
|
Create a initializer that initialize the weight with normal(0, sd) |
|
Create a initializer that initialize the weight with uniform [-scale, scale] |
|
Parameter initialization |
Optimizer updates (NDArray)¶
Update function for Adam optimizer |
|
Check if all the float numbers in the array are finite (used for AMP) |
|
The FTML optimizer described in FTML - Follow the Moving Leader in Deep Learning, available at http://proceedings.mlr.press/v70/zheng17a/zheng17a.pdf |
|
Update function for Ftrl optimizer |
|
Phase I of lamb update it performs the following operations and returns g: |
|
Phase II of lamb update it performs the following operations and updates grad |
|
Mixed Precision version of Phase I of lamb update it performs the following operations and returns g: |
|
Mixed Precision version Phase II of lamb update it performs the following operations and updates grad |
|
Update function for multi-precision Nesterov Accelerated Gradient( NAG) optimizer |
|
Updater function for multi-precision sgd optimizer |
|
Updater function for multi-precision sgd optimizer |
|
Check if all the float numbers in all the arrays are finite (used for AMP) |
|
Momentum update function for multi-precision Stochastic Gradient Descent (SGD) optimizer |
|
Update function for multi-precision Stochastic Gradient Descent (SDG) optimizer |
|
Momentum update function for Stochastic Gradient Descent (SGD) optimizer |
|
Update function for Stochastic Gradient Descent (SDG) optimizer |
|
Update function for Nesterov Accelerated Gradient( NAG) optimizer |
|
Momentum update function for multi-precision Stochastic Gradient Descent (SGD) optimizer |
|
Update function for multi-precision Stochastic Gradient Descent (SDG) optimizer |
|
Momentum update function for Stochastic Gradient Descent (SGD) optimizer |
|
Update function for Stochastic Gradient Descent (SDG) optimizer |
|
Update function for RMSProp optimizer |
|
Update function for RMSPropAlex optimizer |
|
Momentum update function for Stochastic Gradient Descent (SGD) optimizer |
|
Update function for Stochastic Gradient Descent (SGD) optimizer |
|
Update function for SignSGD optimizer |
|
SIGN momentUM (Signum) optimizer |
Optimizer updates (Symbol)¶
Update function for Adam optimizer |
|
Check if all the float numbers in the array are finite (used for AMP) |
|
The FTML optimizer described in FTML - Follow the Moving Leader in Deep Learning, available at http://proceedings.mlr.press/v70/zheng17a/zheng17a.pdf |
|
Update function for Ftrl optimizer |
|
Updater function for multi-precision sgd optimizer |
|
Updater function for multi-precision sgd optimizer |
|
Check if all the float numbers in all the arrays are finite (used for AMP) |
|
Momentum update function for multi-precision Stochastic Gradient Descent (SGD) optimizer |
|
Update function for multi-precision Stochastic Gradient Descent (SDG) optimizer |
|
Momentum update function for Stochastic Gradient Descent (SGD) optimizer |
|
Update function for Stochastic Gradient Descent (SDG) optimizer |
|
Momentum update function for multi-precision Stochastic Gradient Descent (SGD) optimizer |
|
Update function for multi-precision Stochastic Gradient Descent (SDG) optimizer |
|
Momentum update function for Stochastic Gradient Descent (SGD) optimizer |
|
Update function for Stochastic Gradient Descent (SDG) optimizer |
|
Momentum update function for Stochastic Gradient Descent (SGD) optimizer |
|
Update function for Stochastic Gradient Descent (SGD) optimizer |
|
Update function for SignSGD optimizer |
|
SIGN momentUM (Signum) optimizer |