mx.symbol.rmspropalex_update
¶
Description¶
Update function for RMSPropAlex optimizer.
RMSPropAlex is non-centered version of RMSProp.
Define \(E[g^2]_t\) is the decaying average over past squared gradient and \(E[g]_t\) is the decaying average over past gradient.
The update step is
The RMSPropAlex code follows the version in http://arxiv.org/pdf/1308.0850v5.pdf Eq(38) - Eq(45) by Alex Graves, 2013.
Graves suggests the momentum term \(\gamma_1\) to be 0.95, \(\gamma_2\) to be 0.9 and the learning rate \(\eta\) to be 0.0001.
Usage¶
mx.symbol.rmspropalex_update(...)
Arguments¶
Argument |
Description |
---|---|
|
NDArray-or-Symbol. Weight |
|
NDArray-or-Symbol. Gradient |
|
NDArray-or-Symbol n |
|
NDArray-or-Symbol g |
|
NDArray-or-Symbol delta |
|
float, required. Learning rate |
|
float, optional, default=0.949999988. Decay rate. |
|
float, optional, default=0.899999976. Decay rate. |
|
float, optional, default=9.99999994e-09. A small constant for numerical stability. |
|
float, optional, default=0. Weight decay augments the objective function with a regularization term that penalizes large weights. The penalty scales with the square of the magnitude of each weight. |
|
float, optional, default=1. Rescale gradient to grad = rescale_grad*grad. |
|
float, optional, default=-1. Clip gradient to the range of [-clip_gradient, clip_gradient] If clip_gradient <= 0, gradient clipping is turned off. grad = max(min(grad, clip_gradient), -clip_gradient). |
|
float, optional, default=-1. Clip weights to the range of [-clip_weights, clip_weights] If clip_weights <= 0, weight clipping is turned off. weights = max(min(weights, clip_weights), -clip_weights). |
|
string, optional. Name of the resulting symbol. |
Value¶
out
The result mx.symbol
Link to Source Code: http://github.com/apache/incubator-mxnet/blob/1.6.0/src/operator/optimizer_op.cc#L836