mx.opt.adam

Description

Create an Adam optimizer with respective parameters. Adam optimizer as described in [King2014].

[King2014] Diederik Kingma, Jimmy Ba, Adam: A Method for Stochastic Optimization, http://arxiv.org/abs/1412.6980

Usage

mx.opt.adam(

  learning.rate = 0.001,

  beta1 = 0.9,

  beta2 = 0.999,

  epsilon = 1e-08,

  wd = 0,

  rescale.grad = 1,

  clip_gradient = -1,

  lr_scheduler = NULL

)

Arguments

Argument

Description

learning.rate

float, default=1e-3.

The initial learning rate.

beta1

float, default=0.9.

Exponential decay rate for the first moment estimates.

beta2

float, default=0.999.

Exponential decay rate for the second moment estimates.

epsilon

float, default=1e-8

wd

float, default=0.0.

L2 regularization coefficient add to all the weights.

rescale.grad

float, default=1.0.

rescaling factor of gradient.

clip_gradient

float, optional, default=-1 (no clipping if < 0).

clip gradient in range [-clip_gradient, clip_gradient].

lr_scheduler

function, optional.

The learning rate scheduler.