1.Previous
3.Next

first iterations cause large changes in the parameters, while the later ones do only fine-tuning. Such schedules have been known since the work of MacQueen on {{mvar. Practical guidance on choosing the step size in several variants of SGD is given by Spall.

Implicit updates (ISGD)

As mentioned earlier, classical stochastic gradient descent is generally sensitive to learning rate . Fast convergence requires large learning rates but this may induce

1.Previous
3.Next