GPT-2 models of sizes ranging from 125M to 770M, Sophia achieves a 2x speed-upĬompared with Adam in the number of steps, total compute, and wall-clock time. Negligible average per-step time and memory overhead. Sophia onlyĮstimates the diagonal Hessian every handful of iterations, which has It is easy to use for both novices and experts alike. It can execute its tasks per your settings and the physical capabilities of your computer. Of non-convexity and rapid change of Hessian along the trajectory. Wise Memory Optimizer helps enhance your PC's performance by tuning and freeing up the physical memory wasted on useless applications. The clipping controls the worst-case update size and tames the negative impact The moving average of the estimated Hessian, followed by element-wise clipping. The update is the moving average of the gradients divided by
Optimizer that uses a light-weight estimate of the diagonal Hessian as the Second-order Clipped Stochastic Optimization, a simple scalable second-order Adam and its variants have been state-of-the-artįor years, and more sophisticated second-order (Hessian-based) optimizers often Improvement of the optimization algorithm would lead to a material reduction on Download a PDF of the paper titled Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training, by Hong Liu and 4 other authors Download PDF Abstract: Given the massive cost of language model pre-training, a non-trivial
0 kommentar(er)
