If lr 1e-4 gave you a faster decrease of loss than 1e-2 then it would be logical to use 1e-4…

By | September 27, 2018

If lr 1e-4 gave you a faster decrease of loss than 1e-2 then it would be logical to use 1e-4, right? If 1e-4 is “too small to help us escape” then 1e-2 will be even worse because loss decreases slower there.

There is a difference between an assumption that larger LR helps to drive loss down faster and practical observation of how fast loss decreases with different values of LR (that’s what we do with LR finder here).

Read the original article

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.