Donate
Donate
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
Add an identifier
Thank You!
Abstract: Adaptive gradient methods, which adopt historical gradient information to
automatically adjust the learning rate, despite the nice property of fast
convergence, have been observed to generalize worse than stochastic gradient
descent (SGD) with momentum in training deep neural networks. This leaves how
to close the generalization gap of adaptive gradient methods an open problem.
In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are
sometimes "over adapted". We design a new algor ...
11
votes
0
answers
890
views
Recent Votes
Recent Awards •
All
Invitee to
Yuval Filmus
▴
15
Founding Contributor to
FrankS
▴
55
Invitee to
marco.fellous-asiani
▴
15
Founding Contributor to
Ken
▴
45
Recent Replies
Answer: When a question is asked about a paper, are the authors contacted?
by
Andy
46
It depends.
**Authors that are registered users:**
If an author of the paper is
- a registered user,
- known to be the author (se…
Answer: How to write a good question?
by
Andy
46
There are no hard rules criteria for what makes a "good" post. The most important thing is that it either contains **information that's us…
Answer: Implementing an algorithm
by
Andy
46
Use a distibution that's symmetric around the origin and normalize the results so they lie on the sphere. E.g. you can use a Gaussian. He…
Answer: Application of Property RD
by
Dustin
125
It is used to prove the bound in proposition 4.6 which is then used crucially in proposition 4.7 to prove the uniform boundedness of the op…
Traffic: 1 users visited in the last hour