tfw I've been describing all the self therapy I've been doing using gradient descent analogies for months
Conversation
i'm really hoping you or or somebody will one day explain these things to me like i'm five because let's be real here, i'm five
1
3
Gradient descent means "If you're looking to get to the bottom of the hill, head downwards".
This doesn't reliably work, because sometimes hills have dips on them and you get stuck in the bottom of a dip without reaching the bottom of the hill, but it's usually pretty good.
5
10
as a metaphor for navigating life "down" is supposed to be "better"; so you imagine that you're traversing a landscape (the "optimization landscape") and different points in the landscape are better or worse versions of your life, represented by height
2
4
so concretely gradient descent as a life strategy theoretically would mean "figure out a small movement you can make that is as good as possible and then make that movement, then repeat"
practically that "as good as possible" is rough so mostly you settle for "pretty good"
1
3
you can get stuck in a bowl where there are no small moves you can make to go further down / make your life better; this is what people are talking about with "local minima" (or "local maxima" if they flip the analogy so that "up" is "better")
in that situation gradient descent gets stuck and you have to do something else, make some bigger non-local moves, which may temporarily make your life worse. this is the sort of thing mark points to with "nonmonotonicity"
1
5
sometimes people will talk about "simulated annealing" as a metaphor; annealing is a thing you do with a material like metal where you heat it and then cool it and idk anything more about it than that, as a metaphor it means you make large random moves that get smaller over time
1
2
Show replies