Pedro Domingos

The gradient almost never points to the minimum, so descending it is an odd idea.

Of all the directions you can walk towards in a continuous (often high dimensional) space, the gradient offers the highest probability of getting to a minimum, so why would it odd to descend? (I realize that when everyone is descending, a few should explore the mountains, too.)

In an convex problem, there's an enormous range of directions that get you to a minimum with probability 1, and it's not clear the gradient is special in any way. In a nonconvex problem, your statement is false.

and what do you propose for getting to the minimum then? i suspect you will always need an accurate model of the manifold to guess it well, which is where you get into higher order intelligent learning beyond just perception&reward