The fact that evaluating ∇f(x) is as fast as f(x) is very important and often misunderstood http://timvieira.github.io/blog/post/2016/09/25/evaluating-fx-is-as-fast-as-fx/ …https://twitter.com/gabrielpeyre/status/1167663307668373504 …
-
If I recall, given a function f: R^n -> R^m - adjoint (backwards) methods scale with m, forward sensitivity methods scale with n. In almost all of ML, m is a scalar function (a loss function) - so backwards methods dominate.
Yup! And there is a rich space of hybrid forward-reverse methods for the general (n,m) setting depending on the underlying graph.
-
-
Yup! And you can split the underlying computational graph to mix and match - although calculating the optimal combination of forward and backwards passes is NP complete in the general case.
-
You can also optimize for different time-space complexity tradeoffs! I wrote about some of that here for the special case of RNNs http://timvieira.github.io/blog/post/2016/10/01/reversing-a-sequence-with-sublinear-space/ …
- Još 1 odgovor
Novi razgovor -
-
-
Yup! From an API point of view - it means that methods that approximate the Hessian as J^T J (with J as the Jacobian evaluated along the m residuals) are very hard to use efficiently with AD.
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.