The fact that evaluating ∇f(x) is as fast as f(x) is very important and often misunderstood http://timvieira.github.io/blog/post/2016/09/25/evaluating-fx-is-as-fast-as-fx/ …https://twitter.com/gabrielpeyre/status/1167663307668373504 …
If I recall, given a function f: R^n -> R^m - adjoint (backwards) methods scale with m, forward sensitivity methods scale with n. In almost all of ML, m is a scalar function (a loss function) - so backwards methods dominate.
-
-
Yup! And there is a rich space of hybrid forward-reverse methods for the general (n,m) setting depending on the underlying graph.
-
Yup! And you can split the underlying computational graph to mix and match - although calculating the optimal combination of forward and backwards passes is NP complete in the general case.
- Još 2 druga odgovora
Novi razgovor -
-
-
These are pretty much the same method, aren't they? It's only a question of optimal matrix multiplication, which is often right-associative if m is small and left-associative if n is small.
-
It's a sparse matrix, in general.
- Još 1 odgovor
Novi razgovor -
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.