The blackbox style of existing RNN implementations mean even simple modifications such as recovering the hidden state, rather than the output, at each timestep is difficult. One of the biggest advantages of non-recurrent self attention is that the tooling is already sufficient.
-
-
Prikaži ovu nit
-
Even in self-attention you need fancy tricks to be able to perform certain optimizations, such as reversible layers, in a way that's supported and gets maximal advantage from the framework. I'm glad that JAX exists for such cases!
Prikaži ovu nit -
For a link to the PyTorch JIT example, which shows brilliant work but I just wasn't able to convert it into a working solution after trying to work through the code (i.e. not really working result + slower than cuDNN - but maybe all my fault!): https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/ …pic.twitter.com/VJYWKeMYZp
Prikaži ovu nit
Kraj razgovora
Novi razgovor -
-
-
But isn't the pytorch lstm itself implemented in cuda?
-
The LSTM used by most deep learning frameworks (including PyTorch) is the blackbox cuDNN LSTM from
@NvidiaAI. It is likely implemented in CUDA.
Kraj razgovora
Novi razgovor -
-
-
I'd say CUDA and cython for gpu and cpu, jax for tpu. Did you see this?https://github.com/lmnt-com/haste
-
You are really a huge fan of cython, but is it easier to write than C++ with pybind?
- Još 2 druga odgovora
Novi razgovor -
-
-
If it ever reaches mass adoption swift for tensorflow with mlir seems like it’d be a good fit
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
I hope a solution to this is found. There are MANY cases where transformers are simply not optimal at all. And people talking about position coding sounds like someone scratching a chalkboard to me....
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
Julia might be worth a look. I am still trudging through it but it has functional autograd and gpu versions of most nn-primitives. Comparing with cudnn for rnn architectures seems like a good objective (maybe someone has already done it...)
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.
in SF.