Hey @fchollet any plans for transformer models in Keras or mechanisms like attention?
There is an Attention layer in tf.keras. Maybe we'll add multi-head attention too. And we have several reference implementations of Transformer e.g.https://medium.com/tensorflow/a-transformer-chatbot-tutorial-with-tensorflow-2-0-88bf59e66fe2 …
-
-
Yay! Thank you for bring DL to the people
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Is this (https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention …) not available in (normal) keras? I can't find it in https://keras.io/ . Do you think in the long term one of the two (tf.keras or keras) will disappear?
-
Yeh definetely not in the version I have and the tensorflow transformer demo doesn’t mention keras implementation... great news it’s on the horizon though!
End of conversation
New conversation -
-
-
@fchollet for your enjoyment (if you haven’t read it already)https://arxiv.org/abs/1907.06902v1 … -
Thank you for sharing our paper!
End of conversation
New conversation -
-
-
The recurrent attention PR in keras looks dead. Could this be used to implement recurrent attention? I can't find an example of anyone actually using tf.keras.Attention ...https://github.com/keras-team/keras/issues/11172 …
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.