Jason Antic on Twitter: "1/ “Irrational as it seems I didn’t want to use a cluster in the cloud somewhere, watching the dollars leave my bank ac- count as I run various experiments.” That’s exactly how I feel both about doing experiments and even deploying production models."

@citnaj

Obsessively pursuing the perfection of image and video colorization/restoration using deep learning. Creator of DeOldify.

Country	Code	For customers of
United States	40404	(any)
Canada	21212	(any)
United Kingdom	86444	Vodafone, Orange, 3, O2
Brazil	40404	Nextel, TIM
Haiti	40404	Digicel, Voila
Ireland	51210	Vodafone, O2
India	53000	Bharti Airtel, Videocon, Reliance
Indonesia	89887	AXIS, 3, Telkomsel, Indosat, XL Axiata
Italy	4880804	Wind
3424486444	Vodafone
» See SMS short codes for other countries

‏ @citnaj

I saw this paper and immediately dropped what I was doing to read it and got hooked immediately. It’s super interesting, insightful, and FUNNY. I also love his discipline of sticking with a humble desktop to run his experiments- I strongly believe in this.https://twitter.com/Smerity/status/1199529360954257408 …

The SHA-RNN is composed of an RNN, pointer based attention, and a “Boom” feed-forward with a sprinkling of layer normalization. The persistent state is the RNN’s hidden state h as well as the memory M concatenated from previous memories. Bake at 200◦F for 16 to 20 hours in a desktop sized oven.

The attention mechanism within the SHA-RNN is highly computationally efficient. The only matrix multiplication acts on the query. The A block represents scaled dot product attention, a vector-vector operation. The operators {qs, ks, vs} are vectorvector multiplications and thus have minimal overhead. We use a sigmoid to produce {qs, ks}. For vs see Section 6.4.

Bits Per Character (BPC) onenwik8. The single attention SHA-LSTM has an attention head on the second last layer and hadbatch size 16 due to lower memory use. Directly comparing the head count for LSTM models and Transformer models obviously doesn’tmake sense but neither does comparing zero-headed LSTMs against bajillion headed models and then declaring an entire species dead.

Smerity @Smerity

Introducing the SHA-RNN :) - Read alternative history as a research genre - Learn of the terrifying tokenization attack that leaves language models perplexed - Get near SotA results on enwik8 in hours on a lone GPU No Sesame Street or Transformers allowed. https://arxiv.org/abs/1911.11423 pic.twitter.com/RN5TPZ3xWH

1 reply 20 retweets 131 likes