JavaScript is not available.

Oct 26, 2022

12X faster transformer model, possible? Yes, with

@OpenAI

Triton kernels! We release Kernl, a lib to speedup inference of transformer models. It's very fast (sometimes SOTA), 1 LoC to use, and hackable to match most transformer architectures. github.com/ELS-RD/kernl 🧵

115

563

Michaël Benesty

Below all the shapes to keep in mind to get best performances:

Michaël Benesty

kernl/1 - tiled matmul.ipynb at main · ELS-RD/kernl

Regarding tensor cores constrains (related to FMA instructions): docs.nvidia.com/deeplearning/f

Credit for image above goes to Triton tuto website: triton-lang.org/master/getting

If you want to start learning Triton, we have a (very) simple tuto on matmul tiling on Triton here:

github.com

Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackable. - kernl/1 - tiled matmul.ipynb at main · ELS-RD/kernl

My feeling is that with PyTorch 2.0,

@openAI

Triton will be learned and used by many more people. Hopefully, at the end we will get much more "flash attention" like kernels.

The shape of the matrices to be multiplied should be a multiple of a tile shape for best performance. No need for a power of 2, a multiple of 64 usually works well.

So in the end, common tile sizes that work well in today's GPUs are 64x64, 64x128, or 128x128, not bigger. These tiles are hardcoded into libraries such as cutlass or cuBlas and themselves depend on low level GPUs instructions.

A larger tile means better data reuse and faster execution, since the usual bottleneck in the GPU is memory bandwidth, rarely compute speed. But bigger tile means more registers to use, and at some point register spilling which hurts badly performances.

If the shapes of A and B are not multiples of the "tile", you must perform boundary checks, masking, etc. Some of these constraints are heavier when tensor cores are used (vectorization requirements). -> more ops to execute, if/else & divergent branchs, etc. aka slower execution

Basically, the math for matrix multiplication is different for GPUs than what is usually taught at school. When done manually, you manipulate entire rows/columns, but on GPUs you usually multiply blocks (tiles) together to improve data reuse.

It is very interesting that so many people are surprised by this and seem to see it as black magic. This perf boost is most likely due to "tiling", which is the way matmuls are done on GPUs. 🧵

Quote Tweet

Andrej Karpathy

@karpathy

Feb 3

The most dramatic optimization to nanoGPT so far (~25% speedup) is to simply increase vocab size from 50257 to 50304 (nearest multiple of 64). This calculates added useless dimensions but goes down a different kernel path with much higher occupancy. Careful with your Powers of 2.

Naman Goyal

@NamanGoyal21

Feb 3

Its crazy that, at 60% Model FLOPS (FP8) Utilization on H100, original GPT3 configuration can be trained in 3 days on 1024 H100s and PaLM on 12 days on 2048 H100s. That's roughly 50x lesser gpu hours for GPT3 paper 3 years back, and 9x lesser for palm released 9 months back

381

Dan Siroker

@dsiroker

😂 Business Insider thinks OpenAI’s bold vision to bring AGI into reality means “adjusted gross income”

164

516

5,461

Shubham Saboo

@Saboo_Shubham_

Feb 2

Big News 🚨 Microsoft just launched Teams premium powered by ChatGPT at just $7/month 🤯 With ChatGPT, Teams users can generate automatic meeting notes, AI-recommended tasks, personalized meeting templates, and a lot more!!

251

3,731

18.7K

英特爾（Intel）展開自救行動！PC供應鏈傳出，近期英特爾正在積極與特定PC品牌客戶洽談，降價求售上一代的Alder Lake處理器。

Dan Nystedt

@dnystedt

Feb 2

Intel has slashed prices as much as 20% on older generation Alder Lake microprocessors in order to clear inventories, DigiTimes reports, citing supply chain sources. It says the price of an i9 CPU has been cut by US$70-$80. $INTC

digitimes.com.tw

英特爾處理器傳降價求售　最大降幅約2成

Horace He

@cHHillee

Let's talk about a detail that occurs during PyTorch 2.0's codegen - tiling. In many cases, tiling is needed to generate efficient kernels. Even for something as basic as torch.add(A, B), you might need tiling to be efficient! But what is tiling? And when is it needed? (1/13)

130

839

Shayne Longpre

@ShayneRedford

✨New Paper✨What’s the best completely public competitor to #ChatGPT? Flan-T5 beats all public models we tested: Flan-T5 3B ▶️ T0++ 3B ▶️ OPT-IML 175B ▶️ GLM-130B ▶️ Flan 2021 3B ▶️ NIv2 3B We release the

@GoogleAI

🌟Flan Collection🌟data + methods for Instruction Tuning! 1/

269

1,090

Varun Shenoy

@varunshenoy_

Jan 31

Can LLMs extract knowledge graphs from unstructured text? Introducing GraphGPT! Pass in any text (summary of a movie, passage from Wikipedia, etc.) to generate a visualization of entities and their relationships. A quick example:

0:54

133.8K views

637

3,624

Judea Pearl

@yudapearl

Danny's official Yahrzeit is tomorrow, February 1st. Tonight we'll light the Soul Candle (Ner Neshama) and ask him to speak on our behalf, a world gone mad, in places where humanity, wisdom and courage once reigned. Yitgadal V'Yitkadash Shmey Rabba music.youtube.com/watch?v=8Kc6Bg

This Tweet is unavailable.

106

448

2,352

Tianyi Zhang

@Tianyi_Zh

Have large language models solved news summarization? Almost there. Our new study shows that text-davinci-002 is comparable to freelance writers. arxiv.org/abs/2301.13848

536

María de la O Mnez

@mariadelaomar

Jan 31

We are hiring at the #Innovation team of Lefebvre Sarrut within R&D. We are looking for an international Product Manager You will be working directly with Manu Mateo @elsitlab

María de la O Martínez on LinkedIn: #innovation

Thomas Roy ... and myself! and many other colleagues of R&D and…

linkedin.com

We are hiring at the #Innovation team of Lefebvre Sarrut within R&D. We are looking for an international Product Manager You will be working directly with Manu…

Hemant Mohapatra

@MohapatraHemant

Jan 31

At this point frankly if you aren't playing around or trying to integrate some sort of generative AI tech within your product, it's basically 2012 and you are ignoring the cloud.

126

On the architecture of torchdynamo

Jay Hack

@mathemagic1an

Jan 30

What if you could fit an *entire codebase* in an LLM? 🤔 "Efficiently Scaling Transformer Inference" (11/2022) arxiv.org/pdf/2211.05102 Jeff Dean + co break out all the hacks to scale PALM-540B's context length to 43,000 tokens! Here's how 👇

204

1,176

Not PyTorch specific but another recent great read I had, this paper explains in very accessible language how Nvidia tensor cores work and their characteristics. Not recent but most of the fundamentals are the same, at least up to A100. home.cse.ust.hk/~weiwa/papers/

A highly recommended read about Dynamo (the workhorse of PyTorch 2.0). It explains the internals and the why.

docs.google.com

On the architecture of torchdynamo At a high level, torchdynamo takes a sequence of Python bytecodes in a live Python interpreter and produces: An FX graph representing the tensor (and symbolic...

Sravan Kundojjala

@SKundojjala

Jan 29

Intel's finished goods inventory shot up significantly by the end of 4Q22. Inventory as a % of revenue reached 94% and likely be >100% in 1Q23. Like $MU, Intel's inventory dollars are likely to peak in the June quarter. Quarterly revenue rate likely drop to $14b in 2023.

Alex Buraks

@alex_buraks

Jan 28

Wow, I just found the list with initial weights of Yandex ranking factors. Do you need one more thread? 😁 P.S. final weights calculated by AI (matrixnet), but initial values are useful as well.

285

Amin Karbasi

@aminkarbasi

Jan 28

Well, it turned out we cannot parallelize boosting!!! arxiv.org/abs/2301.09627

119

863

Walden Catalyst and 2 others

Nathan Benaich

@nathanbenaich

Jan 27

🛰️Deep tech investing (frontier science e.g. techbio, computing, space, energy) is such an exciting space. Yet, EU startups and VCs are *massively* under-resourced. - EU+UK startups raised *3x less* than US cos in '20-22. - There are *2* dedicated EU deep tech funds >€300m.

Diff Models – A New Way to Edit Code

Carper

@carperai

Jan 27

We're releasing a series of diff models for code generation, trained on millions of commits from GitHub to generate changes to code described by a commit message! carper.ai/diff-model Thanks to

(1/4)

CarperAI is releasing a series of diff models—models trained to predict a code diff, trained on millions of commits scraped from GitHub. We are releasing 3 models of different sizes, all fine-tuned…

118

493

Alex Buraks

@alex_buraks

Jan 27

You probably heard about Yandex, it’s the 4th biggest search engine by market share worldwide. Yesterday proprietary source code of Yandex was leaked. The most interesting part for SEO community is: the list of all 1922 ranking factors used in the search algorithm [🧵THREAD]

172

1,878

5,445

In Sept 2022

released Whisper, the world's most accurate speech recognition (ASR) that can transcribe and translate speech audio from 97 languages! But it had a few shortcomings! Here are the top 7 community enhancements on top of Whisper that you should know about: 🧵

361

Dylan Patel

@dylan522p

Jan 26

Holy moly Intel... Gross margins are now 39.2%! Even worse is outlook of 34.1% for the next quarter! Reminder, Intel use to get ~60% gross margins. They are saying revenue Q1 is ~$11B vs $18.4B the prior year..... -$4B cashflow in 2022, 2023 will be worse CUT THE DIVIDEND! $INTC

416

Hungry Hungry Hippos: Towards Language Modeling with State Space Models

Dan Fu

@realDanFu

Jan 23

Attention is all you need... but how much of it do you need? Announcing H3 - a new generative language models that outperforms GPT-Neo-2.7B with only *2* attention layers! Accepted as a *spotlight* at #ICLR2023! 📣 w/

@tri_dao

📜 arxiv.org/abs/2212.14052 1/n

arxiv.org

State space models (SSMs) have demonstrated state-of-the-art sequence modeling performance in some modalities, but underperform attention in language modeling. Moreover, despite scaling nearly...

308

1,678

Christoph Molnar

@ChristophMolnar

BREAKING: IBM makes game-changing move, invests $42 in ChatGPT Pro to revolutionize IBM Watson capabilities

153

749

9,912

Frank Willett

@WillettNeuro

Our new pre-print is out today! We demonstrate a brain-computer interface that turns speech-related neural activity into text, enabling a person with paralysis to communicate at 62 words per minute - 3.4 times faster than prior work. biorxiv.org/content/10.110

957

3,988

Matt Turck

@mattturck

Jan 21

VCs tweeting about how to operate a startup

0:15

1.7M views

From

Azzu@100

103

353

2,679

Susan Zhang

@suchenzang

Piling on to the pile-on (sorry - it's always easy to criticize 😛), here's a rant about benchmarks for LLMs that are used to back claims of "stronger" or "better" models. Let's start with a tour through GPT-3's Appendix G... 1/8 twitter.com/drjwrae/status

This Tweet is unavailable.

126

Jack Rae

@drjwrae

Seeing a bit of a chinchilla pile-on from this thread. The 'train smaller models longer' paper. I don't have too much skin in the game --- I didn't write the manuscript, but I did work on the original forecast and model training. There seems to be a few misconceptions 1/ twitter.com/suchenzang/sta

This Tweet is unavailable.

109