Click to Follow Cyan4973

Yann Collet

@Cyan4973

Zstandard, LZ4, xxHash, FSE

California, USAfastcompression.blogspot.comJoined June 2010

67 Following

2,023 Followers

Yann Collet’s Tweets

Yann Collet Retweeted

Hardware Unboxed

@HardwareUnboxed

Feb 23

A great illustration of the $1-2 billion impact to Nvidia's "gaming" segment when crypto collapsed in the second half of 2022. Gives you an idea of the portion of GPU sales not going to gamers during that period

Quote Tweet

𝐷𝑟. 𝐼𝑎𝑛 𝐶𝑢𝑡𝑟𝑒𝑠𝑠

@IanCutress

Feb 22

Here are some graphs based on $NVDA @NVIDIA's latest financials. Can really see the big hit gaming had in Q2 FY23 (which is Q3 calendar year 2022) and its effects on margins.

Show this thread

475

Yann Collet Retweeted

Meta Open Source

@MetaOpenSource

Feb 15

Zstandard is a data compression algorithm that balances compression ratio with speed. It's available on every platform, making it a nearly universal tool. The Zstd team at Meta shares how it works, the techniques that make it so fast and why it’s widely used throughout industry.

6:15

2.8K views

Show this thread

Yann Collet Retweeted

Peyman 𝕄𝕀𝕃𝔸ℕ𝔽𝔸ℝ

@docmilanfar

Feb 11

the only people I’ve seen dismissing this insightful article are ones who don’t seem to understand compression very well the only surprise(?) is that it took a skilled writer and thinker, instead of one of us researchers, to make the case crystal clear

newyorker.com

ChatGPT Is a Blurry JPEG of the Web

OpenAI’s chatbot offers paraphrases, whereas Google offers quotes. Which do we prefer?

219

1,068

Zstandard v1.5.4 has been released : github.com/facebook/zstd/ . It features a decent crop of performance improvements, and a new interface preparing an exciting hardware-accelerated future.

github.com

Release Zstandard v1.5.4 · facebook/zstd

Zstandard v1.5.4 is a pretty big release benefiting from one year of work, spread over > 650 commits. It offers significant performance improvements across multiple scenarios, as well as new fea...

Yann Collet Retweeted

Learnk8s — @learnk8s@mastodon.social

@learnk8s

Feb 8

In this post you will learn two strategies to speed up the pod creation time: ➀ zstd compressed container images ➁ Seekable OCI for lazy loading container images ➜

element7.io

A Hidden Gem: Two Ways to Improve AWS Fargate Container Launch Times

Because image caching is not easy to fix on AWS Fargate, AWS is working on two alternative approaches to reduce container launch times

Yann Collet Retweeted

Ives van Hoorne

@CompuIves

Feb 8

It also turns out that memory is very compressible. We compress memory in small chunks of 64kb using zstd, so that we can decompress the memory on-demand when the VM boots from it. This is the compress ratio for a memory snapshot of our editor dev env: 16GiB -> 2034MiB

Quote Tweet

Ives van Hoorne

@CompuIves

Feb 7

People have been asking about the cost of running VMs. There's a cost, but we've also done much work to make it cheaper: - We snapshot & hibernate VMs after 5-10min - We compress memory - We lazy-load memory as-needed Now for 300k weekly VM boots, we run ~380VMs max at a time.

Show this thread

Yann Collet Retweeted

Yann Collet

@Cyan4973

Feb 6

Prof. Abraham Lempel passed away this weekend at the age of 86. Prof. Lempel is the "L" for the Lempel-Ziv compression algorithm (which is the base of most compression algorithms, such as zlib, or zstd) .

en.wikipedia.org

Abraham Lempel - Wikipedia

213

629

Abraham Lempel - Wikipedia

213

629

Yann Collet Retweeted

Mark Callaghan

@MarkCallaghanDB

Feb 2

Performance for crc32c, xxh3, lz4 and zstd using gcc & clang, different optimizer flags and different CPUs (x86, Arm) on AWS (c7g, c6i) and GCP (c3, t2a). Much detail and a few interesting results. smalldatum.blogspot.com/2023/02/rocksd

Yann Collet Retweeted

JannieJo

@JannieJo8

Feb 1

"... America’s so-called paper of record has never apologized for... over-the-top coverage of the Clinton emails or the deeply flawed story about the FBI Trump-Russia probe... [NYT] has shown a stunning lack of curiosity about finding out what went wrong."

inquirer.com

The NYT should tell readers whether it helped crooked FBI agents get Trump elected in 2016

The arrest of a high-level FBI agent on Russia-tied corruption charges raises stunning new questions about how Trump really won in 2016.

Yann Collet Retweeted

Will Bunch

@Will_Bunch

Jan 29

It was the real "October Surprise" - 2016 deceptions that put Trump in the White House, fueled by FBI leaks Knowing now that a top FBI agent went to work for a billionaire Putin ally, the NY Times need to reveal the truth of these stories My new column

inquirer.com

The NYT should tell readers whether it helped crooked FBI agents get Trump elected in 2016

The arrest of a high-level FBI agent on Russia-tied corruption charges raises stunning new questions about how Trump really won in 2016.

273

5,837

13.1K

Yann Collet Retweeted

Jay Rosen

@jayrosen_nyu

Jan 29

The New York Times should "finally apologize for the sins of 2016, expose exactly what went wrong, and then reveal the rest, so this kind of disaster never happens again. They owe it to American democracy." That's

@Will_Bunch

on the McGonigal factor.

inquirer.com

The NYT should tell readers whether it helped crooked FBI agents get Trump elected in 2016

The arrest of a high-level FBI agent on Russia-tied corruption charges raises stunning new questions about how Trump really won in 2016.

169

4,796

13.1K

Yann Collet Retweeted

Phoronix

@phoronix

Jan 28

#LLVM 16.0-rc1 Brings New

@AMD

@Intel

CPU Support, #Zstd Debug Sections, C++17 By Default

phoronix.com

LLVM 16.0-rc1 Brings New AMD & Intel CPU Support, Zstd Debug Sections, C++17 By Default

Following the LLVM 16.0 feature freeze and code branching earlier this week, LLVM 16.0.0-rc1 is now available as the first of at least three planned release candidates.

Yann Collet Retweeted

Robert C. Seacord (@rcs@hachyderm.io)

@RCS

Jan 21

I just read a comment that most programmers don't have access to the C Standard. The last working draft of C23 is right here => open-std.org/jtc1/sc22/wg14 Please help yourself to a download.

212

Yann Collet Retweeted

Ryan Pickering

@ryan_pickering_

Jan 20

"It was clear to us that we couldn't just prevent nuclear power by protesting on the street. As a result, we in the governments ... tried to make nuclear power plants unprofitable by increasing the safety requirements." -October 31, 2022 (translated from German) via

@welt

2/x

468

1,420

Show this thread

Yann Collet Retweeted

Binutils 2.40 Released With #AMD Zen 4 & Upcoming #Intel Instructions, #Zstd Support

phoronix.com

GNU Binutils 2.40 Released With AMD Zen 4 & Upcoming Intel Instructions, Zstd Support

Out today is GNU Binutils 2.40 as the latest feature update to this wide collection of key binary utilities found on Linux systems and other platforms.

Yann Collet Retweeted

Bonjour, profitez bien de votre dimanche. Sachez que les GNU binutils viennent de passer en version 2.40 avec le support de l'architecture AMD Zen 4, le support de la compression zstd, de nouvelles options à des nombreux outils (objdump, ...) ⬇️ lists.gnu.org/archive/html/i

Yann Collet Retweeted

Alexandre Mutel

https://mastodon.social/@xoofx

@xoofx

Nov 1, 2022

Created a PR github.com/dotnet/runtime to improve XxHash3 in the .NET runtime (coming to .NET 8.0!), x2 faster than the current one, and much closer to the C++ version (10%) 🏎️ Love the generic HW .NET intrinsics that allow to target e.g ARM/x64 with the same vectorized code! 😎

github.com

Optimize XxHash3 by xoofx · Pull Request #77756 · dotnet/runtime

Hey there! This is a PR to optimize XxHash3 for large buffers (> 240 bytes) by trying to avoid loading/storing the accumulators on each 64 bytes but instead doing it after a whole batch. Benchma...

Show this thread

Yann Collet Retweeted

Julian Andres Klode (@juliank@mastodon.social)

@JulianKlode

Jan 11

dpkg in unstable now supports zstd compression. Thanks for (rewriting from scratch with multi thread support) and merging, guillem!

Yann Collet Retweeted

Rex Chapman

@RexChapman

Jan 10

This conductor being pranked by the orchestra on his birthday is the Twitter content I’m here for…

0:40

7.5M views

From

Theo

1,078

28.8K

178.7K

Show this thread

Yann Collet Retweeted

Michael Barton

@bioinformatics

Jan 9

It's time we adopted more recent algorithms for FASTQ compression

bioinformaticszen.com

We're wasting money by only supporting gzip for raw DNA files.

Stylised graphics of money with DNA helix in the foreground.

202

Yann Collet Retweeted

Lina Khan

@linakhanFTC

Jan 5

1. One in five US workers is bound by a noncompete clause, which restricts them from freely switching jobs, lowering wages and undermining fair competition. Today

@FTC

proposed a rule to prohibit firms from imposing noncompete clauses on their workers.

ftc.gov

FTC Proposes Rule to Ban Noncompete Clauses, Which Hurt Workers and Harm Competition

The Federal Trade Commission

242

2,612

9,328

Show this thread

Yann Collet Retweeted

#GNU

Binutils 2.40 Branched With #Zstd Support,

@AMD

#Zen4, New

@Intel

Instructions

phoronix.com

GNU Binutils 2.40 Branched With Zstd Support, Zen 4, New Intel Instructions

With the start of the new year also came the branching of GNU Binutils 2.40 ahead of its expected stable release around early February.

Yann Collet Retweeted

Cachix

@cachix_org

Dec 19, 2022

If you're seeing your builds go significantly faster - we've rolled out zstd compression by default as part of Cachix 1.1 release. You can opt-out to XZ via your cache settings. Stayed tuned for more in an upcoming blog post.

Yann Collet Retweeted

Twitter suspends

after Musk promises not to ban it theverge.com/2022/12/14/235

275

1,291

4,754

Yann Collet Retweeted

Last Week Tonight

@LastWeekTonight

Nov 21, 2022

Here’s last night’s story about the Qatar World Cup and one man’s mother’s scarf:

youtube.com

Qatar World Cup: Last Week Tonight with John Oliver (HBO)

John Oliver discusses the Qatar World Cup, the human rights violations happening in the sporting event’s host country, and what perfection really means to Da...

454

996

Show this thread

Yann Collet Retweeted

Percona

@Percona

Nov 17, 2022

Today we are glad to introduce support for a new compression algorithm in Percona XtraBackup 8.0.30 - Zstandard (ZSTD). hubs.ly/Q01syqR00 #MySQL #OpenSource #Percona

percona.com

How To Get Your Backup to Half of Its Size - Introducing ZSTD Support in Percona XtraBackup -...

Introducing support for ZSTD in Percona XtraBackup 8.0.30, a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios.

Yann Collet Retweeted

Gizmodo

@Gizmodo

Nov 8, 2022

Apple Is Tracking You Even When Its Own Privacy Settings Say It’s Not, New Research Says dlvr.it/ScSBM3

113

101

Yann Collet Retweeted

Robᵉʳᵗ Graham

@erratarob@infosec.exchange

@ErrataRob

Nov 15, 2022

One easy way to pick part news stories is when they use the passive voice to hide their craptitude. Who called SBF the next JP Morgan or Warren Buffet? Who was complicit in praising the emperor's fine new clothing instead of doing their job and questioning his nakedness?

Show this thread

Yann Collet Retweeted

David Goldblatt

@davidtgoldblatt

Nov 11, 2022

The Meta layoffs hit a couple of very strong perf engineers I want to recommend, with particular expertise in low-level optimization, scheduling, and concurrency. If you're hiring in those areas please DM me for details.

Yann Collet Retweeted

Marty Stratton,

Studio Director, lied about

@DOOM

Eternal's OST events in a Reddit post that used disinformation to blame me entirely for its failure Later, he offered me a six-figure sum to never speak about it The truth is more important.

link.medium.com

My full statement regarding DOOM Eternal

This statement is issued in response to id Software Studio Director Marty Stratton’s “DOOM Eternal OST Open Letter”, published on Reddit.

2,454

33.4K

106.2K

Show this thread

Yann Collet Retweeted

Dan Luu

@danluu

Nov 6, 2022

NYT coverage of "tech" seems like a plausible example? The story hadn't really broken at the time of the above tweet. The "conspiracy theory" was believed by some, but they were generally dismissed as conspiracy theorists and/or people with sour grapes. twitter.com/mattyglesias/s

For starters, it’s important to accept that the New York Times has always — or at least for many decades — been a far more editor-driven, and self-conscious, publication than many of those with which it competes. Historically, the Los Angeles Times, where I worked twice, for instance, was a reporter-driven, bottom-up newspaper. Most editors wanted to know, every day, before the first morning meeting: “What are you hearing? What have you got?”

It was a shock on arriving at the New York Times in 2004, as the paper’s movie editor, to realize that its editorial dynamic was essentially the reverse. By and large, talented reporters scrambled to match stories with what internally was often called “the narrative.” We were occasionally asked to map a narrative for our various beats a year in advance, square the plan with editors, then generate stories that fit the pre-designated line.

Reality usually had a way of intervening. But I knew one senior reporter who would play solitaire on his c...

read image description

ALT

This Tweet is unavailable.

Show this thread

Yann Collet Retweeted

François Chollet

@fchollet

Oct 30, 2022

Software companies aren't made of code. They're made of processes that produce and maintain code. And the foremost component of these processes are people. The code is just a by-product. More of a liability than an asset.

728

3,914

The sounds in this video is a bit weak, the microphone worn under my neck was likely not enabled. Should it be a concern, Ilya Muraviov made a version passing the sound to a denoiser, resulting in more intelligible speech :

youtube.com

Designing Data Compression Solutions - EE274, Yann Collet (Meta), L10...

Stanford Research Talks

I do have the pleasure to make a lecture to PhD students at Stanford, thanks to Prof. Tsachy Weissman. This one is a bit more practical than usual, and may prove of interest to a larger audience : Deploying Data Compression in Data Centers

youtube.com

Designing Data Compression Solutions - EE274, Yann Collet (Meta), L10...

Show this thread

Yann Collet Retweeted

Jeremy Knauff

@jeremyknauff

Oct 25, 2022

Remember the draconian clause PayPal added to its TOS a couple of weeks ago that would enable them to steal $2,500 from your account every time you post anything they don't like? Well, after the widespread criticism died down, they put it right back in.