Git history gets ridiculously bloated over time. Compacting it all with git gc --aggressive helps a lot. Setting core.compression to 9 gets a little bit more out of the zlib compression. Most size wins come from converting loose objects and smaller packs into unified packs.
Conversation
I'm chronically low on disk space from working on multiple different trees based on Linux, Chromium and the Android Open Source Project. Stuck with this puny 2TB Samsung 980 Pro until they release the 4TB Samsung 990 Pro next year. Have learned lots of ways to save disk space...
1
3
Can also save tons of space with `cp --reflink` if your filesystem has support (XFS and Btrfs but not ext4). It's essentially fork(...) for files. It uses block-based copy-on-write at the destination. Can copy an identical file over another to deduplicate without sharing writes.
2
2
5
I moved to XFS from ext4 for better parallel NVMe performance and started using reflinks a lot. XFS doesn't have transparent compression support yet but I found it hurt build performance too much and most of my space is used by already compressed Git object/pack data anyways.
1
1
4
1
2
actually reducing the underlying disk space used by git wasn't implemented but would be possible by lazily fetching git history and otherwise maintaining a shallow clone. it would be a very large project but it would be fun to work on
1
1
XFS delayed allocation + delayed logging combined with 128GB of memory and a fast NVMe SSD means that barely anything ends up being I/O bound. End up with huge sequential writes as build outputs as flushed out. Reads mostly hit page cache. SSD stats have 10:1 write:read ratio.
1
2
Generating delta updates for GrapheneOS has parts that are extremely I/O bound unless it's done in tmpfs and 128GB memory is not nearly enough without using fewer jobs than CPU cores. It'd already go OOM without tmpfs with 1 job per thread (32) instead of 1 job per core (16).
2
3
this is also where the use case for big corps vs individual devs comes into play; at twitter we offloaded stuff onto remote process executions via the build tool but that's not available to most people + it requires rewriting the entire build for a remote enabled build tool
1
2
I need to do official builds, signing and delta generation locally for security and trust reasons so that's inherently bottlenecked by my local compute resources.
I wanted to make a 64 core Zen 3 Threadripper machine to help with this but Zen 3 Threadripper was never released.
Currently have Ryzen 5950X (16 core Zen 3) with PBO2 enabled and i7-6950X (10 core Broadwell-E). That Intel CPU from 2016 takes 2x longer so the 9 official builds are split 6:3 (previously 9:4 before Android 13).
Part availability is the real blocker rather than it being cost.
1
2
Canadians aren't allowed to buy nice things. It's really hard to get computer parts without a huge price premium especially with the chip shortage. Couldn't get Zen 2 Threadripper parts, then wanted to wait for Zen 3, which got delayed a year then never launched to retail at all.
2
oooh ok yeah hadn't considered signing, can't see a workaround for that. much sadness re threadripper
1
The official builds themselves also need to be done locally, not just the signing part of it.
Development builds tend to take 30 seconds to a couple minutes due to ninja. Release builds are 45 minutes on Ryzen 5950X with PBO2 and 90 minutes on the aging overclocked i7-6950X.
2
3
Show replies