For an example that's important to me, Clang-compiled kernels with LTO and CFI have been shipped since 2018 on Pixel phones. Support for this still isn't in mainline despite many years of them trying to land it. It's strange seeing people talk about that as if it's bleeding edge.
Conversation
At this point, I don't think it's the case that most Linux kernel development is done upstream first. Most people have given up on doing that a long time ago. You get what you need working downstream, ship it, and maybe you try to upstream it but it'll take years to see benefits.
1
Here at AWS, in recent times for features like Nitro Enclaves and Elastic Fabric Adapter getting changes upstream was a launch gating milestone.
Elastic Network Adapter, being "just" a NIC driver, was briefly only available out of tree.
I think "upstream first" is still needed.
1
I'm heavily focused on smartphones where nearly all the drivers are out-of-tree on launch and slowly trickle into the kernel. Literally a majority of the code is not from kernel.org but will gradually trickle there in the years following the launch of an SoC/device.
1
Embedded and phones are a hot mess. I have been lucky to mostly avoid it.
Android's deep kernel fork didn't help. Hopefully that will continue to improve, but it will take time and effort.
3
Android hasn't had a substantial kernel fork for years. It works with mainline kernels.
SoC vendors take latest LTS branch when developing their SoC, implement all their drivers for it, often with a lot of new/rewritten code rather than it just being a minor port from old code.
1
SoC vendor is the upstream for half of the kernel code used on those devices rather than kernel.org.
When Google says they're moving to upstream first they mean giving up on upstreaming remaining few Android common kernel changes and using inferior hacks or eBPF.
1
They defined a stable ABI for kernel modules and hardware support will be done primarily via out-of-tree modules targeting a stable ABI for an LTS branch. ABI can't be at all stable across LTS versions. They won't convince Linux to do anything like that any time soon.
1
github.com/GrapheneOS/ker (just that, no userspace code was dropped) was replaced with literally thousands of lines of code in userspace across multiple components to dynamically manage eBPF rules. What they mean by upstream first is making about a dozen sacrifices like that.
1
2
A particularly painful example is that they switched from ashmem to memfd despite losing the ability to unpin/pin pages similarly to MEM_RESET and MEM_RESET_UNDO on Linux where you can tell the kernel it can free that memory if there's memory pressure but you can ask for it back.
1
1
This was used by web browsers to implement volatile caching aware of memory pressure on Android as they do on Windows since the kernel takes care of it and purges the caches. No such thing on non-Android Linux and now it will be gone on Android, increasing browser memory use.
They got close enough to not having any required Android common kernel code for them to decide to make these final dozen sacrifices to be able to directly use kernel.org kernels with backported bug fixes, performance improvements and in some cases actual features.
2
1
It doesn't change that over half of the kernel code comes from the SoC vendor. It's just not going to be part of the core kernel but rather dynamically loaded kernel modules loaded from boot_vendor / vendor with enforced ABI stability between them with similar tooling as Treble.
1
Show replies