So Firefox 106 triggered a bug on 1st gen Ryzen processors. We found this by noticing a bunch of nonsensical crashes in the painting code that only happened on these processors.
3
24
211
Conversation
How did we fix it? By rebuilding the exact same code as 106. The timing differences caused by using a new profile when doing PGO were enough to cause the generated code to be different enough to avoid triggering the processor bug.
Replying to
How common is it for PGO perturbances to affect stability?
Even having it happen once must terrify anyone thinking of releasing a hotfix, no matter how small the change is.
1
4
Can the PGO process be made more deterministic, while keeping the performance benefits and not adding much more complexity? Like maybe measuring instruction counts rather than time, or measuring w/ a more deterministic thread scheduler, or even using PGO data from a previous run.
1
2
Show replies
Replying to
Terrifying. How do we make sure the next build doesn't reintroduce the issue?
1
4
We don't really know what "the issue" is (AMD's docs are too vague) so testing and watching crash correlations seems the only way.
1
4
Show replies
Replying to
What actually is the processor bug though? Even if a crash only shows on one processor type my first assumption would be unlucky timing exposing an application bug / UB.
1
3
Replying to
We don't know what the actual bug is yet but are talking with AMD about it.
bugzilla.mozilla.org/show_bug.cgi?i has some details about why the crashes don't make sense.
1
2
Replying to
"Before we do anything else, I think we should push the car back up the hill and see if it happens again."
4