Bazel can be set to auto-retry flaky tests. Mixed with good caching it does reduce a lot of the pain from these in my experience at least
-
-
yeah, the tools are pretty nice and helpful but it appears to be far from automatic
1 reply 0 retweets 0 likes -
Are you using bazel? Bazel isn’t perfect but this part really delivers. The caching works and if you have done grain targets your CI is fast. I don’t think flaky tests hurt bazel any more than any CI. Can you explain that?
1 reply 0 retweets 1 like -
I'm currently the person migrating a large mostly-Scala project to Bazel. We're not done but very impressed so far. Flaky tests don't hurt Bazel any more, but what I'm saying is that they appear to really make it harder to enjoy the benefits of cached test results
1 reply 0 retweets 1 like -
We use a merge queue so we only merge to master if the tests are green on merge. With that, flakey tests require rerunning for merge but once in green they are cached.
2 replies 0 retweets 1 like -
Replying to @posco @stephenjudkins and
Sometimes a flaky test is a sloppy test and sometimes it's a buggy feature. What if the green, not the red, is the flake? I worry about caching false positives.
5 replies 1 retweet 3 likes -
Replying to @rossabaker @posco and
Once your having to make a concerted effort to debug/fix flaky tests you can spot/fix them either way. Hooking into the BEP to file issues with teams about their flaky tests can help too
1 reply 0 retweets 1 like -
Another variant is a property test failing after several runs. Deterministic by seed, but takes the generator a long time to find the corner case. Maybe a combination of cached tests and periodic full retests are a happy medium.
1 reply 0 retweets 1 like -
Replying to @rossabaker @0x138 and
You get periodic retesting when you change the upstream code and dependencies.
1 reply 0 retweets 1 like -
Replying to @posco @rossabaker and
We run post-commit tests without caching to spot flakes. People don’t need to eait for them while trying to merge, but the historical stability information is there for anyone who wants to investigate. Works great!
2 replies 1 retweet 6 likes
we’ve been running full periodic uncached builds to flush out flaky tests and to make sure we can still build from scratch in the event the cache is down or goes poof
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.