Replicated and cross checked process randomly fails due to an *offset* (not data) mismatch between both replicas, on the order of once a petabyte. Cause: Google's internal optimized zlib implementation had a buffer overflow read that very rarely caused correct but *longer* output
-
-
Show this thread
-
Service crashes upon receiving a protobuf with wrong schema. Root cause: producer server had kernel paniced, data not flushed to disk (but metadata was). On reboot picked up garbage sectors, happened to contain valid data from another process, format is self-synchronizing.
Show this thread -
Bonus points for debugging that one entirely from logs and records of a single occurrence (and concluding beyond reasonable doubt that that's what had happened based on stuff like suspect offsets lining up on 512 byte boundaries), plus sec impact (data was from another user).
Show this thread -
There was also another Go bug after I left, which someone said was breaking Google production. Cgo callbacks back into Go would transiently break stack tracing after returning, causing the garbage collector to explode if it ran with the right timing. Repro was tricky.
Show this thread -
Can't get away from fixing Google production even after you leave
Show this thread
End of conversation
New conversation -
-
-
I mean, if you ever get bored and want to get paid to go bug hunting again...出来ること山程あるぞ

Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
I hate that my current job is mostly being thrown bugs for : - legacy stuff I'm not allowed to upgrade/port (dependency/version rabbit hole, or ENOTIME) - proprietary stuff I can only work around ...and being told "oh well it's not important anyway, it's just to keep you busy."
-
The second part is especially vexing because it's vendor stuff and either they're dead companies, or I'm not allowed to talk to them/report that their stuff isn't working as intended.
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.