Sarah MeiVerified account

http://HotCRP.com  had an outage this morning. I’m very sorry about that! Here is what happened.

HotCRP uses a Perl script called `banal`, originally by Geoff Voelker, to analyze PDFs for compliance. I rewrote this script about a week ago because it had bugs, especially around leading detection, and lacked features such as reference/appendix detection.

Among the “good” software engineering practices I added: `use warnings`, aka `perl -w`, which causes Perl to emit warnings on dangerous old-school operations like comparing 0 with undefined. I removed a ton of warnings, but missed some.

The PHP interpreter has an amazing feature: if a page takes too much time to render, the interpreter exits cleanly. This eliminates whole classes of failure. But some functions, such as reading a file’s contents, don’t count against the timeout.

We are ready for the outage. 1. There is a funky PDF at e-Energy '19. Analyzing this PDF caused banal to generate lots of warnings. 2. The Perl interpreter wrote these warnings to a pipe back to the PHP interpreter. The Perl interpreter then blocked because the pipe filled up.

3. Meanwhile, the PHP interpreter blocked reading from a *different* pipe: Perl’s *output*, rather than its *error output*. Deadlock.