The TLS protocol had been extended to include a "Heartbeat" extension. It was intended for keep-alives and MTU discovery for DTLS, which uses UDP, but OpenSSL had included it in regular TLS too (which uses TCP).pic.twitter.com/IjhygwcXJV
You can add location information to your Tweets, such as your city or precise location, from the web and via third-party applications. You always have the option to delete your Tweet location history. Learn more
HeartBleed was really well marketed, which is a good thing! Months later in a presentation I showed that it made more headlines and news articles in one day than any war had since Vietnam. Good because people patched. 98% of customers patched within a week.
I know that because on the night of Heartbleed we did something we never did before: we started vulnerability scanning every EC2 IP address and sending customers notifications. We thought it was a big enough deal that the emails would be worth it.
The day after Heartbleed, our core cryptography people met, I remember @pzb was there, and we did a few more things with the OpenSSL package. Amazon's OpenSSL has always been a bit different than the public one, but that day we created a new "hardened" branch.
I won't go into what we did with it here, but quite a bit at the time, Emilia Kasper included some of the changes into base OpenSSL later I think. Our customers mostly upgraded to the latest public version from OpenSSL, which we had in Amazon Linux too.
Unfortunately we had a few customers stuck though; their OpenSSL libraries were embedded in commercial software that they couldn't quickly upgrade. One of our VPs reached out "Is there anything we can do here?"
So at about 2AM, I wrote a Netfilter plugin that could block heart bleed using the Linux Kernel firewall. It's still on GitHub ... https://github.com/colmmacc/nf_conntrack_tls … , it tracks the TLS record layer state machine and would drop any heartbeat messages. Crude but effective.
In our annual planning, we had raised the idea of writing our own TLS/SSL implementation because we thought we could better, but it was a nascent plan. Well that went from nascent to DO IT NOW. I started writing when became Amazon s2n.
It took about 5 weekends, just me, and there's something very special about finally getting a bunch of code together and seeing it work in a browser. It took a little longer, and 3 intense security reviews, to get approval to Open Source it, but our CEO was very supportive.
Now it's widely used across AWS. Blows my mind to think that S3 is using it!https://github.com/awslabs/s2n
s2n is coded specifically in a way to try to avoid the problem heartbleed hit. Rather than parse memory into integers using pointers directly, all across the code, s2n uses a "stuffer" data structure that includes a cursor. Similar to BoringSSL's crypto_bytes, or DJB's stralloc.
Oh BoringSSL! In the months after HeartBleed, the industry rallied to get OpenSSL more funding and support through the core infrastructure initiative. We still take part! And the BoringSSL and LibreSSL forks of OpenSSL happened. Great work from each!
The next year, the amazing @BenLaurie and @trevp__ started an annual High Assurance Cryptography workshop after @RealWorldCrypto, that has also born fruits and helped us produce tools that can analyze cryptography code and find even subtle problems.
I'm almost done, but before I finish, I kind of depressing twist on this whole thing: The Heart Beat extension never really made any sense to begin with. A 0-byte record could have been used as a keep-alive, and ordinary path MTU discovery works for UDP!
All of this trouble for a feature that to this day I can't even think of a good use case for. This is one reason why "Don't do less well. Do less, well." resonates with me as a motto.
That's my story for now, until I remember something I forgot. Thanks to everyone who moved mountains 5 years ago. I'm in JFK waiting to fly to Bucharest, so AMA!
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.