And at bottom, the bug was simple, you send a small amount of data, and ask the server to send you back up to 16k of data, and it would send back 16K of decrypted plaintext from memory. URLs, passwords, cookies, credit cards, just about anything could be in there. Ouch ouch.
-
-
I know that because on the night of Heartbleed we did something we never did before: we started vulnerability scanning every EC2 IP address and sending customers notifications. We thought it was a big enough deal that the emails would be worth it.
Show this thread -
The day after Heartbleed, our core cryptography people met, I remember
@pzb was there, and we did a few more things with the OpenSSL package. Amazon's OpenSSL has always been a bit different than the public one, but that day we created a new "hardened" branch.Show this thread -
I won't go into what we did with it here, but quite a bit at the time, Emilia Kasper included some of the changes into base OpenSSL later I think. Our customers mostly upgraded to the latest public version from OpenSSL, which we had in Amazon Linux too.
Show this thread -
Unfortunately we had a few customers stuck though; their OpenSSL libraries were embedded in commercial software that they couldn't quickly upgrade. One of our VPs reached out "Is there anything we can do here?"
Show this thread -
So at about 2AM, I wrote a Netfilter plugin that could block heart bleed using the Linux Kernel firewall. It's still on GitHub ... https://github.com/colmmacc/nf_conntrack_tls … , it tracks the TLS record layer state machine and would drop any heartbeat messages. Crude but effective.
Show this thread -
In our annual planning, we had raised the idea of writing our own TLS/SSL implementation because we thought we could better, but it was a nascent plan. Well that went from nascent to DO IT NOW. I started writing when became Amazon s2n.
Show this thread -
It took about 5 weekends, just me, and there's something very special about finally getting a bunch of code together and seeing it work in a browser. It took a little longer, and 3 intense security reviews, to get approval to Open Source it, but our CEO was very supportive.
Show this thread -
Now it's widely used across AWS. Blows my mind to think that S3 is using it!https://github.com/awslabs/s2n
Show this thread -
s2n is coded specifically in a way to try to avoid the problem heartbleed hit. Rather than parse memory into integers using pointers directly, all across the code, s2n uses a "stuffer" data structure that includes a cursor. Similar to BoringSSL's crypto_bytes, or DJB's stralloc.
Show this thread -
Oh BoringSSL! In the months after HeartBleed, the industry rallied to get OpenSSL more funding and support through the core infrastructure initiative. We still take part! And the BoringSSL and LibreSSL forks of OpenSSL happened. Great work from each!
Show this thread -
The next year, the amazing
@BenLaurie and@trevp__ started an annual High Assurance Cryptography workshop after@RealWorldCrypto, that has also born fruits and helped us produce tools that can analyze cryptography code and find even subtle problems.Show this thread -
I'm almost done, but before I finish, I kind of depressing twist on this whole thing: The Heart Beat extension never really made any sense to begin with. A 0-byte record could have been used as a keep-alive, and ordinary path MTU discovery works for UDP!
Show this thread -
All of this trouble for a feature that to this day I can't even think of a good use case for. This is one reason why "Don't do less well. Do less, well." resonates with me as a motto.
Show this thread -
That's my story for now, until I remember something I forgot. Thanks to everyone who moved mountains 5 years ago. I'm in JFK waiting to fly to Bucharest, so AMA!
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.