To backtrack a little: once the HeartBleed website went live (which incidentally was hosted on AWS S3! and there was never event a hint of taking it down) we started getting a *lot* of customer contacts.
-
Show this thread
-
HeartBleed was really well marketed, which is a good thing! Months later in a presentation I showed that it made more headlines and news articles in one day than any war had since Vietnam. Good because people patched. 98% of customers patched within a week.
1 reply 18 retweets 134 likesShow this thread -
I know that because on the night of Heartbleed we did something we never did before: we started vulnerability scanning every EC2 IP address and sending customers notifications. We thought it was a big enough deal that the emails would be worth it.
1 reply 6 retweets 129 likesShow this thread -
The day after Heartbleed, our core cryptography people met, I remember
@pzb was there, and we did a few more things with the OpenSSL package. Amazon's OpenSSL has always been a bit different than the public one, but that day we created a new "hardened" branch.2 replies 3 retweets 85 likesShow this thread -
I won't go into what we did with it here, but quite a bit at the time, Emilia Kasper included some of the changes into base OpenSSL later I think. Our customers mostly upgraded to the latest public version from OpenSSL, which we had in Amazon Linux too.
1 reply 2 retweets 63 likesShow this thread -
Unfortunately we had a few customers stuck though; their OpenSSL libraries were embedded in commercial software that they couldn't quickly upgrade. One of our VPs reached out "Is there anything we can do here?"
1 reply 2 retweets 61 likesShow this thread -
So at about 2AM, I wrote a Netfilter plugin that could block heart bleed using the Linux Kernel firewall. It's still on GitHub ... https://github.com/colmmacc/nf_conntrack_tls … , it tracks the TLS record layer state machine and would drop any heartbeat messages. Crude but effective.
1 reply 17 retweets 210 likesShow this thread -
In our annual planning, we had raised the idea of writing our own TLS/SSL implementation because we thought we could better, but it was a nascent plan. Well that went from nascent to DO IT NOW. I started writing when became Amazon s2n.
2 replies 7 retweets 96 likesShow this thread -
It took about 5 weekends, just me, and there's something very special about finally getting a bunch of code together and seeing it work in a browser. It took a little longer, and 3 intense security reviews, to get approval to Open Source it, but our CEO was very supportive.
4 replies 4 retweets 113 likesShow this thread -
Replying to @colmmacc
Impressive, but why weekends? What was happenings on weekdays?
1 reply 0 retweets 1 like
Mostly it’s that I prefer to code for a few hours on weekends when there’s fewer interruptions, no meetings. I say yes to most meetings to help folks out. Every PE is a bit different balance.
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.