It's ten years since we launched Amazon CloudFront! Building it was super fun, my first day at Amazon was spent getting stuck into the CloudFront routing software and making it actually work. A few weeks later, but before we launched, we hit my favorite bug ever ...
-
Show this thread
-
of course it involves YAML, and may be the root of my hatred for that format! So CloudFront is a DNS based CDN, the "magic" happens at the DNS layer, where our DNS servers figure out what the best location to serve your traffic is ...
1 reply 2 retweets 15 likesShow this thread -
we built this gigantic live map of the internet ... for every network out there, we figure out what the correct ordering of CloudFront locations should be. E.g. if I'm Seattle, then it's something like SEA, PDX, SFO ... you get the idea ...
1 reply 2 retweets 10 likesShow this thread -
The map is based on latency measurements and geo-data. For the geo-data, we had import a huge geodb (we have a few of these) and import them into the CloudFront routing later. That import happened in YAML format.
1 reply 1 retweet 10 likesShow this thread -
Anyway, the bug was that the YAML parser ignored the negative signs ... so negative GPS coordinates became positive ones. We caught it before launch, obviously, but it's still my favorite bug because it was so subtle.
4 replies 4 retweets 48 likesShow this thread -
The bug affects 75% of the Earth's surface, but by geographic coincidence or Greenwish winning the meridian wars, it just happened that our entire dev team was unaffected. Both Seattle and Dublin are in the "positive, positive" quadrant of the world!
3 replies 5 retweets 59 likesShow this thread
So we never experienced it ourselves, or our test cases, and we didn't notice it until we really started looking at other locations. I always think of that when I remember that even "dumb" and "obvious" bugs can be subtle and invisible because of your perspective.
-
-
*Greenwich in my previous tweet. Anyway, fun bug! CloudFront had an incredible launch team, including 8 then-or-now Principal Engineers, which is staggering!
3 replies 1 retweet 38 likesShow this threadThanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.