Skip to content
  • Home Home Home, current page.
  • Moments Moments Moments, current page.

Saved searches

  • Remove
  • In this conversation
    Verified accountProtected Tweets @
Suggested users
  • Verified accountProtected Tweets @
  • Verified accountProtected Tweets @
  • Language: English
    • Bahasa Indonesia
    • Bahasa Melayu
    • Català
    • Čeština
    • Dansk
    • Deutsch
    • English UK
    • Español
    • Filipino
    • Français
    • Hrvatski
    • Italiano
    • Magyar
    • Nederlands
    • Norsk
    • Polski
    • Português
    • Română
    • Slovenčina
    • Suomi
    • Svenska
    • Tiếng Việt
    • Türkçe
    • Ελληνικά
    • Български език
    • Русский
    • Српски
    • Українська мова
    • עִבְרִית
    • العربية
    • فارسی
    • मराठी
    • हिन्दी
    • বাংলা
    • ગુજરાતી
    • தமிழ்
    • ಕನ್ನಡ
    • ภาษาไทย
    • 한국어
    • 日本語
    • 简体中文
    • 繁體中文
  • Have an account? Log in
    Have an account?
    · Forgot password?

    New to Twitter?
    Sign up
GergelyOrosz's profile
Gergely Orosz
Gergely Orosz
Gergely Orosz
@GergelyOrosz

Tweets

Gergely Orosz

@GergelyOrosz

Writing http://pragmaticengineer.com , the #1 tech newsletter on Substack & @EngGuidebook. Advisor @mobile__dev. Uber & Skype alumni. Jobs board: http://pragmaticurl.com/jobs 

Amsterdam, The Netherlands
pragmaticengineer.com
Joined April 2009

Tweets

  • © 2022 Twitter
  • About
  • Help Center
  • Terms
  • Privacy policy
  • Cookies
  • Ads info
Dismiss
Previous
Next

Go to a person's profile

Saved searches

  • Remove
  • In this conversation
    Verified accountProtected Tweets @
Suggested users
  • Verified accountProtected Tweets @
  • Verified accountProtected Tweets @

Promote this Tweet

Block

  • Tweet with a location

    You can add location information to your Tweets, such as your city or precise location, from the web and via third-party applications. You always have the option to delete your Tweet location history. Learn more

    Your lists

    Create a new list


    Under 100 characters, optional

    Privacy

    Copy link to Tweet

    Embed this Tweet

    Embed this Video

    Add this Tweet to your website by copying the code below. Learn more

    Add this video to your website by copying the code below. Learn more

    Hmm, there was a problem reaching the server.

    By embedding Twitter content in your website or app, you are agreeing to the Twitter Developer Agreement and Developer Policy.

    Preview

    Why you're seeing this ad

    Log in to Twitter

    · Forgot password?
    Don't have an account? Sign up »

    Sign up for Twitter

    Not on Twitter? Sign up, tune into the things you care about, and get updates as they happen.

    Sign up
    Have an account? Log in »

    Two-way (sending and receiving) short codes:

    Country Code For customers of
    United States 40404 (any)
    Canada 21212 (any)
    United Kingdom 86444 Vodafone, Orange, 3, O2
    Brazil 40404 Nextel, TIM
    Haiti 40404 Digicel, Voila
    Ireland 51210 Vodafone, O2
    India 53000 Bharti Airtel, Videocon, Reliance
    Indonesia 89887 AXIS, 3, Telkomsel, Indosat, XL Axiata
    Italy 4880804 Wind
    3424486444 Vodafone
    » See SMS short codes for other countries

    Confirmation

     

    Welcome home!

    This timeline is where you’ll spend most of your time, getting instant updates about what matters to you.

    Tweets not working for you?

    Hover over the profile pic and click the Following button to unfollow any account.

    Say a lot with a little

    When you see a Tweet you love, tap the heart — it lets the person who wrote it know you shared the love.

    Spread the word

    The fastest way to share someone else’s Tweet with your followers is with a Retweet. Tap the icon to send it instantly.

    Join the conversation

    Add your thoughts about any Tweet with a Reply. Find a topic you’re passionate about, and jump right in.

    Learn the latest

    Get instant insight into what people are talking about now.

    Get more of what you love

    Follow more accounts to get instant updates about topics you care about.

    Find what's happening

    See the latest conversations about any topic instantly.

    Never miss a Moment

    Catch up instantly on the best stories happening as they unfold.

    Gergely Orosz‏ @GergelyOrosz Mar 13

    As it's been ~3 years, figured I'll answer "What caused the Uber Eats glitch that allowed ordering free food for a weekend in India?" This was an outage on my watch. Given Quora is paywalled - can't post the answer w/o a sub - here's the story on idempotency & breaking changes:pic.twitter.com/N6o4zETavF

    1:58 AM - 13 Mar 2022
    • 1,276 Retweets
    • 4,566 Likes
    • Riham Abdillah Hasny Louis Anubhav Patel R George Pu Caroline Domenech Joaquin Grech Abhishek Shree
    119 replies 1,276 retweets 4,566 likes
      1. New conversation
      2. Gergely Orosz‏ @GergelyOrosz Mar 13

        1. What happened? One morning someone in India tried to order food via UberEats in India, using Paytm as a payment method. But they didn't have enough balance. Got an error message. Ordered again. The order went through!! Without having money for it. News spread quick.pic.twitter.com/zPecKAKTuZ

        6 replies 30 retweets 287 likes
        Show this thread
      3. Gergely Orosz‏ @GergelyOrosz Mar 13

        2. This was a payments-related bug. The problem with these is how the bug was in the reconciliation flow. And Uber reconciled with Paytm maybe once a week. How Uber discovered this: restaurants started going offline thanks to huge order quantities in very short times.

        2 replies 12 retweets 245 likes
        Show this thread
      4. Gergely Orosz‏ @GergelyOrosz Mar 13

        3. After it was clear something was up, Uber shut down Paytm as a payment method and started the investigation. My team owner the Paytm payment method at the time, so this was me and my team. We naturally looked at what code changes we've made in the timeframe. None.

        1 reply 7 retweets 225 likes
        Show this thread
      5. Gergely Orosz‏ @GergelyOrosz Mar 13

        4. So if we made zero changes on our end, what happened? Turns out the Paytm team did a change late on a Friday that looked innocent enough. It silently changed an API endpoint from behaving idempotent to non-idempotent. Why does idempotency matter?

        6 replies 32 retweets 368 likes
        Show this thread
      6. Gergely Orosz‏ @GergelyOrosz Mar 13

        5. Idempotency means that you can safely repeat requests as you get the same response every time. I remember the endpoint was charge-related. Before, it always returned the same error when trying to charge a wallet without enough credits. With the change, not anymore:

        3 replies 12 retweets 252 likes
        Show this thread
      7. Gergely Orosz‏ @GergelyOrosz Mar 13

        6. Before 1. "Try to charge wallet X without funds" -> Error1 2. "Try to charge wallet X without funds again" -> Error1 After 1. "Try to charge wallet X without funds" -> Error1 2. "Try to charge wallet X without funds again" -> A Brand New Error

        8 replies 14 retweets 250 likes
        Show this thread
      8. Gergely Orosz‏ @GergelyOrosz Mar 13

        7. Now this might look like a small change, but on Uber's side, the assumption was the endpoint was idempotent, so there was no testing on getting anything else back. The new error was unknown and not mapped to anything. Long story short it was interpreted as "success".

        21 replies 26 retweets 450 likes
        Show this thread
      9. Gergely Orosz‏ @GergelyOrosz Mar 13

        8. So Paytm returned an error never documented before without telling its partners. Some partners assumed idempotency changes are breaking API changes to be communicated: but they were not. Uber was one of these partners. The result? Free food until discovered.

        3 replies 15 retweets 308 likes
        Show this thread
      10. Gergely Orosz‏ @GergelyOrosz Mar 13

        9. So who paid for the free food? Restaurants got paid and customers abusing this functionality were never pursued. The responsible party needed to foot the bill. But who was responsible?

        4 replies 7 retweets 217 likes
        Show this thread
      11. Gergely Orosz‏ @GergelyOrosz Mar 13

        10. I can't share the settlement, so leaving a poll here to decide. Who do you think should have footed the cost for the bug? The API provider changing their API to return a new error? The API consumer not parsing a new error introduced - but not communicated? Who should pay?

        47 replies 11 retweets 181 likes
        Show this thread
      12. Gergely Orosz‏ @GergelyOrosz Mar 13

        Both parties were at fault here, which is why liability is tricky. 1. The API consumer should have coded more defensibly & not assume implicit API behaviors are deliberate. 2. The API provider should have communicated changes ahead of time, and not provide implicit idempotency.

        15 replies 36 retweets 539 likes
        Show this thread
      13. Gergely Orosz‏ @GergelyOrosz Mar 13

        Being in the middle of this outage, a few things I learned: - Don't assume "unknown" means "good". Assume the opposite. - The worst outages make for the best stories later. - College students can eat SO MUCH. They were responsible for the majority of food orders during outage!

        7 replies 85 retweets 1,125 likes
        Show this thread
      14. Gergely Orosz‏ @GergelyOrosz Mar 13

        Just to make things more gray, a correction. The new API behavior was not a clear-cut error if my memory correct: 1. "Try to charge wallet X without funds" -> Error1 (as before) 2. "Try to charge wallet X without funds again" -> A status that is not an error (also not success)

        20 replies 7 retweets 326 likes
        Show this thread
      15. Gergely Orosz‏ @GergelyOrosz Mar 13

        Lots of questions on “why did Uber not handle HTTP error codes?” Because there were none. This API at the time retuned only 200s where the body had a message to be parsed which indicated success / status message / error. Status codes would have made this trivial to catch.

        22 replies 29 retweets 519 likes
        Show this thread
      16. Gergely Orosz‏ @GergelyOrosz Mar 13

        “Did you have tests?” Yes! As always the integration was unit tested with all possible API behaviours *at the time of building the integration*. “Could have you not failed closed vs failing open?” Of course we should have. It’s the morale of the story from consumer side.

        10 replies 11 retweets 295 likes
        Show this thread
      17. Gergely Orosz‏ @GergelyOrosz Mar 13

        Why would you *ever* fail open when there’s something unknown? Growth! You prefer to provide a great experience even if the provider has issues. Reconcile later. This was the case in 2015, when the integration code was written. By 2019, the mentality changed. The code: not yet.

        6 replies 10 retweets 262 likes
        Show this thread
      18. Gergely Orosz‏ @GergelyOrosz Mar 13

        Lots of replies on the payments API design. I don’t want to give Paytm a hard time: they were a lot better vs lots of other PSPs we worked with (my team owned ~15 PSP integrations). We integrated with *much* worse APIs & providers. Paytm - unlike many - kept & keeps improving.

        8 replies 5 retweets 259 likes
        Show this thread
      19. Gergely Orosz‏ @GergelyOrosz Mar 14

        Gergely Orosz Retweeted Willem Spruijt

        Ah, and Willem led writing the postmortem on our side (Uber). Here are takeaways we had (from memory): One thing I *really* appreciated at Uber was how every outage was treated as a learning opportunity. It was a blameless culture and boy, did we learn.https://twitter.com/wspruijt/status/1503316486798168068?s=20&t=PxgmLuwKcqTzHBlyOrxxAg …

        Gergely Orosz added,

        Willem Spruijt @wspruijt
        Replying to @GergelyOrosz
        Ha, I _think_ I handled this outage and wrote the post mortem 🤠 Key takeaways I remember: 1) Producer improvement: communicate (breaking) API changes clearly. 2) Consumer improvement: alert but fail open (for reasons mentioned in this thread) on unknown error codes.
        5 replies 3 retweets 113 likes
        Show this thread
      20. Gergely Orosz‏ @GergelyOrosz Mar 14

        Lots of people saying Uber should have just interpreted the unknown message as “unsuccessful”. Not quite. Here’s a story from a startup that did just that… double and triple charging their customers. Alerting on never-before-seen responses is key over just assuming yay or nay.pic.twitter.com/Ks4jNWUyOM

        10 replies 11 retweets 167 likes
        Show this thread
      21. Gergely Orosz‏ @GergelyOrosz Mar 23

        Some more details in a YouTube video:https://www.youtube.com/watch?v=PVzcWBmN2L0&t=1s …

        1 reply 1 retweet 12 likes
        Show this thread
      22. Gergely Orosz‏ @GergelyOrosz Mar 27

        An update on @Quora: a product manager reached out as it turns out I can answer the question without subscribing, it’s just completely non-obvious. So so answered there as well. But you’d never know. My answer’s invisible to anyone reading Quora… shame.https://www.quora.com/What-was-the-glitch-on-the-Uber-eats-app-that-enabled-users-to-get-free-food-while-paying-for-the-order-through-empty-Paytm-wallet …

        2 replies 0 retweets 11 likes
        Show this thread
      23. Gergely Orosz‏ @GergelyOrosz Mar 27

        Just to show how Quora became the new Experts Exchange: When you view this answer externally, you see 1 answer, and not mine which I took the time to type out. When I log in, I'm the only one to see my answer. It has 5 views: all 5 from me. No idea what @Quora's end game is.pic.twitter.com/4SF2l8suTO

        4 replies 0 retweets 42 likes
        Show this thread
      24. Gergely Orosz‏ @GergelyOrosz Mar 27

        Someone at @Quora looked into it and turns out Quora deliberately hid my answer citing it breaking their Q&A guidelines. "Your answer got collapsed likely because it contained commentary on the question." All I can say: don't bother with the effort answering anything on Quora.

        2 replies 2 retweets 75 likes
        Show this thread
      25. End of conversation

    Loading seems to be taking a while.

    Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.

      Promoted Tweet

      false

      • © 2022 Twitter
      • About
      • Help Center
      • Terms
      • Privacy policy
      • Cookies
      • Ads info