A simple variable name change can often be the difference between a successful new code deployment and things going horribly, horribly wrong.
Last week, India’s biggest fashion shopping app, Myntra inadvertently ended up spamming its 10 million+ users with notifications about purchases they never made.
And its CTO has detailed what went down when the app did that.
On 19th May, at around 2:00 pm, the Myntra notifications team updated their fleet of notification servers with a code change. It took about 3 minutes for the deployment systems to update all servers. It should’ve been a routine change, but as soon as it was deployed, users across the world, including Myntra’s own employees, began seeing a deluge of notifications on their apps. Myntra immediately stopped its notification systems, but by then, lots of notifications had already been sent.
Upon troubleshooting, Myntra realized that the new code that they’d written was okay – but what caused the whole incident was the way code was deployed. Myntra’s notification systems require a set of “transformations” to notifications before they’re sent. This is the stage where things like the recipient’s name, list of recipients, etc are added to notifications. The new code had changed the schema of the database – the old code used to look a field called “recipient”, but the new field used a new variable called “userId”.
In the 2 mins and 37 seconds for which the new code ran before it was shut off, the field recipient had already been populated by the old code, but the new code was looking for the variable called userId. Since the userId variable was empty, these notifications essentially became untargeted notifications, and were hence broadcast to a very large number of users.
Myntra’s tech team says that this situation could’ve been avoided in a couple of ways – all incoming notifications should have been stopped and older notifications been flushed out before the new codebase was deployed. Also, having a simple upper limit on the number of notifications a user can receive in a given period of time could’ve also prevented the mass spamming.
Myntra’s CTO says the team has put systems in place to prevent this from happening again, and that the team has learned from this mistake. Myntra already has one of the best apps around in the Indian tech ecosystem, but as this incident shows, the tiniest of mistakes can have huge ramifications when you’re serving millions of users.