Flipkart’s Big Billion Days is the company’s most anticipated sales event of the year. The company spends millions in ads promoting the event, and makes many months’ worth of revenue during the few days of the sale. But the 2015 edition hadn’t started off exactly as planned.
Exactly at midnight on 14th October 2015, just as the sale was about to go live, Flipkart’s site had crashed. Mobile phones were going to go on sale, and each minute of the site remaining inaccessible meant losses of lakhs for Flipkart. As traffic on the site dropped precipitously, the management looked on in despair.
Flipkart’s employees pull all-nighters during their biggest sales, and this meant that its product and engineering teams were in office as the site had crashed. Punit Soni, who at that point had recently joined the company as its Chief Product Officer, recounted the incident in a recent interview.
“It was the clearest illustration of the paradox of scale. You have hundreds of people at your disposal to attack the problem. But where are you going to point them? What should they actually go do?” Soni remembered. “My immediate instinct was to tell the senior leadership to back off so the engineering team could put their heads down and work — we’d follow their lead,” he said.
Seven minutes later, working under then-Flipkart CTO Peeyush Ranjan, Flipkart’s engineers discovered the problem — a DDoS hack from abroad had directed a giant traffic surge to overload Flipkart’s servers. Flipkart’s engineers believed that someone based outside India was intentionally bringing fake traffic on to Flipkart, which meant that real users couldn’t access the site.
Flipkart’s engineers couldn’t immediately stop the attack — so they improvised. “What we decided to do was actually pretty crazy,” Soni says. “We shut down all traffic coming into Flipkart from outside of India. We would, of course, lose some revenue, but it was worth it to cordon off the country where the majority of purchases would be coming from,” he says. Flipkart’s decision meant that for that moment, no user outside India could access the site and make purchases — but it also stopped the DDoS attack.
“It was an insane, weighty decision to make in the moment, but I’m glad we pulled the trigger, because if we went down and ran it by the senior executives first, we would have debated it for an hour,” Soni recalls. Soon enough, Flipkart’s site was again live, and customers, though irate at the delay, resumed their purchases. Flipkart’s engineering team’s quick thinking had stemmed the damage, and made sure its biggest sale hadn’t been a complete washout.