Flipkart’s big billion days sale in 2014 was plagued with problems, so much so that the founders had to issue a public apology. This sale in 2015 was also not a smooth affair. Consumers were complaining of the same old problems of app not loading, products not showing etc and even the competitors were taking a dig at the dismal performance.
Hey folks,@Snapdeal apps, msite & website are working as smoothly as ever.Come shop w/ us while others figure out how to make their app work
— Kunal Bahl (@1kunalbahl) October 14, 2015
When billion dollar companies fail to ensure optimum performance on big sales days, you know that the problem is complicated.
As per a report in the Economic Times, Flipkart was preparing to ship more than 100,000 packages daily (it ships ~ 65000 on a normal day).
Such yearly big-sale days (from festive season in India to Black Friday deals in the US) are a tough test for any e-commerce venture. The traffic & orders go up exponentially and test the robustness of everything from the website infrastructure, inventory position and fulfilment processes. So how do ecommerce companies prepare for the big sale?
Various technical infrastructure experts shared their views on the topic.
Brandon Elliot (Chief Technologist, Digital at Rackspace) believes that you can either add more resources to the stack or ask the existing resources to do less by focusing only on the core business tasks. According to him, dry runs and a skilled devops team can take away much of the pain.
The team at EPAM Systems (provider of software product development services – link) gave in-depth answers on how to prepare your website for such big sale days.
According to them testing is the key: “Testing, testing and more testing. Cyber Monday will become Black Monday for your company, so high traffic volumes means exceptional site performance. Testing is key to superior performance and it is imperative the testing is conducted well in advance of any major sale or event.
We recommend testing not only the eCommerce site but also all downstream systems and how they interact and respond to the site requests (i.e. warehouse systems and accuracy of inventory levels); test the speed and responsiveness for any given location, monitor the content load times and ensure optimization, ensure the site performs well across all platforms and devices. Test how long switchover takes from a user perspective (like browser DNS caching). Maintain failover capability on DNS change / routing level as well as load balancing / forwarding level. Lock-down the site, trying to rush to roll-out new functionality and/or any code changes can run serious risks, a code freeze should be in place at least 3-4 days advance of any major sale or event. This includes a freeze on deployments to any of the environments that could impact the site.
On sale day, testing and monitoring is even more imperative. Make sure to test all promotions with dummy orders as soon as the sale goes live; increase monitoring performance of all systems, not just eCommerce, but downstream systems such as fulfillments, customer service, etc. Success is in the hands of the team, increase staff and make sure they are prepared to jump into action. The marketing team should monitor traffic and sales performance and identify any issues as well as watch the competition. The technology team should be on stand-by through the full duration and always have developers ready to respond to any last minute changes. Your competitor just put out a better promotion? Make sure your marketing teams and technology teams are ready to respond and can quickly make changes to the site if needed.“
The Argyle Technology Group (link) has consulted with startups that faced such sudden high traffic days after being featured on the extremely popular TV show ‘Shark Tank’. Each episode has a viewership of ~7 million, and featured businesses often go from a normal traffic of 100,000 to 3 million people in just a minute or two. They manage the scalability by using Amazon Web Services (AWS). Michael Canzoneri (Partner) says, “1) Everything in AWS has a limit. By default you are only allowed to have 20 servers in your account. You need to submit a support request to get your limit increased. This is a really bad thing to find out when you are trying to go from 15 servers to 30. This request needs to be put in well before any event that will cause you to increase the number of servers you use.
2) If you use Amazon’s Elastic Load Balancing (ELB) service then it’s important to know that, while it’s an excellent service, it doesn’t handle sudden dramatic bursts in traffic well. The system is built to handle gradual increases in traffic. To alleviate this you need to put in a support ticket and ask them to prewarm your ELBs. If you provide them an estimate of traffic and a date range they will set things up so that you will be able to handle the sudden impact of users.
3) Using a Content Delivery Network (CDN) is extremely important for this sort of event. A CDN will handle requests for content that doesn’t change much. This includes images, movies, documents, and static html pages. Most e-commerce sites really can get by on being entirely static content until the point that someone makes a purchase. Getting your site using Amazon’s CloudFront or another CDN like Akamai really sets those services up to take the brunt of the traffic which is what they are built to do. You may need to make adjustments to your site in what gets passed to the back end servers and reduce that to almost nothing until the point that the user becomes valuable by choosing to make a purchase.
4) Lastly, what’s the right capacity? Well if you’ve done all of what I’ve mentioned so far then the real capacity you need to worry about is the percentage of people who are going actually make a purchase. This can be a simple multiplier. If you typically have a 3% purchase rate then you’ll be able to figure out what that is a 100,000 users or a million and then adjust your server capacity and AWS support requests accordingly.“
Casey Halloran (Co-Founder & CEO, VacationsCostaRica.com) faces this issue often in the holiday season. “Since most of our prospects are from the Northeast of the U.S., it makes sense that they want to plan a vacation to the tropics while enduring the bitter cold of winter holidays. So how do we prep?
– We review AWS server logs, get an average of previous peak traffic
– We forecast 3x load of highest useage previous date
– If server has capacity for that estimate, we’re OK
– If not, we increase storage capacity
– Last, we optimize the website for load time. We love Pinggdom tools and Google Webmaster Tools site speed for this.”
Anthony Watson from ShipBob, a startup that helps e-commerce companies with their shipping needs, lent a different perspective to preparing for such high sales days. “The biggest hurdle to overcome once the Holiday season really hits is underestimating how busy all facets of your business will truly get. Even though orders might be up 30% and you accommodate with extra staff and supplies, 30% order increase means 30% more unexpected customer service, 30% more back-end claim support, 30% more time spent managing personnel, etc. When those extra duties start to pile up, the accuracy and speed of shipping out the holiday orders falls dramatically and you can lose potential return customers with a bad shipping experience. All the little problems that come up on a day-to-day basis increase exponentially over the holidays and if you aren’t able to delegate appropriately to your staff with 100% trust you’ll be putting out small fires for the entire month of December and dishing out refunds / hand-holding angry customers after the holidays are over.
Whether you choose to take advantage of cloud based services like AWS or keep everything in-house, big traffic days need a lot of preparation if things are to go smoothly.