How free food brought down Guzman Y Gomez

When it launched an online ordering app late in July, the IT team at Mexican fast food chain Guzman Y Gomez never fully comprehended just how compelling a free burrito could actually be.

The battle scars of the restaurant's CTO Julian Dabbs were still fresh when he fronted the Future IT Infrastructure forum in Melbourne this week.

He said he didn’t realise quite how far people - mainly university students - would go to get their hands on free food when the marketing team came up with the idea of offering the first burrito ordered through the app at no charge.

“Some people figured out how to give themselves multiple coupons by changing certain details in the account - so we patched that up,” he said.

“But then we had students going out and buying new $2 SIM cards because we had phone level verification, and a $2 SIM card is cheaper than a $10 burrito.”

But the real challenge faced by Dabbs and his team was the traffic that swamped the new app overnight, which peaked at some 20 times the volume GYG had provisioned for.

“Previously about one percent of our customers made purchases online, and after our app launch this jumped to 10 percent pretty much overnight, peaking at 20 percent,” Dabbs said.

“Our infrastructure was really only built for one percent, allowing for a little bit of growth on top of that.

“We assumed this would be a pretty successful campaign, but in retrospect we went went into it blind."

The Guzman Y Gomez app became so popular it topped the iOS trending charts  - “even above Pokemon Go”, according to Dabbs - but behind the scenes everything was not okay.

The Oracle-hosted online ordering platform crashed in the middle of the lunch rush just days into the campaign period, leaving the GYG team and their Oracle counterparts furiously working to migrate the whole platform from the single multi-tenanted server the IT giant had provided to three dedicated servers.

They managed the task in an hour, but by then lunch was over and the reputational damage was already done.

Just days later, Guzman’s SSL certificates expired, once again sending the system to the ground.

“Again we brought it back up in the same morning, but in the middle of this sort of campaign that equals lost customers, and lost investment,” Dabbs said.

The experience has left the CTO with some important lessons and ten times the capacity in the Oracle data centre that he started with.

“Anything that can go wrong, will go wrong,” he warned.

“If I could start over again I would make sure we had simplified our core infrastructure before we even started with our UI, UX, feature maps and marketing.”

Dabbs also cautioned against assuming anyone is too big to fail.

“What we wanted to think was that Oracle is such a giant that we don’t have to worry about them, we’ll just assume their systems will stay up,” he said.

“But to be fair on them we had grossly underestimated the volume forecasts."