Telstra discovers '50 Shades Darker' DevOps sweet spot

When the DevOps team charged with optimising and maintaining Telstra’s streaming content play, Telstra TV, first went looking for a lull in traffic volume to exploit as a maintenance window they knew roughly around what time slot to look.

But they didn’t know what people might be watching.

With the carrier cranking out Game of Thrones to millions of viewers over April, one of Telstra’s recently promoted Agile coaches, Grant Simonds, knew that as the company was getting on board with continuous delivery for software and microservices, there were some moments you just don’t pick.

Having helped build the user experience and software development for Telstra Media, Simonds was familiar with traffic patterns.

“With a movie service all the peak traffic is on a Friday night, and a Saturday night at about 7pm,” Simonds told the Sydney DevOps Talks conference in Sydney on Wednesday, adding that “during the week… was actually the best time for us to do upgrades.”

“11am on a on the first three days a week, Monday, Tuesday, Wednesday,” continued adding that “not having to get up during the night” wasn’t so bad either.

So what do Australian’s enjoying late morning leisure time with a spot of bingeing prefer? Well, a bit of trashy fetishism if Telstra’s set top boxes are to be believed.

“Really the only thing that people are watching on weekdays at 11am is 50 Shades Darker anyway,” Simonds let slip, much to the amusement of the audience.

“So if we interrupt… Actually, it's probably important isn't it.”

One of the advantages of running peak loads out of normal business hours is that your dev team gets part of its life back.

Midnight maintenance windows and pizza are effective, but they also take their toll on families as well as running the risk of sleepy eyes on the job.

Having lucid people around helps too, Simonds noted of the lucky slot.

“It’s easier to debug, it’s more flexible and you can do a few more things in parallel,” he said

While Telstra Media has adopted microservices and can make changes much faster, it and still make sense to push changes at a reasonable rather than full throttle given the number of stakeholders that can be involved.

But the shift to microservices has also meant more to manage, even with automation, because a monolithic application is now around 44 smaller ones. Simonds reckons the trade-off is well worth it.

“This actually enabled us to do changes much faster in production. Two hours is about is about [our] record, from writing code to having it in production,” Simonds said.

“So compared to four months, that's not terrible.”

Vendor stew

Telstra is no stranger to agile, having bet heavily on the development rehab program to reduce its massive legacy backlog. 

In terms of Telstra Media’s architecture, Simonds describes it as “sort of classic AWS architecture.”

So the load balancers would be a ELBs (elastic load balancing) the API gateway is an Apigee gateway, and then microservices, which running on a Kubernetes stack with a little bit of Rancher rancher there as well.

Simonds described the architecture behind Telstra TV 2, launched September 2018, and Telstra TV 3, launched March 2019 as both essentially fitting into the same architecture.

“They both have connectors for their respective services. It's a pub/sub architecture using SQS [simple queue service].

“It gives us gives us resilience. So now it means if microservice fails and it dies, meh, whatever, hopefully Kubernetes will take care of getting us another one.”

The removal of the growth impediments is also paying off.

“It allows us to scale now, we don’t even think about scaling,” Simonds said.

“We used to have to dimension our servers and virtual machines up to at least four months before we knew the traffic was coming. Now we don't care.”

“They take about five minutes to spin up completely for some new VM or a few seconds if it's just another container spinning up.”

“It really allows us to learn, to get much faster, and put new features out really fast.”