I had the chance to talk to Ticketmaster‘s Victor Gajendran who will be attending (and speaking) for the first time at SCaLE 14x this year, which is taking place on January 21 and 22 in Pasadena, California. He’ll speak to attendees about how his company uses open source and how to empower your small teams to be part of a large, effective whole.

Find out more about his talk, Lessons learned (the hard way) by doing DevOps at scale, in this interview.

I know people would love to hear about the free or open source technologies you’re using. What can you tell us?

The fact that Ticketmaster has been around for 40 years is reflected in its tech stack. We use a variety of technologies spanning from VAX emulators to modern Docker containers. It is fair to say that we are a open source based company. Our primary cloud infrastructure is mostly Xen with a bit of OpenStack. Our primary operating system is Linux. Our software delivery platform is full of open source technologies like Git, Jenkins, Rundeck, Docker, and so on.

Is the decision to use free and open source tools relatively new, or would you say Ticketmaster was an early adopter?

Ticketmaster is definitely an early adopter of open source technologies and has been for a very long time. We truly believe that the vast open source community can solve technical problems way faster than any individual organization can. I’d even venture to say that this open source based philosophy has given us the agility required to be the leader in the ticketing industry.

When you made the decision to start on a DevOps journey, did you bring in new people with experience in a DevOps environment or focus on supporting training for your existing people?

Both. Our tech organization is over 1,000 employees. While we continue to bring in new and current talent, we are also investing heavily in our existing engineers because they are the ones that can ensure that we don’t repeat the same mistakes Ticketmaster has made in the past.

What’s the single most important piece of advice you’d give to another company embarking on a DevOps journey tomorrow?

Create a culture of learning where everyone feels safe to make honest, new mistakes while they push the boundaries of technology.

How did you train yourselves to break large tasks into smaller tasks? It sounds challenging!

We practice the principle of promises. For example, one of Ticketmaster’s goals is to empower the fans so that they are entertained the way they want to be. The tech organization would translate that to a promise to build applications that empower the fans. Now, the TechOps organization would break it down as, “Make applications accessible and highly available so that the fans are empowered through our applications.” Then, individual teams would further break down or translate this business promise into a sub-promise that is in alignment with the “super” promise, yet is deliverable within the context of that team. This decentralized approach not only helped us empower our teams, but also align the entire organization.

Who was in charge of rearranging the task into smaller chunks? Was there any resistance or pushback when things got moved around?

The leaders at every level were in charge. The senior leadership team and the leaders at various levels translated the high-level business promise into a promise that’s relevant for their group, keeping in mind the constraints of reality. By ensuring that everyone felt connected to the bigger purpose of the business/company, our staff was very motivated. Don’t get me wrong—there was pushback. However, it was manageable because of the approach we took.

What do you think is the most unique challenge in Ticketmaster’s systems?

The scale. There’s no guarantee that even the very well-solved problems will work for our scale. We want to be extremely agile in solving business challenges with open source solutions. However, we must somehow make sure that it will work when it is fully scaled out. This challenge has been keeping us hungry and on our toes.