Lessons learned (the hard way) by doing DevOps at scale

Ticketmaster is a 40-year old giant yet a savvy business leader in the ticketing industry. While innovative business strategies and initial technology breakthroughs brought the company to where it is now, there has not been a significant shift in how it’s technical operations were run until last year. Leadership change at the C-level resulted in strategic investments towards modernizing our ways of delivering software. Obviously, the much discussed DevOps model was launched. While we have come a long way in the DevOps journey, our retrospectives are filled with scars and missteps. This talk covers all the key lessons learned during this journey with real-life examples. More than a talk, I’d call this a knowledge sharing session that aims to help other companies - big and small - to avoid the mistakes that we did while trying to implement DevOps model. 

For example, we all know that Dev and Ops team members should work together in creating and operating the products. So, we picked this “obvious” principle and insisted that our Dev and Ops teams adopt it. Well, they did exactly that. In fact, they went a step further and created a cross-functional team that had skills represented from networking, cloud infrastructure, systems engineering, systems reliability engineering, development, testing, and business analysis. And oh! for redundancy, they added two of each, created a team, and called it a “pod”. This pod ended up having 22 resources, their stand ups would run for 45 mins, their story card walls were a mile high, left hand didn’t know what the right hand was doing, and so on. Because of the team size, we couldn’t even have frequent celebrations of our mini-victories. After losing a bunch of time and resources, we learned that we should adopt Jeff Bezos’ 2-pizza rule even for our team size and not just for meetings. 

Another example is an incident that taught us that simple problems like log management become extremely complicated when done in scale. So, we trained ourselves to break giant problems into smaller manageable chunks and used multiple mini-pods to focus sharply yet be aware of the big game plan. 

We are even learning about the process of learning itself! We recently proved that learning is an on-going process as we seem to learning consistently over time - even more so during recent times (Yes, we track the volume of lessons learned!). 

Of course, I wouldn’t want to give it all but it's worthless if not shared. So, see you there! :)