To accrue additional technical debt, turn to page 52! A choose-your-own-adventure system design case study.

This talk focuses on the implementation of large running environments that support the entire suite of products at a company - specifically for the use of sales engineers to demo any and all of the products. In the past the environments had been managed in a manual fashion and were open to changes from anyone - leading to difficulty maintaining parity and ensuring working environments. With the opportunity to roll out an entirely new kit, the responsible team at Grafana Labs set out to bring the best practices and tools to designing their new demo environments. 

Key technologies in the implementation revolve around running tools and products on kubernetes clusters, and automation of the deployment of these tools, as well as the clusters and  underlying infrastructure itself. Decision points will be discussed around CI/CD, Security, Cloud Providers, etc. Our final implementation involves OSS such as Terraform, Flux, Bitnami Sealed Secrets, Atlantis, Helm, Kubernetes, Prometheus, and OpenTelemetry, however along the way other tools were evaluated such as Vault, kubernetes secrets, TF Controller, Argo, 

As we work through the presentation, we aim to be interactive with the audience. The format will discuss design decisions each step of the way, and especially focus on the choices that were made between different OSS options. There will not be a set right answer, and we want to work through the choices made by the audience to reach a final example state of the system. To work in a true “choose your adventure” format, different options will lead to different discussions in the presentation, and various “perils” and “successes” will arise as particular options are chosen.

We hope that this presentation will help attendees learn skills for critical thinking around design choices of large systems with many components, and how to better assess OSS options and the caveats that they might hold.