FBInstance to MetaInstance: Our Journey from Long-Running Mutable to Immutable Instances

In this talk, we will share our experiences and lessons learned from transitioning from long-running to immutable instances. We will discuss the challenges we faced when running Chef in the cloud and how these experiences can provide valuable insights for others.

We dive into the issues we encountered, such as managing build time dependencies: the "6 month upgrade problem" vs "15 minute upgrade problem". We will also discuss the difficulties we faced when trying to share cookbooks across different environments, and the constraints we faced due to being locked into specific versions of CentOS.

We will also share our experiences with DNF Repos, and the challenges we faced when trying to reuse well-tested RPMs from internal repos due to version and dependency mismatches. We will discuss the challenges in catching failures early in this model, and the limitations of testing options.

We then introduce how we turned these challenges around. We made our host truly immutable, meaning once it's bootstrapped, it stays unchanged during its runtime. We updated our images with a complete new build philosophy, accomplished through CI/CD pipelines, and kept Chef around for emergency patching.

Later we cover how we made internally written host daemons think they were running on CentOS hosts by using docker containers, allowing us to make our daemons portable to different distros with relative ease. This expanded our fleet adoption, which was previously limited to CentOS.