Increase Configuration Re-use with Service Oriented Design & TDD

Tools such as Puppet and Chef provide a foundation for the re-use and sharing of application and service configuration components.  The goal for an IT organization should be to take a component "off the shelf" and use it as a building block to compose an overall  service. Providing for re-use facilitates the trend of getting everyone involved, including application developers, in using automation to promote and support a Self-Service IT environment.  Although the foundation for re-use gives us a starting point, there are several barriers to realize a vision of truly composable infrastructure. In this talk we will discuss some of these barriers and propose some design patterns and approaches that can facilitate the continual improvement of re-use and composition.

Two main areas covered in this presentation will be:

1. Applying service & object oriented design principals to configuration code
2. Using TDD practices w/ a focus on functional and integration testing of configuration components

Applying Service & Object Oriented Design Principals

Much like how application developers create classes and libraries that are shared, infrastructure developers aim to achieve the same by composing configuration components together like lego blocks.  In practice it is rare to be able to use 3rd party configuration components un-altered, usually requiring the infrastructure developer to understand what they are doing and fix/enhance where needed. This could be for several reasons, such as hard coding configuration values or assumptions and external dependencies directly in the components.

Using a service and object oriented design analogy for configuration components can be an advantage that can provide a more formalized separation between the interface and implementation of native configuration management representations. First, with an encapsulated interface, the developer has well defined place to read about and plug into third-party common components. Second, recognizing that one is working in an environment where components can be upgraded on their own release cycle, interoperability critically depends on whether the interface changes or not and if it does whether it follows backward compatibility practices.  An example of a compatibility change would be when a new parameter is added making sure it has a default value.

When looking at Chef and Puppet as examples, there is an intuitive notion as to what serves as the interfaces, such as Puppet Class, Definition and Resource signatures. Also, there is growing best practices in how the documentation of modules in for example git readme files are given that indiacte how to use the capabilities of the module. In lacking a more "formal" interface some issues could surface:

• The "global variable" side-effect with capabilities such as Chef search and Puppet exported resources which represent external dependencies
• The "public"  vs. "private" resource issue where components deemed private are not to be referenced outside of the Puppet Module or Chef Cookbook 
• Type checking and validation being done on an adhoc basis and within the "implementation" as opposed to being (maybe optional) part of the "object signature"

A related problem discussed is issues caused w/ exchanging components of a similar type, such as an application connected to a "database" which could plug in/out a Postgres or Mysql component.  We explore loose coupling of cross serviece attributes references where developers do not have standardize on common names (ie: port vs database_port vs db_port).

Importance of Functional and Integration Testing

As operations engineers have adopted advanced practices around automation more and more their day to day jobs are mirroring that of application developers.  While creating, managing and sharing libraries of automation scripts and tooling within a project and across an organization a growing issue becomes the change management and release process between environments, much like a developer who is delivering parts of their applications and platform updates.  Questions need to be answered before releasing infrastructure changes:

• If I change this "base" configuration component, will it break any components/services that depend on it?
• What are possible side effects of the infrastructure changes on the application(s) that are running on it?
• Are these changes written in an idempotent manner such that repeated execution will not make unwanted changes or cause harm to the environment?
• Are the changes violating constraints or version dependencies when multiple applications are depending on the infrastructure?

Testing practices such as lint checking automation code, pre-checking attribute/config values and dry runs are fine to utilize at the front of your testing pipeline but are just the tip of the iceberg.  We explore some tools and patterns for tests and checks to perform inline & post configure functional and integration testing.