Scaling Environments in Kubernetes with OpenTelemetry and Service Mesh

Modern applications are composed of increasing numbers of microservices which talk to each other using APIs. They are also increasingly composed of heterogeneous components such as managed databases, message queues, data warehouses, serverless infrastructure, third party APIs, and so on. This has made it all the more important to provide developers with rich feedback and signals from environments that resemble production.

An emerging way to scale these test environments is using OpenTelemetry and Service Meshes (like Istio & Linkerd) in conjunction. Using request isolation, it becomes possible to construct very lightweight but high fidelity test environments that turn up in seconds. This approach has been built and is being used at scale in companies like Uber, Lyft and Doordash across hundreds of developers, and in this talk, the audience will learn how they can design and implement such a system in their own organization.

The target audience is platform engineers, developers and distributed systems engineers that are dealing with complex applications featuring tens or even hundreds of microservices. This talk will explain the fundamental concepts behind such a system and how such a system can be built by combining together open-source components and emerging open standards like the baggage spec (https://www.w3.org/TR/baggage/).

This talk will cover:

• Why build microservice environments using OpenTelemetry and Service Meshes (Istio, Linkerd, etc)
• Concept: Baseline environment
• Concept: Request Tenancy
• Concepts: Data isolation for databases & message queues
• Practical details & live demo of environments isolated from one another using request tenancy in Kubernetes using Istio
• Case studies of similar systems and considerations when building such a system in practice.