Charmed HPC on OpenStack Sunbeam

The worlds of Cloud Computing and High-Performance Computing (HPC) are rapidly converging with the rise of AI and complex scientific workloads; however, many HPC users and system administrators are experiencing the limitations of classical HPC systems. Traditional HPC workload schedulers, like Slurm and OpenPBS, are excellent choices for running short-term batch jobs on multi-tenant compute nodes, but they are not good for running indefinite services such as data warehouses and inference servers. Modern HPC systems need to become more cloud-like where they are able to support running both short-term workloads and indefinite services, be elastic with their resource usage, and autonomize common lifecycle operations.

OpenStack and Kubernetes are great platforms for building these modern HPC systems, but there's one problem: the limitations of your system operations team. 

The user experience for deploying and managing an HPC system is often overlooked. It is already hard enough to build and deploy a classical HPC system, and now you are adding the additional complexity of deploying two new layers. One layer for OpenStack and another layer on top of OpenStack that contains both the core services of the HPC cluster and the Kubernetes cluster for hosting indefinite services. This architectural complexity can quickly bog down any experienced team, but it doesn’t have to be this way. 

In this talk, you will learn how to use Sunbeam – a cloud-native OpenStack powered by Kubernetes – and Charmed HPC to streamline the deployment and management of modern HPC clusters. You will learn how to use Sunbeam to set up OpenStack, and then you will learn how to deploy your HPC system on top of OpenStack using Charmed HPC. Lastly, you will learn how to set up Kubernetes next to your HPC system to provide cluster services such as identity management and observability metrics. Setting up a modern HPC system does not need to be a dark and stormy process - it can be as clear as day with Sunbeam!