Building for Scale With Kubernetes and Pulumi

Scale is tough, and using Infrastructure as Code (IaC) practices along with tools like Pulumi and Kubernetes has been crucial to our growth. We’ve had a big couple years here at Hydrolix, and it doesn’t look like it’s going to let up, so we are constantly looking for ways to make our infrastructure more efficient and easier to use.

Our customers generally prefer an easy, built-in solution, which means we take the primary responsibility of provisioning and maintaining the stateless Kubernetes clusters that power Hydrolix. At the same time, we also want to make it easier and faster for our teams—from engineering to customer success—to deploy and manage our architecture, too.

In this post, we’ll talk about some of the approaches we’ve used to automating and building our platform, and what we’ve learned along the way. This post is based on 3 Tips to Power Explosive Growth with Automated Deployments, a talk we gave at KubeCrash.

Why We Built On Kubernetes

In the very early days of Hydrolix, we used CloudFormation to orchestrate resources in AWS. However, we soon realized that we needed a multi-cloud solution. We chose Kubernetes for a few reasons. We were already using containerization with CloudFormation, so Kubernetes was both a natural fit and a quick win because we went from having to directly implement certain functionality in CloudFormation to getting that same functionality out of the box with Kubernetes. We also liked the best practices and community around how best to integrate Kubernetes. While it hasn’t always been our experience that all of the best practices are immediately implementable, having that community knowledge has been extremely useful—and it’s a big part of the reason we want to share what we’ve learned with the community and give back.

The next image shows the general architecture of Hydrolix. The major subsystems, including ingest, merge, and query, are all built on stateless Kubernetes.

Our infrastructure uses the Kubernetes operator pattern, which gives us flexibility to adapt to environments we deploy in and to any customer requirements driving deployments. There are dozens of services involved in building out our custom Hydrolix cluster, and so you can imagine that the process for getting that built out was (and is) pretty involved. Kubernetes doesn’t really build all the cloud resources we needed for these clusters, and we really needed something to fill the gap. We wanted a way to automate as much of it as possible. We’ll be writing more about the Kubernetes operator pattern and the Hydrolix Kubernetes operator in future posts.

Why We Chose Pulumi and an Infrastructure as Code (IaC) Approach

Our goal was to use a GitOps/Infrastructure as Code (IaC) approach, where we use code (via configs) to describe what our deployed system is going to look like. In an ideal world, our site reliability engineers (SREs) get the request and general specs needed, put together the matching config, run some checks to see that what’s getting built matches the request, and then check the config into the git repository. The automation then applies the config in a predictable way. With IaC, you’re not just building new things, but thinking about how infrastructure components can go away, or need to be updated, or need a configuration change. You really have to consider the whole life cycle of your running infrastructure.

The two obvious options for us were Terraform and Pulumi, because most other options were limited to a single cloud. We chose Pulumi because, at the time, it had better practical implementations for managing both Kubernetes and other cloud provider resources in a single tool. The truly winning feature of Pulumi is that we could use a general purpose programming language to do what Terraform does with a domain-specific language. (That has changed somewhat in Terraform now, but we’ve invested in Pulumi and are happy with it.) We also wanted to give customers the option to use our tooling and provisioning processes to deploy and manage their own Hydrolix instances. Terraform would’ve led to larger expenses, more fiddling to get it to work with Kubernetes, and more investment from users in terms of additional tooling rather than something we could simply package for them and hand over.

Implementing IaC made deployment easier and allowed us to even out the differences between our Kubernetes managed services. For instance, with Pulumi we can decide we want to deploy the metrics server and then easily bring it up to the level of other services. Without that automation, it would be more challenging to consistently automate deployment. With Pulumi, we can say: “I want a [new] Hydrolix cluster” and “I want my Hydrolix cluster that’s already running to be version x.y.z.” with a single command.

It’s worth noting that configuration in Pulumi versus what should be configured in Kubernetes operators can be kind of an arbitrary decision. We have found that Pulumi is not always the best fit for directly creating Kubernetes resources, so we use it to wrap the Kubernetes operator. We then have the operator deploy the described managed Kubernetes resources and service, and then Pulumi deploys the Hydrolix custom resource, and it’s a little less awkward overall.

Ultimately, there’s a whole world of automation that wraps around the lifecycles of infrastructure and software. We optimized for tooling that enabled us to be meaningfully cross-cloud, while automating as much of our workflows as possible.

What We’ve Learned From Implementing IaC

First off, scaling is hard. If we’ve learned anything, it’s that incremental improvements, whether to the underlying tech, our processes, or even our teams, is the key to actually making progress and being able to respond to whatever surprises come up. As we discussed at KubeCrash, we have a few pieces of advice we’d like to share:

1. Find the IaC tools that best work for you. For us, that meant choosing Pulumi over Terraform.
2. Build a team dedicated to your automation effort, and empower them to deliver value quickly by giving them focused milestones. IaC takes time and commitment—it’s not a one-and-done project.
3. Don’t forget about change management, and build it into your plans from the beginning.

We quickly realized that this wasn’t the kind of work that would be a one-and-done project. We began breaking it down into an MVP and then incremental milestones to get there, and we continue to do that as we discover new work that needs to be done. This has allowed us to deliver value quickly, which we can then assess and pivot on as needed.

Building on that point, we embraced this work as expensive but ultimately worth it—meaning we dedicated a team to it from the beginning. Thanks to that, we’ve seen massive returns on manual time saved, operational costs lowered due to less error and toil, and our team isn’t burnt out. And as an extra bonus, we’ve been able to really investigate and plan for the interconnectedness of the entire system, rather than doing piecemeal work that has to be stitched together later.

A dedicated team also enabled us to consider and reject an ultimately untenable alternative: needing to hire for every customer and every cluster we spin up. While we love giving folks work, we also pride ourselves on that work being meaningful, which means our SREs and customer success engineers (CSEs) work on a broad range of projects and build lots of skills, rather than narrowly focusing only on a customer or a cluster (or small group of clusters). We like being a company that can help people grow even as we’re scaling up. 

What’s Next: ArgoCD and Full Automation

There’s an entire universe of automation around deployment. We’re just at the beginning of a project to use ArgoCD to help manage Hydrolix cloud resources in a GitOps fashion. Our first step will be taking some of the configuration out of Pulumi, where we’re overextending Pulumi right now, and moving it into another directory to be consumed by ArgoCD. In particular, we’re looking to ArgoCD to act as an interface between our Pulumi wrapper and the Hydrolix Kubernetes operator.

We’ll then be looking at ways to wire everything together in the most optimal way, so that each config that produces a dependent piece of information is consumed by another part of the process in the correct order. Ideally, we’ll create fully-automated fleet-wide rollouts! But we’re just in the design phase right now.

Next Steps

Interested in working at Hydrolix? Check out our careers page.