As the old adage goes, "when you have a hammer, everything looks like a nail." Adapting this to the past few years of my life, "when you have Go, use it for everything." So when I began my BYOC project a few years ago, I immediately reached for the Go language.

After a bit of prototyping with already familiar tools like Terraform and Ansible, I wanted to streamline my workflow by keeping as much of it as possible in code. This way, I could keep all of my business logic in reusable Go packages and use a single config file as the infrastructure's source of truth. I used this config file to auto-generate Terraform module files (using the hclwrite package), along with a bit of light Go templating to generate some Ansible files. But I found myself without writing various translation code between these layers, and I was starting to lose the benefit of static typing that comes with Go.

Hello, Pulumi

Of course, I've heard about Pulumi before, but hadn't given it too much thought since I'd become deeply familiar with production-grade Terraform and HCL tooling. I had originally assumed that Pulumi was simply a wrapper over Terraform, similar to CDK. Turns out that I was mistaken. Pulumi has its own native execution engine and provider paradigm. While you can bridge Terraform providers into Pulumi through a wrapper, this actually has nothing to do with Terraform directly as the providers are acting more as service APIs that are used by the Pulumi engine.

Since Pulumi has first-class support for Go, I figured that I would give it a shot in my new project to remove my Terraform dependency.

So far, I've been impressed with Pulumi's execution engine and the ease in which it integrates into traditional software applications. Pulumi is especially powerful when using its Automation API to programmatically control infrastructure provisioning. This API allows you to work with Pulumi stacks themselves as primitive objects inside whichever programming language you are using. This unlocks a world of possibilities. For example, you can create and destroy infrastructure in a statically-typed programming language as part of a CI/CD pipeline. Or, you can create a custom config that is specific to your own business case and use that to generate infrastructure that is managed by Pulumi.

The Bootstrapping Problem

One of the pain points that I've encountered using Terraform is how to bootstrap an S3 Bucket state backend. This traditionally requires multiple steps:

  1. Manually create an S3 bucket using the console or API
  2. Add that bucket to a provider block and run terraform init
  3. Optionally terraform import the bucket into your terraform state to manage everything in one place

You could also use a separate Terraform module to manage the state bucket, but now you have 2 separate modules instead of 1. This problem is compounded when working across different AWS accounts. And it isn't completely solved by using S3 bucket subpaths; there will always be multiple sources of truth.

Pulumi To The Rescue

Because the Pulumi auto package allows us to work with stacks (modules) in code, we can actually automate the entire bootstrap process into a single function.

  1. Create a new stack with state stored locally
  2. Update the stack to create an S3 bucket and record its URI as an output
  3. Create a second stack using that bucket's URI as its backend
  4. Export the first stack's state and import it into the second stack. Or now, just use a pulumi state move to handle the whole thing.

Now you have an S3 bucket that can manage itself, solving this classic chicken-and-egg infra problem for the last time.

Now, with AI!

I started writing this article over 2 years ago, and had a semi-working GitHub Gist with the code. But the code wasn't "perfect", so I forgot about the Gist (and this article soon after).

But now that writing code seems to be a thing of the past, I've just decided to release this article and get the idea out there.

This practice has come in handy over the past few months and has allowed my coding agents to bootstrap full e2e workflows including a database benchmark runner, multi-cloud tests for a Kubernetes Operator, and regression testing for the same BYOC project that I was writing when I first conceived of this article.

Stay tuned for more AI-centric writing! It took me a little while to get there, but I'm now a true believer.