Ovation Ventures Ovation Developer Docs

Proposals

2026-01-29 Kubernetes Proposal

By Ben C. Forsberg

The Problem

While AWS Amplify is the fastest way to get a single repo project on-line, it has several drawbacks:

As our project grows beyond Balance Web to include reporting pipelines and microservices such as quiz APIs, certificate generators, and third-party integrations, we have outgrown the capabilities of Amplify. Indeed, Amplify and SOC2 compliance may be incompatible.

AWS offers “serverless” solutions such as Lambdas and Fargate tasks. Lambdas are difficult to develop, because they require a specific operating environment which can only partially be emulated on a developer’s workstation. Fargate is rather complicated to set up and configure, involving a bewildering depth of abstractions and connected resources.

Our current Terraform configuration treats the entire AWS environment as one, atomic whole. Deployments become difficult because any change to infrastructure involves re-analyzing the entire environment, possibly breaking or reverting changes to unrelated components.

Proposed Solution: Terraform + EKS + Helm + Helmsman

Terraform would be used to configure AWS directly. Resources configured directly via Terraform would include:

Individual components would be built and shipped as Docker containers, and their deployment described in Helm Charts.

Each service or microservice would have its own Helm Chart, which would live in the same Git repo as the code. GitHub Actions can build and publish the Docker (or Containerd) image as well as build, package, and publish the Helm Chart. The main infrastructure repository would contain the Terraform configuration—which would only contain the organization-wide configuration, not individual services—as well as the Helmsman configuration—which would list the charts, their versions, and their environment override configuration variables. Deploying a configuration change to a service would involve bumping its version number.

Explanation

Kubernetes

Kubernetes (K8s) is a “container orchestrator”. It is like “Docker Compose”, but able to schedule containers to run on any number of distributed “nodes” (individual computers or virtual machines) and handles the networking between them. Kubernetes offers many of the same features as EC2, but with a simpler model.

Helm Charts

A Helm Chart is a collection of resource files, much like Terraform. Instead of Hashicorp Configuration Language, Helm Charts are Go template files that render to YAML files. Much like a Terraform module, a Helm chart exposes configuration variables that may have defaults. Unlike Terraform modules, Helm charts are versioned and pushed to a repository that manages multiple versions for each chart. So a deployment gone wrong can always be reverted to the previous version. Each Helm Chart will contain a description of all the resources it requires as a single, atomically-deployable unit.

Helmsman

Helmsman is analogous to a package.json file. It maps Helm Charts and their overriding value files and versions.

Advantages:

Cloud Vendor Independence

All the major cloud vendors offer managed Kubernetes services, and Kubernetes can be run without a managed service on virtual machines or even rack hardware. So the more infrastructure that exists within Kubernetes, the more your cloud infrastructure becomes cloud-agnostic.

Development Isomorphism

A single-node Kubernetes cluster can easily be started on a development workstation with Minikube or Colima, allowing a developer to reproduce the entire cloud infrastructure (or the parts of it configured within Kubernetes) on their own development workstation without deploying anything to the cloud.

Simplified Model

Kubernetes resource model is not simple, but it is much simpler than Amazon’s. For example, K8s’ permissions system is similar to Unix’ user/group/other system with a small handful of verbs, compared to AWS bewildering forest of a permission model.

Network routing also involves fewer components.

An Ingress handles HTTP rules to route traffic to a named Service, which groups a Deployment or a StatefulSet, which is a scaleable collection of Pods, each of which is running one or more Containers. AWS would replace just the Ingress and Service part of the chain with a load balancer, target, target group, vpc, security group, and policy.

I believe it is a much more tenable idea to have developers develop and maintain their own Helm Charts to live along with their microservice code than it is to do the same with Terraform modules.

Here is an example of a YAML file containing several Kubernetes resources. This is sufficient to describe the hosting of my little maze-drawing app, which has a working example of a simple Helm Chart.

# an "Ingress" contains rules for routing HTTP requests.
# In my cluster, a blanket, wildcard cert is fetched from
# Let's Encrypt and used site-wide. In AWS, it would be
# configured on the single Application Load Balancer that
# would proxy the Kubernetes cluster.

# See https://github.com/unquabain/mazer/blob/main/chart/mazer/templates/ingres.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  # Annotations are used as arguments to the underlying plug-in.
  # In this case, the Nginx load balancer.
  annotations:
    # This annotation tells Nginx that there is a service
    # located in this cluster called "letsencrypt-prod" that
    # issues Certificates.
    cert-manager.io/cluster-issuer: letsencrypt-prod
  name: mazer
spec:
  # the ingressClassName links this to an IngressController,
  # which is an integration point between the virtual world
  # of the K8s cluster and the real world.
  #
  # Whenever K8s needs to connect to its outside environment,
  # it does so via a "plugin". Resources such as "StorageClass"
  # and "IngressController" configure these plugins.
  #
  # On my cluster, I am running an NGinx load balancer, whose
  # plugin comes with the default K8s distribution. On AWS, this
  # would be the ALB class, which is built-in to EKS.
  ingressClassName: nginx
  rules:
  # An example host rule. You can have many per Ingress
  # or many Ingresses.
  #
  # Note that K8s resources let you nest things in the YAML,
  # while AWS resources typically use one flatly-defined
  # resource per abstraction, which is more difficult to
  # remember and use.
  - host: mazer.4zb.org
    http:
      paths:
      - backend:
          # The name and port of the service, defined below.
          service:
            name: mazer
            port:
              number: 80
        path: /
        pathType: Prefix
  # TLS/SSL is also configured here.
  tls:
  - hosts:
    - mazer.4zb.org
    # K8s has its own secrets management, similar to AWS SecretsManager
    # This secret contains the private key corresponding to the
    # certificate obtained from Let's Encrypt.
    secretName: mazer
---
# A Deployment describes a "Pod" along with auto-scaling rules.
# Even if you are only running a single container, putting it
# in a Deployment allows K8s to manage self-healing and
# zero-downtime deployments.
#
# Deployments are for stateless services. The alternative
# is the "StatefulSet", which adds a "PersistentVolumeClaim"
# template that makes sure that attached storage is provisioned
# and attached correctly. If the "Pods" are restarted because
# of a problem or an upgrade, the "StatefulSet" makes sure that
# the existing volumes are recycled and re-attached appropriately.
#
# A "Pod" is pretty silimar to what Docker Compose creates:
# it's a collection of containers all running on the same
# "Node" (machine) and sharing physical resources as well
# as a virtual network connection.
#
# This deployment makes sure that at LEAST 1 replica is always
# running. When I publish an update, it will spin up the new
# one and verify its health before shutting down the old one.
#
# See https://github.com/unquabain/mazer/blob/main/chart/mazer/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  # This is used to match the service with the deployment
  name: mazer
spec:
  replicas: 1
  selector:
    matchLabels:
      # This is used the match the pod with the deployment
      app: mazer
  template:
    metadata:
      labels:
        # This is used to match the pod with the deployment
        app: mazer
    spec:
      containers:
      - name: mazer
        image: unquabain/mazer:latest
        ports:
        - containerPort: 8080
---
# A Service is the abstraction layer between a "Deployment"
# or "StatefulSet" and an "Ingress". The connections are
# made flexible through "selectors" and "labels". For example,
# if you have a database with a master and a read replica,
# you can create a write service that only targets the master
# and a separate read service that balances the load between
# both.
#
# See https://github.com/unquabain/mazer/blob/main/chart/mazer/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  # This is used to match the Ingress with the service
  name: mazer
spec:
  # There are different types of services, and the documentation is
  # a little confusing.
  # 
  # ClusterIP means that the service is only
  # accessible via K8s internal routing, and so can be deployed to any
  # node.
  # 
  # Traffic will be routed here from the IngressController, which
  # is a service of type LoadBalancer.
  #
  # A third type is NodePort, which means that it responds to the real,
  # externally visible IP address of a particular node and is bound to
  # one of its externally-visible ports.
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
  selector:
    # This links the Service with the Deployment
    app: mazer

This example doesn’t include resources that are used cluster-wide, such as the IngressController and various services related to certificate management, like letsencrypt-prod, which is mentioned. It also doesn’t include the Secret resource, for obvious reasons, which is nonetheless pretty simple.

The above example could be all in one, multi-document YAML file (as it is presented here) or in multiple files. To turn it into a Helm Chart, you can replace specific values with Go-style template variables, {{ .Like.This }}. The Helm tool includes many useful template functions for manipulating text and structured data.

Versioned Deployments

Helm Charts are published to a registry. (ECR can contain both OCR containers and Helm Charts.) Within that registry, the each chart is published with a SemVer version. When Helm Charts are deployed, a specific version is targeted. Also, the variables that are used to configure the chart are versioned. So you can always roll back to a previous version if your deployment goes wrong.

Rapid Development of Developer-Enablement Tooling

Kubernetes offers very easy role-based access controls based on secure certificates rather than a persistent user pool. Deploying a service as simple as a shell script that does not need to be exposed to the public, but to a controlled group of developers, becomes practically trivial.

Disadvantages

Learning Curve

Kubernetes’ resource model is different from Amazon’s, and so would require developers—who would be in charge of their own, several charts—to learn this new system.

Redundant Functionality

There isn’t anything specific that Kubernetes provides that isn’t provided by bare AWS functionality. It would be adding an abstraction layer to the overall architecture.

Multiplied Configuration

There would now be two separate configuration languages in the global infrastructure configuration. Since the Hashicorp Configuration Language files would only be used for organization-wide configuration changes, it might make code reviews lower-quality as people would become less familiar with it.

Migration Headaches

We already have one system and CI/CD pipeline built up in AWS Amplify, a few services running on Fargate, and many Lambdas. Migrating some or all of these services to K8s without downtime will be delicate. It will doubtless involve running two deployment pipelines in parallel until we are satisfied that we have reached parity and can cut DNS records over.

We may choose to leave some in place—such as the reporting pipeline lambdas—and only migrate them when we publish major changes.