Ovation Ventures Ovation Developer Docs

Tools

EKS/Kubernetes Cluster

EKS/Kubernetes Cluster

Contents

Motivation

While AWS Amplify is the fastest way to get a single repo project on-line, it has several drawbacks:

As our project grows beyond Balance Web to include reporting pipelines and microservices such as quiz APIs, certificate generators, and third-party integrations, we have outgrown the capabilities of Amplify. Indeed, Amplify and SOC2 compliance may be incompatible.

AWS offers “serverless” solutions such as Lambdas and Fargate tasks. Lambdas are difficult to develop, because they require a specific operating environment which can only partially be emulated on a developer’s workstation. Fargate is rather complicated to set up and configure, involving a bewildering depth of abstractions and connected resources.

Our current Terraform configuration treats the entire AWS environment as one, atomic whole. Deployments become difficult because any change to infrastructure involves re-analyzing the entire environment, possibly breaking or reverting changes to unrelated components.

Explanation

Kubernetes

Kubernetes (K8s) is a “container orchestrator”. It is like “Docker Compose”, but able to schedule containers to run on any number of distributed “nodes” (individual computers or virtual machines) and handles the networking between them. Kubernetes offers many of the same features as EC2, but with a simpler model.

Helm Charts

A Helm Chart is a collection of resource files, much like Terraform. Instead of Hashicorp Configuration Language, Helm Charts are Go template files that render to YAML files. Much like a Terraform module, a Helm chart exposes configuration variables that may have defaults. Unlike Terraform modules, Helm charts are versioned and pushed to a repository that manages multiple versions for each chart. So a release gone wrong can always be reverted to the previous version. Each Helm Chart will contain a description of all the resources it requires as a single, atomically-deployable unit.

Helmsman

Helmsman is analogous to a package.json file. It maps Helm Charts and their overriding value files and versions.

Cookbook

Adding a Domain to an EKS cluster

In the terraform-infrastructure repo, find the main.tf file for the environment you are adding the domain to.

Toward the end, there will be a block for the EKS Cluster, which will be recognizable by the # EKS Cluster comment, the -k8s suffix to the module name, and the source parameter, which should be ../../modules/eks_cluster.

Find the array of domain names, and add the new domain as a string.

When you apply the configuration, it will seem to hang while creating the certificate validation for the new domain. It needs a little outside assistance.

If you registered the domain in Route53, there will be a Hosted Zone. The module will also create a Route53 Hosted Zone. You will need to delete one of them.

Comment Action
HostedZone created by Route53 Registrar Delete this one
Managed by Terraform Keep this one

Whether the domain was registered in Route53 or an external registrar, you will need to update the NS records with the registrar. Examine the “Managed by Terraform” Hosted Zone, and find the four NS servers configured there. Copy these four domain names into the NS or Name Server records for the domain, removing the final . when pasting. You should remove any existing NS records.

The certification may take up to 24 hours to be created, and it is safe to CTRL-C the Terraform plan. However, in practice, it usually resolves in a few minutes.

CAUTION Only press CTRL-C once. Terraform will catch this and attempt to clean up after itself. If you cancel out of this “graceful” cleanup, you will be left with a stale lock, and will block further attempts to use Terraform until you do some simple, but awkward and scary cleanup.

Updating an EKS Deployment in Helmsman

  1. Run task choose-env to fetch credentials for the cluster and ECR registry.
  2. Confirm the OCI image has been pushed to a registry the cluster can access.
  3. Run task utils:update-deployment-tag to interactively update the image tag, or task edit to update the helmsman.yaml manually.
  4. Run TARGET=your_deployment task plan to review changes, then TARGET=your_deployment task apply to deploy.

For full details, see Deploying an Update to an Existing App in the Helmsman Config section.

Adding/Updating a Secret to an EKS Deployment

The Helmsman setup uses sops and age to encrypt secrets with asymmetrical keys, so even if you don’t have access to the decryption key for an environment, you can still add or update the values using the sops tool.

sops set environments/prod/my-app-secrts.yaml '["connection"]["password"]' '"Sw0ordfish"'

You can use a similar syntax to remove a key if it is no longer needed.

sops unset environments/prod/my-app-secrets.yaml '["connection"]["password"]'

If you do have access to the private key (stored in 1Password and installed on your workstation with the task setup command), you can use the sops edit command to edit the plaintext version of the secrets file.

EDITOR="$(which codium) --wait" sops edit environments/prod/my-app-secrets.yaml

Your choice of editor has to block the command line until the file is saved, because sops will re-encrypt the file when it is done.

Description of Security, Secrets and Authentication

Administrators of the cluster log in using their AWS accounts, which must have MFA enabled, and must have a specific Role attached to their account.

User Authentication

Users who have run the setup script should have a user in their ~/.kube/config like this:

- name: arn:aws:eks:us-west-2:614522506954:cluster/dev-k8s
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - us-west-2
      - eks
      - get-token
      - --cluster-name
      - dev-k8s
      - --output
      - json
      - --role
      - arn:aws:iam::614522506954:role/KubeUser
      command: aws
      env:
      - name: AWS_PROFILE
        value: dev
      interactiveMode: IfAvailable
      provideClusterInfo: false

This uses the AWS CLI tool to provide an authentication token to the cluster. The profile in this example is dev. The corresponding entry in ~/.aws/config looks like this:

[profile dev]
credential_process = op_credential_process --account 44LNRCZQE5D4RI6QWR34XDZJ5Y --item frqist4nocaclkbvpyyffahvva --mfa-field mfa' device' --otp-field one-time' password' --access-key-field access_key_id --secret-key-field secret_access_key --region-field region
region = us-west-2

This uses op_credential_process to fetch security credentials from 1Password, including a single-use time-based one-time code. 1Password will do its own user authentication using a combination of passwords and biometrics.

The user belongs to the group KubeUsers, which has two policies, UseKube and KubeUserAssumer. The UseKube policy allows the user to fetch the authentication token (using the aws eks get-token command listed in the .kube/config file). The KubeUserAssumer allows the user to assume the KubeUser role (also listed in the .kube/config file command).

That Role ARN is also mentioned in a special config map on the cluster.

# Contents edited
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::614522506954:role/KubeUser
      username: "{{SessionName}}"
      groups:
      - system:masters

This allows the user who can assume the KubeUser role to be in the system:masters built-in group and attaches their IAM username as their Kubernetes username, in case there are additional, custom RoleBindings or ClusterRoleBindings for that user.

In order to gain access to the Kubernetes cluster, therefore, a user must:

To revoke access to an individual, one may do any one of:

Furthermore, each environment (infra-test, dev, test, and prod) has its own cluster and its own, redundant set of users. Access can therefore be granted and denied on a cluster-by-cluster basis.

Ways this system might be circumvented.

A user with Kubernetes access granted in this way has admin privileges on the cluster. They might, therefore, create for themselves a signed certificate that places them into the system:masters group, and use that without reference to IAM or needing an MFA second credential. This is a necessary risk, because users who are granted access to the cluster are administrators, and need to have the ability to create additional users within Kubernetes in order to create service accounts and grant access to external tools, if appropriate.

Suggested Improvements

Create separate Developers group Ticket 868jh3vmt

Access to the cluster should be separated into cluster administrators (the current configuration), and users who have access only to the apps namespace for deploying, updating, and deleting code. These apps users should have the ability to edit most resources in the apps namespace, but not have access to cluster resources, like certificates.

App Secrets

Secrets needed by the app (e.g. database passwords, API keys) are handled by Helm’s secrets plugin, which uses sops and age to store encrypted secrets in GitHub, but uses 1Password to gate developer access to the encryption keys.

The encryption private keys for each environment are stored in 1Password in separate vaults in order to provide role-based access. A setup script uses your 1Password access to download these keys and store them in a system location on your workstation. 1Password uses a combination of passwords and biometric validation to make sure that you have been granted access to these keys.

The Helmsman tool coordinates calls to the Helm tool, which decrypts these values on-the-fly at deployment-time.

Because asymmetric encryption is used, and sops encrypts each value individually, it should be possible for developers to add new secrets to the encrypted files without being able to decrypt old ones. NB Needs verification.

In Helmsman and Helm, secrets that are decrypted in this way are never printed on the screen when listing the changes that are to be applied. So running the planning phase in a shared Zoom session or sharing the output is safe. However, if the --dry-run option is supplied, these secrets are emitted on the console, so this should always be a separate step.

Ways this system might be circumvented.

Because the private keys are located on the developer’s workstation, there is not an easy way to remove a developer’s access once granted short of rotating and redistributing the keys. The sops tool supports this, and the setup tool makes downloading new keys easy for the developers who retain access to the rotated keys. However, a better solution would involve fetching the keys from 1Password when needed, rather than pre-sharing the keys to selected developers.

Suggested Improvements

Move Keys to KMS Ticket 868jh48yt

The sops tool supports a number of alternate key retrieval schemes besides storing the key on a file on disk. These include using key services provided by AWS, GCP, Azure and Hashicorp. sops also provides its own key service with a bespoke protocol. One improvement might be to use AWS KMS to provide these keys, and use IAM permissions to grant role-based access to developers. Another solution might be to run the sops keyservice on an SSH server, and use the SSH servers’ user access controls to gate developer access.

The private keys should be moved to AWS KMS, and be gated with IAM policies and groups. The drawback is that it de-centralizes role-based permissions, which are currently managed by 1Password.

Investigate how complicated it is to grant access to users of other accounts (within the same umbrella business account) to secrets. The aim is to keep keys and permissions together in one place and grant controlled access to environments from there.

Apps

By “app” I mean a single web site, micro-service, cron-job, or any other self-contained, individually launchable unit of code.

For apps, it is suggested that the developer consider how the app is to be deployed, and include in each code repository:

For the build script, I recommend Taskfile.yaml, though there are certainly many options.

It is recommended that you install a single-node Kubernetes cluster on your workstation, and deploy your chart to that rather than using Docker Compose. The reason is that, although Kubernetes with a single node is functionally identical to Docker Compose, it will allow you to deploy your app and all its dependencies to your local “cluster” using the same tools and configurations as we will use to deploy it to cloud environments.

For local development, you can create a Helmsman file pointing at your local cluster, and use local file paths to point to your Helm chart. When you deploy to a cluster, including “development”, you will need to publish your Docker image to the development ECR registry. You should also package and deploy your versioned Helm chart to the same ECR registry, and use its oci:// address in the development Helmsman file. The reason for this is that published charts are versioned, and you can roll-back or publish multiple versions of your chart onto the shared cloud resource (with different subdomains).

The Multi-Stage Dockerfile

The following is an example of a simple Dockerfile that uses a multi-stage build to create a web service.

FROM golang:1.25 AS build
ENV CGO_ENABLED=1
RUN go install -tags extended github.com/gohugoio/hugo@latest
RUN mkdir -p /build
WORKDIR /build
COPY . .
RUN hugo build --baseURL "https://unquabain.com/" --environment "production"

FROM nginx:stable
COPY --from=build build/public /usr/share/nginx/html

It has two FROM lines: the first installs the tools needed to build the code (Go, from the base image, and Hugo from the install command), and the second copies over just what is needed in the production image. The image that is deployed, therefore, will not have Go or Hugo or any of the source files (in this case, a system of Markdown files).

The Helm Chart

The best way to write a Helm chart is to copy one that exists that is similar. You can start with this very simple HTTP server from the k8s-infrastructure repository.

It begins with a Chart.yaml. The most important parts of this file are the name, and the version. These are used by Helmsman to track all the resources it deploys to your cluster.

Next is the values.yaml, which is the interface to your chart. This should include all the values that need to change from environment to environment. The helm command can display this file, or individual keys from this file along with their comments, so make sure to document this file well. The values here don’t have to follow the structure of the Kubernetes resources that will be created. You should structure this file to make sense for the app you are creating.

The values.yaml file contains all the default values. Helm and Helmsman overlay this with other values files supplied at different points. In the helmsman.yaml configuration file, you can point to: other YAML files that override the defaults; secrets YAML files, which are encrypted and will be decrypted and override the other YAML files at deployment; and individual value overrides, which will override the secrets files. You can also specify values right on the command line, which will override all of those.

I recommend making the “tag” field of your image URI (in your “Pod” spec, which is in your “Deployment” or “StatefulSet” resource) into a separate field in your values.yaml, because this allows you to override it in your helmsman.yaml file, for easy updates when your image needs to be updated.

apps:
  my-service:
    chart: oci://...
    version: 0.1.23
    set:
      "image.tag": "2026-04-24-16-abcdef"

The templates/ folder contains YAML files that are each Go templates. The values will be replaced when you install your chart on your cluster. In the case of the “Simple HTTP Server”, there are three files, each creating a different type of resource.

Some other resources you may want to create in your chart include:

You can use the helm tool to install a Chart into any Kubernetes cluster you have access to, including your local minikube or Colima cluster. When your chart is working on your local cluster, the helm package and helm push commands can be used to publish it to an OCI registry. Remember that for every change, you need to update the version field in your Chart.yaml file. Helm tries to do the least disruptive thing it can, so if it doesn’t know your chart changed, it won’t push any updates. For development, you probably want to use the SemVer “build” field, like this: 0.1.0-dev01. That gives you something to update while you test locally, and you can remove that when you publish to the registry.

The Build Script

The build script should know how to build and publish both your app image and your chart as OCI images. I recommend Taskfile.yaml, because:

It is by no means the only tool that can do this job.

There are currently several versions of Taskfiles in various repos that can be used as models. It is a goal to make a library version that can be used to bootstrap new systems.

One recommendation is to use this shell command to generate tags for your OCI images:

git log -1 --date=format:%Y-%m-%d-%H --pretty=format:%cd-%h

This produces a tag like 2026-04-27-15-26210f1, that includes the date and hour of the last commit, and the short commit hash. This allows you to see at a glance whether a tag you are trying to deploy is newer or older than what’s already deployed, and still unambiguously targets a specific commit. Another suggestion would be to use the current ClickUp ticket number in your image tags.

# Assuming your branch is 868blahblah-do-some-task
git branch | awk '/\*/{print $2}' | cut -d- -f1

Helmsman Config

There are examples of working Helmsman configs for each environment. For example, here is the one for dev.

An example “app” looks like this one, for the Balance Stripe API:

apps:
# ...
  balance-stripe-api:
    !!merge <<: *app-template
    enabled: true
    chart: oci://614522506954.dkr.ecr.us-west-2.amazonaws.com/charts/balance-stripe-api
    version: "0.1.1"
    valuesFile: "balance-stripe-api-values.yaml"
    secretsFile: "balance-stripe-api-secrets.yaml"
    set:
      "image.tag": "2026-04-27-15-26210f1"
    depends:
      - gateway

The name of the deployment, balance-stripe-api, is available throughout the chart as {{ .Release.Name }}, and most charts use this liberally to tie their resources together.

The !!merge <<: *app-template allows the app to re-use common values defined earlier in the file.

enabled: true allows you to easily turn the deployment off, removing all related resources from the cluster in one go.

The chart key tells Helmsman where to find your chart. For deployment to your local cluster, this can be a file path relative to the directory where the helmsman.yaml file is located. For third-party charts, this can be the full https path to the chart, or you can register the chart repo in the helmRepos section at the top of the YAML file. Unfortunately, as of the most recent version of Helm, you cannot register OCI repos here, and have to type out the full domain name.

The version key must match the version key in your Chart.yaml exactly, if it is on disk. Otherwise, when you publish your chart to an OCI registry, Helm will look up that version in a “manifest” and download the matching version. In this way, you can have several versions of your chart deployed simultaneously to the same cluster, and you can manage upgrades and downgrades easily.

valuesFile points to a path, relative to the helmsman.yaml file, of your values overrides. You can also pass an array of valuesFiles if you want to break it up. For example, we may in the future standardized some parts of the values of our in-house charts and have a common.yaml file with per-environment values like “image.registry” and some networking values.

The secretsFile (or secretsFiles if you have more than one) key works just like valuesFile, but it uses a plugin to decrypt the file at deployment time. The k8s-infrastructure repo uses Sops/Age to encrypt the secrets files in place so they can be checked in to version control. The encryption uses asymmetrical keys which allow devs to add new values to the files, even if they don’t have access to decrypt the existing values.

The set key allows you another level of overrides over the values files. You can use this to pull forward the most frequently-changed values, such as the image tag, in order to make updates to existing deployments easier.

Deploying an Update to an Existing App

There is a Taskfile.yaml in the k8s-infrastructure repository that has some helpful scripts to make this routine task easier.

First, make sure you are working in the correct environment.

task choose-env

Make sure that the OCI image has been pushed to a registry that the cluster has access to. The cluster has implicit access to the ECR registry in the same account and AZ as itself. For other clusters, look for deployed ecr-secrets-updater instances.

Then, if your app uses a single Deployment, you might be able to use the utils:update-deployment-tag task.

task utils:update-deployment-tag

This interactively asks you to choose an eligible app, then lists out all the image tags that have been pushed to the registry.

This is a very simple script, though, and is only meant to help out in the most common of development use cases. In other events, you can run

task edit

to open the helmsman.yaml file in the appropriate environment folder using your preferred editor ($VISUAL if set, then $EDITOR if that’s set, otherwise falling back to vi, which I’m sure is your favorite editor as well as mine.)

Find the app entry and edit it as appropriate: usually just the "image.tag" override. Best practice: update the comment next to the image tag to indicate what ticket is being deployed.

Run TARGET=my-application task plan to check what updates Helm will make, then TARGET=my-application task apply to make the changes.

You can remove the TARGET=my-application bit to plan or apply all the changes that Helmsman can detect. The protected: true attribute is useful here for preventing Helm from deploying certain apps, even if changes are detected.

If you have any questions about what a task does, the Taskfile.yaml file is easy to read. Look for the section with the task name, and look for the cmd: or cmds: key under that.