The k3s Home Lab, Part 4: Ephemeral GitHub Actions Runners with ARC

Flux had already made the cluster reproducible. The next gap was the path into it. Application manifests reconciled from Git, but container images were still built outside the platform and pushed by hand. For a small journaling application, I wanted a commit to produce an immutable image, publish it to the private Gitea registry, and let Flux promote that image without giving a CI job direct control of the cluster.

GitHub Actions Runner Controller, usually shortened to ARC, supplied the missing execution layer. ARC watches GitHub's Actions service, translates queued work into Kubernetes demand, and creates self-hosted runner pods only when jobs exist. The runner completes one job and disappears. At idle, the listener remains connected but the expensive Docker builder scales to zero.

The result is two related control loops with a deliberate boundary between them. CI builds and publishes an artifact. GitOps selects and deploys it. GitHub never receives cluster-admin credentials, and the workflow never patches the live Deployment.

From Commit to Ephemeral Runner

The CI path begins before ARC is involved. A developer opens a feature branch, the change passes review, and merging to the default branch queues the GitHub Actions workflow. Only then does ARC translate the queued job into temporary Kubernetes capacity.

---
config:
  layout: dagre
  theme: dark
  flowchart:
    nodeSpacing: 45
    rankSpacing: 65
    curve: basis
---
flowchart LR
    subgraph GITHUB["Development and GitHub"]
        direction TB
        DEV["Developer<br/>writes code"]
        BRANCH["GitHub<br/>feature branch"]
        PR["Pull request"]
        REVIEW["Review and checks"]
        MERGE["Merge to main"]
        WORKFLOW["Actions workflow<br/>is queued"]

        DEV --> BRANCH
        BRANCH --> PR
        PR --> REVIEW
        REVIEW --> MERGE
        MERGE --> WORKFLOW
    end

    subgraph ARC["ARC on K3s"]
        direction TB
        LISTENER["Listener receives<br/>the job"]
        SCALE["Scale set requests<br/>one runner"]
        POD["Ephemeral runner pod<br/>starts"]
        BUILD["Build and test<br/>the application"]
        PUSH["Push image to<br/>Gitea registry"]
        DELETE["Job completes<br/>and pod is deleted"]
        ZERO["Runner capacity<br/>returns to zero"]

        LISTENER --> SCALE
        SCALE --> POD
        POD --> BUILD
        BUILD --> PUSH
        PUSH --> DELETE
        DELETE --> ZERO
    end

    WORKFLOW --> LISTENER

    classDef github fill:#242f49,stroke:#93c5fd,color:#ffffff,stroke-width:2px
    classDef arc fill:#172f50,stroke:#60a5fa,color:#ffffff,stroke-width:2px
    classDef runner fill:#123e42,stroke:#2dd4bf,color:#ffffff,stroke-width:2px
    classDef registry fill:#402d17,stroke:#f59e0b,color:#ffffff,stroke-width:2px
    classDef lifecycle fill:#18283d,stroke:#64748b,color:#e2e8f0,stroke-width:2px

    class DEV,BRANCH,PR,REVIEW,MERGE,WORKFLOW github
    class LISTENER,SCALE arc
    class POD,BUILD runner
    class PUSH registry
    class DELETE,ZERO lifecycle

ARC itself has three visible layers. The controller is the Kubernetes operator that reconciles ARC custom resources. A listener maintains a long-poll connection to GitHub for one runner scale set. Ephemeral runner pods are the actual job capacity. Keeping those roles distinct is important during both sizing and troubleshooting.

Prerequisites and Trust Boundaries

The cluster already had k3s, Flux, ingress, persistent storage, and access to the private Gitea registry. ARC added several new dependencies: outbound HTTPS and DNS access to GitHub and GHCR, enough node capacity for image builds, a GitHub identity allowed to manage repository runners, and registry credentials that could push the journaling application's images.

I separated credentials by purpose. ARC's GitHub credential registers and manages runners. The workflow's Gitea credential pushes images. Cluster workloads use a different, read-only pull credential. Flux uses its own registry credential for scanning tags and a separate Git identity if image automation writes commits. Reusing one powerful secret would have been convenient, but it would also have made every component part of the same compromise path.

Docker-in-Docker was the largest security decision. It made existing Docker-based Actions straightforward, but the Docker daemon runs privileged. An ephemeral pod limits how long state survives; it does not make arbitrary workflow code harmless. Runner nodes should be treated as a build trust zone, isolated from sensitive workloads and internal services wherever the cluster topology allows it.

Repository Layout

ARC is installed from two Helm charts: one for the shared controller and one for each runner scale set. Both charts are published as OCI artifacts in GitHub Container Registry rather than through a traditional Helm repository. I kept their Flux sources, releases, authentication material, and dependency order together under infrastructure.

clusters/
  home-lab/
    infrastructure/
      actions-runner-controller/
        namespace.yaml
        controller-oci.yaml
        controller-release.yaml
        runner-scale-set-oci.yaml
        journal-runner-release.yaml
        github-auth.sops.yaml
        kustomization.yaml
    apps/
      journal/
        deployment.yaml
        image-repository.yaml
        image-policy.yaml
        image-update-automation.yaml
        registry-pull-secret.sops.yaml
        kustomization.yaml

The names describe responsibilities rather than implementation accidents. The journaling application owns its deployment and image policy. ARC remains shared infrastructure even if the first scale set serves only one repository.

Namespace and Secret

GitHub recommends keeping controller pods and runner pods in separate namespaces. A compact homelab can start with one namespace, but a stronger layout places the controller in arc-systems and listeners, runner secrets, and runner pods in github-actions. Namespace separation supports narrower RBAC and policy, though it is not isolation by itself.

apiVersion: v1
kind: Namespace
metadata:
  name: github-actions
  labels:
    app.kubernetes.io/part-of: actions-runner-controller

The ARC authentication secret belongs in the same namespace as the runner scale-set release. The checked-in version is encrypted with SOPS; a normal Kubernetes Secret containing base64 data is only encoded and should not be committed as if it were protected.

apiVersion: v1
kind: Secret
metadata:
  name: github-arc-auth
  namespace: github-actions
type: Opaque
stringData:
  github_token: ENC[AES256_GCM,data:...,type:str]
sops:
  # SOPS metadata omitted

OCI Helm Sources

ARC's current scale-set mode is delivered through OCI Helm charts. That detail matters with Flux: the sources are OCIRepository resources, not entries in a conventional HelmRepository. I pinned both charts to an explicitly tested release rather than allowing an unreviewed upgrade to change the controller and runner behavior.

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: arc-controller-chart
  namespace: flux-system
spec:
  interval: 1h
  url: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
  ref:
    tag: "<tested-arc-version>"
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: arc-scale-set-chart
  namespace: flux-system
spec:
  interval: 1h
  url: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
  ref:
    tag: "<tested-arc-version>"

API versions evolve, so the committed manifests must match the Flux controllers actually installed in the cluster. The durable design is the OCI URL, explicit version, reconciliation interval, and visible dependency—not a particular beta or stable API label copied from an article.

Installing the Controller

The controller is shared infrastructure. One installation can reconcile multiple runner scale sets, so its release name should not be tied to the journaling application.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: arc-controller
  namespace: github-actions
spec:
  interval: 30m
  chartRef:
    kind: OCIRepository
    name: arc-controller-chart
    namespace: flux-system
  values:
    replicaCount: 1
    flags:
      logLevel: info
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 512Mi

One replica is reasonable for a small cluster where CI availability is not critical. A larger environment would add controller availability, topology spread, disruption policy, metrics, centralized logs, and a tested upgrade path. Availability should be designed around real failure domains; two controller pods on one physical host are not independent.

Defining the Runner Scale Set

The scale set is repository-scoped and named journal-runner. That name becomes the workflow's runs-on target. A minimum of zero removes idle job pods, while a maximum of two permits limited concurrency without allowing builds to consume the entire cluster.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: journal-runner
  namespace: github-actions
spec:
  interval: 15m
  dependsOn:
    - name: arc-controller
  chartRef:
    kind: OCIRepository
    name: arc-scale-set-chart
    namespace: flux-system
  values:
    githubConfigUrl: https://github.com/example/journal
    githubConfigSecret: github-arc-auth
    runnerScaleSetName: journal-runner
    minRunners: 0
    maxRunners: 2
    containerMode:
      type: dind
    template:
      spec:
        containers:
          - name: runner
            image: ghcr.io/actions/actions-runner:latest
            resources:
              requests:
                cpu: "1"
                memory: 2Gi
              limits:
                cpu: "4"
                memory: 8Gi
        nodeSelector:
          kubernetes.io/os: linux

The runner container must remain named runner; ARC uses that name when it configures the pod. In a long-lived repository I would pin the runner image by version or digest, then automate reviewed updates. The latest value above keeps the example readable but should not be mistaken for a supply-chain policy.

The Authentication Failure That Looked Like Kubernetes

The controller deployed successfully, but the listener never appeared. The controller logs showed a 403 while requesting a registration token from GitHub. Because the Kubernetes resources and secret existed, it was tempting to keep debugging Helm and RBAC. The failure was actually at the GitHub API boundary.

A repository-scoped fine-grained personal access token needs Repository permissions → Administration: Read and write. Granting Actions read and write sounds correct, but runner registration is an administration operation. Without that permission, ARC can start in Kubernetes but cannot create or authenticate the repository's runner scale set.

After changing a fine-grained token, I also verify its resource owner, selected repositories, expiration, organization approval, and SSO state. A permission shown in the token editor is not proof that the organization has accepted or authorized the token.

Why One Runner Appeared as Two Containers

When the first job arrived, Kubernetes reported the runner pod as 2/2 Running. That was not two runner pods. Docker-in-Docker mode places the Actions runner and a Docker daemon in the same pod. Kubernetes readiness counts containers inside a pod, so both containers being ready produces 2/2.

kubectl get pod -n github-actions <runner-pod> \
  -o jsonpath='{.spec.containers[*].name}{"\n"}'

# runner dind

The controller and listener are also pods, but neither executes the application build. ARC's custom resources provide the useful capacity view: current runners, pending runners, running runners, finished runners, and deleting runners. With one active job, the expected state is one runner even though the namespace contains multiple pods.

The Ephemeral Runner Lifecycle

1. The listener maintains an HTTPS long-poll connection to GitHub's Actions service.
2. A workflow targets runs-on: journal-runner, and GitHub assigns the job to that scale set.
3. The listener updates desired capacity. ARC creates an EphemeralRunnerSet and an EphemeralRunner.
4. Kubernetes schedules the runner pod. In DinD mode, both the runner and Docker daemon containers initialize.
5. The runner registers with a short-lived token, accepts one job, checks out the repository, and builds the image.
6. The workflow pushes immutable and convenience tags to the Gitea registry.
7. The runner exits. ARC removes the GitHub runner record and deletes the Kubernetes resources and pod.
8. Flux detects the new registry tag, commits the selected version to Git, and reconciles the journaling application.

Scale-to-zero changes the definition of healthy. At idle, no runner pod is expected. The persistent listener is the signal that the scale set can receive work; runner pods are short-lived evidence of demand.

Building and Publishing the Image

The workflow produces an immutable tag derived from the commit and may also publish latest for convenience. Flux promotes the immutable version. That preserves provenance and makes rollback a Git change rather than a guess about what a mutable tag contained.

name: Build journal image

on:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  image:
    runs-on: journal-runner
    steps:
      - name: Check out source
        uses: actions/checkout@v5

      - name: Log in to the private registry
        uses: docker/login-action@v4
        with:
          registry: git.example.internal
          username: ${{ secrets.GITEA_REGISTRY_USERNAME }}
          password: ${{ secrets.GITEA_REGISTRY_TOKEN }}

      - name: Generate image metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: git.example.internal/apps/journal
          tags: |
            type=sha,prefix=sha-
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          provenance: mode=max

I upgraded the JavaScript-based actions as their runtimes changed rather than ignoring runner deprecation warnings. Major action tags are readable, but high-assurance workflows should pin third-party actions to reviewed commit SHAs and use dependency automation to propose updates.

Private Registry Details

The runner needs a write-capable Gitea credential only for the login and push steps. The journaling application's namespace receives a separate image-pull secret with read-only registry access. Flux's ImageRepository also needs permission to enumerate tags.

If Gitea uses a private certificate authority, every K3s node that may pull the image must trust it. K3s configures containerd registry mirrors and TLS settings through /etc/rancher/k3s/registries.yaml. That file must be consistent across eligible nodes, and changing it requires a planned K3s restart. A publicly trusted internal certificate is simpler and safer than disabling TLS verification.

The job can successfully push an image while Kubernetes still fails to pull it. Registry publication, Flux scanning, and node image pulls are three separate authentication and trust paths; each needs independent verification.

Flux Image Automation

Flux image automation closes the loop without giving the GitHub workflow Kubernetes credentials. The delivery path has three ownership boundaries: registry discovery and policy, Git desired state, and cluster reconciliation. An ImageRepository scans Gitea, an ImagePolicy selects an allowed tag, and an ImageUpdateAutomation writes that selection back to the cluster repository before the normal Flux reconciliation deploys it.

---
config:
  layout: dagre
  theme: dark
  flowchart:
    nodeSpacing: 45
    rankSpacing: 65
    curve: basis
---
flowchart LR
    subgraph DISCOVERY["Registry and image selection"]
        direction TB
        REGISTRY["Gitea registry<br/>contains new image"]
        SCAN["ImageRepository<br/>scans tags"]
        CHECK["ImagePolicy<br/>checks versions"]
        SELECT["Newest eligible<br/>image selected"]

        REGISTRY --> SCAN
        SCAN --> CHECK
        CHECK --> SELECT
    end

    subgraph GIT["Git desired state"]
        direction TB
        AUTOMATION["ImageUpdateAutomation<br/>updates image reference"]
        COMMIT["Flux creates<br/>a Git commit"]
        STATE["K3s repository records<br/>the desired image"]

        AUTOMATION --> COMMIT
        COMMIT --> STATE
    end

    subgraph DEPLOYMENT["Reconciliation and deployment"]
        direction TB
        SOURCE["Flux detects<br/>the Git revision"]
        RECONCILE["Kustomization builds<br/>and applies resources"]
        ROLLOUT["Kubernetes performs<br/>a rolling deployment"]
        APP["Journaling application<br/>runs the new image"]

        SOURCE --> RECONCILE
        RECONCILE --> ROLLOUT
        ROLLOUT --> APP
    end

    SELECT --> AUTOMATION
    STATE --> SOURCE

    classDef registry fill:#402d17,stroke:#f59e0b,color:#ffffff,stroke-width:2px
    classDef image fill:#28244d,stroke:#a78bfa,color:#ffffff,stroke-width:2px
    classDef git fill:#242f49,stroke:#93c5fd,color:#ffffff,stroke-width:2px
    classDef flux fill:#172f50,stroke:#60a5fa,color:#ffffff,stroke-width:2px
    classDef workload fill:#123e42,stroke:#34d399,color:#ffffff,stroke-width:2px

    class REGISTRY registry
    class SCAN,CHECK,SELECT image
    class AUTOMATION,COMMIT,STATE git
    class SOURCE,RECONCILE flux
    class ROLLOUT,APP workload

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: journal
  namespace: flux-system
spec:
  image: git.example.internal/apps/journal
  interval: 5m
  secretRef:
    name: gitea-registry-pull
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: journal
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: journal
  filterTags:
    pattern: '^main-(?P<ts>[0-9]{8}T[0-9]{6}Z)-[a-f0-9]+$'
    extract: '$ts'
  policy:
    numerical:
      order: asc

A raw Git SHA is immutable but not chronological, so alphabetical sorting does not identify the newest commit. A sortable tag such as main-20260814T134501Z-a1b2c3d gives Flux an ordering field while retaining the commit identity.

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
  name: journal
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: flux-bot
        email: flux-bot@example.invalid
      messageTemplate: "chore(images): update journal image"
    push:
      branch: main
  update:
    path: ./clusters/home-lab/apps/journal
    strategy: Setters

image: git.example.internal/apps/journal:main-20260814T134501Z-a1b2c3d # {"$imagepolicy": "flux-system:journal"}

The image controllers update Git; the normal Flux Kustomization remains responsible for deploying the result. This distinction preserves the same review, reconciliation, drift correction, and recovery model used by the rest of the cluster.

Verification from the Outside In

I verified each boundary independently rather than treating a green workflow as proof of a complete deployment.

# Did Flux retrieve and install the OCI charts?
flux get sources oci -A
flux get helmreleases -A

# Are the controller, listener, and ARC resources healthy?
kubectl get pods,autoscalingrunnersets,ephemeralrunnersets,ephemeralrunners \
  -n github-actions

# What does the scale set believe its capacity is?
kubectl describe autoscalingrunnerset journal-runner -n github-actions

# Watch one job create and retire one runner pod.
kubectl get pods -n github-actions -w

# Did Flux scan, select, write, and reconcile the image?
flux get image repository -A
flux get image policy -A
flux get image update -A
flux get kustomizations -A

At idle, I expect the controller and listener to remain, the current runner count to be zero, and no ephemeral runner pod to exist. During a job, pending capacity should become one running runner. Afterward, finished and deleting counters may appear briefly before the scale set returns to zero.

The final proof is the running workload's image digest, not the tag shown in Git. A successful registry push does not prove Flux selected the tag; a Flux commit does not prove reconciliation succeeded; a Deployment update does not prove the node pulled the intended image.

Troubleshooting by Symptom

403 while requesting a registration token

Confirm the runner scope, token resource owner, selected repository, expiration, organization approval, and SSO state. For a repository-scoped fine-grained PAT, verify Administration: Read and write. Inspect the HTTP failure in controller logs before changing Kubernetes resources.

The controller runs, but there is no listener

Verify githubConfigUrl, the referenced secret name, the github_token key, and the secret namespace. Then inspect controller logs, HelmRelease conditions, Kubernetes API access, DNS, egress policy, and proxy behavior. A missing listener means the control plane has not reached the point where runner pods can be useful.

The scale set exists, but jobs remain queued

Match the workflow's runs-on value to runnerScaleSetName exactly. Check that maxRunners is not zero, inspect AutoscalingRunnerSet status, and read listener logs for assignment decisions. If an EphemeralRunner exists, move down the stack to scheduling events, image pulls, resource pressure, taints, volumes, and Pod Security admission.

The runner starts, but Docker fails

Inspect both containers. A broken DinD daemon, insufficient ephemeral storage, registry certificate error, MTU mismatch, or unreachable registry can all surface as a generic Docker client failure in the Actions step.

An extra runner briefly appears after cancellation

Cancellation reaches the active runner and listener on different paths. The runner can terminate before the listener observes the canceled job, causing a short-lived replacement before desired capacity settles. Persistent excess capacity is a problem; a brief correction during cancellation can be normal.

Security Considerations

Self-hosted Actions runners execute repository code on infrastructure I own. That statement, rather than the fact that the pods are temporary, defines the security model.

Future Improvements

The first improvement is GitHub App authentication. GitHub recommends it for repository- and organization-scoped ARC installations. Installation tokens are short-lived and permissions are explicit, eliminating a long-lived personal token from the normal runner lifecycle.

The second is build caching. Scale-to-zero discards local Docker layers, so cold builds pay the full dependency and layer cost. A registry-backed BuildKit cache can preserve ephemerality while improving startup time. The cache needs its own retention, size, integrity, and trust policy; “make builds faster” should not quietly create an unbounded artifact store.

The third is separating trusted release work from untrusted validation. An organization-scoped runner can serve multiple repositories through runner groups, but sharing capacity also shares a trust domain. I would use different scale sets, node pools, credentials, and network policies for release builds and pull-request checks.

Finally, the delivery path should mature toward signed images, SBOM generation, vulnerability policy, digest promotion, admission verification, and metrics for job startup latency, listener availability, stuck runners, and Flux reconciliation failures.

Lessons Learned

First, the API permission name matters more than intuition. Runner registration is repository administration, not merely Actions access. The 403 was accurate; I was debugging the wrong boundary.

Second, a missing listener is a high-value diagnostic signal. If the controller exists but the listener does not, authentication, configuration, or controller access failed before runner scheduling entered the picture.

Third, Kubernetes readiness counts containers, not runners. A DinD runner showing 2/2 is one pod containing the runner and Docker daemon. ARC's custom-resource status is the authoritative capacity view.

Fourth, scale-to-zero changes what healthy looks like. The absence of an idle runner pod is the desired state. The listener is durable; runners are disposable.

Finally, CI and GitOps should meet at the registry and Git. The workflow produces an immutable artifact. Flux decides which artifact the cluster should run. That boundary keeps deployment state reviewable, reversible, and recoverable without handing the build job control of production.