# Argocd & ARC-github-runner on EKS with Terraform

# Day 7 — `argocd` & `github-runner`: How Code Reaches Production

> **Series: Zero to Production on AWS EKS with Terraform** — Part 9 · Day 7 Layer 4 of 5 — delivery. Add-ons of the day: **ArgoCD** and the **GitHub Actions Runner Controller**, both inside `eks-addon`.
>
> **Last updated:** 26 October 2026

**In short:** ArgoCD and the GitHub Actions Runner Controller (ARC) are the delivery layer of an EKS platform. ArgoCD continuously syncs the cluster to what Git declares, so merges become deploys and drift is visible. ARC runs CI jobs on self-hosted runner pods inside the cluster that scale to zero. Both are toggles in the `eks-addon` Terraform module.

We have a network, a cluster, storage, certs, load balancing, secrets, and smart scaling. The cluster is fully usable. But how does *your app* actually get onto it — and stay up to date — without someone running `kubectl apply` by hand at 11pm?

That's today: **delivery**. Two add-ons.

1. **ArgoCD** — pushes app changes from Git to the cluster, automatically (**GitOps**).
2. **GitHub Actions Runner Controller (ARC)** — runs your CI/CD pipelines *inside* the cluster.

---

## Part 1 — ArgoCD: GitOps delivery

### Why replace manual deploys with GitOps?

GitOps makes the Git repo the single source of truth, so what's in Git is what runs in the cluster. Manual deploys are fragile: someone forgets a step, dev and prod drift, and there's no clean record of "what's actually running right now." You want the **Git repo to be the single source of truth** — what's in Git *is* what's in the cluster, always.

### What is ArgoCD?

**ArgoCD** is a robot that continuously compares **"what Git says should be running"** against **"what's actually running"** and fixes any difference. Merge a change to your manifests → ArgoCD notices → rolls it out → reports health and sync status. Someone manually changed something in the cluster? ArgoCD flags the drift (or reverts it). The repo is the truth.

```plaintext
   git push (new manifest / new image tag)
            │
            ▼
   ArgoCD detects diff ──► applies to cluster ──► reports Synced/Healthy
            ▲                                            │
            └──────────── reconcile loop ◄───────────────┘
```

### Which inputs does ArgoCD need in `eks-addon`?

ArgoCD needs an on/off toggle, install basics, and the ingress hostname and ACM certificate for its UI.

| Variable | Meaning |
| --- | --- |
| `enable_argocd` | turn it on |
| `argocd_chart_version`, `argocd_namespace` | install basics |
| `argocd_enable_ingress` | expose the ArgoCD UI via an ALB |
| `argocd_server_host` | hostname for the UI (e.g. `argocd.example.com`) |
| `argocd_acm_certificate_arn` | the **Day-5 ACM cert** for https on that host |
| `argocd_enable_applicationset` | manage many apps/environments from templates |
| `argocd_enable_notifications` | Slack/webhook deploy notifications |
| `argocd_enable_dex` | SSO / external identity integration |
| `argocd_values_override` | Helm escape hatch |

```hcl
enable_argocd               = true
argocd_enable_ingress       = true
argocd_server_host          = "argocd.example.com"
argocd_acm_certificate_arn  = module.acm.certificate_arn   # ← Day 5 wiring
argocd_enable_applicationset = true
```

### Why is ArgoCD token-minting in a separate `argocd-token` module?

It keeps the shared `eks-addon` module GitOps-engine-agnostic. ArgoCD often needs an **API token** (to register itself, or for automation to create apps). Minting that token is split into a **dedicated** `argocd-token` **module**, *not* baked into the shared `eks-addon`. Why? So that the shared add-on stack stays **GitOps-engine-agnostic** — a pure monitoring cluster that doesn't run ArgoCD has no reason to carry token-minting logic. Keep the common module clean; isolate the product-specific bit.

---

## Part 2 — GitHub Actions Runner Controller (ARC)

### Why not just use GitHub-hosted runners?

Hosted runners cost money per minute, can't reach private resources, and don't give you control of their environment. CI/CD jobs (build, test, deploy) need machines to run on. GitHub's hosted runners cost money per minute, can't reach your *private* resources, and you don't control their environment.

### What is the GitHub Actions Runner Controller (ARC)?

**ARC** runs **self-hosted GitHub Actions runners as pods inside your own cluster**. When jobs queue up, it spins up runner pods; when the queue empties, it scales back down (even to zero). You get cheaper CI, full control of the runner image, and direct private access to your VPC resources.

```plaintext
  GitHub job queued
        │  ARC sees it
        ▼
  Runner pod spawned in cluster ──runs the job──► pod terminated when done
        (scales with demand, down to zero when idle)
```

### Which inputs does ARC need in `eks-addon`?

ARC needs the GitHub org/repo URL, a token, and a map of runner scale sets.

| Variable | Meaning |
| --- | --- |
| `enable_github_runner` | turn it on |
| `github_config_url` | the org/repo the runners register to |
| `github_token` | auth (supply via External Secrets / a secret, not plaintext) |
| `github_runners` | map of runner scale-sets (names, sizes, labels) |
| `runner_container_mode` | e.g. `dind` (Docker-in-Docker) for jobs that build images |
| `arc_controller_chart_version` / `arc_runner_chart_version` | chart versions |

```hcl
enable_github_runner = true
github_config_url    = "https://github.com/<org>"
github_token         = var.github_token   # sourced from a secret, never hardcoded
runner_container_mode = "dind"
github_runners = {
  default = { min_runners = 0, max_runners = 5 }
}
```

### What are the gotchas with self-hosted runners on EKS?

The main ones are memory sizing for `dind`, Karpenter evictions, and token handling.

- `dind` **runners are memory-hungry.** Docker-in-Docker needs real headroom. If a runner pod's memory limit exceeds the node's actual capacity, it **OOM-kills or never schedules** — and the node can flap to `NotReady`. Size runner pods *and* their nodes together. (This bit us: 6Gi runner pods on ~3Gi-usable nodes.)
- **Protect runners from Karpenter consolidation.** A runner mid-job that gets evicted by node consolidation = a failed pipeline. Give runner pods a `karpenter.sh/do-not-disrupt` annotation so they finish their job before the node can be reclaimed (ties straight back to Day 6).
- **Never hardcode** `github_token`**.** Pull it through External Secrets (Day 6) or a Kubernetes secret. It's a credential.

---

## How do ArgoCD and ARC tie the platform together?

They form the push-to-live loop: ARC builds and pushes the image, and ArgoCD deploys the updated manifests.

```plaintext
  Developer ──git push──► GitHub
        │                    │
        │                    ├─► ARC runner pod builds image ──push──► ECR (Day 3)
        │                    │   and updates the manifest's image tag
        ▼                    ▼
   Manifests repo ◄──── ArgoCD watches ────► deploys to EKS ──► ALB + ACM serve it (Day 5)
```

This is the full loop: **push code → CI builds it on your own runners → ArgoCD ships it → it's live behind https.** No manual `kubectl`, fully auditable, identical in every environment.

---

## How do you verify ArgoCD and the runners?

Check that ArgoCD applications show `Synced`/`Healthy` and that the runner scale set appears in GitHub.

```bash
# ArgoCD
kubectl -n argocd get pods
kubectl -n argocd get applications        # Synced / Healthy
# open https://argocd.example.com (the ingress + ACM cert)

# Runners
kubectl get pods -n <runner-namespace>
# check Settings → Actions → Runners in GitHub: scale set shows Idle/Active
```

---

## Frequently asked questions

### What does ArgoCD do when someone changes the cluster by hand?

ArgoCD continuously compares what Git says should be running with what is actually running. A manual change shows up as drift, which ArgoCD flags or reverts, because the repo is the source of truth.

### How do I expose the ArgoCD UI over HTTPS on EKS?

Set `argocd_enable_ingress = true`, give it a hostname with `argocd_server_host`, and pass the Day-5 ACM certificate as `argocd_acm_certificate_arn`. The ALB controller then serves the UI over HTTPS.

### Can self-hosted GitHub runners scale to zero?

Yes. ARC spins up runner pods when jobs queue and scales back down, even to zero, when the queue empties. Set `min_runners = 0` on the scale set in `github_runners`.

### Why do my dind runner pods get OOM-killed or never schedule?

Docker-in-Docker is memory-hungry, and a runner pod whose memory limit exceeds the node's usable capacity either OOM-kills or never schedules. Size runner pods and their nodes together.

### Where should the GitHub token for ARC come from?

Never hardcode `github_token`. Supply it through External Secrets or a Kubernetes secret, because it is a credential.

---

## What's the takeaway?

- **ArgoCD** makes Git the source of truth — merges become deploys, drift is visible/auto-corrected. Token-minting is isolated in a dedicated `argocd-token` module to keep the shared add-on engine-agnostic.
- **ARC** runs CI/CD on self-hosted runners inside the cluster — cheaper, private-reachable, controllable — but size them and shield them from consolidation.
- Together with Day 3 (ECR) and Day 5 (ALB + ACM), they form the complete **push-to-live** pipeline.

**Next — Day 8 (finale):** eyes on everything. The observability stack — **Promtail, Loki, Grafana, Prometheus agent, Tempo** — plus the `ec2` module and a full **one-command environment** wrap-up. 📊