# Observability Stack on EKS via TERRAFORM....!!

# Day 8 (Finale) — Observability, the `ec2` Module & One-Command Environments

> **Series: Zero to Production on AWS EKS with Terraform** — Part 10 · Day 8 (finale) Layer 5 of 5 — observability — plus the wrap-up. Add-ons of the day: **Promtail**, **Loki + Grafana**, **Prometheus agent**, **Tempo**; module: `ec2`.
>
> **Last updated:** 26 October 2026

**In short:** Observability on EKS means collecting logs, metrics and traces. Here Promtail ships logs to Loki, a Prometheus agent remote-writes metrics to a central Prometheus, Tempo stores traces in S3, and Grafana shows all three. All of it is installed by one Terraform `eks-addon` call, and the whole platform then rebuilds a full environment with one `terraform apply`.

We've built everything that *runs* apps. The last layer is the one that lets you **sleep at night**: knowing what's happening inside the cluster, and seeing problems before your users do.

Then we tie the whole 8-day journey together: one library, one command, a full environment.

---

## Part 1 — The observability stack

Observability has three pillars: **logs** (what happened), **metrics** (how much / how fast) and **traces** (where the time went inside one request). Our `eks-addon` ships all three.

### What does Promtail do in an EKS cluster?

Promtail collects every container's logs on every node and ships them to Loki. **Promtail** runs as a **DaemonSet** (one pod on every node). It tails the logs of every container, labels them (namespace, pod, app), and ships them to Loki. Think of it as a courier on every machine, collecting mail and forwarding it to the central post office.

| Variable (in `eks-addon`) | Meaning |
| --- | --- |
| `enable_promtail` | turn it on |
| `promtail_namespace`, `promtail_chart_version` | install basics |
| `loki_host`, `loki_port` | where to ship logs (the Loki endpoint) |

> Note the cross-cluster wiring: a workload cluster's Promtail can ship to a Loki running in the **shared monitoring VPC** — reachable privately thanks to the **Transit Gateway** from Day 2.

### What do Loki and Grafana do?

Loki stores and indexes the logs; Grafana is where you search them, chart them and alert on them.

- **Loki** stores logs cheaply and makes them queryable (LogQL). It's like a search engine for your logs — and it backs its storage with **S3** and uses the **gp3 PVCs** from Day 5's `ebs-csi`.
- **Grafana** is the dashboard layer — live charts, log explorers, and alerts. It's the single pane of glass where on-call engineers actually look.

| Variable (in `eks-addon`) | Meaning |
| --- | --- |
| `enable_loki_grafana` | turn it on |
| `loki_chart_version` | chart version |
| `grafana_admin_password` | initial admin login (source from a secret!) |
| `loki_acm_certificate_arn` | **Day-5 ACM cert** for the Grafana/Loki https endpoint |
| `loki_s3_bucket_name`, `loki_s3_region` | durable log storage in S3 |

### What is Prometheus agent mode?

Prometheus agent mode collects metrics and forwards them without storing them locally. **Prometheus agent** scrapes metrics (CPU, memory, request rates, custom app metrics) and **remote-writes** them to a central Prometheus/store. "Agent mode" means it doesn't store locally — it just collects and forwards, keeping workload clusters light.

| Variable (in `eks-addon`) | Meaning |
| --- | --- |
| `enable_prometheus_agent` | turn it on |
| `prometheus_agent_namespace`, `prometheus_agent_chart_version` | install basics |
| `prometheus_remote_write_url` | central endpoint to ship metrics to |
| `prometheus_agent_node_selector` | pin the agent to specific nodes |

### What is Tempo used for in distributed tracing?

Tempo stores distributed traces in S3 so you can see where the time went inside one request.

Logs tell you *that* a request failed; metrics tell you *how often*. **Traces** tell you *where*: one request is followed across every service it touches, with the time spent in each hop. When checkout is slow, a trace shows you it's the third downstream call, not the database you were about to blame.

**Tempo** stores those traces cheaply in **S3** (no heavy database to run) and Grafana shows them next to the logs and metrics for the same moment. Your apps send spans with OpenTelemetry; Tempo keeps them for as long as you choose.

| Variable (in `eks-addon`) | Meaning |
| --- | --- |
| `enable_tempo` | turn it on |
| `tempo_chart_version` | chart version |
| `tempo_retention_days` | how long traces are kept (drives both the S3 lifecycle rule and Tempo's compactor) |
| `tempo_iam_role_name` | the IAM role Tempo uses to read/write its S3 bucket — no static keys |

> In practice, tracing is what cut our troubleshooting time the most: instead of grepping logs across five services, you open one trace and see the slow hop.

### How does observability data flow from the cluster to Grafana?

Logs, metrics and traces each take their own path, and all three end up in Grafana.

```plaintext
  Every node ──Promtail──► Loki ──► Grafana (logs + dashboards, behind ACM https)
  Every node ──Prometheus agent ──remote-write──► central Prometheus ──► Grafana (metrics)
  Every app  ──OpenTelemetry spans──────────────► Tempo (S3) ──────────► Grafana (traces)
                         │
        (cross-VPC ships travel privately over the Day-2 Transit Gateway)
```

```hcl
enable_promtail        = true
loki_host              = "loki.monitoring.internal"
loki_port              = 3100

enable_loki_grafana       = true
grafana_admin_password    = var.grafana_admin_password   # from a secret
loki_acm_certificate_arn  = module.acm.certificate_arn
loki_s3_bucket_name       = "prod-loki-logs"
loki_s3_region            = "us-east-1"

enable_prometheus_agent     = true
prometheus_remote_write_url = "https://prometheus.monitoring.internal/api/v1/write"

enable_tempo         = true
tempo_retention_days = 14
```

### Which optional observability add-ons are in the `eks-addon` bundle?

The same `eks-addon` call carries a few optional extras, each behind its own `enable_*` flag:

- **Alerting** — Alertmanager routes alerts to Slack, split by severity.
- **Robusta** — enriches alerts with pod logs and events, so the Slack message already contains the first clue.
- **OpenCost** — cost per namespace and workload, so "who is spending what" has an answer.
- **Blackbox exporter** — probes your public URLs and TCP endpoints from the outside, like a user would.
- **CloudWatch exporter** — pulls AWS-side metrics (load balancers, managed databases) into the same Grafana.

---

## Part 2 — the `ec2` module (the occasional exception)

The `ec2` module creates plain virtual machines for the few things that don't belong on EKS. 99% of our compute is pods on EKS. But sometimes you need a **plain virtual machine** — a bastion/jump host, a one-off tool, a license server, a legacy thing that won't containerize. That's the `ec2` module.

It's **map-driven** like `security-group` and `alb`: describe a map of instances, get them all with consistent tags. Subnets and security groups are passed in as **lookup maps**, so each instance refers to them by a friendly name instead of a raw ID.

| Variable | Meaning |
| --- | --- |
| `ec2_instances` | map keyed by name → `ami`, `instance_type`, `subnet` (a key in `subnet_ids`), `security_groups` (keys in `security_group_ids`), plus optional `iam_instance_profile`, `user_data`, `root_block_device`, … |
| `subnet_ids` | lookup map of subnet lists, e.g. `{ private = module.vpc.private_subnets }` |
| `security_group_ids` | lookup map of group IDs — the Day 2 module's `ids` output fits straight in |
| `tags` | merged onto every instance |

**Outputs:** `instance_ids`, `private_ips`.

```hcl
module "ec2" {
  source = "git::https://github.com/<org>/devops-terraform-modules.git//modules/ec2?ref=v1.4.0"

  subnet_ids         = { private = module.vpc.private_subnets }
  security_group_ids = module.security_group.ids

  ec2_instances = {
    bastion = {
      ami                  = data.aws_ami.al2023.id
      instance_type        = "t3.micro"
      subnet               = "private"
      security_groups      = ["bastion"]
      iam_instance_profile = "ssm-managed-instance"   # connect with SSM Session Manager, no SSH key
      root_block_device    = { volume_type = "gp3", volume_size = 20, encrypted = true }
    }
  }

  tags = { Project = "platform", Role = "bastion" }
}
```

> **Default to EKS.** Reach for `ec2` only when something genuinely doesn't belong in the cluster. Every VM is a thing you have to patch and babysit — the opposite of why we built all this.

---

## Part 3 — the finale: it all comes together

Eight days, a dozen-plus modules, five layers. Here's the whole platform on one page:

```plaintext
   ┌──────────────────────────────────────────────────────────────┐
   │  devops-terraform-modules  (one versioned library)             │
   └──────────────────────────────────────────────────────────────┘
        Day 1  vpc ............... private network, subnets, endpoints
        Day 2  security-group .... firewalls
               tgw / tgw-attach .. private bridges between VPCs
        Day 3  iam-policy ........ least-privilege permissions
               ecr ............... container image registries
        Day 4  eks ............... Kubernetes control plane + OIDC/IRSA
        Day 5  ebs-csi ........... durable storage (gp3 default)
               acm ............... TLS certificates (the 🔒)
               alb ............... infra-level LBs
               + ALB controller .. Ingress → real load balancers
        Day 6  external-secrets .. AWS secrets → K8s, no value in Git
               karpenter ......... right-sized nodes on demand
        Day 7  argocd ........... GitOps delivery (Git = truth)
               github-runner ..... CI/CD inside the cluster
        Day 8  promtail/loki/
               grafana/prometheus/
               tempo ............. logs, metrics, traces, dashboards, alerts
               ec2 .............. the occasional plain VM
```

### How do you spin up a new environment with one command?

You copy a consumer stack, edit a small `tfvars`, and run `terraform apply`.

Because every layer is a module wired by **outputs → inputs**, spinning up a brand-new environment (a new `dev`, `staging`, or whole new product) is:

1. Copy a consumer-stack folder, point its backend at a fresh state key.
2. Edit a small `tfvars`: name, region, CIDR, sizes, hostnames.
3. `terraform init` → pulls the **pinned** module versions.
4. `terraform plan` → read the diff.
5. `terraform apply` → ☕ (~15–20 min, mostly AWS building the EKS control plane).
6. A full, production-grade environment is alive — network, cluster, storage, certs, secrets, autoscaling, GitOps, CI, and observability — **identical to its siblings** because it came from the identical modules at the identical version.

No console clicking. No drift. A security fix is **one PR to the library + a version bump** in each stack — and every environment inherits it.

---

## Frequently asked questions

### What are the three pillars of observability?

The three pillars are logs (what happened), metrics (how much and how fast) and traces (where the time went inside one request). In this platform Loki stores logs, a central Prometheus stores metrics and Tempo stores traces, and Grafana shows all three.

### Why run Prometheus in agent mode on workload clusters?

Agent mode doesn't store metrics locally; it scrapes them and remote-writes them to a central Prometheus. That keeps workload clusters light while all metrics end up in one place.

### Where does Tempo store traces?

Tempo stores traces in S3, so there is no heavy database to run. `tempo_retention_days` sets how long traces are kept, and Tempo reaches its bucket through an IAM role instead of static keys.

### Can a workload cluster ship logs to Loki in another VPC?

Yes. A workload cluster's Promtail can ship logs to a Loki running in a shared monitoring VPC, reached privately over the Transit Gateway from Day 2.

### When should you use the `ec2` module instead of EKS?

Only when something genuinely doesn't belong in the cluster, such as a bastion host, a license server or a legacy app that won't containerize. Every VM is something you have to patch and babysit, so the default is EKS.

---

## What's the whole-series takeaway?

- Infrastructure is **code**: versioned, reviewed, reproducible.
- It assembles in **five wired layers**, each a focused module with clear inputs and outputs.
- **OIDC/IRSA** (Day 4) is the security keystone; **outputs → inputs** is the wiring that auto-orders the build.
- One **library** powers **many** environments — so they stay identical by design and upgrade by intent, not by luck.
- The payoff: a fresh, full-stack environment is `terraform apply` **+ a coffee** away, not a two-week project — and that's the whole point of doing infrastructure *well*.

That's the build. Thanks for following along — go build something. 🚀

**But wait — there's more.** Building a platform is the easy half; running it is where the real lessons are. The next three posts are **bonus war stories**: real production incidents from running this exact platform, what the symptoms looked like, what actually caused them, and the fixes. Starting with networking in Bonus 1.

---

*Everything in this series comes from a real, production-tested module library and the per-environment stacks that consume it. Names, IDs and addresses in the examples are placeholders.*