Day 8 (Finale) — Observability, the ec2 Module & One-Command Environments

Series: Zero to Production on AWS EKS with Terraform — Part 10 · Day 8 (finale) Layer 5 of 5 — observability — plus the wrap-up. Add-ons of the day: Promtail, Loki + Grafana, Prometheus agent, Tempo; module: ec2.

Last updated: 26 October 2026

In short: Observability on EKS means collecting logs, metrics and traces. Here Promtail ships logs to Loki, a Prometheus agent remote-writes metrics to a central Prometheus, Tempo stores traces in S3, and Grafana shows all three. All of it is installed by one Terraform eks-addon call, and the whole platform then rebuilds a full environment with one terraform apply.

We've built everything that runs apps. The last layer is the one that lets you sleep at night: knowing what's happening inside the cluster, and seeing problems before your users do.

Then we tie the whole 8-day journey together: one library, one command, a full environment.

Part 1 — The observability stack

Observability has three pillars: logs (what happened), metrics (how much / how fast) and traces (where the time went inside one request). Our eks-addon ships all three.

What does Promtail do in an EKS cluster?

Promtail collects every container's logs on every node and ships them to Loki. Promtail runs as a DaemonSet (one pod on every node). It tails the logs of every container, labels them (namespace, pod, app), and ships them to Loki. Think of it as a courier on every machine, collecting mail and forwarding it to the central post office.

Note the cross-cluster wiring: a workload cluster's Promtail can ship to a Loki running in the shared monitoring VPC — reachable privately thanks to the Transit Gateway from Day 2.

What do Loki and Grafana do?

Loki stores and indexes the logs; Grafana is where you search them, chart them and alert on them.

• Loki stores logs cheaply and makes them queryable (LogQL). It's like a search engine for your logs — and it backs its storage with S3 and uses the gp3 PVCs from Day 5's ebs-csi.
• Grafana is the dashboard layer — live charts, log explorers, and alerts. It's the single pane of glass where on-call engineers actually look.

What is Prometheus agent mode?

Prometheus agent mode collects metrics and forwards them without storing them locally. Prometheus agent scrapes metrics (CPU, memory, request rates, custom app metrics) and remote-writes them to a central Prometheus/store. "Agent mode" means it doesn't store locally — it just collects and forwards, keeping workload clusters light.

What is Tempo used for in distributed tracing?

Tempo stores distributed traces in S3 so you can see where the time went inside one request.

Logs tell you that a request failed; metrics tell you how often. Traces tell you where: one request is followed across every service it touches, with the time spent in each hop. When checkout is slow, a trace shows you it's the third downstream call, not the database you were about to blame.

Tempo stores those traces cheaply in S3 (no heavy database to run) and Grafana shows them next to the logs and metrics for the same moment. Your apps send spans with OpenTelemetry; Tempo keeps them for as long as you choose.

In practice, tracing is what cut our troubleshooting time the most: instead of grepping logs across five services, you open one trace and see the slow hop.

How does observability data flow from the cluster to Grafana?

Logs, metrics and traces each take their own path, and all three end up in Grafana.

  Every node ──Promtail──► Loki ──► Grafana (logs + dashboards, behind ACM https)
  Every node ──Prometheus agent ──remote-write──► central Prometheus ──► Grafana (metrics)
  Every app  ──OpenTelemetry spans──────────────► Tempo (S3) ──────────► Grafana (traces)
                         │
        (cross-VPC ships travel privately over the Day-2 Transit Gateway)

enable_promtail        = true
loki_host              = "loki.monitoring.internal"
loki_port              = 3100

enable_loki_grafana       = true
grafana_admin_password    = var.grafana_admin_password   # from a secret
loki_acm_certificate_arn  = module.acm.certificate_arn
loki_s3_bucket_name       = "prod-loki-logs"
loki_s3_region            = "us-east-1"

enable_prometheus_agent     = true
prometheus_remote_write_url = "https://prometheus.monitoring.internal/api/v1/write"

enable_tempo         = true
tempo_retention_days = 14

Which optional observability add-ons are in the eks-addon bundle?

The same eks-addon call carries a few optional extras, each behind its own enable_* flag:

• Alerting — Alertmanager routes alerts to Slack, split by severity.
• Robusta — enriches alerts with pod logs and events, so the Slack message already contains the first clue.
• OpenCost — cost per namespace and workload, so "who is spending what" has an answer.
• Blackbox exporter — probes your public URLs and TCP endpoints from the outside, like a user would.
• CloudWatch exporter — pulls AWS-side metrics (load balancers, managed databases) into the same Grafana.

Part 2 — the ec2 module (the occasional exception)

The ec2 module creates plain virtual machines for the few things that don't belong on EKS. 99% of our compute is pods on EKS. But sometimes you need a plain virtual machine — a bastion/jump host, a one-off tool, a license server, a legacy thing that won't containerize. That's the ec2 module.

It's map-driven like security-group and alb: describe a map of instances, get them all with consistent tags. Subnets and security groups are passed in as lookup maps, so each instance refers to them by a friendly name instead of a raw ID.

Outputs: instance_ids, private_ips.

module "ec2" {
  source = "git::https://github.com/<org>/devops-terraform-modules.git//modules/ec2?ref=v1.4.0"

  subnet_ids         = { private = module.vpc.private_subnets }
  security_group_ids = module.security_group.ids

  ec2_instances = {
    bastion = {
      ami                  = data.aws_ami.al2023.id
      instance_type        = "t3.micro"
      subnet               = "private"
      security_groups      = ["bastion"]
      iam_instance_profile = "ssm-managed-instance"   # connect with SSM Session Manager, no SSH key
      root_block_device    = { volume_type = "gp3", volume_size = 20, encrypted = true }
    }
  }

  tags = { Project = "platform", Role = "bastion" }
}

Default to EKS. Reach for ec2 only when something genuinely doesn't belong in the cluster. Every VM is a thing you have to patch and babysit — the opposite of why we built all this.

Part 3 — the finale: it all comes together

Eight days, a dozen-plus modules, five layers. Here's the whole platform on one page:

   ┌──────────────────────────────────────────────────────────────┐
   │  devops-terraform-modules  (one versioned library)             │
   └──────────────────────────────────────────────────────────────┘
        Day 1  vpc ............... private network, subnets, endpoints
        Day 2  security-group .... firewalls
               tgw / tgw-attach .. private bridges between VPCs
        Day 3  iam-policy ........ least-privilege permissions
               ecr ............... container image registries
        Day 4  eks ............... Kubernetes control plane + OIDC/IRSA
        Day 5  ebs-csi ........... durable storage (gp3 default)
               acm ............... TLS certificates (the 🔒)
               alb ............... infra-level LBs
               + ALB controller .. Ingress → real load balancers
        Day 6  external-secrets .. AWS secrets → K8s, no value in Git
               karpenter ......... right-sized nodes on demand
        Day 7  argocd ........... GitOps delivery (Git = truth)
               github-runner ..... CI/CD inside the cluster
        Day 8  promtail/loki/
               grafana/prometheus/
               tempo ............. logs, metrics, traces, dashboards, alerts
               ec2 .............. the occasional plain VM

How do you spin up a new environment with one command?

You copy a consumer stack, edit a small tfvars, and run terraform apply.

Because every layer is a module wired by outputs → inputs, spinning up a brand-new environment (a new dev, staging, or whole new product) is:

1. Copy a consumer-stack folder, point its backend at a fresh state key.
2. Edit a small tfvars: name, region, CIDR, sizes, hostnames.
3. terraform init → pulls the pinned module versions.
4. terraform plan → read the diff.
5. terraform apply → ☕ (~15–20 min, mostly AWS building the EKS control plane).
6. A full, production-grade environment is alive — network, cluster, storage, certs, secrets, autoscaling, GitOps, CI, and observability — identical to its siblings because it came from the identical modules at the identical version.

No console clicking. No drift. A security fix is one PR to the library + a version bump in each stack — and every environment inherits it.

Frequently asked questions

What are the three pillars of observability?

The three pillars are logs (what happened), metrics (how much and how fast) and traces (where the time went inside one request). In this platform Loki stores logs, a central Prometheus stores metrics and Tempo stores traces, and Grafana shows all three.

Why run Prometheus in agent mode on workload clusters?

Agent mode doesn't store metrics locally; it scrapes them and remote-writes them to a central Prometheus. That keeps workload clusters light while all metrics end up in one place.

Where does Tempo store traces?

Tempo stores traces in S3, so there is no heavy database to run. tempo_retention_days sets how long traces are kept, and Tempo reaches its bucket through an IAM role instead of static keys.

Can a workload cluster ship logs to Loki in another VPC?

Yes. A workload cluster's Promtail can ship logs to a Loki running in a shared monitoring VPC, reached privately over the Transit Gateway from Day 2.

When should you use the ec2 module instead of EKS?

Only when something genuinely doesn't belong in the cluster, such as a bastion host, a license server or a legacy app that won't containerize. Every VM is something you have to patch and babysit, so the default is EKS.

What's the whole-series takeaway?

• Infrastructure is code: versioned, reviewed, reproducible.
• It assembles in five wired layers, each a focused module with clear inputs and outputs.
• OIDC/IRSA (Day 4) is the security keystone; outputs → inputs is the wiring that auto-orders the build.
• One library powers many environments — so they stay identical by design and upgrade by intent, not by luck.
• The payoff: a fresh, full-stack environment is terraform apply + a coffee away, not a two-week project — and that's the whole point of doing infrastructure well.

That's the build. Thanks for following along — go build something. 🚀

But wait — there's more. Building a platform is the easy half; running it is where the real lessons are. The next three posts are bonus war stories: real production incidents from running this exact platform, what the symptoms looked like, what actually caused them, and the fixes. Starting with networking in Bonus 1.

Everything in this series comes from a real, production-tested module library and the per-environment stacks that consume it. Names, IDs and addresses in the examples are placeholders.