Ingress topology is a decision, not a default: Caddy vs ALB, and how we let it change later

Every deployment tool has an opinion baked in about how traffic reaches your app. Usually it’s “there’s a load balancer,” full stop. That default costs real money for small apps and, worse, treating it as a default rather than a decision means nobody builds the machinery to change it later. We learned this the painful way, building automation that infers infrastructure from a repo and provisions it on AWS.

The $20 question #

Take the most common shape we see: a SPA frontend plus an API backend. One container serves static files, the other serves /api. On AWS, the reflexive answer is an Application Load Balancer — about $20/month base before LCUs, before you’ve served a single request. For a side project or an early-stage product running on a single ECS host, that’s often the single biggest line item on the bill, paying for capabilities (multi-target routing, health-checked autoscaling pools) the app doesn’t use yet.

The cheaper topology: run a Caddy reverse proxy as another container on the same host. Caddy terminates TLS (it does ACME automatically), serves the frontend at /, proxies /api to the backend over localhost networking. One public IP, zero ALB. Same-origin, so no CORS circus either.

But that topology has a ceiling. The moment you want multiple tasks, autoscaling, or zero-downtime deploys with target-group draining, you want the ALB: path-based listener rules sending /api* to the backend target group and everything else to the frontend, both behind one hostname with an ACM cert.

So ingress is genuinely a decision with a crossover point — and crucially, a decision that should be revisitable. An app that starts on Caddy should be able to move to an ALB without being rebuilt, and (less obviously, but it happens) move back.

The trap: topology logic trapped inside the analyze pipeline #

Our pipeline has two phases. Analyzereads the repo, classifies services (frontend, backend_api, worker…), and decides topology: should this be a Caddy host-edge, or same-origin ALB path routing? The result is frozen into per-task rows — ingress_mode, ingress_id, subdomain. Deploy reads those rows and applies them verbatim. By design, deploy does not re-think anything; re-analysis on every deploy would re-derive frameworks, env vars, and commands the user may have hand-edited.

The topology decision lived inline inside analyze’s task-enrichment step — about 80 lines deep in a function doing a dozen other things. Which meant:

Changing topology required a full re-analyze. A user flips the project from Caddy to ALB in settings, hits deploy, and deploy faithfully re-applies the stored rows: frontend now points at the new ALB, but the API task is still ingress_mode=none(it used to be reached through Caddy), and the now-unrouted Caddy container is still in the task list. We shipped exactly this hybrid to a real project: frontend on the ALB, API orphaned, a zombie Caddy service running and routing nothing. The logs were damning in their simplicity — deploy resolved the stored tasks and applied; there is no enrichment step in that path, so there was no code that could have fixed it.

Two code paths started to drift.The obvious patch — copy a bit of topology logic into the deploy path — is how you end up with two almost-identical decision procedures that disagree in edge cases. We could see that future and didn’t want it.

The fix: one pure function, multiple callers #

We extracted the entire decision into a single pure, idempotent function:

def reconcile_ingress_topology(
    tasks: list[dict],
    *,
    is_aws: bool,
    is_single_host_aws: bool,
    prefer_public_ip: bool,
    alb_ingress_rows: list[dict],
    caddy_ingress_id: str | None,
    force: bool = False,
) -> IngressTopologyResult:
    """Pick Caddy vs ALB vs none for the SPA+API pair and mutate
    ingress_mode / ingress_id / subdomain in place. No-op if already correct."""

Three properties were non-negotiable:

Pure. No I/O. Ensuring the Caddy ingress row exists in the database is async and project-specific, so the caller does it and passes caddy_ingress_idin. If the Caddy branch is selected but no id was supplied, the function falls through to the ALB branch. This makes the thing unit-testable with zero mocks — our tests are literally “here’s the task list from the real incident, run the function, assert the rows.”

Idempotent.Running it on already-correct tasks changes nothing. That property is what makes it safe to call from places that don’t know whether reconciliation is needed.

One implementation, every caller. Three call sites now share it:

1. Analyzecalls it during enrichment, exactly as before (behavior-preserving refactor — the existing test suite had to pass unchanged).
2. A dedicated ingress-migration task, triggered over NATS when the user changes the ingress kind in settings. It loads the storedtask rows — no framework or command re-derivation, so it’s milliseconds, not a full analyze — reconciles, persists only the ingress columns, and deletes orphaned ingress rows of the losing kind so a later switch-back can’t re-bind to a stale edge. It does not auto-deploy; the user sees the diff and deploys when ready.
3. A deploy-time guardthat runs the same function in-memory before generating Terraform. Because it’s idempotent, this is free when topology is already correct, and it means a stale hybrid structurally cannot reach apply.

One subtlety in the force flag: when a user hand-edits tasks in the architecture editor, rows get marked source='user'and the topology functions normally refuse to touch them — user intent wins. But an ingress switch is user intent, so the migration consumer passes force=True, while the deploy guard keeps force=False so a routine deploy never clobbers a deliberate per-task edit. Same function, calibrated authority per caller.

Derive, don't store #

The second bug class was subtler. The ALB topology mounts the API at /api — a value, edge_path_prefix, that we originally computed during analyze and carried along with the task. But it wasn’t a persisted column, so it evaporated on the analyze→DB→deploy round-trip: even a correct re-analyze would lose the /api mount by the time Terraform variables were built.

The fix was not “add a column.” The fix was to derive it at deploy time from the service classification: a backend_api task in an attached same-origin topology gets /api, period. Same for ingress_kind— derived from the ingress row the task points to, not stored next to it. A derived value can’t go stale, can’t disagree with its source, and doesn’t need a migration when the rule changes.

That rule generalizes: persist decisions, derive everything decidable from them.The decision “this project uses ALB same-origin routing, API attached to ingress X under subdomain Y” is state worth storing. The path prefix is a consequence; storing consequences is how you manufacture split-brain.

The general lesson #

The first version’s deep flaw wasn’t the Caddy/ALB logic — that was fine. It was where the logic lived: embedded in one phase of a pipeline, so the decision could only be revisited by re-running that whole phase. The moment topology became a user-changeable setting, that architecture was wrong.

If you’re building infrastructure automation: make every topology decision a pure function of current state — current tasks, current ingress rows, current capacity config — and let anything that needs the decision call that function. Reconciliation stops being a special recovery mode and becomes the only mode. Kubernetes controllers figured this out years ago; it took shipping a Caddy/ALB chimera to a real customer project for us to apply it one level up the stack.

This is from building Vylara — connect your repo, and it figures out and provisions the infrastructure in your own AWS account.