Skip to content

ADR-016: One explicit browser TLS certificate per PR preview

Context

On 2026-08-13 every preview browser Ingress independently carried the letsencrypt-prod ingress-shim annotation and a distinct TLS Secret. The seven public preview hosts therefore produced up to seven single-SAN certificates per preview. Let’s Encrypt returned HTTP 429 after 50 certificates for syrf.org.uk had been issued in its preceding 168-hour window. PR #2809 then served the ingress controller's fake certificate for API, Project Management, and User Guide while Argo applications still reported Synced and Healthy.

The same failure class recurred on 2026-08-30 in PR #2992. Five public single-host Certificates became Ready, while Docs failed separately and the Web TLS Secret was absent after an ACME finalization race. Nginx served its built-in fake certificate for Web even though the workflow and Argo health had already reported success. This proves that application health and independent per-service issuance are not a browser-readiness contract.

The production issuer is verified HTTP-01 only. DNS-01 and wildcard issuance are not available infrastructure capabilities.

Decision

Each preview will own one explicit cert-manager.io/v1 Certificate in the preview-infrastructure chart. It has exactly these seven DNS names:

  • pr-{N}.syrf.org.uk
  • api.pr-{N}.syrf.org.uk
  • project-management.pr-{N}.syrf.org.uk
  • identity.pr-{N}.syrf.org.uk
  • quartz.pr-{N}.syrf.org.uk
  • docs.pr-{N}.syrf.org.uk
  • help.pr-{N}.syrf.org.uk

The Certificate owns deterministic Secret pr-{N}-tls. The preview infrastructure chart enables it by default with the established letsencrypt-prod ClusterIssuer. That issuer is Ready and uses the nginx HTTP-01 solver; the existing preview DNS records already route all seven names to that ingress class. This decision does not change production host Certificates, production DNS, or issuer configuration.

Every service Application already receives the authoritative numeric syrf.prNumber. The shared and bespoke Ingress templates use that preview identity to replace legacy per-service secretName inputs with pr-{N}-tls and remove cert-manager/ACME issuance annotations. This makes Certificate creation and consumption one chart-revision contract rather than a separately timed GitOps opt-in, and prevents ingress-shim from creating competing per-service Certificates. The same rule applies to optional secondary Ingresses such as Docs OAuth2 proxy and PDF Agent path routing. Multiple Ingresses may share one of the seven hosts, but every matching Ingress must use the shared Secret without issuer annotations. The Web chart's optional legacy redirect Ingress is suppressed for previews because its additional hostname is outside the exact seven-host contract. When no valid preview identity exists, staging and production rendering is unchanged.

On migration, removing an issuer annotation does not delete the Certificate previously created by ingress-shim. The readiness hook therefore deletes only legacy Certificates whose controller owner matches the name and UID of a current networking.k8s.io/v1 Ingress, target a non-shared Secret, and contain no SAN outside this preview's exact seven-host inventory. The hook first proves every Ingress has already dropped all issuer/ACME annotations, then re-lists and rejects any remaining competitor. It never selects the explicit shared Certificate, manually owned Certificates, or the unrelated OpenIddict key Certificates, and it never deletes a Secret or a preview namespace.

Preview success is an end-to-end contract, not an Argo health inference. The PostSync notifier must verify the Certificate's Ready condition, its expected Secret name and exact SAN set, absence of competing browser Certificates, exact seven-host coverage across all browser Ingress rules, and absence of issuer annotations. It then validates the Secret type/key shape, the leaf's validity and exact SAN set, trusted HTTPS for every public host, and that every endpoint serves the same leaf stored in the shared Secret. No Secret or private-key data is logged. Only after all checks pass may it mark that attempt's unique GitHub Deployment successful. The still-running workflow waits for that exact Deployment ID and is the sole writer of shared PR metadata and the preview/deploy commit status. Build completion remains awaiting TLS verification; a failed or incomplete verifier reports failure or remains pending, never Ready. Disabling TLS while the readiness notifier is enabled is a Helm render error, so there is no fake-certificate success path. Each deployment attempt first resets preview/deploy to pending even when the commit SHA is unchanged. The workflow writes the generated PR status block before GitOps activation and keeps it pending until its own attempt-scoped Deployment reports success. A cancelled or stale same-SHA cluster hook can update only its old Deployment ID, not shared readiness. Creating a build attempt immediately replaces any prior Ready PR row with attempt-scoped pending state, before versioning or image work begins. TLS polling uses a ten-minute elapsed wall-clock deadline with every Kubernetes and network operation bounded by the remaining time. Service Deployment checks likewise share one ten-minute wall-clock deadline rather than receiving ten minutes per Deployment. The hook owns all bounded retries in one Pod (backoffLimit: 0), so a terminal result cannot be contradicted by a Kubernetes Job retry.

Every GitOps sync that can disrupt a ready preview owns the same attempt contract. Metadata-only edits and /reseed-db create a fresh GitHub Deployment, reset shared readiness to pending, write that exact Deployment ID into GitOps, and remain alive until the PostSync verifier reports its terminal result. A final latest-Deployment fence prevents an older build, edit, reseed, or cancellation from overwriting a newer attempt. Closed or unlabelled PRs and inactive Deployments defer to teardown, which remains authoritative. The PR-body action revalidates that exact ownership immediately before its API write, including both the expected Deployment ID and full PR head SHA, exposes whether publication occurred, and the following commit and Deployment status writers repeat the latest-attempt check immediately before their writes. The shared commit-status publisher also rechecks after its write and repairs from the newest Deployment if ownership changed during the API call. The cluster notifier performs the same post-publication check and restores its own Deployment to inactive if teardown or a successor won the race. Build and reseed pollers detect an attempt-labelled failed notifier Job, so dependency bootstrap failure becomes terminal without consuming the full readiness budget. The build workflow gives the hook a 95-minute polling budget covering bounded dependency bootstrap, authentication/publication, and all readiness phases. The reseed path uses a 100-minute end-to-end polling budget because it begins before the Argo handoff. Database readiness uses one 45-minute elapsed deadline with bounded Kubernetes API calls and sleeps, rather than 270 potentially unbounded iterations. Continuation attempts perform the same full ownership fence before resetting shared readiness; if a successor appears during that API sequence, the stale continuation repeatedly revalidates and restores the successor's observed Deployment state to only the generated preview block and shared commit context, then inactivates itself.

This is intentionally not feature-flagged: it is a preview-only deployment safety contract with a GitOps revert boundary, not user-facing behaviour.

Consequences

  • Normal preview steady state changes from up to seven browser certificate issuances to one. Chart/workflow iteration uses renders and unit tests. A deliberately authorized trusted-chain proof may use one real preview issuance through the established issuer; it must not create extra preview Certificates or touch production hosts.
  • Wildcards are rejected: HTTP-01 cannot issue them, no DNS-01 capability is verified, and *.syrf.org.uk does not cover multi-label preview hosts.
  • Preview, staging, and production never share a browser TLS Secret. That would violate namespace/key isolation and widen renewal and credential blast radius.
  • Ordinary rebuilds reuse the deterministic Secret and Certificate. cert-manager owns renewal in place. A real namespace teardown/recreation necessarily reissues once, and readiness remains closed until the replacement Secret and all served leaves agree. Concurrent previews use disjoint PR-scoped names.
  • Rollback is a reviewed Git revert of the SyRF chart revision. It must not delete Certificates, Secrets, or previews manually and must preserve the no-premature-Ready gate.

Follow-up

Stale previews are a separate governance concern. Existing scheduled sweep paths are dry-run and their imperative mutation mode must not be enabled as a shortcut. A subsequent reviewed policy will use GitOps preview-directory reconciliation and approval-gated deletion, with an explicit lease/retention decision for old open labelled PRs.