Bulk PDF Agent ARRNC Hosting Correction — Migration Plan¶

Design: 2026-08-30-bulk-pdf-agent-arrnc-hosting-design.md. Decision: ADR-015.

This is the executable, multi-repository Phase 2 plan. Phase 1 is documentation only and stops at Gate A for explicit architecture approval. No PR below is authorized by the existence of this plan.

1. Safety invariants¶

Every implementation PR and operation preserves these invariants:

Production is untouched until a distinct promotion plan, review, and Gate C approval.
The Kubernetes Application, ingress, PVC, and reviewed rollback configuration stay intact until ARRNC staging has passed authenticated end-to-end proof. Its Deployment may scale to zero only after the Gate B intake/queue/claim/storage drain; no resource is removed at that cutover.
A queue has at most one active environment consumer. There is no dual-consumer soak.
An ordinary agent sees one exact storage child and no SMB credential or sibling path.
PR code receives no production/staging credential, mount, network, broker account, or Docker control surface.
PR-controlled jobs receive no ARRNC activation identity; privileged activation runs only from protected base-branch code and never executes the PR checkout.
Desired state is declarative. Imperative tools may execute an approved record but do not become a second authority.
All destructive cleanup is canonical-child-scoped, no-follow, idempotent, and evidenced.
A failed mount, sentinel, provenance, capacity, queue-ownership, or restore proof is a STOP, never permission to weaken the contract.
Secrets and participant/project/file identifiers are absent from PRs and proof bundles.
Each live apply has a named operator, rollback owner, observation window, and explicit gate.

2. Dependency graph and Phase 2 waves¶

flowchart TD
    A[Gate A<br/>approve and merge planning PR] --> W1A
    A --> W1B
    A --> W1B0
    W1A[Wave 1A server-config<br/>storage + persistent staging] --> W2
    W1B0[Wave 1B0 staging IAM<br/>existing notifier boundary only] --> W3
    W1B[Wave 1B SyRF<br/>runtime + identity contracts] --> W2
    W1A --> W1C[Wave 1C registrar + gatekeeper<br/>preview family and exact-PR IAM]
    W1B --> W1C
    W1B0 --> W1C
    W1C --> W2[Wave 2<br/>preview lifecycle automation]
    W1A --> W3[Wave 3<br/>ARRNC staging dry-run, paused]
    W1B --> W3
    W3 --> B{Gate B<br/>approve live staging cutover}
    B --> W4[Wave 4<br/>single-owner staging cutover + E2E]
    W4 --> W5[Wave 5<br/>Kubernetes drain/retirement + doc cleanup]
    W2 --> W6[Wave 6<br/>preview acceptance]
    W4 --> W6
    W5 --> C{Gate C later<br/>production promotion approval}
    W6 --> C

Wave 1 repository PRs may be developed in parallel, but their contracts are merged in the dependency order shown. Wave 3 is the first ARRNC environment mutation and requires reviewed implementation plus an authorized operator. Wave 4 cannot start without Gate B.

The reduced staging critical path is Gate A → Waves 1A, 1B0 and 1B → Wave 3 → Gate B → Wave 4. Wave 1B0 is staging-only: it extends the already-required notifier permissions boundary for exact staging resources and grants no preview-family or production authority. Waves 1C and 2 prepare previews in parallel; Wave 1C owns every preview IAM grant, depends on the reviewed staging boundary shape rather than inheriting its resources, and approved preview activation still waits for successful staging proof in Wave 4. No notifier-network-infrastructure wave is required. Wave 5 remains after staging proof, and production remains outside this programme until Gate C.

Wave	Repository PRs	Outcome
1A	server-config	exact staging child mount, isolated network/scanner, paused persistent slot, quotas/alerts/verification
1B0	camarades-infrastructure	add only exact staging Bulk PDF grants plus exact staging log and legacy `ReferenceUpload`/`ReferenceUpdate` read replacements to the existing `SyRFS3NotifierLambdaBoundary`; preserve shared wildcard caps until exact-N cutover proves zero legacy consumers, then retire them in Wave 1C; add no preview Bulk PDF, production, VPC, DNS, endpoint, ENI, or egress authority
1B	syrf	immutable-digest ARRNC deployment inputs, parity health/config, queue-owner controls, notifier release gate, dedicated issuer client/scope, exact-SHA credentialed-artifact gate, authenticated proof harness
1C	server-config, camarades-infrastructure for protected IAM bootstrap/guardrails, arrnc-api-deploy only where transport changes are required, and university storage owner	bounded dynamic preview-family registrar, protected canonical-PR identity/provenance, exact per-PR IAM, per-PR storage/allocation, rootless container isolation, atomic fleet admission, and exact gatekeeper execution contract
2	syrf plus server-config	approved-PR activation/teardown, digest provenance, configurable concurrent capacity, queueing, and orphan reconciliation
3	server-config operational apply	ARRNC staging reconciled paused; mount/ACL/ClamAV/broker/rollback dry-run evidence
4	cluster-gitops plus controlled ARRNC operation	intake freeze, queue/claim drain, Kubernetes scale-to-zero, ARRNC sole consumer, authenticated E2E and soak
5	cluster-gitops then syrf	snapshot-backed Application/ingress/PVC retirement; remove obsolete chart/workflow/docs authority
6	syrf plus server-config operational proof	multiple concurrent previews from approved PRs, isolation/E2E, independent cancellation/teardown, orphan recovery
future	separately planned multi-repo production promotion	no action until Gate C

3. Numbered migration plan¶

Step 1 — Lock the cross-repository contract¶

Owner: SyRF and ARRNC platform maintainers.

Before implementation branches diverge, add shared test vectors/fixtures for:

canonical roots production, staging, previews/pr-N and derived container names;
invalid zero, zero-padded, negative, overflow, traversal, separator, Unicode, and sibling inputs;
persistent versus preview secret/network/storage profiles;
paused/enabled consumer states;
image repository, digest, source commit, PR, and trust provenance;
teardown state-machine idempotency.

The fixtures contain no credentials or live paths beyond approved non-secret mount names. Each repository consumes the same cases or an exact generated copy with checksum.

Proof: contract suites in all affected repositories agree. Rollback: test-only; no environment effect. STOP: any repository needs a different canonical root or container naming rule.

Step 2 — Provision child-scoped storage contracts in server-config¶

Repository: camaradesuk/server-config. Owner: ARRNC platform.

Expected files:

vars/projects.yml;
roles/cifs_mounts/defaults/main.yml, tasks/main.yml, handlers/main.yml, and restart unit template;
roles/docker_volumes/tasks/main.yml;
roles/verify/tasks/check_cifs_mounts.yml, check_cifs_restart_units.yml, check_volumes.yml, and a focused PDF-agent verification task;
filter_plugins/tests/test_syrf_pdf_agent_hosting.py.

Preview-storage implementation dependency and owner:

the university Camarades storage owner must provide, before preview activation, an audited operation or isolated allocation pool that accepts only canonical PR number and idempotency key, then atomically creates/allocates the exact previews/pr-N child with its distinct principal, ACL, quota, immutable allocation record, and server-side connection/bandwidth/IOPS limits, and retires only that child after fencing and unmount. It must not preseed the ARRNC readiness sentinel;
its scoped capability remains in the ARRNC root-owned control plane, never GitHub, SyRF workflows, or preview code. A pre-provisioned pool is acceptable when allocation, reset/fencing, identity rotation, and audit prevent prior or sibling PR exposure. This missing automation is an implementation dependency to solve, not an alternative-hosting decision or a blocker for the persistent staging PR.

Implement declarative staging and preview-family storage templates:

create exact host roots for staging and previews/pr-N scratch/output;
require direct child SMB shares and distinct root-owned credential scopes;
pin UID/GID and least-privilege ACLs;
verify the immutable storage-owner allocation record, then create the sole structured ARRNC readiness sentinel only after active mount/source/ACL/quota proof;
add per-child byte/inode quota and ARRNC aggregate capacity preflight;
restart only the matching stopped/failed environment container after mount recovery;
prohibit the common Camarades root, path traversal, sibling access, symlink/reparse-point targets, local mount underlays, and any preview deletion target outside previews/pr-N;
record that production is unchanged and the ARRNC host backup excludes its mount; defer the remote-share backup attestation/restore proof to Gate C; retain staging-delivered bytes while any durable reference can use them, applying 30 days only to proven unreferenced/orphan test data after application-state retirement; set the preview no-backup/24-hour teardown SLO.
verify the atomic storage-owner allocation/audit record, exact principal/ACL/quota, and server-side connection/bandwidth/IOPS limits before mounting a preview child; reject a preseeded/stale readiness sentinel and create the ARRNC-owned sentinel only after mount proof.

Tests:

Ansible syntax and check mode;
negative structural tests for broad share roots, reused credentials, missing allocation record, missing quota/readiness sentinel, preseeded or stale readiness sentinel, wrong UID, cross-environment restart, and unsafe cleanup paths;
Molecule/rehearsal-host mount-loss and local-underlay refusal where available;
external ACL tests proving each identity can access only its child.
simultaneous bounded preview I/O tests that create/delete/rename/fsync and prove per-preview and aggregate connection/bandwidth/IOPS limits preserve production and staging service health.

Proof before apply: university SMB export accepts each exact child; distinct ACLs deny siblings; quota and backup classifications are approved. Rollback: remove only the new inactive staging/preview declarations; production diff must be empty. STOP before preview activation: an exact child/allocation, distinct identity, ACL, quota, I/O limits, reset/fencing, and audit cannot be delivered without broader authority. Persistent staging implementation and proof continue independently.

Step 3 — Add the persistent staging runtime to server-config¶

Repository: camaradesuk/server-config. Owner: ARRNC platform.

Expected files:

vars/projects.yml;
vars/portal-env-overrides.yaml;
deployments/arrnc-api.yml;
roles/clamav;
roles/gatekeeper/templates/services.conf.j2 and container.env.j2;
roles/verify/tasks/check_clamav.yml, check_gatekeeper.yml, check_containers.yml;
config-service record/reconcile tests.

Declare syrf-pdf-agent-staging and its scanner/network with:

the same agent UID, /app/data/output and /app/data/scratch contract;
a staging-only broker secret/vhost/queue, storage values, and exact PublicBaseUrl derived from a separately rendered, deployment-owned TrustedExistingPublicBase plus the stable staging/ child. That independent authority fixes the scheme, host, effective port, and already-served root path and is not inferred from PublicBaseUrl or supplied by an application/PR caller;
immutable image/digest, resource/PID/log/scratch limits, restart policy, and structured health;
managed/gated/paused state that cannot consume during the dry run;
separate network and scanner from production;
no changes to syrf-pdf-agent-production.

Tests assert manifest exhaustiveness, env completeness, exact-root/URL/trusted-base pairing, identical independently rendered trusted-base authority in API, Project Management, and registrar, rejection of wrong scheme/host/effective-port/base-path or arbitrary URL prefixes, immutable digest, no secret values in rendered output, isolated network, no ports/privilege/Docker socket, exact volume, paused consumer, and production byte-for-byte semantic stability.

Proof: a no-op production diff and a staging-only check-mode plan. Rollback: remove the inactive staging desired state, including its derived URL, and its scanner/network if unused. STOP: config-service cannot represent paused consumption independently of container readiness.

Before activation, the serving/storage owner proves that the existing Camarades public static root continues to serve the untouched production/ child, serves the stable staging/ child and lifecycle-owned previews/pr-N/ children, and denies directory listing, scratch, partial/quarantined files, sentinels, and unscanned content. Cross-child denial applies to storage/mount traversal and application link derivation, not to a public client that already knows a valid delivered URL under another child. “Sibling denial” never means removing the existing production public child or inventing an authenticated serving boundary. This wave has no authority to create or change IIS bindings, virtual directories, hostnames, certificates, or other IIS state. If content/mount ownership beneath the existing root cannot meet the contract, STOP for architecture review; do not restore the agent-chart static server.

Step 4 — Complete the IAM cap, then SyRF's runtime, identity, and notifier contract¶

Repositories: camaradesuk/camarades-infrastructure, camaradesuk/syrf, and lifecycle rendering in camaradesuk/cluster-gitops. Owners: the existing notifier permissions-boundary owner, SyRF maintainers, SyRF identity owner/Auth0 tenant administrator, environment secret owner, the AWS bucket/policy owner, and GKE platform maintainers. The narrow infrastructure PR must merge before either new Lambda role is activated; SyRF implementation may be reviewed in parallel.

Expected files:

src/services/pdf-agent/SyRF.PdfAgent/Program.cs and Configuration/PdfAgentOptions.cs;
storage, clamd-readiness, broker-binding, lifecycle, and health tests under src/services/pdf-agent/SyRF.PdfAgent.Tests;
src/services/pdf-agent/SyRF.PdfAgent/Dockerfile and appsettings.json;
.github/workflows/ci-cd.yml and reusable workflow/scripts;
src/libs/kernel/SyRF.SharedKernel/Settings/BulkPdfStorageSettings.cs and focused settings tests;
shared environment mapping for BulkPdfStorageSettings and exact public child URLs;
deletion-independent Project Management BulkPdfUploadStorageBinding, release/cleanup-authority, and BulkPdfUploadQuarantineFence models, repositories, and indexes under src/libs/project-management/SyRF.ProjectManagement.Core and SyRF.ProjectManagement.Mongo.Data/Repositories, storing immutable environment/job/bucket/key/version/size/hash/metadata outside Project BSON. The server-created binding is the pre-quarantine deletion fence from completed-object transition until cleanup disposition; the candidate-Project fence set then survives any prepared or finalized transfer;
src/services/project-management/SyRF.ProjectManagement.Endpoint/Consumers/ object-ready, cleanup-registration, storage-binding-resolution, quarantine-fence registration/transfer, hold-retired-confirmation, receipt-drain reconciliation, and claim consumers that persist/read the release record before command acknowledgement and on every claim, plus their request/response contracts in src/libs/project-management/SyRF.ProjectManagement.Messages/ and SyRF.ProjectManagement.Endpoint.Tests durability/retry, transfer-boundary, and deletion-fence tests;
BulkPdfMultipartSweepProcessor.cs, BulkPdfUploadController.CancelUpload, and focused tests replacing both completed-object DeleteObjectAsync paths for cancellation/abandonment races with terminal cleanup-pending handoff to durable notifier capture and the exact-version reclaimer;
ProjectManagementService.DeleteProjectAsync, ProjectRepository, and their tests enforcing the nonterminal-session/job, terminal-cleanup, and active-release-record deletion fences, cleanup-ACK conversion of the separate release record to a receipt without removing embedded terminal history, transactional ordinary history-pruning handoff, and fail-closed Project deletion/history pruning while any candidate-bound quarantine fence or transfer remains active;
BulkPdfUploadController/new-session admission configuration and active-session tests in the PM API, so maintenance can reject initiation without disabling completion/cancellation endpoints.
dedicated notifier release-state and no-job-identity storage-binding resolver read routes, plus quarantine-fence registration/transfer, cleanup-acknowledgement, and hold-retired-confirmation write routes, in src/services/api/SyRF.API.Endpoint/Controllers/BulkPdfUploadController.cs (or a focused controller) and exact-route, retry, transfer-receipt, and authorization/integration tests in SyRF.API.Endpoint.Tests;
src/services/api/SyRF.API.Endpoint/Auth/DirectBearerSchemeSelector.cs, API authentication registration, and DirectBearerSchemeSelectorTests.cs so each exact notifier scope and audience can authenticate only its dedicated route without acquiring syrf_api;
BulkPdfLinkService and API authorization tests proving an approved preview resolves only its derived public child prefix while ordinary preview users cannot initiate the synthetic harness;
the S3 notifier's durable hold-store/release-gate boundary and its object-version/job reconciliation tests, so preview notification cannot outrun the committed Uploaded job;
src/services/s3-notifier/SyRF.S3FileSavedNotifier.Endpoint handlers/repository and tests;
src/services/s3-notifier/.chart/templates ACK resources for the environment-scoped DynamoDB hold/retirement-outbox table, reconciliation Lambda/EventBridge schedule, least-privilege IAM, and configuration; the scheduled reconciler owns both release publication and the terminal exact-version reclamation phase described below;
camarades-infrastructure/terraform/lambda/ack-iam.tf, which owns SyRFS3NotifierLambdaBoundary, for the staging-only IAM Wave 1B0 cap described below; terraform fmt, terraform validate, and a reviewed plan must prove no unrelated IAM or network change;
src/services/s3-notifier/.chart/templates/env-vars-job.yaml for the complete Lambda environment, including the secret-backed notifier machine identity and exact HTTPS release-state, storage-binding-resolver, quarantine-fence, cleanup-acknowledgement, and hold-retired-confirmation endpoints;
src/services/s3-notifier/.chart/values.yaml, src/services/s3-notifier/template-local.yaml, .github/workflows/ci-cd.yml, and preview/staging values rendering for store/schedule settings.
src/services/identity/SyRF.Identity.Endpoint/Program.cs, Services/OpenIddictClientSeeder.cs, Controllers/AuthorizationController.cs, appsettings.json, SyRF.Identity.Endpoint.Tests/OpenIddictClientSeederTests.cs, and AuthorizationControllerTests.cs for the notifier-only client, release-state/resolver read and cleanup/retirement write admission under exactly two scopes, exact route admission, and client-credential resource/audience mapping;
.github/workflows/pr-preview.yml plus protected activation/cleanup workflow and scripts for an immutable package object-version/SHA-256 approval binding covering both notifier functions and for moving exact preview Mongo database deletion after cleanup acknowledgement, zero active release records, zero hold/outbox rows, notifier shutdown, retention/revocation of the protected synthetic-harness cleanup identity/path, and safe purge of every retained receipt;
src/charts/syrf-common/env-mapping.yaml, the Project Management chart/environment values, and the AWS principal behind the referenced aws-s3 secret. Wave 1 must trace that principal to its owning policy; if it is currently unmanaged, the named AWS account owner must record and review the minimum environment bucket/prefix grant before activation. That policy adds exact-prefix s3:GetObjectVersion only for the version-specific release HEAD and version-pinned agent-claim capability, plus strongly consistent dynamodb:GetItem only on the exact environment hold/outbox table for receipt drain proof. Git contains references, never keys;
cluster-gitops/syrf/services/s3-notifier/values.yaml, cluster-gitops/syrf/environments/staging/s3-notifier/values.yaml, and cluster-gitops/syrf/environments/preview/services/s3-notifier/values.yaml for the validated terminal-payload-retention policy and removal of age-based completed Bulk PDF expiration in staging/previews, retaining only the separate incomplete-MPU safety net. Completed-version expiry is suspended and its rendered absence proved before durable hold capture is enabled in that environment. Production lifecycle values are unchanged before Gate C.

Implement only differences necessary for one ARRNC runtime contract:

explicit consumer paused/enabled desired state with no queue subscription while paused;
health output for root/sentinel/mount capability, clamd/signature age, broker identity, consumer ownership, scratch/free space, and journal recovery;
orderly fence/drain/stop and idempotent restart behavior;
immutable digest deployment input and previous-digest rollback;
environment-scoped queue/user/vhost validation and exact root propagation;
resource-compatible read-only/non-root container behavior;
authenticated staging E2E harness using synthetic project/file names and redacted proof output.
a maintenance admission gate that rejects only new multipart sessions while already accepted sessions retain signing, acknowledgement, heartbeat, completion, cancellation, and abandonment until they quiesce.
a trusted, durable notifier release gate keyed by environment, immutable object identity/version, and job ID. The S3 invocation conditionally writes the validated release envelope, including bucket, key, and S3 version ID but no presigned URL, to an environment-scoped DynamoDB hold table before attempting publication. A separately named reconciliation Lambda, invoked on a bounded EventBridge schedule and manually invocable only by the existing protected deploy/operator role, leases due rows and calls a dedicated read-only endpoint on the existing authenticated SyRF API boundary. A notifier-only machine credential and scope may read only the minimum environment/job/object-key state: committed Uploaded for release candidates, and terminal/nonterminal disposition plus active/retriable-claim presence for any held cleanup candidate, including a row that becomes terminal before publication. It returns no S3 version or URL. The endpoint is backed by the Project Management job store; the Lambdas get no Mongo connection and the split API/PM rollout adds no object-version field to Project BSON. Instead, after exact version-specific HEAD and before acknowledging the object-ready command, Project Management idempotently persists a separate BulkPdfUploadReleaseRecord keyed by environment/job/stable-message identity with the immutable bucket/key/version/size/hash/metadata binding and release state. This registration is a reversible pre-claim handoff: the Project job remains cancellable until the agent claim atomically persists its processing claim token. If cancellation wins, a later claim emits no capability and one Project Management transaction converts the release to CleanupOnly while terminalizing the job. If the claim wins, later cancellation returns processing_already_started. Only after the accepted claim commit does Project Management repeat exact HEAD and mint a fresh version-pinned URL. A retry after claim commit reuses the stable claim and remints the URL without double- processing; cooperative mid-processing cancellation is out of scope. The record contains no URL and survives bounded Project history independently. For a terminal-before- publication hold, the reconciler emits no agent work. It instead sends a stable IBulkPdfUploadCleanupRegistrationCommand with the same immutable binding; Project Management validates the live terminal job and exact version-specific HEAD, then idempotently persists a non-claimable CleanupOnly record before acknowledging registration. Reclamation cannot start until either a processing-capable or cleanup-only record is durable.

The existing env-vars-job.yaml must either target both the S3 event-handler function and the separately named reconciler with their respective complete environment replacements, or a dedicated reconciler configuration job must do so. The event-handler replacement contains only its event identity, exact hold-table/object-prefix configuration, publication-pause state, and existing broker dispatch configuration; it contains no release-state or cleanup-acknowledgement endpoint, issuer, token endpoint, audience, client ID, or client secret. Only the reconciler replacement renders the exact API release-state, storage-binding-resolver, quarantine-fence, cleanup-acknowledgement, and hold-retired-confirmation endpoints plus the protected issuer, token endpoint, audience, client ID, and secret-backed client secret.

The selected issuer owns a dedicated confidential notifier client per environment with client credentials and exactly two constant scopes: release-state read and cleanup-acknowledgement write. The read scope authorizes only the job-bound release-state route and the separate no-job-identity storage-binding resolver route. The latter accepts only the authenticated client's mapped environment plus bucket/key/version, queries only server-created multipart bindings, and returns the bounded zero/one/multiple result needed for quarantine handling; it cannot enumerate Projects, jobs, buckets, or sibling prefixes. The write scope authorizes only idempotent quarantine-fence registration/transfer, cleanup acknowledgement, and subsequent hold-retired confirmation for the same environment/object binding and immutable resolver, transfer, or delete/ack digest. It grants no general Project or job mutation. The API configuration maps each authenticated client identity (client_id/azp, plus subject where available) to exactly one environment and fails closed when a valid token is presented to any other environment's route. OpenIddict registers the scope in Program.cs and creates/updates the client in OpenIddictClientSeeder; the tests prove it cannot mint syrf_api, admin, provisioning, or user scopes. If the selected environment issuer is Auth0, the named tenant administrator creates the equivalent dedicated M2M client and exact API permissions before activation. The environment secret owner records only a secret reference in deployment configuration. Rotation creates a next-generation client with the same two scopes, moves only the reconciler to its ID/secret, proves both read routes and all three cleanup-write operations, and only then revokes/removes the old client. Preview teardown keeps the client admitted and usable by the reconciler until zero active holds, zero QuarantinedHold items, zero retirement outboxes, and zero active release records are proved, unless every quarantine has a fully finalized transfer whose PM-link, source-tombstone, and active-destination receipts all verify. Only then does it deny the preview client ID at the API, revoke issuer token entries where supported, disable/delete the client, and keep the API fence until the maximum lifetime of any already-issued self-contained token has elapsed. Only then may it claim credential absence. The preview cleanup debit and dynamic cleanup record remain charged, without occupying an agent container slot, until the later receipt-retention and database-decision gate completes. A preview client is unique to its PR environment and never shares staging or production credentials. Production identity configuration is unchanged before Gate C.

OpenIddict token issuance and API admission are one contract. Identity maps both notifier scopes to the exact API/introspection audience in AuthorizationController.SetClientCredentialResources. The API bearer selector/validator accepts that audience plus the release-state read scope only on the dedicated job-state and storage-binding-resolver routes, and the cleanup-acknowledgement write scope only on the exact idempotent quarantine-fence, cleanup-acknowledgement, and hold-retired-confirmation routes; it continues to require syrf_api everywhere else. Each route rejects the other notifier scope, and the resolver rejects a job ID or a caller-selected environment. A notifier token cannot authenticate any broader API controller, and a syrf_api token without the route policy is not a substitute for the notifier machine policy. Each admitted API operation sends a stable Project Management message: the API overwrites the environment from its authenticated client mapping, the storage-binding consumer owns the server-side lookup, the quarantine-fence consumer owns the durable deletion fence, and the hold-retired consumer owns the conditional receipt update. Neither API nor notifier receives direct Mongo authority. Tests present each staging and preview client to its own route and then prove cross-client and cross-environment combinations are denied even when audience and scope are otherwise valid.

Both notifier functions retain the repository's existing supported Lambda network path. The required rollout boundary is application and identity configuration: protected absolute HTTPS token and job-state URLs, exact scheme/host/effective-port/path validation, redirects disabled, direct Mongo access absent, and environment-scoped S3/DynamoDB/broker/API permissions. A dedicated notifier VPC, private subnets, endpoint routing, DNS firewall, stateful egress firewall, or ENI permissions are not staging, preview, or production activation prerequisites. Those may be proposed later as optional security hardening in a separately reviewed programme. The staging infrastructure prerequisite is narrower: the existing Terraform-owned SyRFS3NotifierLambdaBoundary must add only the exact staging DynamoDB/SQS/S3 actions below before staging chart roles can receive them. Wave 1C separately creates each exact-N preview policy under the protected registrar; it cannot inherit a preview-family wildcard from Wave 1B0. Wave 1B0 does not remove the boundary's legacy preview log-group or syrfapp-uploads-* read caps; they remain until Wave 1C proves every admitted preview uses its exact-N replacement and zero shared/current legacy consumers remain. Before retirement, the boundary gains permanent exact staging replacements: logs:CreateLogGroup on only arn:aws:logs:eu-west-1:318789018510:log-group:/aws/lambda/syrfAppUploadS3Notifier-staging, stream/event writes on only that group's :* streams, and the existing version-specific ReferenceUpload/ReferenceUpdate metadata reads on only arn:aws:s3:::syrfapp-uploads-staging/Projects/*. Simulation proves those staging paths remain effective with the wildcard statements removed; only then does Wave 1C retire the wildcards. Neither IAM change may add or imply a VPC/network programme.

Missing configuration, authentication, binding, or availability fails closed and leaves the row held with retry/alert state. The reconciler sends the stable-identity object-ready command to Project Management only after the response proves committed Uploaded. Project Management validates the stable job/environment/key/metadata binding and performs an exact version-specific S3 HEAD for the command's bucket, key, and immutable version before accepting release, then persists the BulkPdfUploadReleaseRecord before acknowledging the command; absence, replacement, or size/hash/metadata mismatch fails closed. No notifier or durable hold/command creates, stores, or publishes a download URL. Project Management mints a fresh version-pinned presigned URL only after atomically accepting the agent's processing claim. Until that claim commit the object-ready release remains cancellable. Cancellation that wins registers cleanup-only and terminalizes without emitting a URL; a claim that wins makes later cancellation return processing_already_started. Idempotent retry after claim commit or capability expiry obtains a new URL only after the same exact-version check, without a second processing claim, and the agent receives no AWS credentials. Cooperative cancellation after accepted claim is outside this rollout.

A completed object that races cancellation or abandonment follows the same single cleanup owner. Both BulkPdfMultipartSweepProcessor and BulkPdfUploadController.CancelUpload stop calling unversioned DeleteObjectAsync; they record an explicit terminal cleanup-pending obligation but do not release active upload capacity yet. Durable notifier capture supplies the immutable exact version; the reconciler registers it with Project Management, whose one Mongo transaction persists/reuses CleanupOnly, terminalizes the job, and releases capacity. Only that durable cleanup-authority commit permits release. The reconciler then waits the snapshotted retention, and performs the exact-version delete and acknowledgement without ever emitting agent work. Delayed S3 notification, redelivery, and recovery tests prove this handoff. Rollout first deploys and proves the new sweeper/registration behavior while the legacy narrow PM delete grant still exists; only then may declarative policy remove that grant. Rollback keeps behavior and permission as a compatible set, so an old sweeper is never run after its required permission has been removed.

The capability move is wire-compatible. First deploy one compatibility agent image to the retained Kubernetes consumer: it accepts queued legacy commands carrying ZipDownloadUrl and the new claim-response capability, preferring the claim response when available. Prove the compatibility agent against the legacy producer, then against the new producer, before any new-format command is emitted. The old agent/new producer direction is forbidden because the new command deliberately carries no URL. Then deploy the new producer/claim path and use the same immutable compatibility image on ARRNC. Rollback may return to Kubernetes only at that compatibility floor while any new-format command is queued; services roll back as a mutually compatible set, and legacy-command support is removed only after Kubernetes retirement. The new durable command and notifier never mint or store a URL.

The target Project Management execution principal receives s3:GetObjectVersion only on the selected environment upload bucket and exact Bulk PDF object-key prefix, through its role, permissions boundary, and bucket policy where applicable. That grant covers the version-specific HEAD and version-pinned presign operation; it does not grant list, write, delete, unversioned sibling, cross-environment, or notifier access. Missing/wrong-version and out-of-prefix requests remain denied before release or agent claim. The same traced principal receives dynamodb:GetItem only on the exact environment hold/outbox table for strongly consistent receipt-drain proof. It receives no Query, Scan, transaction, write, sibling/cross-environment table, or production-table authority before Gate C.

The implementation first resolves the actual AWS principal behind Project Management's SYRF__S3Settings__KeyId secret reference without printing the key or secret. Its principal ARN, policy owner, environment, bucket/prefix, and exact hold/outbox table become non-secret deployment evidence. If no declarative owner can be established, staging activation stops; an ad-hoc console grant is not an accepted migration step.

Stable message identity plus the existing consumer journal makes retry/redelivery idempotent; the design does not claim transport exactly-once. Rows retain status, lease, and attempt/error fields. Pending, leased, published, retention-waiting, and receipt-retirement-pending active holds, plus QuarantinedHold items, have no TTL and cannot age out during an outage. Successful publication starts terminal-state observation but does not assign TTL. After proved exact-version deletion and cleanup acknowledgement, the reconciler conditionally marks the matching DynamoDB row ReceiptRetirementPending with the immutable delete/acknowledgement result. One DynamoDB TransactWriteItems then atomically creates a HoldRetirementOutbox item containing that binding/result digest and atomically converts the stable-event-keyed hold item in place to an immutable CaptureTombstone; it cannot report retirement while an active hold exists. The tombstone retains the exact binding, immutable delete/ack digest, outbox key, event time, and a deployment-owned snapshotted replay deadline. A duplicate event or DLQ redrive conditionally finds that same capture key: an exact binding match is success without a new hold, while a mismatch poisons and alerts. The outbox is the durable retry source for the idempotent Project Management hold-retired confirmation. The worker first conditionally persists the returned immutable retirement acknowledgement ID/time in the outbox and only then deletes it; an acknowledged outbox resumes directly at conditional deletion. A crash resumes from either the pending hold row or outbox, and already-confirmed replay is success. Neither item has TTL. Protected cleanup deletes a tombstone only after its replay deadline, the exact failure queue has zero visible/in-flight messages and an oldest-message timestamp newer than the tombstone, and authoritative version inventory has checkpointed beyond it; redrive is forbidden after that fence. TTL is never teardown proof. The immutable S3 object version referenced by such a row is equally durable: staging and preview buckets have no age-based lifecycle expiration for completed Bulk PDF object versions. The incomplete-multipart lifecycle safety net remains separate. Therefore the handler must classify the reserved Bulk PDF key shapes (Projects/_bulk-staging/ in staging and singular preview/pr-N/Projects/_bulk-staging/ in previews) before trusting metadata. After a successful exact-version metadata read, malformed, missing, or unrecognized immutable metadata beneath one of those reserved prefixes creates a non-publishable RejectedPreHold quarantine row before the invocation may succeed. That row contains only environment, bucket, key, version, bounded reason code/digest, observation time, snapshotted invalid-object retention, lease, and exact-delete result; it carries no caller-derived job identity and cannot authorize agent work.

Protected values render reconciler.invalidObjectRetention: PT24H for staging and PT1H for previews, with the same reject-missing/zero/negative/malformed/lower and snapshot-no-acceleration rules as terminal retention. Before deletion, the reconciler calls the dedicated authenticated Project Management storage-binding resolver under the same environment-bound release-state read scope, with only environment/bucket/key/version evidence and no job identity. Project Management matches against its deletion-independent server-created multipart job/object-key binding and expected size/hash, never against malformed object metadata or caller-derived job identity. The completed-object transition persists that binding (without an S3 version field in Project BSON) before capacity may release or the job may terminalize. Ordinary pruning/Project deletion must check it and cannot remove it; objectless cancellation retires it only in the same transaction as the persisted MPU/object-absence proof, while completed-object cleanup retires it only through the acknowledgement transaction below. The read-only resolver therefore returns no candidate identity and cannot open a deletion window before the later quarantine-fence write. Zero matches proves an unowned object. After retention, the reconciler conditionally marks the exact row UnownedDeletePending with the resolver-decision digest, idempotently deletes only that version, and persists the immutable delete result. Authoritative version inventory must then observe that version absent and persist its checkpoint ID/time before a full-binding conditional delete retires the RejectedPreHold. A crash resumes from the row at every boundary; this branch fabricates no Project Management acknowledgement and creates no hold-retirement outbox. Exactly one matching Uploaded or authoritative terminal completed-object job causes an idempotent IBulkPdfUploadRejectedObjectCommand: Project Management repeats exact version-specific HEAD. For Uploaded, one transaction persists a non-claimable CleanupOnly release record, terminalizes failed-cleanup-pending, and releases active capacity. For a terminal binding with no active/retriable claim, it preserves that terminal disposition/history and idempotently persists the same CleanupOnly authority without emitting agent work; an existing matching cleanup record is replay success. The normal receipt-retirement path then owns deletion. A terminal binding that claims object absence while this version exists, multiple matches, mismatch, active/retriable claim, or ambiguity first calls the narrow idempotent quarantine-fence route. Project Management re-resolves the server-owned binding and, in one Mongo transaction, persists a no-TTL BulkPdfUploadQuarantineFence for every bounded candidate Project/job plus one immutable fence-set receipt, without exposing candidate identities. Only after that receipt is durable does the reconciler conditionally transition the same stable capture item in place from an active hold to a durable QuarantinedHold; unavailable state remains an active retryable hold. That atomic transition retains the complete immutable binding, evidence/digests, delete intent/result if any, retry authority, cleanup-only client requirement, source event identity, aggregate cleanup-budget debit, and alert/escalation state before the active-hold state ends. QuarantinedHold has no TTL, suppresses exact-binding duplicate capture at the same key, and remains eligible for the bounded reclaimer/operator repair path. It can leave the preview ledger only through proved exact-version cleanup plus the normal acknowledgement/tombstone path, or the following durable, idempotent cross-store transfer protocol; no Mongo/DynamoDB cross-service transaction is assumed:

Prepare fence: using a stable transfer ID derived from source environment, capture key, source event, fence-set ID, destination ledger, and immutable binding digest, Project Management transactionally marks the existing fence set TransferPending. The original candidate fences remain active and continue denying Project deletion/pruning; retry returns the same prepare receipt and conflicting destination/binding input fails closed.
Accept dormant destination: the separately owned non-production ledger conditionally stores PreparedQuarantineTransfer with that prepare receipt, stable capture key/source event, binding/evidence, retry authority, duplicate-suppression obligation, cleanup-client requirement, and the same stable aggregate-debit ID. It is not yet permitted to acknowledge cleanup.
Link fence: only after reading the destination acceptance receipt does the reconciler ask Project Management to add that exact destination/transfer ID as the prospective cleanup owner. Project Management retains every original candidate fence, persists the acceptance digest, and returns an idempotent linkage receipt; it never replaces or releases the old owner in this step.
Finalize ledgers: the source conditionally converts its unchanged QuarantinedHold in place to QuarantineTransferTombstone only when the prepare, destination-acceptance, and PM-linkage receipts all match. The destination then conditionally promotes its prepared item to active cleanup ownership using the source-tombstone receipt, and PM records transfer finalization only after receiving both receipts. The same aggregate-debit ID is counted once across source and destination; the replay-fenced source tombstone retains it until terminal cleanup acknowledgement.

Every state transition is queryable by stable ID, so crashes, timeouts, lost responses, and retries resume rather than compensate by deleting either copy. Only the linked destination cleanup client may submit the final exact-version acknowledgement containing the transfer/fence-set IDs; PM then clears the candidate fences transactionally. Exact duplicates remain suppressed at the source key. The source tombstone has no TTL and must first satisfy the same replay-deadline, failure-queue watermark, and inventory-checkpoint fence as CaptureTombstone. It has three additional hard purge conditions: the linked destination's matching exact-version cleanup acknowledgement, PM's durable final fence-set-clearance disposition, and authoritative release of the shared aggregate-debit ID. A pending, failed, or unavailable destination cleanup therefore keeps the source tombstone, table, purge schedule, and dynamic record. Evidence-only copying is forbidden.

Metadata/config/S3 read failures remain retryable invocation failures, but bounded Lambda retry is not the durability boundary. src/services/s3-notifier/.chart/templates/function.yaml, a new chart-owned failure-queue template, chart schema, and protected environment values configure an encrypted exact-environment SQS on-failure destination for exhausted event-handler invocations, maximum supported retention, alarms before retention risk, and idempotent reconciler redrive. Independently, the scheduled reconciler checkpoints an authoritative ListObjectVersions inventory of only the exact reserved Bulk PDF prefix and compares every current/noncurrent version with normal holds, RejectedPreHold rows, and the trusted Project Management storage-binding resolver. A missing durable row is reconstructed before the inventory checkpoint advances. Queue loss/expiry, destination failure, handler outage, pagination crash, and duplicate/redrive therefore converge without requiring a second S3 event. Unknown objects outside the reserved Bulk PDF prefixes retain the shared handler's existing skip behavior. Completed-object expiry cannot be suspended until the failure destination, redrive, authoritative version inventory, checkpoints, lag alarms, rejected-object resolution, and empty-difference proof are deployed and proved.

Protected cluster-gitops values own terminal payload retention: syrf/environments/staging/s3-notifier/values.yaml renders reconciler.terminalPayloadRetention: PT24H, while syrf/environments/preview/services/s3-notifier/values.yaml renders PT1H. Chart schema and render validation reject missing, zero, negative, malformed, or lower durations for an active staging/preview reconciler. Production remains unset and inactive until Gate C defines its policy. At the first authenticated terminal/no-active-or-retriable-claim observation, a conditional row update snapshots the policy value/version, terminal-observed time, and retention-eligible time. Later configuration changes apply only to rows first observed afterward and never recompute or accelerate an existing row.

The same bounded EventBridge schedule invokes a distinct terminal-reclamation phase in the environment's reconciler. The DynamoDB row is the immutable locator and cleanup ledger: it keeps environment, job, bucket, key, version, publication state, terminal-observed time, retention-eligible time, cleanup lease, and delete result. For any held candidate, the reconciler re-reads the exact authenticated job-state route; that response exposes only the bound job's terminal/nonterminal disposition and whether an active or retriable claim remains, never a URL or caller-selected object. The separate BulkPdfUploadReleaseRecord is also the deletion-independent cleanup authority. Before the existing per-search or per-project history bounds may prune a terminal job, a processing-capable or cleanup-only record must already hold the immutable binding. Project.PruneBulkPdfUploadHistory and PruneBulkPdfUploadHistoryAcrossProject participate in a Mongo session transaction that records terminal disposition and claim state there before removing embedded history; a terminal job with a completed-object cleanup obligation that is awaiting cleanup registration is skipped and alerted. A completed-object cleanup obligation exists after any successful multipart-complete transition or whenever a matching notifier hold or release record exists. Cancellation or abandonment that reaches terminal without multipart completion must atomically persist the sweeper's exact MPU-absent and object-absent proof with the terminal transition; that objectless proof is cleanup-complete and neither fabricates a hold/release record nor requires an impossible cleanup receipt. Whole-Project deletion fails closed while any nonterminal Bulk PDF session/job, terminal job with a completed-object cleanup obligation but no matching completed cleanup receipt, terminal objectless job without the persisted absence proof, or active processing/cleanup record, unresolved completed-object BulkPdfUploadStorageBinding, or BulkPdfUploadQuarantineFence exists. Project-owned pruning applies the same fence. Cleanup acknowledgement carries the immutable binding, exact delete/ack digest, and optional fence-set/ transfer IDs. In one idempotent conditional Mongo transaction PM converts the active release record to its receipt, marks the matching storage binding cleanup-complete, and, when a fence-set ID is present, verifies the caller is the current cleanup owner, records one immutable final-disposition receipt, and retires every member BulkPdfUploadQuarantineFence row in that complete set. Missing, extra, mismatched, already-reassigned, or partially updated members fail closed; replay returns the same receipt. No Project deletion or history pruning can observe a partially cleared set. This does not remove the matching embedded terminal history entry. That job and its report remain visible until the existing bounded-history policy prunes them; a matching receipt satisfies the completed-object fence for later Project deletion. The route reads the live job or release record, so pruning/deletion cannot erase cleanup authority. A row that becomes terminal before publication is marked with the authoritative terminal disposition and is never published. The reconciler conditionally leases reclamation only after a published row reaches a terminal downstream job with no active/retriable claim and its snapshotted retention, or after an unpublished row's explicit terminal disposition proves no claim/retry can require it, its cleanup-only record is durable, and its retention elapses. The exact bucket, key, and version come only from the durable row. It then performs version-specific absence/identity proof and DeleteObjectVersion and records the returned version-specific result conditionally. It then calls an idempotent authenticated cleanup-acknowledgement operation. The acknowledgement executes the complete storage-binding/fence-set transaction above and returns the same non-authorizing receipt on retry. The reconciler then conditionally persists ReceiptRetirementPending plus the exact delete/acknowledgement result in that same DynamoDB row, uses one conditional DynamoDB transaction to put the immutable HoldRetirementOutbox item and convert that stable capture key in place to CaptureTombstone. Only the outbox worker calls the dedicated hold-retired-confirmation route under the same environment-bound cleanup write scope with the immutable binding and delete/ack digest. Project Management conditionally records HoldRetiredAt on the receipt and returns the same immutable acknowledgement ID/time on retry. The worker conditionally persists that result in the outbox before deleting it. Notifier absence requires zero active release records, zero active holds, zero QuarantinedHold items, and zero retirement outboxes, unless every quarantine has that fully finalized replay-fenced transfer proof. Audit receipts own no processing/cleanup authority or container slot; an unlocked retained database instead carries a bounded cleanup debit until its later purge/drop. A lost response is retried idempotently against that same version; it cannot select another version.

Protected deployment values also render releaseRecord.cleanupReceiptRetention: P90D for staging and P7D for previews; production remains unset until Gate C. The acknowledgement snapshots ReceiptCreatedAt, policy version, and ReceiptExpiresAt, and a later setting cannot accelerate an existing receipt. The Project Management-owned release-record purge job may remove a receipt only when its expiry has elapsed, HoldRetiredAt proves the atomic hold-to-outbox transition and acknowledged retirement, and OutboxDrainedAt proves a later exact-key, strongly consistent DynamoDB GetItem found no matching outbox, no active processing/cleanup record exists, and the matching embedded job has already left history through the ordinary last-20-per-search pruning rule or its Project was deleted. Thus cleanup never shortens history/report visibility, retained history always has deletion-fence proof, and the deletion-independent collection is bounded by retained history plus the snapshotted retry tail. A duplicate acknowledgement within retention returns the stored result. After safe purge, a late request fails closed with deterministic 410 CleanupReceiptRetired and cannot recreate cleanup authority or mutate a Project; a missing hold-retired acknowledgement or drain proof prevents purge and alerts.

The Project Management receipt reconciler owns that drain proof. It enumerates only its receipts with HoldRetiredAt set and OutboxDrainedAt absent, derives the exact environment table and outbox key from the immutable receipt binding, and performs a strongly consistent exact-key dynamodb:GetItem. Absence conditionally records OutboxDrainedAt; presence leaves the receipt ineligible and the no-TTL outbox retryable. Its traced declarative AWS principal receives only GetItem on the exact staging table or admitted preview table, with no Query, Scan, transaction, write, sibling, cross-environment, or production authority. Missing ownership or an ambiguous table/key mapping stops activation. Thus a Project Management commit plus lost response and an outage beyond receipt expiry cannot become 410: the undrained receipt remains present until the outbox obtains/persists its replayed acknowledgement, deletes itself, and the exact absence proof completes.

Wave 1B0 is staging-only and first changes only camarades-infrastructure/terraform/lambda/ack-iam.tf, the declarative owner of the existing SyRFS3NotifierLambdaBoundary. The boundary cap adds dynamodb:PutItem, dynamodb:GetItem, and dynamodb:UpdateItem for event-handler capture, and dynamodb:GetItem, dynamodb:Query, dynamodb:UpdateItem, and dynamodb:DeleteItem for reconciliation. IAM has no dynamodb:TransactWriteItems action: the reconciler's exact-table PutItem and UpdateItem permissions authorize outbox creation plus in-place hold-to-tombstone conversion when the SDK calls TransactWriteItems; DeleteItem remains separately required for acknowledged outbox removal, replay-fenced capture/transfer tombstone removal, and full-binding conditional RejectedPreHold retirement. Add dynamodb:ConditionCheckItem only if implementation introduces a separate condition-check item; conditions on the transactional put/update do not require it. Where supported, dynamodb:EnclosingOperation = TransactWriteItems further constrains those reconciler permissions. The event handler remains denied DeleteItem and therefore cannot execute the retirement transaction. The boundary also caps handler sqs:SendMessage and reconciler sqs:ReceiveMessage/sqs:DeleteMessage/sqs:GetQueueAttributes to the exact staging failure queue. The table naming contract is exact: syrf-bulk-pdf-holds-staging plus its /index/* resources. Wave 1B0 adds no preview-family Bulk PDF ARN pattern, wildcard pr-* table, preview queue, preview role/session, or preview object namespace. It preserves only the pre-existing preview log-group and syrfapp-uploads-* read caps until the exact-N cutover/zero-legacy-consumer proof above, while adding the exact staging log group/streams and arn:aws:s3:::syrfapp-uploads-staging/Projects/* version-metadata-read replacements. Those replacements preserve the staging handler's current ReferenceUpload and ReferenceUpdate paths after the shared wildcard statements are retired. Production table authority also remains absent until Gate C.

The boundary adds s3:DeleteObjectVersion only for the rendered staging bucket's Projects/_bulk-staging/* objects. The staging reconciler alone receives s3:ListBucketVersions on that exact bucket ARN with an exact s3:prefix condition for the same reserved namespace, solely for authoritative recovery inventory. Preview authority is deferred in full to Wave 1C, including table/index, failure-queue, bucket, key-prefix, version-delete, and version-inventory grants. Production object authority remains absent until Gate C. The staging cap grants no unversioned delete, unscoped or sibling-prefix list, write, sibling object prefix, or package cleanup. It is only a maximum-permission cap: the staging chart inline role policy must still name its one rendered table ARN/indexes and exact bucket/ prefix. Terraform plan evidence must show only that staging managed-policy delta and prove the effective policy contains no preview or production resource. No staging Lambda role, hold capture, reconciler, or publication is enabled until the boundary update is applied and its effective staging policy is proved.

The shared event-handler code path retains version-specific metadata-read behavior for every environment bucket/prefix already handled by ReferenceUpload and ReferenceUpdate, and adds the Bulk PDF prefix where needed; the migration must not narrow those existing flows before the handler dispatches by upload kind. Each deployed function still uses an environment-specific execution role, and the handler has no delete authority. For staging, the named reconciler cleanup role, the Wave 1B0 staging boundary, and the staging bucket policy grant exact-prefix s3:GetObjectVersion, s3:DeleteObjectVersion, and prefix-conditioned s3:ListBucketVersions. For a preview, the exact-N event-handler/reconciler execution roles use only their Wave 1C exact per-preview permissions boundary, inline policy, and bucket policy; they do not attach, reuse, or cause broadening of the Wave 1B0 staging boundary. Both environment paths deny unversioned delete, unscoped/sibling list, write, sibling/cross-environment prefixes, and every event-handler or Project Management delete. Preview roles are additionally limited to the S3 upload prefix returned by SyrfSettings.GetS3PathPrefix(), exactly preview/pr-{n}/ (singular). That S3 namespace is not the public filesystem/link child previews/pr-N/ (plural), and neither may be derived from the other. The separate teardown role remains responsible for delete markers, noncurrent versions, and incomplete multipart uploads when the entire preview is destroyed. Missing auth, nonterminal or active/retriable state, unmet retention, lease races, delete failure, and missing version before a recorded delete intent all fail closed with the hold retained and no TTL. Long auth/API/broker/consumer outages therefore retain both row and payload; a missing version is an alerted failed invariant, never a reason to discard the hold.

Enabling durable hold capture has an explicit lifecycle order gate. First close new Bulk PDF session admission for the short configuration window, quiesce accepted sessions, inventory legacy completed versions without durable rows, and assign them a retention/quarantine owner. Then suspend completed-object age expiration and prove the rendered bucket rule is absent. Only after that proof may the event handler create a durable row that references an exact version. Deploy and prove the route, hold table, scheduled reconciler, exact principal/policies, alarms, and synthetic terminal-version reclamation before publication is unpaused. Failure leaves hold capture disabled and the rollout paused; it never restores age expiry over captured rows. Production lifecycle remains unchanged before Gate C. An environment-scoped, audited Bulk PDF pause control stops only publication of held Bulk PDF object-ready commands, not durable Bulk PDF capture/reconciliation inventory and not the shared dispatcher's existing ReferenceUpload or ReferenceUpdate flows. Those unrelated recognized event kinds continue their current success/retry semantics throughout cutover. Table, schedule, IAM, alarms, terminal retention, preview teardown, and production-no-change rendering are part of the same reviewed deployment contract. 10. an exact canonical HTTP(S) PublicBaseUrl for every environment, derived from a separately rendered, deployment-owned TrustedExistingPublicBase authority plus production/, staging/, or previews/pr-N/. The authority fixes exact scheme, host, effective port, and already-served root path independently of PublicBaseUrl and caller input. API, Project Management, and registrar render and compare it exactly; invalid, absent, caller-supplied, cross-environment, wrong-origin/port/path, or traversal-bearing combinations fail closed. 11. a protected synthetic-harness initiation path, authorized by a dedicated non-human policy, plus the same policy on GetStudyPdfPaths, signing, acknowledgement, heartbeat, completion, cancellation, abandonment, status, report, retry, and GetBulkPdfUploadHistory for only the matching synthetic Project/search/session. Before snapshot, the harness creates the Project, creates and attaches its SystematicSearch, seeds bounded synthetic Study/PDF-path rows, and proves its exact authorization and cleanup ownership. GetStudyPdfPaths then supplies the non-empty initiation fingerprint; bounded history is readable only to the harness for that synthetic scope and exposes no broader preview/user history. These paths validate the exact preview storage root. Ordinary users remain unable to initiate or operate preview Bulk uploads, while BulkPdfLinkService returns the intended public link only after successful delivery beneath the exact previews/pr-N/ prefix.

The release image is identical for staging and production. Environment configuration remains external. The PR does not deploy or enable any environment.

Tests:

unit and integration suites for paused startup, single queue binding, duplicate/redelivery, journal replay, mount/clamd/broker loss, disk threshold, and shutdown with an active claim;
API tests proving the admission gate rejects initiation but does not turn accepted-session lifecycle endpoints into 404s, plus a cutover drain with an active multipart upload;
staging cutover tests proving pre-freeze validation accepts a self-consistent old Kubernetes/PVC mapping and a separately self-consistent ARRNC target mapping without requiring equality, rejects early target activation, and requires exact API/Project Management/registrar/agent equality only after the initial copy and atomic URL switch; any mixed mapping rolls back before activation;
storage-authority/controller/link tests proving every environment requires an exact canonical PublicBaseUrl and independent TrustedExistingPublicBase, and that API, Project Management, and registrar agree exactly; a matching previews/pr-N root and dedicated harness policy permit the authorized harness to create and attach its SystematicSearch, seed synthetic Study/PDF-path rows, take a non-empty fingerprint, complete the whole lifecycle, and resolve the final public link. Missing search membership, empty/out-of-scope fixtures, or absent cleanup authority fail before initiation. An absent or caller-supplied URL, wrong host/root, sibling/cross-preview prefix, traversal, or ordinary-user initiation/lifecycle access fails closed before any multipart or job side effect;
notifier integration tests that force object completion before job persistence, prove the event is persisted before handler success, remains held, and is later reconciled after Uploaded even with no second S3 event. Tests cover conditional-write replay, lease expiry/crash recovery, exact machine-scope authorization, environment/job/object mismatch, and exact version-specific S3 HEAD absence/replacement/metadata mismatch without a new BSON version field. They prove the hold and new-format object-ready command never contain a URL, every accepted/idempotent agent-claim retry mints a fresh version-pinned URL (including after the prior capability lifetime), and the Project job remains cancellable until the processing claim CAS commits. Cover cancel before object-ready send, after send but before claim, concurrent claim/cancel in both winner orders, crash before and after claim commit, broker redelivery, cleanup-only idempotency, and the exact processing_already_started response after claim acceptance. A cancellation winner must issue no URL; a post-commit retry must remint without double-processing, and the agent has no AWS credentials. A dual-read compatibility agent first accepts both queued legacy commands carrying ZipDownloadUrl and new claim-response capabilities, preferring the claim response when available. Deploy that compatibility floor to the retained Kubernetes consumer before producing new-format commands; prove the compatibility agent first with the legacy producer and then with the new claim-response producer, queued-message replay, and rollback. Explicitly reject any old-agent/new-producer deployment. No rollback may go below the compatibility floor while a new-format command can remain queued; legacy support is removed only after Kubernetes retirement. Tests also cover token/API outage, both event-handler and separately named reconciler complete environment replacement, protected exact endpoint configuration only on the reconciler plus proof that the handler environment contains no OAuth credential or cleanup route, denial of direct Mongo, stable message identity/redelivery, Bulk-PDF-only publication pause with continued capture, unchanged ReferenceUpload/ReferenceUpdate dispatch success/retry behavior, manual/scheduled reconciliation, quarantine-ledger alarms, IAM denial, encrypted exact-environment on-failure queue delivery/redrive and pre-retention alarms, plus authoritative exact-prefix version-inventory reconstruction after exhausted Lambda retries, destination failure/expiry, pagination crash, and duplicate events; a long outage beyond terminal-retention duration and policy-version snapshot proving pending/leased/poisoned/ published/retention-waiting rows have no TTL and cannot age out, and proving their exact S3 object versions also survive with no completed-object lifecycle expiry; scheduled terminal reclamation from the durable row's exact bucket/key/version; authenticated terminal/no-active-or-retriable-claim revalidation; retention and conditional-lease races; cancellation/abandonment/recovery making a held row terminal before publication, proving it is never published, the reconciler sends an idempotent cleanup-registration command, Project Management validates exact HEAD and persists a non-claimable cleanup-only record, and the row is reclaimed only after registration plus explicit terminal disposition and retention; a completed-object cancellation/abandonment race proving BulkPdfMultipartSweepProcessor reaches terminal cleanup-pending without DeleteObjectAsync, tolerates a delayed/redelivered S3 event, retains capacity until CleanupOnly is durably committed in the same transaction that terminalizes the job, and completes through the same exact-version reclaimer; persistence of the deletion-independent release record before command acknowledgement, plus a crash/redelivery followed by an accepted retry claim after the prior capability expires, proving the durable immutable binding is reused and a fresh URL is minted; forced per-search and per-project history pruning while a hold remains, proving the release record continues to authorize reclamation and an unregistered terminal job cannot be pruned; attempted whole-Project deletion proving it fails closed for nonterminal work, terminal work without a completed cleanup receipt, and active processing/cleanup records; cleanup acknowledgement preserving the matching embedded terminal entry and report until ordinary bounded-history pruning while its receipt satisfies the Project-deletion fence; version-specific delete/retry/absence proof; idempotent cleanup acknowledgement converting the active release record to a non-authorizing audit receipt followed by durable ReceiptRetirementPending, atomic conditional hold-to-tombstone plus HoldRetirementOutbox transition, hold-retired confirmation only after the active hold has become a non-authorizing tombstone, and acknowledgement persistence in the outbox before its deletion. Crash/lost-response tests cover every boundary, including a Project Management commit plus lost response and outage beyond receipt expiry; duplicate acknowledgement/confirmation, permanent hold-to-tombstone transition failure, and already-confirmed success remain safe; snapshotted P90D staging/P7D preview receipt expiry; purge denial before expiry, acknowledged retirement, exact-key strongly consistent outbox-absence proof recorded as OutboxDrainedAt, active-record absence, and ordinary history pruning/Project deletion; missing-confirmation/outbox alerting; retained-result retry and deterministic fail-closed 410 CleanupReceiptRetired after safe purge; cleanup-authority absence proving zero active release records, zero active holds, zero QuarantinedHold items, and zero retirement outboxes (or fully finalized PM-linked, replay-fenced transfer proof for every quarantine). CaptureTombstone retains no authority or capacity; QuarantineTransferTombstone retains the source replay evidence and shared debit until matching destination cleanup acknowledgement, PM fence-set clearance/final disposition, and authoritative debit release. Final notifier/table/dynamic-record absence additionally requires conditional purge of every tombstone by its retained protected owner; exact-prefix GetObjectVersion/DeleteObjectVersion plus prefix-conditioned ListBucketVersions allow and unversioned/unscoped-list/sibling/cross-environment/ event-handler/Project-Management denial; per-preview teardown; and production configuration remaining unchanged;
Identity tests that create/update the dedicated notifier client with only client credentials, release-state read, and cleanup-acknowledgement write scopes; reject missing/default/duplicate client IDs and secrets; prove each narrow route rejects the other scope and every broader API route denies both; bind each authenticated client identity to exactly one configured environment and deny every cross-client/cross-environment combination despite otherwise-valid audience and scope; and exercise create, next-client rotation, old-client revocation/removal, preview teardown revocation, and the equivalent Auth0 contract for any environment that selects Auth0. Configuration and effective-permission tests prove only the reconciler receives a client/secret and acknowledgement route, while the event handler can write the exact hold and its exact failure queue but cannot obtain a token or acknowledge cleanup. Synchronize proves the old full client is denied/revoked only after the protected cleanup-only generation is minted, injected into the retained reconciler, proved, and switched in. Teardown proves that cleanup generation remains usable only by the bounded reconciler through final cleanup acknowledgement and the zero-hold/zero-outbox/zero-active-record gate; only afterward may it deny, revoke, and delete that cleanup generation, keep the deny fence through maximum token lifetime, and claim credential absence or release its reservation. OpenIddict resource-mapping and API bearer-selector tests prove each exact notifier scope/audience authenticates only its dedicated route, is rejected by the other notifier route and every broader route, and does not weaken the existing syrf_api requirement;
Helm/cluster-gitops contract tests that preserve the existing Lambda network shape, render exact protected token/API endpoints only for the reconciler and a credential-free event-handler environment, reject wrong hosts/ports/paths and redirects, grant no Mongo or broad AWS credential, and do not introduce a notifier-VPC dependency. They render staging PT24H and preview PT1H payload retention plus P90D/P7D cleanup-receipt retention, reject missing/zero/negative/malformed/lower active retention, leave production unset, and prove a later setting change cannot accelerate rows or receipts that already snapshotted their policy version and eligibility/expiry time;
notifier S3-IAM tests proving a versioned staging/preview event handler can read metadata only via its existing handled prefixes plus the Bulk PDF prefix where required, preserving ReferenceUpload and ReferenceUpdate before dispatch while every event-handler delete is denied; reconciler tests separately prove exact-version delete and exact-prefix version-list grants plus unscoped-list, sibling, cross-preview, cross-environment, and production denial;
Project Management deployment-policy tests proving exact-prefix s3:GetObjectVersion permits version-specific HEAD and version-pinned presign, while unversioned sibling, wrong-prefix, cross-environment, and list/write/delete are denied; prove that grant is not attached to notifier principals, whose separate handler/reconciler policies permit only the exact operations above;
workflow tests proving a PR-head package upload alone cannot update either credentialed function or obtain a secret; credentialed deployment references an immutable S3 object version and package SHA-256 bound to repository/PR/exact head SHA/maintainer approval; synchronize invalidates that binding; both function deployments revalidate it, and reconciler-only secret injection separately revalidates it; and teardown disables invocations and revokes the preview client before deleting functions/package references; then a prefix-scoped cleanup role aborts every incomplete multipart upload and enumerates/deletes every current/noncurrent object version and delete marker beneath only that preview's data prefix and exact notifier-package key, repeating until version/marker/MPU listings are empty before bucket deletion or object-storage-allocation release;
container image scan/build and non-root/read-only checks;
workflow validation that deploy resolves and records a digest and cannot target production from a staging action.

Rollback: retain the dual-read agent compatibility floor, pause Bulk PDF publication, drain or hold new-format commands, and roll SyRF services back only to a mutually compatible set. Do not restore completed-version expiry while any durable row exists. Revert inactive configuration normally; there is no notifier-VPC resource rollback in this programme. STOP: the agent cannot become ready without consuming, cannot drain/fence deterministically, or the notifier cannot authenticate and read the exact bound job state without broad datastore authority, or an unapproved credentialed artifact.

Step 5 — Build the bounded preview-family registrar¶

Primary repository: camaradesuk/server-config. Supporting repositories: camaradesuk/camarades-infrastructure for protected registrar IAM bootstrap/guardrails, and camaradesuk/arrnc-api-deploy only if the existing execution protocol cannot carry the typed operation. Owners: ARRNC config-service, IAM, and gatekeeper maintainers.

Expected server-config files:

config-service/src/ConfigService/Api/ApiEndpoints.cs and authorization;
config-service/src/ConfigService/Operations and Record;
config-service/src/ConfigService/Schemas/deployment-manifest.schema.json;
config-service/src/ConfigService/Provenance;
config-service/src/ConfigService/Reconcile;
config-service tests for API, authorization, records, provenance, races, and reconciliation;
deployments/arrnc-api.yml and root-controlled templates.

Expected arrnc-api-deploy files if needed:

scripts/container-web-deploy and container-web-apply;
scripts/install-gatekeeper;
config/services.conf.example;
shell/PowerShell contract tests.

Expected camarades-infrastructure files:

terraform/lambda/ack-iam.tf for the staging boundary and protected registrar IAM management guardrails; the existing broad per-PR role/tag mutation patterns must not remain an unverified positive-authority path;
terraform/lambda/main.tf to retire the shared preview execution role and direct caller-set-driven preview resources only after ACK adoption; it must not create the replacement exact per-PR roles/policies;
terraform/lambda/variables.tf so preview_prs, commit SHAs, and versions are reconciliation inputs only and cannot establish canonical PR identity or authorize resources;
terraform/lambda/github-actions-iam.tf to scope role/policy/pass-role authority to the protected ACK/registrar bootstrap path and deny PR-controlled trust, policy, boundary, or tag mutation;
Terraform policy-simulation/contract tests for exact-N binding and negative sibling cases.

Expected SyRF chart files, preserving ADR-010 ownership:

src/services/s3-notifier/.chart/templates/iam-role.yaml (and Policy resources/helpers as required) to render distinct exact-N handler, reconciler, Project Management receipt-reconciler, and teardown cleanup roles from only the protected registrar record;
src/services/s3-notifier/.chart/templates/function.yaml to bind the exact handler and reconciler roles only to their corresponding functions;
src/services/s3-notifier/.chart/values.yaml and templates/_helpers.tpl to reject a caller- selected N and render the exact table, queue, bucket, prefix, camarades-terraform-state-aws/lambda-packages/pr-N.zip package object, log-group, role, and function names;
src/services/s3-notifier/.chart/templates/serviceaccount.yaml, permission-job.yaml, and env-vars-job.yaml, plus their values.yaml controls, so preview rendering disables the shared setup service account/jobs and routes exact-N function environment and invoke-permission changes through the protected registrar;
Project Management deployment identity/configuration files discovered in Wave 1C to bind its independently identified runtime principal to only the exact-N receipt-reconciler role.

Companion authority documents are executable Wave 1C inventory, not historical advice:

docs/platform/lambda-ack-gitops/technical-plan.md states that exact-N previews do not use SyRFS3NotifierLambdaBoundary or a pr-* setup-job trust path;
docs/planning/s3-notifier-iam-migration.md supersedes its shared-boundary requirement for previews and requires protected registrar ownership of preview Lambda configuration and invoke permission;
camarades-infrastructure/terraform/lambda/ack-iam.tf removes the shared syrf-ack-setup-job role's pr-* trust and preview-family Lambda mutation resources while preserving separately scoped staging/production behavior.

These files are Wave 1C for every preview-family change. They must not reintroduce preview data- plane authority into the Wave 1B0 staging boundary, and no apply belongs to this planning PR.

Implement typed create/pause/resume/retire operations whose only caller-controlled deployment values are canonical PR number, verified SyRF digest/source, desired operation, and idempotency key. Derive every name/path/network/secret/resource value and the PublicBaseUrl from the separately protected TrustedExistingPublicBase authority plus canonical previews/pr-N/; callers cannot select or override a host, scheme, port, path, mount, or URL. The registrar compares that independent authority and derived URL exactly with the separately rendered API and Project Management values before creating any record or resource. Verify trusted repo/workflow, open PR, digest attestation, fleet capacity, record ownership, and explicit authorized-maintainer approval bound to the exact current head SHA and digest. Invalidate activation on every synchronize event, recheck the binding immediately before registration, serialize by PR, and audit each transition. Before creating any state, atomically reserve CPU, memory, PIDs, block I/O, Docker/runtime disk, scratch, storage quota/allocation, ports/routes, network, RabbitMQ, and scanner capacity from a fleet-wide ledger. The configurable concurrency ceiling derives from measured ARRNC capacity and may exceed one; requests queue without partial resources when the aggregate safe budget is unavailable. The registrar uses only the storage owner's scoped provisioning/allocation capability and verifies its audit result. Immediately before a queued request reserves resources and provisions anything, one atomic admission check must revalidate the exact head repository, open/enabled PR state, current head SHA, verified digest, and current authorized-maintainer approval; a stale or ineligible request is discarded with no partial state.

Wave 1C also owns every preview-family IAM grant; no preview role may reuse or depend on resource authority from Wave 1B0. The protected registrar derives a canonical positive, non-zero-padded PR number N from an independent provider/base-repository lookup or independently verified signed attestation that binds the open same-repository PR, exact head SHA, verified digest, and maintainer approval. Caller-supplied N, resource names, or session tags are untrusted inputs. The registrar uses the independently established N once to render the exact syrf-bulk-pdf-holds-pr-N table plus /index/*, exact failure queue, exact syrfapp-uploads-pr-N bucket, and singular preview/pr-N/Projects/_bulk-staging/* object namespace, including the identical ListBucketVersions prefix condition, plus the exact package object arn:aws:s3:::camarades-terraform-state-aws/lambda-packages/pr-N.zip. The bucket, key prefix, and package key must carry the same N as the table, queue, distinct handler/reconciler roles and functions, exact-N Project Management receipt-reconciler grant, exact-N teardown cleanup role, and control-plane session. The handler and reconciler effective policies include logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents only on their corresponding exact-N CloudWatch log groups.

Prefer a policy-variable design only when effective-policy simulation and negative tests prove that one protected N binds every resource simultaneously. Otherwise the registrar must render an exact per-preview permissions boundary and inline role policy from its independent canonical N, validate their effective intersection before activation, and retire only that exact policy set. A preview-family wildcard combined with a caller-selected tag is forbidden.

The current Terraform preview_prs set, preview commit/version maps, shared syrfS3NotifierPreviewLambdaRole, and wildcard syrfS3NotifierPr*LambdaRole management paths are inventory to migrate, not proof of N or the replacement owner. ADR-010 remains authoritative: chart/ACK Role, Policy, and Function resources own the preview lifecycle. Wave 1C retires or constrains Terraform to ACK/registrar bootstrap and guardrails; Terraform must not create the replacement per-PR roles. Terraform plan and effective-policy evidence must show that removing or changing the registrar record cannot retag an existing role, preserve a stale session, or redirect a different PR's table, queue, bucket, or prefix.

For previews, this plan supersedes the older ACK instructions that require every notifier role to attach SyRFS3NotifierLambdaBoundary and that trust system:serviceaccount:pr-*:ack-setup-job. The registrar-rendered exact per-preview boundary is the only boundary on each exact-N preview role. Preview chart values set setupJob.enabled: false, so no preview ack-setup-job ServiceAccount, permission hook, or environment hook is rendered. The shared setup role loses pr-* trust and preview-family Lambda mutation resources. After re-fetching provider/base-repository provenance and the immutable registrar record, the protected registrar alone applies the exact-N function environment and S3 invoke permission, binding the same N into function and bucket. PR jobs, PR-head code, images, and service accounts receive no lambda:UpdateFunctionConfiguration, lambda:AddPermission, or lambda:RemovePermission authority.

Only the protected registrar or its protected identity broker may mint the immutable canonical record and stamp the canonical PR tag. Protected chart/ACK reconciliation consumes that record. The ACK Lambda controller's bootstrap principal may narrowly iam:PassRole only the exact-N handler execution role to the exact-N handler function and the distinct exact-N reconciler execution role to the exact-N reconciler function. It cannot cross-pass or assume either execution role. Each per-PR execution role separately retains only the AWS Lambda service principal (lambda.amazonaws.com) in its trust policy so its corresponding function can obtain runtime credentials. The exact-N Project Management receipt-reconciler role trusts only the independently identified Project Management runtime principal and grants exact-key strongly consistent dynamodb:GetItem; the exact-N teardown cleanup role trusts only the protected registrar/teardown principal and grants only the already-defined exact-prefix cleanup actions. PR jobs and preview code can assume none of these roles. Trust, boundary, and organization controls deny missing, duplicate, malformed, non-positive, or zero-padded tags; direct caller sts:TagSession, iam:TagRole/iam:UntagRole, arbitrary sts:AssumeRole, and iam:PassRole; and mutation of role trust, inline/attached policies, permissions boundary, session duration, or transitive-tag rules. Create, queued retry, reconcile, and retire re-fetch or independently verify provenance and audit the exact effective N-bound grant. Retirement retains the receipt role until all drain proofs are recorded and the teardown role until exact version/marker/MPU absence is proved, then removes the complete exact-N role/policy set. Active/cutover preview Role/Policy CRs keep iam.deletionPolicy: retain. After the proofs, a one-way registrar deletion authorization bound to the exact N changes only that preview set to iam.deletionPolicy: delete; ACK deletion and exact AWS role/policy absence are required before removing the dynamic record. Staging and production remain retain. Tests must prove a PR job, image, or stale approval cannot retag, untag, assume, pass, broaden, persist, or redirect that authority, and that changing N in any table, queue, bucket, prefix, principal, role, function, log group, or session fails closed without sibling access. Negative tests must substitute N independently in the upload prefix and package key and prove both fail closed. They also render preview serviceaccount.yaml, permission-job.yaml, and env-vars-job.yaml absent; prove the shared setup role cannot be assumed from pr-* and cannot mutate any preview-family function; and prove the protected registrar rejects a function/bucket with a sibling N. Policy simulation must additionally prove the handler cannot use reconciler, receipt, or teardown authority; the reconciler cannot use receipt or teardown authority; and no exact-N role can access a sibling PR.

Security tests must prove rejection of arbitrary repositories, tags, commands, paths, ports, networks, env names, secret scopes, every head repository other than the SyRF base repository, stale/closed PRs, per-slot profiles or requested concurrency above hard limits, digest drift, record takeover, traversal, replay with conflicting input, and persistent-slot retirement. They must separately prove that transient aggregate-capacity exhaustion within valid limits queues with no partial state and later admits after capacity is released only after a fresh atomic eligibility and provenance check. Tests close, disable, synchronize, change the digest, and revoke approval while queued and prove none can reserve or provision. Host network tests must also prove deny-by-default preview egress: allow only the exact preview scanner plus minimum controlled DNS, presigned-S3 HTTPS, and preview RabbitMQ destinations; deny host gateways, metadata endpoints, ARRNC/UoE internal CIDRs, persistent environments, and arbitrary internet routes. Root-controlled per-preview tests enforce ingress/egress rate and burst ceilings even for allowed presigned-S3 traffic and prove simultaneous bounded transfers cannot exhaust the ARRNC uplink or degrade production/staging health.

Each explicitly approved same-repository PR receives its own disposable rootless Docker container on ARRNC. Root-controlled templates enforce the pinned digest, non-root UID, dropped capabilities, no-new-privileges, read-only root filesystem where supported, seccomp/AppArmor, private network namespace, no Docker socket, no host namespace/device/admin access, and no production/staging mount, credential, route, or service identity. Any pull request whose head repository is not exactly the SyRF base repository, including every fork, is rejected, as is a stale, closed, or unapproved same-repository head. VM/microVM isolation and /dev/kvm are not prerequisites for this approved-workload policy.

Root applies immutable per-preview and aggregate CPU, memory, PID, block-I/O, runtime-disk, scratch, storage-quota, port/route, network, broker, and scanner ceilings. Bounded write/discard/fsync and simultaneous-preview tests prove those ceilings and acceptable production/staging health.

ARRNC root mounts the exact SMB child and bind-mounts only that child into the matching rootless container. The registrar/storage-owner operation atomically creates the exact child, principal, ACL, quota, and immutable allocation record, but never the readiness sentinel. After ARRNC has mounted the exact child and verified its source fingerprint, ACL, quota, and write/rename/fsync capability, the ARRNC operator creates the sentinel and verifies its structured content before activation. A preseeded sentinel fails closed. Tests prove exact source, no-follow behavior, traversal/symlink/reparse-point/sibling/common-root denial, mount-loss fail-closed behavior, cross-preview denial, and ordered container stop and fence before host unmount. SMB credentials never enter any preview container.

The scanner updater has a distinct trusted identity and an allowlist only to a pinned internal signature mirror. Tests prove refresh, mirror failure, stale-signature fail-closed behavior, and that the PR agent cannot reach the mirror.

Registry/host tests must reject a manifest compressed total or signed build/verifier unpacked-size attestation above reviewed ceilings, insufficient Docker data-root headroom, and teardown of a layer referenced by another slot. Reconciliation must reclaim proven preview-only layers, detect leakage, and avoid broad pruning when ownership is ambiguous. Broker tests must drive simultaneous preview clients through connection/channel, message-size, publish-rate, queue-length/bytes, and storage limits and prove persistent-environment health is unchanged; requests queue when aggregate capacity is unavailable, or a separately reviewed dedicated preview broker supplies the boundary.

arrnc-api-deploy must remain a constrained actuator. If the typed policy cannot stay in server-config, STOP for architecture review rather than duplicate it.

Rollback: disable preview-family API and reconcile no new records; existing preview records remain paused for explicit safe teardown. Persistent slots are unaffected.

Step 6 — Wire trusted PR preview lifecycle¶

Repositories: camaradesuk/syrf and camaradesuk/server-config. Owners: SyRF CI and ARRNC platform.

Expected SyRF files:

.github/workflows/pr-preview.yml;
.github/workflows/_preview-gitversion.yml;
.github scripts that detect service changes and clean preview resources;
CLAUDE.md preview runner/security boundary.

Activation requires an eligible same-repository PR and explicit preview configuration. The workflow that executes PR code builds the PDF-agent image, resolves its digest, and publishes only the attestation. It receives no ARRNC, registrar, storage, broker, or protected-environment credential. A PR-head job may likewise build and attest a notifier package, but it cannot deploy either notifier function or read/inject the notifier credential. The mutable lambda-packages/pr-N.zip key is not a credentialed deployment source. The package selected for activation is an immutable S3 object version whose SHA-256 is included in the approval record. A separate protected workflow loaded from the base branch, using pinned dependencies and a short-lived preview-family-only identity, re-verifies PR state, capacity, and an approval/configuration record for the exact head SHA and digest before calling the registrar. Every synchronize event invalidates that record. The exact-head generation is held on a protected non-PR-controlled approval boundary, so invalidation is also an immediate runtime fence, not merely a ban on the next package update. The protected base-branch controller first closes new Bulk session admission while the last-approved API/Project Management runtime, notifier client, held publication, and scheduled reconciliation remain active but fenced from replacement. Accepted multipart sessions retain only their lifecycle paths and must complete, cancel, abandon, or policy-expire while the old approved event handler continues durable S3 capture and the old approved reconciler can publish and drive accepted Uploaded jobs to terminal state. After every accepted job is terminal, the controller pauses held publication, inventories and reconciles late holds, then disables S3 invocation only after no accepted session can create another object. It then creates a publication-disabled cleanup-only client generation under the protected issuer, injects it only into the retained last-approved reconciler, proves both read routes and all three cleanup-write operations, and switches reconciliation to it. Only then does it deny the old preview client at notifier API admission, revoke it at the non-PR-controlled issuer, drain in-flight calls, and wait the maximum issued-token lifetime. The resulting receipt proves both credential fencing and zero remaining object-creation paths. The PR-head build can still produce an unprivileged artifact, but the protected deployment/registrar gate refuses that head until it holds the fence receipt plus a new exact-head approval. The cleanup-only generation remains admitted until zero holds, zero retirement outboxes, and zero active release records are proved. Reapproval then replaces it with a fresh exact-head client generation before re-enabling calls. If the PR closes or remains unapproved, the protected controller keeps the last-approved API/Project Management runtime and that already-issued cleanup-only generation fenced from replacement, with publication still disabled, solely to finish hold acknowledgement/teardown; it never targets the unapproved head. Thus an unapproved synchronized API/Identity cannot receive the old secret or bearer token even if its ordinary preview build trigger fires; this control-plane fence does not add protected PDF serving, VM isolation, or IIS authority. Activation rechecks the binding immediately before registration. The same protected workflow rechecks repository, PR, exact head SHA, notifier object version/package SHA-256, and maintainer approval immediately before updating both the S3 event-handler and reconciler functions and immediately before injecting the unique preview-only client secret into the reconciler alone. An unapproved synchronize cannot replace either approved function or let its credential-free event handler inherit approval; the reconciler alone can receive the new client. It never checks out or executes PR code, then waits for paused readiness/isolation proof before enabling the preview's synthetic-only Bulk consumer. Forks never reach the ARRNC path. Approval of the exact same-repository head and digest is the code-trust decision: the image is trusted application code, and the scan/type/atomic-copy and exact-child tests contain accidental defects rather than claiming hostile-code isolation. Do not add protected preview serving, a second promotion tree, VM isolation, or IIS mutation as an activation prerequisite.

The registrar record and root-controlled server-config templates render the preview container with the exact PublicBaseUrl derived from the independent TrustedExistingPublicBase authority and canonical previews/pr-N/ child. Schema/render tests require exact API/Project Management/registrar agreement and reject any caller-supplied, absent, wrong-origin/port/base-path, wrong-root, sibling, or traversal-bearing URL. The protected workflow verifies the derived URL and exact mount before enabling the consumer; teardown removes the child-specific configuration with the record. No workflow is authorized to mutate IIS bindings, virtual directories, hostnames, certificates, or other IIS state.

The protected harness alone uses a synthetic preview identity and protected API path to create the matching Project, create and attach its SystematicSearch, seed bounded synthetic Study/PDF-path rows, prove exact authorization and cleanup ownership, obtain the resulting non-empty GetStudyPdfPaths fingerprint, initiate the persisted upload session/job, upload and acknowledge every approved non-sensitive multipart fixture part, complete the session, and read only that synthetic scope's bounded history. Only after the durable Uploaded job records the exact environment, size, hash, and policy required by BulkPdfUploadClaimConsumer may the trusted notifier release gate publish the exact object/version event. The gate must already have durably captured an early S3 event, survive a crash and the absence of another S3 event, and converge one logical publication through stable-identity idempotency after the committed job is visible. External policy disables human upload initiation for previews. PR code receives no selector or credential for production, clinical, participant, or other sensitive objects.

Cleanup on close/config removal invokes the idempotent registrar teardown. It first closes only new session admission while accepted sessions retain lifecycle paths; completes, cancels, abandons, or policy-expires them while durable notifier capture, publication, and scheduled reconciliation continue; proves every accepted job terminal; then pauses publication while capture continues, inventories every late event, and disables new S3 event invocation only after no accepted session can create another object. Only then may it fence/drain, stop the container and scanner, unmount the exact child, and let the storage owner no-follow delete only that child or quarantine it after rejecting traversal, symlink/reparse-point, sibling, or local-underlay targets. It removes the exact preview image reference and only proven unreferenced preview layers, disposable rootless runtime state, per-preview network, routes, ports, scanner allocation, and exact storage allocation, then waits for durable absence. The scheduled reclaimer, preview client, broker identity, and dynamic record remain until every hold, including retention-waiting rows, has reached conditional exact-version deletion plus authenticated cleanup acknowledgement. A poisoned row first becomes the in-place durable QuarantinedHold described above; it is never force-discarded. The preview cleanup runtime, client, aggregate debit, and dynamic ledger remain until zero active holds, zero QuarantinedHold items, zero retirement outboxes, and zero active Project Management release records are proved, unless the durable transfer handshake's PM-link, source-tombstone, active-destination, and PM-finalization receipts prove a non-production quarantine owner accepted every stable capture key, source event, duplicate-suppression obligation, cleanup authority, and debit. Only then may teardown disable normal EventBridge reclamation, revoke the client, remove the capture function, and switch the reconciler to a credential-free tombstone-only purge schedule. The exact table, that protected purge owner, and a tombstone-only dynamic record remain until every CaptureTombstone and QuarantineTransferTombstone conditionally clears its replay/queue/inventory fence; a transfer tombstone additionally requires matching destination exact-version cleanup acknowledgement, PM fence-set clearance/final disposition, and authoritative shared-debit release. Only then may the final reconciler, table, and dynamic record be removed. The existing early database decision in .github/workflows/pr-preview.yml must move behind this gate: PM state remains available for every release-state read and cleanup acknowledgement. Then the workflow applies its snapshotted close-time label decision. Without lock-db, it removes every PR-head, agent, notifier, and user-facing preview access path but keeps the exact Mongo database, the last-approved API/Project Management cleanup-only path, and the protected synthetic-harness identity in cleanup custody until every snapshotted P7D receipt expiry. No new admission, upload, claim, or arbitrary Project route is enabled. The server derives the deletable Project set from the protected activation manifest; the workflow cannot supply a Project ID, and the identity is authorized only for those teardown-owned synthetic Projects in that exact preview database. The protected base-branch cleanup job then invokes the ordinary idempotent Project-deletion contract for each teardown-owned synthetic Project, using the synthetic manifest identity and only after zero nonterminal work, completed cleanup receipts, zero active records, zero active holds, zero QuarantinedHold items, and zero retirement outboxes are proved. Project deletion is the normal history-removal branch and makes those expired receipts purge-eligible without requiring future uploads. The Project Management-owned purge path removes eligible receipts and proves zero retained receipts. The workflow then revokes the synthetic-harness cleanup identity, disables the cleanup-only API/PM path, waits the maximum token lifetime, deletes only that exact database, and records absence. Any failed Project deletion or purge retains/quarantines the cleanup debit and database; it never force-drops them. The dynamic cleanup record remains charged, without an agent slot, until this completes. With lock-db, it preserves that exact database as promised, proves it was not dropped or mutated by teardown, removes all preview runtime/client access to it, and records the existing preservation handoff; preservation never bypasses the zero-hold/zero-active-record gate. Teardown may then revoke the broker identity. The tombstone-only dynamic record remains until the final replay-fenced purge described above. A protected prefix-scoped cleanup role then uses only ListBucketVersions/ListBucketMultipartUploads, AbortMultipartUpload, and DeleteObjectVersion authority to abort all incomplete multipart uploads and enumerate/delete every current or noncurrent version and delete marker beneath the preview's exact singular S3 upload prefix preview/pr-{n}/ and exact lambda-packages/pr-N.zip package key. The public output child previews/pr-N/ is never used as an S3 selector. The cleanup role repeats version/marker/MPU listings until empty; aws s3 rm --recursive or an unversioned key delete is not absence proof. Only after ledger cleanup may the ACK bucket/package reference be removed. It atomically releases only resources whose absence is proved. Any unsafe remainder stays charged to the owning reservation, or moves atomically to an explicit quarantine debit against the same aggregate fleet budget, until reclamation is proved. An hourly server-side reconciler covers missed GitHub events and alerts at one hour; the 24-hour SLO is a hard incident threshold. A later preview creates a fresh container from its verified immutable digest with fresh identities and allocations; runtime state is never reused across PRs without the reviewed reset, fencing, and identity-rotation contract.

Tests:

workflow trigger/permission/secret-flow static validation;
trusted versus fork event matrices;
proof that a same-repository PR modifying either workflow cannot obtain the activation identity or replace privileged base-branch code/dependencies;
proof that uploading or overwriting a PR package cannot alter either credentialed Lambda; the protected deployment uses only the approved immutable object version/SHA-256 for both handlers, injects only the exact approved full client after revalidation, and on synchronize mints/proves a publication-disabled cleanup-only generation before revoking that old full client. The cleanup generation alone remains until teardown ledger reclamation and cleanup acknowledgement complete;
synchronize/stale-approval tests proving a new head SHA cannot reuse a prior opt-in, review, or digest binding;
proof that PR code cannot alter the host egress policy and cannot reach host gateways, metadata, ARRNC/UoE internal CIDRs, persistent-environment endpoints, or arbitrary internet destinations;
simultaneous bounded-transfer proof that root-controlled per-preview and aggregate ingress/egress rate and burst ceilings protect the shared ARRNC uplink and persistent-service health;
simultaneous bounded-I/O proof that host-enforced per-preview and aggregate block-device bandwidth/IOPS ceilings cover writable runtime data and scratch without degrading production/staging storage health;
simultaneous-client proof that broker connection/channel, message-size, publish-rate, queue-length/bytes, storage, and scanner limits fail closed without degrading persistent environments or another preview;
oversized-image and accumulated-layer proof for activation ceilings, reserved Docker headroom, preview-only reclamation, and refusal to prune persistent/shared layers;
container removal/recreation proof showing that a surviving process, stale identity, network, mount, message, route, allocation, or runtime state cannot cross PR assignments;
synthetic-only input tests rejecting human upload initiation and every object/prefix outside the protected preview fixture set, proving multipart upload/acknowledgement/completion produces the matching synthetic Project and durable Uploaded job with environment/size/hash/policy before the release gate publishes; force the S3 event to arrive first and prove durable hold, crash recovery, and one logical stable-identity post-Uploaded publication despite redelivery, with no sensitive content in proof artifacts;
proof that the existing public base serves a clean delivered PDF only at the exact previews/pr-N/ prefix, EICAR/unscanned/partial/quarantined output produces no link, and listing, storage/mount traversal, cross-preview link derivation, or credential exposure remains impossible, with no per-preview IIS object or mutation. A client that already knows another preview's valid delivered public URL can fetch it; no authenticated serving boundary is claimed;
create/close/reopen/retag/reseed races and duplicate events;
synchronize-race tests proving approval generation invalidation immediately closes new-session admission and fences replacement while the retained approved handler, reconciler, publication, and old-client API admission remain active for accepted sessions. They terminalize accepted work, then pause publication, inventory/reconcile late holds, disable S3 invocation only after zero object-creation paths, mint and prove a publication-disabled cleanup-only client for the retained reconciler before revoking the old client, drain calls, wait maximum token lifetime, and prevent the new PR head from deploying or receiving any old secret/token until a protected fresh-head approval replaces the cleanup generation and reopens the gate. Open-without-reapproval and close- without-reapproval fixtures prove that already-issued cleanup-only generation drains the retained last-approved runtime without enabling publication or exposing credentials to the unapproved head;
close/config-removal with an accepted multipart session and delayed S3 notification, proving session quiescence, terminal-job/hold reconciliation, publication pause, consumer fence, and exact-child removal occur in that order without late work or orphaned notifier state;
objectless cancellation/abandonment tests proving a terminal transition without multipart completion atomically records MPU/object absence, creates no fictional hold/release record, can be pruned, and does not block Project deletion; completed-object jobs still require their matching cleanup receipt. Separate controller-cancel and sweeper-race fixtures prove neither path calls unversioned DeleteObjectAsync, creates a delete marker, or records object absence for a surviving noncurrent version; both hand the exact version to durable cleanup-pending reclamation. A delayed-capture/outage fixture proves active capacity remains charged until Project Management atomically persists/reuses the deletion-independent storage binding plus CleanupOnly, terminalizes the job, and releases capacity. Resolver-versus-Project-delete/prune races prove the pre-existing storage binding blocks deletion before any quarantine-fence call returns;
malformed/missing/unrecognized-metadata tests beneath each reserved Bulk PDF key shape proving a successful metadata read creates one non-publishable RejectedPreHold row before handler success and never trusts metadata job identity. An unowned exact version is reclaimed without fabricated Project Management authority: tests crash before/after delete-result persistence, require a later authoritative absence checkpoint, reject stale/full-binding mismatches, and conditionally retire the row without an outbox. A unique server-owned storage-binding match instead repeats exact version-specific HEAD; one Mongo transaction persists CleanupOnly, terminalizes failed-cleanup-pending, and releases capacity before completing the normal receipt path. A cancellation-before-resolution fixture proves a unique authoritative terminal completed-object binding preserves terminal history, creates/reuses CleanupOnly, and emits no agent work. Objectless contradiction, active/retriable claim or ambiguous/mismatched matches first persist a PM-owned BulkPdfUploadQuarantineFence set before QuarantinedHold; unavailable resolver state remains a retryable active hold. Ordinary history pruning and user-initiated Project deletion fail closed on every fence, including after a protected quarantine transfer, until exact cleanup clears it. Multi-candidate fixtures prove one exact delete/ack digest conditionally retires the complete fence set and records its immutable final disposition in the same PM transaction; missing/extra/mismatched members, wrong cleanup owner, partial-write failure, and replay cannot partially unblock any Project. Exhausted async failures enter the exact DLQ, while authoritative paginated version inventory reconstructs missing rows after DLQ loss/expiry or destination failure and proves an empty difference before age expiry is removed. Non-Bulk objects keep existing skip behavior;
notifier route-policy tests proving the environment-bound release-state read scope admits both the job-bound state route and only the no-job-identity storage-binding resolver, while the cleanup write scope admits only quarantine-fence registration/transfer, cleanup acknowledgement, and hold-retired confirmation for the immutable binding/evidence digest. Cross-route, caller-selected environment, job-bearing resolver, enumeration, broader mutation, sibling-prefix, and duplicate/uncertain-response cases fail closed;
Terraform/ACK IAM contract evidence proving Wave 1B0 changes only the staging resources in SyRFS3NotifierLambdaBoundary, permits only the staging hold-table action sets including reconciler exact-table PutItem/UpdateItem constituents for TransactWriteItems, plus separate DeleteItem for acknowledged outbox removal, replay-fenced capture/transfer tombstone cleanup, and full-binding conditional RejectedPreHold retirement, exact failure queues, exact Bulk PDF version-delete namespaces, and exact prefix-conditioned version inventory; policy validation rejects the nonexistent dynamodb:TransactWriteItems IAM action and permits ConditionCheckItem only if a separate condition-check item is implemented. It continues to deny handler DeleteItem/retirement-transaction access, every preview-family and production ARN, broad DynamoDB, unversioned/unscoped-list/sibling S3, package-cleanup, VPC, DNS, endpoint, and ENI authority; effective-policy tests prove chart inline staging roles remain pinned to the one rendered staging table and prefix. The plan must prove the pre-existing preview log-group and syrfapp-uploads-* read caps remain unchanged until every preview exact-N cutover and zero legacy consumers. Before Wave 1C retires them, tests prove exact staging log-group/stream grants and version-metadata reads on only arn:aws:s3:::syrfapp-uploads-staging/Projects/* preserve existing ReferenceUpload and ReferenceUpdate behavior after the wildcard statements are removed;
Wave 1C preview-IAM contract evidence proving a protected registrar independently establishes one canonical positive, non-zero-padded PR number and binds that same N into the exact table and indexes, failure queue, syrfapp-uploads-pr-N bucket, singular preview/pr-N/Projects/_bulk-staging/* prefix, version-inventory prefix condition, exact camarades-terraform-state-aws/lambda-packages/pr-N.zip package object, distinct handler/reconciler roles and functions, exact-N Project Management receipt-reconciler grant, exact-N teardown cleanup role, corresponding handler/reconciler log groups, and control-plane session. Evidence proves chart/ACK is the lifecycle owner and Terraform creates no replacement per-PR role. The handler and reconciler boundaries and inline policies permit logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents only on their corresponding exact-N log groups. Policy simulation must reject mismatched N, caller-supplied or malformed tags, sibling resources, retag/untag, arbitrary assume/pass-role, cross-pass between functions, receipt/teardown authority from a Lambda, trust/policy/boundary mutation, stale provenance, and replay after retirement. Retirement tests retain the receipt principal through drain proof and the teardown principal through exact-prefix version/marker/MPU absence, then require a one-way exact-N preview-only transition from iam.deletionPolicy: retain to iam.deletionPolicy: delete, ACK deletion, and AWS absence before the dynamic record disappears; staging and production remain retain. Package tests substitute a sibling N and prove denial. Rendering tests prove preview setup jobs and their ServiceAccount are absent, the shared setup role has no pr-* trust or preview-family Lambda resource, and only the protected registrar can perform an exact same-N function-environment/invoke-permission operation; sibling-N and PR-head callers are denied. The two companion ACK documents are inputs to the same contract and cannot restore the shared preview boundary or setup-job path. If one policy variable cannot prove the cross-resource equality, acceptance requires a validated exact per-preview boundary and inline policy; no family wildcard escape is accepted;
Project Management principal-policy tests proving only strongly consistent exact-key dynamodb:GetItem on its rendered environment hold/outbox table is admitted for drain proof; Query, Scan, transaction, every write, sibling/cross-environment tables, and production remain denied;
duplicate/redrive tests proving the hold-to-retirement transaction converts the stable capture key to CaptureTombstone while creating the outbox; exact-binding duplicates before the snapshotted replay fence are suppressed, mismatches poison, and conditional tombstone deletion requires the replay deadline, empty visible/in-flight failure queue with a newer oldest-message watermark, and authoritative inventory checkpoint. Transfer crash/retry, lost-response, and delayed-event tests cover every prepare-fence, dormant-destination acceptance, PM-link, source-tombstone, destination-activation, and PM-finalize boundary. They prove conflicting retries fail closed, the original candidate fence never disappears, only the linked destination can acknowledge cleanup, the stable key/source event and duplicate fence survive, the shared debit ID is counted once, and the source tombstone retains the ordinary replay/queue/inventory boundary plus its stricter destination-cleanup-acknowledgement, PM-fence-clearance, and shared-debit-release boundary. Delayed, failed, unavailable, and lost-response destination-cleanup tests prove the tombstone, exact table, purge schedule, and tombstone-only dynamic record cannot disappear first. Crash tests also cover every ordinary transaction/outbox/tombstone boundary;
credentialed-notifier teardown proof that new S3 invocation stops only after session quiescence, while the schedule and OAuth client remain active through snapshotted retention, quarantine-ledger remediation, exact-version deletion, and cleanup acknowledgement for every hold. Poison fixtures prove the active item atomically becomes QuarantinedHold without losing binding, retry authority, duplicate suppression, or aggregate debit, and that an evidence-only copy cannot clear teardown. Only after active-hold, quarantine-ledger, retirement-outbox, and active Project Management release- record sets are empty (or every quarantine has matching PM-link, source-tombstone, active-destination, and PM-finalization receipts) may normal reconciliation stop, the client be revoked, and capture be removed. Tests retain the exact table, a credential-free tombstone-purge schedule, and tombstone-only dynamic record until zero conditionally purged capture/transfer tombstones, then prove final notifier/table/record absence; only the cleanup-authority gate controls the snapshotted lock-db decision. Tests prove the unlocked branch first isolates the exact preview Mongo database, retains it through all snapshotted P7D receipt windows, invokes ordinary deletion for every teardown-owned synthetic Project through the retained protected synthetic-harness identity and cleanup-only API/PM path, rejects caller-selected/user/persistent Projects, safely purges every receipt, revokes that last identity/path, and only then deletes the database. Project-delete/purge failure retains the cleanup debit; the locked branch preserves it unchanged but removes every preview runtime/client access path and records the preservation handoff. Neither branch may use the current earlier workflow ordering. Only then does the approved exact package key become reclaimable;
cleanup-identity authorization tests proving the protected base-branch workflow can reissue only short-lived tokens for the exact preview synthetic-harness principal, the server derives Project targets only from the protected manifest, and caller-selected IDs, user/persistent Projects, sibling previews, staging, and production are denied. The last cleanup-only API/PM path and identity remain until Project deletion/receipt purge succeeds, then revocation plus maximum token lifetime precedes database deletion;
versioned-S3 teardown proof that prefix-scoped ListObjectVersions, multipart enumeration/abort, version-ID deletion, and delete-marker deletion remove every version beneath singular S3 prefix preview/pr-{n}/ plus the exact package key while denying the plural public child as an S3 selector, sibling prefixes, and persistent buckets; repeated listings must prove zero before bucket removal or object-storage-allocation release;
atomic fleet-admission contention proving requests queue with no partial state when any safe aggregate budget is unavailable;
at least two simultaneous previews proving distinct mounts, credentials, names, networks, routes, messages, URL prefixes, and cleanup authority;
independent failure, cancellation, and orphan reconciliation proving one preview cannot read, stop, revoke, unmount, delete, or release resources owned by another preview;
configuration tests proving the concurrency ceiling derives from measured capacity rather than a hard-coded single slot;
exact cleanup denial for another PR and persistent environments;
broker/secret revocation and mounted-child absence.

Rollback: disable new activation, enumerate and fence/retire every admitted preview record, retain quarantine debits until absence is proved, and restore the previous workflow. STOP: PR code can reach production/staging secrets, networks, mounts, scanner, or Docker control.

Step 7 — Review implementation before any apply¶

All Wave 1 and 2 PRs must be green and review-clean. Attach a redacted cross-repo matrix showing each ADR requirement, implementation diff, automated test, operator proof, and rollback command.

The user reviews:

exact server-config check-mode diff;
production no-change evidence;
capacity/quota/backup owner sign-off;
registrar threat model and negative tests;
staging cutover and restore runbooks;
proposed maintenance window and named operators.

Merging code/config does not authorize apply. Gate B is requested only after Step 8 dry-run evidence exists.

Step 8 — Apply ARRNC staging paused and collect dry-run proof¶

Repositories: no new design change; execute the reviewed server-config workflow. Owner: authorized ARRNC operator. Environment: staging slot only.

Preflight:

verify hostname and inventory target;
record current production semantic state and live Kubernetes staging state;
verify queue/claims without changing them;
verify exact child/ACL/quota/backup classifications externally;
confirm that pre-cutover rollback stops/removes the inactive staging slot; identify a previous digest only if it is independently proven to support paused startup with no queue subscription.

Apply the staging mount, scanner/network, secrets, and container with consumer paused and Bulk intake still served by Kubernetes. Run mount/source/sentinel/sibling denial, quota, EICAR, signature freshness, broker-auth-with-zero-consumer, public URL isolation, resource, restart, log, alert, and stopped/absent rollback. Do not start a legacy digest during this dry run. A previous-digest rehearsal is allowed only after that digest has separately passed the paused, zero-subscription contract.

Required evidence:

production semantic diff empty;
ARRNC staging desired equals reconciled digest/config;
queue consumer remains the existing Kubernetes consumer only;
no writes outside the staging child;
no production credential reachable by the staging container;
rollback removes or stops the inactive slot without changing the sole Kubernetes consumer; if a compatible prior digest exists, its optional rehearsal remains paused with zero subscriptions.

STOP and roll back the new inactive slot on any mismatch. Do not touch Kubernetes.

Step 9 — Gate B: approve staging activation¶

The user explicitly approves or rejects the live handover using Step 8 evidence, the final cluster-gitops drain diff, the maintenance window, and named rollback owners.

Approval covers only:

temporary staging Bulk intake freeze;
Kubernetes staging PDF-agent scale-to-zero with PVC/Application/ingress retained;
ARRNC staging consumer activation;
authenticated staging proof and soak;
later retirement steps if their evidence gates pass.

It does not cover production.

Step 10 — Perform the single-owner staging cutover¶

Repositories/operations: cluster-gitops drain PR plus reviewed ARRNC state transition. Owners: GKE and ARRNC operators together.

Before the freeze, rendered cluster-gitops evidence must validate the current Kubernetes API/Project Management/PVC/static-server mapping as one internally consistent old root/URL, and rendered server-config plus candidate SyRF values must separately validate the ARRNC child and existing-public-root URL as one internally consistent target mapping. They must not be equal yet: the target child has not received its initial copy and activating its URL early would break new links. Preview API, Project Management, agent, and notifier templates must agree on their exact derived previews/pr-N/ authority and scoped hold-table/schedule configuration. The notifier hold store and scheduled reconciler must be deployed healthy, prove durable capture while publication is paused, and provide a complete environment inventory. A missing or contradictory root/URL/store/schedule is a STOP before any queue, route, or consumer change; this cutover grants no IIS mutation authority.

Close only new staging Bulk session admission while existing multipart sessions retain sign/acknowledge/heartbeat/complete/cancel/abandon paths.
Prove active upload sessions quiescent while notifier publication and scheduled reconciliation continue. Release and drain eligible held events until every accepted job is terminal; Uploaded or another known nonterminal state is not a drain.
Only after that terminal proof, set the reviewed environment-scoped Bulk PDF publication pause control while durable capture remains enabled, and capture a hold-table inventory binding S3 objects to event/job/publication state. Preserve any late captured event for reconciliation.
Drain ready/unacknowledged messages, claims, and journal with exactly one Kubernetes consumer, then enumerate/checksum and perform an initial PVC transfer through the reviewed controlled path into the ARRNC staging child, and prove canonical-path, checksum, collision, stored-reference, and candidate public-static-root-read equivalence. Stop on any unexplained difference.
After the initial copy/equivalence proof, atomically switch the reviewed staging PublicBaseUrl content ownership to the stable ARRNC staging child beneath the already-served root while the Kubernetes static server still runs, without changing IIS configuration. At this point—not before the freeze—require exact target URL/root equality across API, Project Management, registrar, and ARRNC agent, then prove existing and newly generated references resolve from ARRNC. Any mixed old/target mapping rolls back the URL switch before consumer activation.
Merge/sync a narrowly scoped cluster-gitops change that sets the staging PDF-agent Deployment to zero while retaining its Application, ingress, Service, PVC, and values.
With the old consumer fenced, perform a final delta transfer, prove no post-fence PVC writes, and reconcile every S3 object/hold-table entry with terminal state or a preserved durable queue event. Prove zero consumers and unchanged queue/claim/read invariants.
Enable ARRNC staging consumer, prove exactly one consumer and correct environment identity, then clear the reviewed Bulk PDF publication pause control and prove scheduled reconciliation drains every eligible held row with stable-identity redelivery safety.
Reopen staging new-session admission and execute the authenticated proof suite.
Observe through the agreed soak with queues, claims, errors, mount, clamd, resources, and public reads monitored.

Authenticated proof covers clean and EICAR uploads, S3/notifier/Rabbit/agent/PM/SignalR/CSV/Study/ existing-public-root path, duplicate delivery, process kill, clamd outage, broker outage, mount outage, disk threshold, queue recovery, and idempotency. It also proves previous-digest rollback only when an independently verified compatible ARRNC digest exists; for the initial activation it instead proves stopped/absent pre-cutover rollback and the retained Kubernetes rollback path after cutover.

Rollback:

close only new-session admission while accepted sessions retain lifecycle paths;
complete, cancel, abandon, or policy-expire accepted sessions while notifier publication and scheduled reconciliation continue, then release/drain held events until every accepted job is terminal;
set the reviewed Bulk PDF publication pause while durable capture continues, inventory any late event as terminal or durably preserved, and keep the Kubernetes consumer off;
fence ARRNC and drain or establish safe recoverability;
prove zero ARRNC consumers;
enumerate ARRNC output created since cutover and reconcile it back to the retained PVC through the reviewed inverse transfer, with checksum/collision and stored-reference proof;
scale Kubernetes back to one by reverting GitOps and prove the static server ready against the reconciled PVC;
restore the previous staging API and Project Management root/PublicBaseUrl configuration and content ownership without IIS mutation;
clear or restore the reviewed Bulk PDF publication pause only after the Kubernetes consumer is sole owner, then prove hold-table reconciliation, PVC health, and old/new link continuity;
intake on and monitor.

STOP: any ambiguity in consumer, claim, journal, queue, storage, or rollback ownership.

Step 11 — Retire Kubernetes staging reversibly¶

Repository: camaradesuk/cluster-gitops. Owner: GKE platform. Dependency: Step 10 accepted soak.

PR 11A keeps the Deployment at zero, disables/removes ingress exposure, and preserves the PVC and rollback metadata. Before merge, enumerate/checksum PVC contents and reconcile them against ARRNC staging. Create a VolumeSnapshot or equivalent controlled archive and prove restoration into an isolated claim. Record owner, expiry, and restore instructions.

PR 11B, after the retention/soak gate, removes:

syrf/environments/staging/pdf-agent/config.yaml and values.yaml;
the generated Argo CD Application and ingress/Service/Deployment ownership;
obsolete preview/production PDF-agent Kubernetes configuration;
the PVC only after snapshot/archive proof and the explicit expiry checkpoint.

ApplicationSet effects from argocd/applicationsets/syrf.yaml and syrf-previews.yaml must be shown in rendered diff. Argo orphan warnings are reconciled; no manual delete is used.

Tests/proof:

Helm/ApplicationSet render before/after;
Argo dry-run/diff and sync health;
queue/claim/ARRNC consumer evidence before and after;
snapshot restore and file checksum proof;
Git revert plus restore recreates the prior staging shape without queue consumption until explicitly enabled.

Rollback: execute the Step 10 admission/session/notifier quiescence and final pause/inventory gate, then fence ARRNC, revert GitOps, and restore the tested artifact without enabling the old consumer or route. Inventory every ARRNC file written after artifact capture, perform a collision-safe checksummed inverse transfer into the restored claim, and prove every current BulkPdfDeliveredPath/public reference resolves there before enabling the old static route and sole Kubernetes consumer. STOP: no restorable artifact, unexplained file difference, queued/in-flight work, or Argo prune outside the expected resource set.

Step 12 — Remove obsolete SyRF Kubernetes authority¶

Repository: camaradesuk/syrf. Owner: SyRF maintainers. Dependency: Step 11 accepted.

Remove the PDF-agent Helm deployment target and PVC/static-server tests/templates if no longer used, delete preview GitOps deployment outputs for this service, and update:

src/services/pdf-agent/.chart;
.github/workflows/pr-preview.yml and CI chart packaging paths;
CLAUDE.md;
ADR-012 and the 2026-08-11 design/plan supersession notices;
feature and operator documentation.

Keep the Docker image, runtime tests, ARRNC deployment, and application storage-authority contracts. Documentation validation and a repository-wide search must find no operative statement that directs a PDF agent to Kubernetes/PVC.

Rollback: restore documentation/chart only if Kubernetes rollback is explicitly selected. STOP: another active workflow still consumes the chart or cluster-gitops still references it.

Step 13 — Prove concurrent dynamic previews end to end¶

Repositories/operations: trusted SyRF PR and ARRNC preview-family registrar. Owners: SyRF CI and ARRNC operator. Dependency: Waves 1C and 2 green; staging accepted.

Use at least two non-sensitive, explicitly approved same-repository PR fixtures concurrently:

opt in, approve, build, attest, and register each immutable digest while proving atomic reservation and queueing when the configured aggregate budget is exhausted;
prove each exact previews/pr-N allocation record, ARRNC-created post-mount readiness sentinel, ACL/quota, and cross-preview storage/mount denial;
prove isolated network/scanner/broker credentials, per-preview and aggregate resource caps, and host-enforced block-device bandwidth/IOPS ceilings under simultaneous bounded I/O;
use the protected harness for each fixture to create the matching synthetic Project, create and attach its SystematicSearch, seed bounded Study/PDF-path rows, take the non-empty fingerprint, initiate/upload/acknowledge/complete every fixture part, force notification to reach the durable release gate before job persistence, prove it remains held until Uploaded and then converges one logical publication with stable-identity redelivery safety, and run clean and EICAR authenticated preview uploads; prove each clean delivered PDF is publicly readable at the existing base plus its exact previews/pr-N/ prefix; prove the wrong preview cannot derive that link or access its storage/mount, while EICAR, malformed, partial, and quarantined output produces no link or read. A valid delivered public URL remains fetchable when known, regardless of which preview's client requests it;
while one preview remains healthy, fail the other and prove isolation, then recover it; while the recovered preview remains healthy, cancel the first and prove that containers, mounts, credentials, names, networks, routes, messages, URL prefixes, queues, storage, and cleanup effects remain confined to their owning PR and persistent staging remains healthy, then recreate the cancelled preview from its immutable digest and fresh identities;
deliberately suppress cleanup for one preview, prove the reconciler completes only that orphaned preview while the other remains healthy, and prove any quarantined remainder stays charged to the fleet budget;
close/disable every remaining fixture and prove consumer fence, queue/claim convergence, hold-to-outbox retirement, credential revocation, public-child disappearance, container/network/runtime removal, unmount, and exact-child retirement. The cleanup record/debit remains without an agent slot through the real P7D receipt window; after ordinary deletion of the teardown-owned synthetic Project and safe receipt purge, prove exact database and record absence and only then the corresponding atomic fleet-reservation release.

No production/staging secret-name material may appear inside the preview container even if values are redacted. The preview must be unable to connect to persistent agent/scanner networks.

Rollback: disable preview activation globally; enumerate, fence, and safely retire every fixture admitted by the concurrent rehearsal; retain any quarantined resource debit until absence is proved. STOP: active runtime/session/job/hold/outbox/release-record reclamation, container/mount/public-child removal, or quarantine handoff exceeds 24 hours; the intentional non-authorizing P7D receipt and isolated-database cleanup debit is excluded from that clock but must purge and release within 24 hours after its latest snapshotted receipt expiry. Also stop if cross-preview or persistent-environment isolation fails, fleet admission creates partial state, or the registrar needs broader authority.

Step 14 — Prepare production as a separate future phase¶

Do not execute under this plan. After staging and preview acceptance, write a new production promotion plan using current evidence. It must include production child/backup proof, queue/claim handover, existing-file/public-root compatibility, capacity, maintenance/rollback owners, previous digest, observability, and an explicit Gate C request.

No prior approval, merged implementation, staging apply, or preview success implies Gate C.

4. Cross-repository acceptance matrix¶

Requirement	Automated proof	Operator proof	Activation blocker
Exact child isolation	canonical/path/ACL negative tests	direct mount, sibling denial, no-follow write/delete	any root/sibling access
No production credential leakage	rendered-secret scope and workflow tests	container env/mount/network inspection	any persistent secret reachable by PR code
Immutable image parity	digest/provenance tests	reconciled digest; compatible prior-digest rollback when one exists, otherwise stopped/absent and retained-Kubernetes rollback	mutable tag or unverified commit
ClamAV parity	readiness/EICAR/signature tests	clean + EICAR, signature-age alert	stale/unreachable scanner
RabbitMQ isolation	vhost/queue/user and paused-state tests	consumer count, ready/unacked/error/claim evidence	more than one or wrong consumer
Resource/capacity	declared-limit and concurrency tests	ARRNC free capacity, quota and failure alert	unbounded slot or missing quota
Failure recovery	journal/redelivery/idempotency tests	process/mount/broker/clamd/disk rehearsals	duplicate effect or ambiguous claim
Kubernetes rollback	rendered GitOps/snapshot tests	isolated snapshot restore and checksum	untested restore or unexpected prune
Preview teardown	race/idempotency/path tests	event and orphan-reconciler teardown	residue, credential, mount, or record remains
Production untouched	semantic-diff test	before/after inventory evidence	any production change

5. Required PR metadata and evidence¶

Every implementation PR states:

its wave, dependencies, repository owner, and exact environment effect;
files and contracts changed;
tests and negative tests run;
whether merge triggers an apply (default: no);
rollback and STOP conditions;
production impact, which must be none before Gate C;
links to evidence using redacted identifiers.

Operational evidence records timestamps, commit/digest, desired/reconciled state, consumer/queue/ claim status, checks performed, outcome, and named operator/rollback owner. It never records credentials, participant information, project names, or uploaded filenames.

6. Phase gates and completion¶

Phase 1 is complete only when this planning PR:

contains the superseding ADR, design, numbered plan, and stale-authority cleanup;
passes documentation/contract checks;
has no unresolved valid review findings;
is green and review-clean;
is not merged;
is presented for explicit Gate A approval with the exact Phase 2 waves.

Phase 2 begins only after Gate A. It stops again at Gate B before live staging cutover. Production is a future separately planned phase that stops at Gate C.