Bulk PDF Agent ARRNC Hosting Correction — Design

Decision: ADR-015. Executable plan: 2026-08-30-bulk-pdf-agent-arrnc-hosting-plan.md.

This document supersedes every PDF-agent hosting/storage-deployment statement in the 2026-08-11 Bulk PDF design. It does not supersede that design's upload, security, processing, or domain behavior.

1. Objective and non-goals

All SyRF Bulk PDF processing environments use separate Docker containers on ARRNC API and exact environment-owned children of the Edinburgh Camarades share. Production and staging are persistent, while PR preview slots are bounded and lifecycle-managed.

This planning change does not:

• change runtime code, server configuration, cluster GitOps, credentials, storage, queues, or live containers;
• authorize an ARRNC apply or a Kubernetes deletion;
• merge itself or approve Phase 2;
• promote or alter production.

2. Evidence and precedence

The correction resolves four facts in this order:

1. The owner's later explicit ARRNC-for-all-environments decision is the product authority.
2. The preserved Juniper rewrite corroborates child-scoped production, staging, and preview roots, bounded preview slots, and teardown, but is not copied wholesale and remains untouched.
3. Current repositories define the implementation surface: SyRF has the agent and application authority; server-config has static production ARRNC foundations; arrnc-api-deploy has the gatekeeper transport; cluster-gitops owns the live staging Kubernetes workload.
4. Read-only live evidence proves the stale recovery is active: Argo CD reports staging Healthy/Synced with one Deployment, three containers, ingress, and a bound 20 Gi PVC.

Operational approval of that recovery was not an explicit architectural reversal. Therefore the live workload is preserved as the rollback source until ARRNC staging is proven.

3. Target architecture

flowchart LR
    WEB[Web / API] -->|multipart ZIP upload| S3[Environment temporary S3<br/>exact bucket/prefix]
    S3 -->|immutable version event| N[Notifier hold/reconciler]
    N -->|version-pinned release, no URL| APP
    APP -->|exact version HEAD| S3
    APP[SyRF API / PM<br/>environment authority] -->|durable environment job| MQ[RabbitMQ]
    MQ --> A[Selected environment agent]
    APP -->|fresh presigned GET in<br/>accepted claim response| A
    A -->|temporary ZIP GET| S3

    subgraph ARRNC[ARRNC API host]
        REG[Config service<br/>bounded registrar]
        P[production agent<br/>persistent, untouched]
        T[staging agent<br/>persistent]
        R[PR-N agent<br/>dynamic]
        CP[production clamd]
        CT[staging clamd]
        CR[PR-N clamd]
        MP[exact child mount<br/>production]
        MT[exact child mount<br/>staging]
        MR[exact child mount<br/>previews/pr-N]
        REG --> P
        REG --> T
        REG --> R
        P --> CP
        T --> CT
        R --> CR
        P --> MP
        T --> MT
        R --> MR
    end

    A -->|validate + ClamAV scan + extract<br/>atomic copy| M[Exact environment child mount]
    M --> IIS[Existing public static root<br/>plus exact child prefix]

Only one environment agent receives a given queue. The diagram shows possible slots, not simultaneous consumers for one environment. Production, staging, and each approved preview use separate container identity/configuration, broker authority, temporary S3 bucket/prefix access, and exact child mount. Preview code receives neither production storage credentials nor broad AWS credentials. The rollout creates no IIS binding, hostname, certificate, or virtual-directory change.

4. Repository and ownership boundaries

Repository	Owner	Authoritative responsibilities	Expected implementation surfaces
camaradesuk/syrf	SyRF maintainers	image, runtime configuration, application storage authority, broker naming, notifier release gate, notifier OAuth client/scope, workflows, health and E2E harness	src/services/pdf-agent; src/services/s3-notifier; src/services/identity/SyRF.Identity.Endpoint/Program.cs and Services/OpenIddictClientSeeder.cs; src/services/identity/SyRF.Identity.Endpoint.Tests/OpenIddictClientSeederTests.cs; src/libs/kernel/SyRF.SharedKernel/Settings/BulkPdfStorageSettings.cs; .github/workflows/ci-cd.yml; .github/workflows/pr-preview.yml; .github/workflows/_preview-gitversion.yml; CLAUDE.md; docs
camaradesuk/server-config	ARRNC platform maintainers	users/ACLs, child mounts, credentials, Docker networks, ClamAV, persistent slots, dynamic registrar policy, reconciliation, verification	vars/projects.yml; vars/portal-env-overrides.yaml; deployments/arrnc-api.yml; roles/cifs_mounts; roles/clamav; roles/docker_volumes; roles/gatekeeper; roles/verify; config-service; filter_plugins/tests/test_syrf_pdf_agent_hosting.py
camaradesuk/arrnc-api-deploy	ARRNC gatekeeper maintainers	constrained deploy/apply transport and installer contract	scripts/container-web-deploy; scripts/container-web-apply; scripts/install-gatekeeper; config/services.conf.example; tests
camaradesuk/cluster-gitops	GKE platform maintainers	order notifier retention configuration safely; later perform reversible drain and retirement of the old staging Application, ingress, and PVC	syrf/services/s3-notifier/values.yaml; syrf/environments/staging/s3-notifier/values.yaml; syrf/environments/preview/services/s3-notifier/values.yaml; syrf/environments/staging/pdf-agent; syrf/environments/production/pdf-agent/config.yaml; syrf/services/pdf-agent; argocd/applicationsets/syrf.yaml; argocd/applicationsets/syrf-previews.yaml
University Camarades storage service	storage owner; implementation interface is a preview-activation dependency, not a persistent-staging merge gate	audited per-PR child/allocation, principal, ACL, quota, I/O-limit provisioning, reset/fencing, and exact retirement	scoped provisioning operation or isolated allocation pool; audit and simultaneous bounded-I/O proof

Secrets stay in their existing repository- and host-owned secret systems. Plans, PR descriptions, logs, and proof bundles use secret names and redacted fingerprints only.

5. Storage contract

5.1 Canonical layout

The application value object remains the canonical naming authority:

Runtime	Application root	Host mount	Container output	URL/read contract
Production	production	/srv/data/syrf-pdf/production/output	/app/data/output	/production/
Staging	staging	/srv/data/syrf-pdf/staging/output	/app/data/output	/staging/
Preview N	previews/pr-N	/srv/data/syrf-pdf/previews/pr-N/output	/app/data/output	/previews/pr-N/

Scratch is a private host directory under the environment slot, never a share path and never served by IIS. Every public URL must end in the same canonical root accepted by BulkPdfStorageSettings: production/, the stable staging/ child, or lifecycle-owned previews/pr-N/. The existing public base and static-root mapping are reused; no per-preview IIS binding, virtual directory, hostname, certificate, or other IIS mutation is required or authorized.

BulkPdfStorageSettings keeps storage and link authority aligned by requiring an exact canonical HTTP(S) PublicBaseUrl for all three runtime kinds. For a preview, the registrar derives that URL from the fixed existing public base plus the canonical previews/pr-N/ suffix; neither PR code nor the caller can choose a host, scheme, port, or path. Public preview PDF links are intentional for explicitly approved trusted same-repository previews, after the normal scan/type/copy boundary.

The fixed base is a separately rendered, deployment-owned TrustedExistingPublicBase authority, not a value inferred from or supplied alongside PublicBaseUrl. It fixes the exact scheme, host, effective port, and already-served root path. API, Project Management, and registrar each compare their canonical child URL against that independent authority and one another before any upload, job, registration, or link side effect. Missing, arbitrary-prefix, wrong-origin, wrong-port, or inconsistent values fail closed.

5.2 Isolation and permissions

Each environment uses a distinct SMB principal, root-owned credential file, child mount, Unix UID/GID mapping, and server-side child ACL. Production, staging, and preview identities cannot list or traverse sibling children. The agent is non-root and receives only the exact output bind and its own scratch.

Preview PR code receives neither credential files nor mount commands. Root-controlled server-config mounts the child before the approved rootless container starts. No container gets CAP_SYS_ADMIN, host PID/network namespaces, privileged mode, the Docker socket, or a writable host root.

The storage owner, not GitHub or the PR container, atomically creates or allocates and retires the remote previews/pr-N child with its distinct principal, ACL, quota, immutable allocation record, and server-side connection/bandwidth/IOPS policy through an audited operation or equivalently isolated pre-provisioned pool. The operation rejects traversal, symlink/reparse-point, sibling, and local mount-underlay targets. Allocation, reset/fencing, identity rotation, and audit prevent prior or sibling PR exposure. The storage owner does not create the runtime readiness sentinel. The registrar reserves the allocation in the same atomic fleet admission transaction, verifies the allocation record, mounts and proves the exact source/ACL/quota plus sibling denial and I/O limits, and only then creates the sole ARRNC-owned readiness sentinel. The missing operation is an implementation dependency to deliver before preview activation; it does not change the approved ARRNC/container architecture or block persistent staging work.

ARRNC root mounts the exact remote child and bind-mounts only that child into the disposable rootless container. SMB credentials, sibling mounts, and the common root are absent from the container. Lifecycle tests cover exact source, no-follow path handling, sibling denial, mount loss, ordered container stop/unmount, and exact allocation retirement.

The preview bridge is combined with root-controlled deny-by-default egress enforcement. Only its own scanner and the minimum controlled DNS, presigned-S3 HTTPS, and preview RabbitMQ destinations are allowed. Host gateways, metadata endpoints, ARRNC/UoE internal CIDRs, persistent-environment endpoints, and arbitrary internet egress are denied. PR code cannot change the policy. Both positive routes and denied host/internal routes are activation tests; a bridge without this enforcement is not an isolation boundary. Root-controlled shaping applies per-preview ingress/egress rate and burst ceilings to allowed destinations, including presigned S3. Simultaneous bounded transfers must leave persistent services healthy and the ARRNC uplink within its reserved aggregate budget.

The trust boundary admits only explicitly authorized, same-repository SyRF heads. Each gets its own disposable rootless Docker container on ARRNC with a pinned digest, non-root UID, dropped capabilities, no-new-privileges, read-only root filesystem where supported, seccomp/AppArmor, private network namespace, no Docker socket, no host namespace/device/admin access, and no production/staging mount, credential, route, or service identity. A pull request whose head repository is not exactly the SyRF base repository, including every fork, cannot use this path; neither can a stale, closed, or unapproved same-repository head. VM/microVM isolation is optional future hardening, not an activation prerequisite for this approved workload policy.

The same approval gates the credentialed notifier artifact. The ordinary PR workflow may build and attest PR-head code, but its mutable lambda-packages/pr-N.zip key is never referenced by a function that has the notifier credential. A protected base-branch workflow binds an immutable S3 object version and package SHA-256 to the repository, PR, exact head SHA, and maintainer approval; immediately before each function deployment it revalidates that binding, and it separately revalidates before injecting the client secret into the reconciler alone. Synchronize invalidates the approval, and new PR code cannot replace either function or receive a preview credential until re-approved. This protects credential delivery before code becomes trusted; it does not add protected serving, a promotion tree, VM isolation, or IIS authority.

Fleet admission atomically reserves CPU, memory, PIDs, block I/O, Docker/runtime disk, scratch, storage quota, ports/routes, network, RabbitMQ, and scanner capacity before creating any slot. The configured concurrency limit derives from measured ARRNC capacity and can exceed one; requests queue without partial state when the safe aggregate budget is unavailable. Bounded write/discard/fsync and simultaneous-preview tests prove the reservations and acceptable persistent-slot storage latency/health. Immediately before a queued request reserves resources, the registrar atomically revalidates the exact repository, open/enabled state, current head SHA, verified digest, and current authorized-maintainer approval; stale requests are discarded without provisioning.

Preview output is generated by code under test, but an approved trusted preview uses the same malware scan, PDF type validation, and atomic-copy contract as staging and production before output is delivered. The existing public static root serves that delivered output at the derived previews/pr-N/ prefix. It must not expose scratch, partial, quarantined, sentinel, sibling, or unscanned content through link derivation or directory listing. This is a public static boundary: anyone who already knows a valid delivered preview URL may fetch it, and no caller-environment authentication boundary is claimed. Storage principals, mounts, and application link builders still cannot list, traverse, or derive another preview's child. No host-side output-inspection API or second promotion tree is required. Explicit maintainer approval of the exact same-repository head and digest is the code-trust decision; these safeguards contain accidental defects and do not claim to isolate deliberately malicious approved code from its own writable child.

Preview input is synthetic-only. A protected base-branch harness outside PR execution uses a protected synthetic identity/API path to create a matching Project, create and attach its SystematicSearch, seed bounded synthetic Study/PDF-path rows, prove its authorization and cleanup ownership, obtain the resulting exact GetStudyPdfPaths fingerprint, initiate the persisted upload session/job, upload and acknowledge every approved non-sensitive multipart fixture part, complete the session, and read only that synthetic scope's bounded history. A trusted notifier release gate conditionally persists the validated immutable release envelope, including bucket, key, and S3 version ID but no presigned URL, to an environment-scoped DynamoDB hold table before handler success. A bounded EventBridge schedule invokes a separately named reconciliation Lambda that leases due rows, uses a notifier-only machine identity against a dedicated read-only endpoint on the existing authenticated SyRF API boundary, and publishes the stable-identity object-ready command to Project Management only when the minimal response proves the same environment, job, and object key are committed as Uploaded. For any held cleanup candidate, including one that becomes terminal before publication, the same route returns only terminal/nonterminal disposition and active/retriable-claim presence for that binding; it never returns a version or URL. The endpoint is backed by the Project Management job store but grants no direct Mongo access and does not require a new object-version field in Project BSON during the split API/PM rollout. Project Management validates the stable job binding and performs an exact version-specific S3 HEAD for the release command's bucket, key, and immutable version before accepting it; absence, replacement, or metadata/size/hash mismatch fails closed. Before acknowledging the object-ready command, Project Management idempotently persists a separate BulkPdfUploadReleaseRecord keyed by environment/job/stable-message identity with the immutable bucket/key/version/size/hash/metadata binding and release state. This record is not embedded in Project BSON and contains no URL. Object-ready registration remains reversible and the Project job remains cancellable until Project Management atomically persists the agent's processing claim token. Cancellation that wins that race forces a later claim to emit no URL and transactionally convert the record to CleanupOnly while terminalizing the job. If the claim wins, cancellation returns processing_already_started. Only after an accepted claim commit does Project Management repeat the exact version-specific HEAD and mint a fresh version-pinned URL; a retry after commit reuses the stable claim and remints the capability without double-processing. Cooperative mid-processing cancellation is outside this rollout. If the authenticated state is terminal before publication, the reconciler never emits agent work. It instead sends a stable IBulkPdfUploadCleanupRegistrationCommand carrying the same hold binding; Project Management validates the live terminal job and exact version-specific HEAD, then idempotently persists the record in non-claimable CleanupOnly state before acknowledging registration. Exact-version reclamation cannot start until either a processing-capable or cleanup-only record exists.

The selected issuer owns one dedicated confidential notifier client per environment, with the client-credentials grant and exactly two scopes: release-state read and cleanup-acknowledgement write. The read scope admits only the job-bound release-state route and the separate no-job-identity storage-binding resolver, whose request is limited to the client's mapped environment plus bucket/key/version and whose response is a bounded zero/one/multiple server-owned binding result. The write scope admits only idempotent quarantine-fence registration/transfer, cleanup acknowledgement, and subsequent hold-retired confirmation for the same environment/object binding and immutable evidence/delete digest. Neither scope permits enumeration or general Project/job mutation. The API keeps an explicit fail-closed mapping from the authenticated client identity (client_id/azp, plus subject where available) to exactly one environment and rejects that otherwise-valid token on every sibling environment's route. OpenIddict environments seed it in OpenIddictClientSeeder; an environment still using Auth0 receives an equivalent dedicated machine-to-machine client from the named tenant administrator. The environment secret owner creates and stores the secret; rotation creates a next-generation client with the same two scopes, moves only the reconciler to its ID/secret and proves both read routes plus all three cleanup-write operations before revoking/removing the old client. Preview teardown keeps that client admitted and usable by the reconciler until zero active hold rows, zero quarantine-ledger rows, zero retirement outboxes, and zero active release records are proved. Only then does it deny the client ID at the API, revoke token entries where supported, disable/delete the client, and retain the deny fence until the maximum already-issued token lifetime has elapsed before claiming credential absence. A preview never receives a staging or production client; production identity is untouched before Gate C.

For OpenIddict, Identity also maps both notifier scopes to the exact API/introspection audience. The API bearer selector admits the read scope only to the release-state and storage-binding-resolver routes and the write scope only to quarantine-fence, cleanup-acknowledgement, and hold-retired-confirmation routes, while retaining the existing syrf_api requirement everywhere else. The resolver rejects a job ID or caller-selected environment. Route-authentication tests cover each narrow success, cross-scope, cross-environment, enumeration, sibling-prefix, and broader-controller denial. The admitted API routes overwrite environment from the authenticated client mapping and send stable messages to named Project Management storage-binding, quarantine-fence, and hold-retired consumers; those consumers alone own server-side lookup, durable deletion fencing, or conditional receipt update. Neither API nor notifier receives direct Mongo authority.

Protected configuration targets both the S3 event handler and separately named reconciler with their respective complete but different environments. The handler gets only event identity, exact hold-table/object-prefix, publication-pause, and existing broker settings; it gets no API/auth endpoint, audience, client ID, or secret. Only the reconciler gets the release-state and storage-binding-resolver, quarantine-fence, cleanup-acknowledgement, and hold-retired-confirmation API endpoints, issuer, token endpoint, audience, client ID, and secret-backed client secret. Both retain the repository's existing supported Lambda network shape. The required boundary is exact protected HTTPS URL configuration, redirects disabled, direct Mongo access absent, and environment-scoped S3/DynamoDB/broker/API permissions. A dedicated notifier VPC, private subnets, DNS or stateful egress firewall, VPC endpoints, and ENI-policy work are optional future hardening only; they are not a rollout gate. The existing SyRFS3NotifierLambdaBoundary is different: its declarative owner, camarades-infrastructure/terraform/lambda/ack-iam.tf, is a narrow staging-only IAM prerequisite because the current cap cannot authorize staging hold-table writes/reconciliation, its exact failure queue, staging version deletion, or authoritative exact-prefix inventory. Before staging activation, that boundary adds only those staging actions, including reconciler exact-table PutItem/UpdateItem constituents for the TransactWriteItems call, plus separate DeleteItem for acknowledged outbox removal, replay-fenced capture/transfer tombstone removal, and full-binding conditional RejectedPreHold retirement, and prefix-conditioned ListBucketVersions; IAM has no dynamodb:TransactWriteItems action. ConditionCheckItem is present only if a separate condition-check item is implemented, and the event handler remains denied DeleteItem. Each chart role remains further pinned to its one rendered staging table ARN/indexes and exact staging prefix. This dependency adds no preview, production, or VPC/network authority. All preview-family IAM is created and retired only by Wave 1C under the registrar contract below.

Missing configuration, authentication, binding, or availability fails closed and remains held. Pending, leased, published, retention-waiting, and receipt-retirement-pending active holds, plus QuarantinedHold items, carry no TTL. Publication starts terminal-state observation, not row expiry. After proved exact-version deletion and cleanup acknowledgement, the reconciler marks the row ReceiptRetirementPending. One conditional DynamoDB transaction atomically puts an immutable HoldRetirementOutbox item and converts the same stable- event-keyed hold in place to CaptureTombstone; it cannot report retirement while an active hold exists. Exact-binding duplicate capture is success against the tombstone, while mismatch poisons. The outbox calls the dedicated hold-retired-confirmation route under the same environment-bound cleanup-write scope, conditionally stores the returned immutable acknowledgement ID/time, and is deleted only afterward. An uncertain response resumes from the pending row or outbox. The tombstone retains the immutable binding/delete digest through a snapshotted replay deadline and is conditionally removed only after that deadline, an empty visible/in-flight failure queue with a newer oldest-message watermark, and authoritative inventory checkpoint. Redrive after that fence is forbidden. None has TTL; TTL is not teardown or absence proof. Lease-expiry recovery plus downstream idempotency tolerate crashes and redelivery; no second S3 event or transport exactly-once assumption is required.

Protected cluster-gitops configuration owns the terminal payload retention: staging renders reconciler.terminalPayloadRetention: PT24H and every preview renders PT1H. Schema/render checks reject a missing, zero, negative, malformed, or lower duration; production remains unset and inactive until Gate C. On the first authenticated terminal/no-active-or-retriable-claim observation, the reconciler conditionally snapshots the policy value/version, terminal-observed time, and retention-eligible time in the row. A later configuration change affects only rows first observed afterward and can never recompute or accelerate an existing row.

Their exact S3 object versions share that durability boundary. Staging and preview buckets apply no age-based lifecycle expiration to completed Bulk PDF objects; the incomplete-MPU lifecycle rule remains only a safety net. Before that expiry is removed, the handler classifies reserved Bulk PDF key shapes before trusting metadata. A successful exact-version metadata read that proves malformed, missing, or unrecognized immutable metadata creates a non-publishable RejectedPreHold row with no caller-derived job authority. Its snapshotted invalid-object retention is PT24H staging or PT1H preview. Before cleanup, the reconciler uses the dedicated authenticated resolver under its environment-bound release-state read scope; Project Management resolves only its server-owned environment/bucket/key/version storage binding and accepts no job identity. The deletion-independent BulkPdfUploadStorageBinding is persisted at completed-object transition without a Project BSON version field and already blocks pruning/Project deletion. It retires only with objectless absence proof or the cleanup-acknowledgement transaction, so this read cannot open a race before quarantine fencing. Zero matches permits unowned cleanup: after retention the reconciler records the resolver-decision digest, exact-version delete result, and a later authoritative-inventory absence checkpoint in the row, then retires it with a full-binding conditional delete. Crashes resume from the row and this branch creates neither fabricated acknowledgement nor retirement outbox. One matching Uploaded or authoritative terminal completed-object job repeats exact HEAD; Uploaded terminalizes failed-cleanup-pending and releases capacity in the same transaction that persists CleanupOnly, while the terminal branch preserves disposition and history. Both idempotently persist/reuse CleanupOnly with no agent work. Objectless contradiction, active/retriable claim, or ambiguity first uses the narrow write route: Project Management re-resolves candidates and atomically persists a no-TTL BulkPdfUploadQuarantineFence for each candidate plus an immutable fence-set receipt without exposing identities. Only then does the stable capture item become QuarantinedHold; unavailable state remains a retryable active hold. The quarantine retains binding, evidence/delete state, retry authority, duplicate suppression, cleanup client requirement, source event identity, fence-set receipt, and aggregate-budget debit. It has no TTL and leaves cleanup custody only after normal reclamation or a durable idempotent cross-store handshake; there is no assumed Mongo/DynamoDB transaction. PM first records TransferPending for a stable transfer ID while every candidate fence remains active. The destination conditionally accepts a dormant prepared hold carrying the same stable capture key/source event, immutable binding/evidence, retry and duplicate obligations, cleanup-client requirement, and stable debit ID. PM then durably links that receipt without replacing the original fences. Only after the PM linkage receipt may the source become QuarantineTransferTombstone; the destination activates against the source receipt, and PM finalizes only after both receipts. Stable IDs make each step queryable and retryable after crashes/lost responses, conflicting retries fail closed, and capacity counts the shared debit once. Only the linked destination may acknowledge exact cleanup and clear the PM fences. The source tombstone retains the debit and suppresses duplicates. Purge requires the ordinary CaptureTombstone replay/queue/inventory fence plus matching destination exact-version cleanup acknowledgement, PM's durable fence-set-clearance/final disposition, and authoritative shared- debit release. Pending/failed destination cleanup retains the tombstone, table, purge schedule, and dynamic record. Evidence-only copying cannot clear teardown. Read/config failures still retry, while an encrypted exact-environment on-failure queue receives exhausted invocations. The scheduled reconciler also checkpoints exact-prefix version inventory and reconstructs missing holds after queue loss/expiry or destination failure. Unknown objects outside the reserved prefixes keep existing skip behavior, and expiry cannot be removed before this recovery proof. The environment's bounded EventBridge-scheduled reconciler has a distinct terminal-reclamation phase. Its durable row is the immutable locator and cleanup ledger, retaining the exact environment/job/bucket/key/version plus publication, terminal-observed, retention, conditional cleanup-lease, and delete-result state. For any held candidate it re-reads the authenticated exact-job route for terminal disposition and active/retriable-claim absence. The separate BulkPdfUploadReleaseRecord is also the deletion-independent cleanup authority. Before bounded history may prune a terminal job, a processing-capable or cleanup-only record must already hold the immutable binding; the same Mongo session transaction records terminal disposition and claim state there and only then removes embedded history. A terminal job with a completed-object cleanup obligation that is awaiting cleanup registration is skipped by pruning and alerted. That obligation exists after multipart completion or whenever a matching hold/release record exists. An objectless cancellation/abandonment instead atomically persists the sweeper's MPU-absent and object-absent proof with terminal state; it creates no fictional release record and needs no receipt. Whole-Project deletion fails closed while any nonterminal Bulk PDF session/job, completed-object terminal job without a matching completed cleanup receipt, objectless terminal job without its persisted absence proof, active processing/cleanup record, or an unresolved BulkPdfUploadStorageBinding or BulkPdfUploadQuarantineFence exists. Bounded history pruning applies the same fence. Cleanup acknowledgement conditionally verifies the exact delete/ack digest and current owner, converts the active record to its receipt, completes the storage binding, and retires every member of an optional fence set with one immutable final-disposition receipt in the same PM transaction. Missing/extra/mismatched/partial members fail closed; replay returns the same receipt. It does not remove the matching embedded terminal history entry. The job and report remain visible until ordinary bounded-history pruning, while the receipt satisfies the completed-object Project-deletion fence. The route reads the live job or this record, so aggregate pruning/deletion cannot erase cleanup authority. A row that terminalizes before publication is never published; after its snapshotted retention, either that explicit disposition or a published terminal job conditionally leases exact-version cleanup, but only after the corresponding cleanup-only or processing-capable record is durable. The version-specific delete result is recorded conditionally before the reconciler sends an idempotent authenticated cleanup acknowledgement. The acknowledgement executes the complete binding/fence-set transaction above and returns the same non-authorizing receipt on retry. The reconciler then persists ReceiptRetirementPending and the immutable delete/ack result in the hold row, atomically creates the immutable retirement outbox and converts the stable capture key to CaptureTombstone, and only the outbox obtains retirement confirmation through the dedicated authenticated route described above. The outbox persists that acknowledgement before deletion. Notifier absence proves zero active release records, zero active holds, zero QuarantinedHold items, and zero retirement outboxes. Capture tombstones own no cleanup authority or container capacity; transfer tombstones retain only the shared debit/replay evidence until their stricter purge fence clears. The exact table, credential-free protected tombstone-purge owner, and tombstone-only dynamic record remain until every tombstone conditionally clears its replay/queue/inventory fence; each transfer tombstone also requires matching destination cleanup acknowledgement, PM fence-set final clearance, and authoritative shared-debit release. Only then is final notifier/table/record absence true. An unlocked retained database carries only a bounded cleanup debit. Protected values set receipt retention to P90D staging and P7D preview, snapshot receipt creation/policy/expiry, and cannot accelerate existing receipts; production remains unset before Gate C. The Project Management purge owner removes a receipt only after expiry, acknowledged retirement, OutboxDrainedAt from a later strongly consistent exact-key DynamoDB absence proof, zero active record, and ordinary pruning of its embedded job or Project deletion. A duplicate within retention returns the stored result; after safe purge it fails closed with 410 CleanupReceiptRetired, while missing retirement confirmation or drain proof blocks purge and alerts. A Project Management receipt reconciler derives the exact outbox key from each immutable receipt and uses strongly consistent GetItem on only the mapped environment table; absence conditionally records OutboxDrainedAt. Its traced principal has no Query, Scan, transaction, write, sibling/cross-environment, or production-table authority. This keeps a PM-commit/lost-response outbox replayable even past receipt expiry. Uncertain responses retry idempotently against the same version. The environment reconciler role is the named principal and receives only exact-prefix GetObjectVersion/DeleteObjectVersion and prefix-conditioned ListBucketVersions through its role, boundary, and bucket policy. Unversioned delete, unscoped/sibling list, write, siblings, cross-environment access, event-handler and target Project Management delete are denied. To make that target executable, BulkPdfMultipartSweepProcessor and BulkPdfUploadController.CancelUpload no longer call unversioned DeleteObjectAsync when completion races cancellation or abandonment. Both record the cleanup-pending obligation without releasing active capacity. Durable notifier capture supplies the exact version; Project Management atomically persists/reuses CleanupOnly, terminalizes the job, and releases capacity. Only that cleanup-authority commit permits release; the same exact-version reclaimer later deletes. No delete marker, false object-absence proof, or agent command is emitted. Deploy and prove that behavior while the legacy narrow PM delete grant still exists, then remove the grant declaratively. Rollback retains the behavior/permission pair; an old sweeper never runs without its required permission. Missing state, auth, retention, lease, or deletion proof fails closed without TTL, so an extended outage cannot leave a live hold pointing to an expired payload. Preview IAM selects the singular S3 upload prefix preview/pr-{n}/ returned by SyrfSettings.GetS3PathPrefix(), never the plural public child previews/pr-N/; neither namespace is derived from the other. Preview-wide version/marker/MPU cleanup remains a separate fenced teardown responsibility. The shared event handler retains GetObjectVersion for every environment bucket/prefix it already handles for ReferenceUpload and ReferenceUpdate, adding the Bulk PDF prefix where required, because metadata is read before upload-kind dispatch. All handler deletes remain denied. For staging/previews, close new admission briefly, quiesce accepted sessions, inventory legacy versions, suspend completed-version expiry, and prove the rendered rule absent before enabling durable hold capture. Then prove the route, ledger, paused scheduled worker, exact policies, and synthetic terminal-version reclamation before unpausing publication. Failure leaves capture disabled and never restores expiry over captured rows; production lifecycle is untouched before Gate C.

The Project Management principal has exact-bucket/prefix s3:GetObjectVersion for the version-specific HEAD and version-pinned claim capability only. Role/boundary/bucket-policy tests deny sibling prefixes, other environments, and list/write/delete. That Project Management grant is not attached to notifier principals; their separate exact handler/reconciler grants are listed above. The current aws-s3 secret reference is resolved to a non-secret principal ARN and declarative policy owner before activation; an unowned/manual grant is a stop condition.

No notifier or durable command creates, stores, or publishes a presigned download URL. Project Management mints a fresh version-pinned URL only in every accepted or idempotently accepted agent claim response, after the same release and exact-version checks; the agent receives no AWS credentials. Before new-format commands are emitted, a compatibility agent is deployed to the retained Kubernetes consumer and proved first against the legacy ZipDownloadUrl producer and then against the new claim-response producer, preferring the latter, plus queued replay. The old agent/new producer direction is forbidden because a new command deliberately carries no URL. The same immutable image is used on ARRNC. Kubernetes rollback remains valid only at that compatibility floor while new-format commands may be queued; legacy support is removed only after Kubernetes retirement. An audited environment pause stops only held Bulk PDF publication without stopping durable Bulk PDF capture/inventory. The shared dispatcher's existing ReferenceUpload and ReferenceUpdate flows continue unchanged with their current success/retry behavior. Human upload initiation is disabled by an external preview control. PR code cannot select production, clinical, participant, or other sensitive objects.

Controller and link tests require a matching previews/pr-N root, an exact PublicBaseUrl suffix, and exact agreement with the independently rendered TrustedExistingPublicBase, then prove the dedicated harness can complete the whole multipart lifecycle and resolve the final public link. They reject absent or caller-supplied URLs, wrong hosts or roots, sibling/cross-preview prefixes, traversal, and inconsistent root/URL variants before multipart creation or job persistence. The same dedicated policy protects GetStudyPdfPaths, signing, acknowledgement, heartbeat, completion, cancellation, abandonment, status, report, retry, and bounded history for only the matching synthetic Project/search/session from ordinary preview users.

5.3 Sentinels and readiness

The ARRNC operator creates .syrf-storage-ok only after verifying the storage owner's immutable allocation record and the active mounted child. It is the sole readiness-sentinel creator; the storage owner never preseeds it. Its structured, non-secret content includes schema version, canonical root, expected mount source fingerprint, and environment identity. The agent validates content, not mere presence.

The existing Camarades public static root must be proven read-only to preserve the untouched production child and expose the stable staging child and lifecycle-owned preview children while denying directory listing, scratch, cross-environment traversal, partial/quarantined files, sentinels, and unscanned content. This plan grants no authority to add or change IIS bindings, virtual directories, hostnames, certificates, or other IIS state. If the existing mapping cannot serve exact child prefixes by content/mount ownership alone, activation stops for architecture review; an ad-hoc static server is not a fallback.

Readiness fails closed unless all of these agree:

• configured application root and independent runtime/active environment;
• expected direct child mount source;
• sentinel schema and environment;
• effective UID/GID write and rename/fsync capability;
• canonical no-follow resolution beneath the child;
• minimum free bytes and quota headroom.

The mount unit restarts only the matching environment container after recovery. A missing or stale mount never falls through to the local directory underneath it.

5.4 Capacity, backup, and retention

• Production: no Phase 2 staging change may alter its mount, credentials, files, slot, queue, image, or flag. The ARRNC host backup excludes this mount; Gate C therefore requires the remote share owner to attest its actual backup/retention and a restore proof before any promotion.
• Staging: non-production and excluded from production backup. Delivered bytes remain while any durable job, Study, report, or downstream reference can generate or consume their URL. The 30-day policy applies only to proven unreferenced/orphan test data after coordinated application-state retirement; there is no blanket age-delete.
• Preview: no backup. Closure/disablement has a 24-hour convergence SLO for active runtime/session/job/hold/outbox/release-record reclamation, runtime/public-child removal, or quarantine handoff. The intentional non-authorizing P7D receipt plus isolated-database cleanup debit is outside that clock and must purge/release within 24 hours after the latest snapshotted receipt expiry.
• Quotas: server-enforced per child. The implementation PR records proposed byte/inode limits from measured workload evidence and ARRNC free capacity; activation fails if limits and alerts are absent.
• Preview concurrency: a configurable value computed from measured CPU, memory, PID, block-I/O, runtime-disk, scratch, storage-quota, port/route, network, RabbitMQ, and scanner budgets. The registrar may admit multiple previews only when its atomic fleet ledger can reserve every required resource; otherwise it queues the request without partial state.

No cleanup process may receive production-root delete authority. Preview cleanup first fences and unmounts its container, derives one canonical previews/pr-N child, refuses traversal, symlinks/reparse points, siblings, and local mount underlays, and records what it deleted or quarantined.

6. Container and runtime contract

6.1 Images

Staging and production select the same release image by GHCR digest. Preview N selects a PR-specific digest built from the same Dockerfile. A tag can be input to resolution but never the reconciled identity. Provenance binds repository camaradesuk/syrf, commit SHA, workflow, and PR.

Protected activation reads the registry manifest's compressed total and verifies a signed build/verifier attestation of unpacked size before host pull. It enforces reviewed ceilings plus Docker data-root headroom for pull, unpack, and rollback margin. Teardown removes only preview image references/layers proven unused by every live and persistent slot. Reconciliation finds leaked preview layers and headroom drift; ambiguous ownership quarantines and alerts instead of invoking a broad prune.

6.2 ClamAV

Every slot uses the same ClamAV 1.4 image/configuration and limits required for the agent's maximum entry size. Staging and production use separate networks and scanner containers. A preview uses a per-preview scanner and network so one approved workload cannot reach a persistent scanner or consume capacity reserved for another slot. clamd has no published host port.

A trusted updater with a separate network identity may reach only a pinned internal signature mirror. It verifies and publishes a read-only signature set to the scanner. The PR-controlled agent receives no mirror route; refresh, stale-signature fail-closed, and mirror-failure tests prove that updates do not widen preview egress.

Readiness proves the socket, engine version, current signature database age, and EICAR detection. Signature freshness has an alert threshold and fails new work closed once the approved maximum age is exceeded.

6.3 RabbitMQ

Each environment has a distinct virtual host, queue, user, and least-privilege permissions. Preview credentials are minted for one PR and revoked during teardown. Broker credentials are environment-scoped secrets delivered only to the matching slot.

For approved preview clients, broker policy additionally enforces per-preview connection/channel, message-size, publish-rate, queue-length/bytes, and storage limits, with shared-broker free-space alarms. Activation includes simultaneous-client limit tests. If the shared RabbitMQ service cannot isolate aggregate preview limits from staging and production, requests queue or previews use a separately reviewed dedicated broker.

The agent uses its existing durable retries, journal, MassTransit error transport, and environment-root messages. Queue health records ready, unacknowledged, consumer count, oldest age, and dead/error transport depth. Readiness requires exactly one intended consumer when enabled and zero while paused.

6.4 Resource and lifecycle controls

server-config declares CPU, memory, PID, scratch/quota, restart, and log-rotation limits. Preview containers additionally receive host-enforced block-device bandwidth/IOPS ceilings covering their writable runtime data and scratch, with persistent-slot health included in simultaneous bounded-I/O proof. All environments use the same health vocabulary:

• storage-authority and sentinel valid;
• clamd ready and signatures fresh;
• broker connected and exact queue identity;
• consumer state matches desired paused/enabled state;
• scratch journal recoverable and free space above threshold.

Stop is ordered: close new-session admission while existing multipart sessions retain their sign/acknowledge/heartbeat/complete/cancel/abandon paths; complete, cancel, abandon, or policy-expire them while notifier capture, publication, and scheduled reconciliation continue; prove every accepted job terminal; pause publication while capture continues and inventory every late notification and hold, with the independent scheduled reclaimer remaining active; then pause/fence the consumer, wait for or safely abandon the active claim, flush the journal, and stop. Restart replays idempotently from durable broker and journal state. Rollback pins the previous verified digest; it never starts a second consumer.

7. Declarative ARRNC control plane

7.1 Persistent records

server-config adds a static staging project, mount, env scope, network/scanner, verification, and deployment-manifest entry beside the existing production foundation. Both remain gated while implementation and dry-run proof are assembled. Production remains managed/paused exactly as its pre-migration state requires; it is not adopted incidentally.

7.2 Preview family registrar

The config service gains a typed preview-family operation rather than accepting free-form DeploymentEntry fields. Input is limited to:

• canonical positive PR number;
• verified SyRF PDF-agent image digest and source commit;
• requested desired state create, pause, resume, or retire;
• authenticated repository/workflow identity and idempotency key.

It derives container name, app/environment, storage root, mount, network, scanner, broker scope, secret scope, resource profile, and retention record from root-controlled templates. Structural validation checks the derived record both before commit and before reconcile.

Create first executes one atomic fleet-admission transaction that reserves CPU, memory, PIDs, block I/O, Docker/runtime disk, scratch, storage quota/allocation, ports/routes, network, broker, and scanner capacity. The configured concurrency ceiling is computed from measured budgets and may permit multiple simultaneous previews; a request queues with no partial resources when the aggregate safe budget is unavailable. Create and retire are auditable state transitions. A per-PR lock prevents duplicate operations while the fleet ledger serializes competing reservations. Reconciliation compares open explicitly enabled trusted PRs with desired records, repairs or quarantines orphaned partial state, and removes neither persistent slot. A failed reconciliation alerts but does not widen a path or secret scope.

7.3 Trust and provenance

Same-repository branch code is eligible only after explicit approval by an authorized maintainer bound to its exact head and digest. The branch build job may publish only the image digest and attestation and receives no ARRNC, storage, broker, registrar, or protected-environment credential. A protected workflow sourced only from the base branch performs activation with a short-lived preview-family-only identity. It does not check out or execute PR code and pins every privileged action/reusable workflow dependency. A fork cannot receive secrets or ARRNC execution. Manual promotion of a fork result, if ever needed, is a separate admin-reviewed workflow that rebuilds the exact commit in a trusted context.

The registrar verifies the digest's repository/commit attestation, the PR is open, the enabling approval/configuration is recorded against the exact current head SHA and digest, every required fleet resource can be atomically reserved, and no slot for N points at another digest. Every synchronize event invalidates activation approval on a protected non-PR-controlled generation boundary. Before the new head may deploy, that boundary closes new Bulk session admission while the last-approved event handler, reconciler, publication, and client remain active but fenced from replacement until accepted sessions terminalize. It then pauses publication, inventories and reconciles every late hold, and disables S3 invocation only after no accepted session can create another object. It next creates and proves a publication-disabled cleanup-only client generation, injects it only into the retained last-approved reconciler, and switches reconciliation to it. Only then does it deny the old client at API admission, revoke it at the protected issuer, drain calls, and wait maximum token lifetime. The deployment/registrar gate then requires that fence receipt plus a fresh exact-head approval and client generation, so unapproved synchronized code receives no old secret or bearer token. The cleanup-only generation remains through zero holds/outboxes/active records; reapproval replaces it with the fresh exact-head generation. If reapproval does not occur, teardown retains the last-approved runtime and already-issued cleanup generation; it never targets the unapproved head. This control-plane fence adds no protected PDF-serving mode. The SHA/digest binding is rechecked immediately before registration. Queued requests repeat the full repository, open/enabled, head/digest, and approval check immediately before reservation and provisioning. Mutable tag-only requests fail.

7.4 Preview IAM is a registrar-owned exact-PR resource

Wave 1B0 is staging-only and adds no preview-family Bulk PDF policy resource or wildcard preview ARN. It preserves the existing boundary's legacy preview log-group and syrfapp-uploads-* read caps until every admitted preview has an exact-N replacement and zero legacy preview consumers are proved. Before Wave 1C retires those shared caps, Wave 1B0 installs exact staging replacements: logs:CreateLogGroup on only arn:aws:logs:eu-west-1:318789018510:log-group:/aws/lambda/syrfAppUploadS3Notifier-staging, stream/event writes on only that group's :* streams, and the version-specific metadata reads used by existing ReferenceUpload and ReferenceUpdate dispatch on only arn:aws:s3:::syrfapp-uploads-staging/Projects/*. Simulation proves staging still works after the wildcards are removed; only Wave 1C then retires them. Wave 1C owns every preview IAM grant. A protected registrar derives one canonical positive, non-zero-padded PR number from independently fetched provider and base-repository state, the open PR, exact head SHA, verified digest, and maintainer approval. It does not accept caller-supplied PR fields or session tags as provenance. The independently established N is the single input to every derived preview resource.

One effective preview policy must bind that same N to all of these resources: the exact syrf-bulk-pdf-holds-pr-N table and /index/*, exact failure queue, exact syrfapp-uploads-pr-N bucket, singular preview/pr-N/Projects/_bulk-staging/* object namespace, and the identical prefix condition for ListBucketVersions; the exact package object arn:aws:s3:::camarades-terraform-state-aws/lambda-packages/pr-N.zip; distinct handler and reconciler roles/functions; an exact-N Project Management receipt-reconciler grant; and an exact-N teardown cleanup role. The handler and reconciler effective policies include the three required CloudWatch actions (logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents) on only their corresponding exact-N log groups. Policy simulation and negative tests must prove that changing N in any one table, queue, bucket, key prefix, principal, role, function, log group, or session cannot authorize a sibling, including substituting a different N in the package key. If IAM policy variables cannot prove the cross-resource equality, the registrar renders an exact per-preview permissions boundary and inline role policy from its independently verified N, validates the effective policy, and only then activates the preview. A family wildcard plus caller-selected tag is not equivalent evidence.

The current Terraform preview_prs set, commit/version maps, shared preview Lambda role, and wildcard per-PR role-management paths are migration inputs only. Consistent with ADR-010, the service chart's ACK Role/Policy and Function resources remain the lifecycle owner. Wave 1C retires or constrains the Terraform paths so the independently verified registrar record is the sole positive authority and Terraform is limited to ACK/registrar bootstrap and guardrails; it cannot create the replacement per-PR roles or turn a caller-selected set member or tag into N.

The older ACK plans are correspondingly superseded for previews: an exact-N preview role uses the registrar-rendered exact per-preview boundary, never the shared SyRFS3NotifierLambdaBoundary. Preview values set setupJob.enabled: false, disabling the chart's permission-job, env-vars-job, and ack-setup-job service account. The shared syrf-ack-setup-job role loses its system:serviceaccount:pr-*:ack-setup-job trust and preview-family Lambda resources. Only the protected registrar may configure exact-N preview function environment and S3 invoke permission; it re-verifies provenance and same-N function/bucket binding immediately before each operation. No PR-head workload or setup job receives lambda:UpdateFunctionConfiguration, lambda:AddPermission, or lambda:RemovePermission authority.

Only the protected registrar or its protected identity broker may mint the canonical record and stamp a PR tag. Protected chart/ACK reconciliation consumes that record. The ACK Lambda controller's bootstrap principal may narrowly iam:PassRole only the exact-N handler execution role to the exact-N handler function and the distinct exact-N reconciler execution role to the exact-N reconciler function. It cannot cross-pass or assume either execution role. Each per-PR execution role separately trusts only the AWS Lambda service principal (lambda.amazonaws.com) so its function can obtain runtime credentials. The exact-N receipt-reconciler role trusts only the independently identified Project Management runtime principal; the exact-N teardown role trusts only the protected registrar/teardown principal. PR jobs and preview code can assume none of these roles. The trust path rejects missing, duplicate, malformed, non-positive, and zero-padded values and denies caller sts:TagSession, role iam:TagRole/iam:UntagRole, arbitrary sts:AssumeRole, and iam:PassRole. Preview builders and containers cannot edit trust or role policies, attach/detach policies, change a permissions boundary, mutate tags, extend session duration, or nominate transitive tags. Protected create, retry, reconcile, and retire operations re-fetch or independently verify provenance; retirement fences the runtime, retains the receipt role through drain proof and the teardown role through exact-prefix absence proof, then revokes and deletes the complete exact-N IAM set. Preview Role/Policy CRs use iam.deletionPolicy: retain through activation, migration, and those proofs. The registrar then records a one-way exact-N deletion authorization, changes only that preview set to iam.deletionPolicy: delete, lets ACK delete it, and proves AWS absence before removing the dynamic record. Staging and production remain retain. This control does not add IIS authority, a protected publication tree, VM isolation, or a different public-link model: approved same-repository previews remain trusted workloads served beneath the existing public root.

8. Staging coexistence and cutover

8.1 Build without consumption

ARRNC staging is first applied with the Bulk feature disabled and agent consumption paused. Host proof covers mount/ACL/sentinel, container limits, ClamAV/EICAR, broker authentication without a consumer, public URL isolation, logs/metrics, and stopped/absent rollback. Previous-digest rollback is additionally required only when a prior ARRNC digest has independently proven paused startup with zero queue subscriptions.

The Kubernetes staging workload remains healthy and authoritative during this work. Before Gate B, the current Kubernetes/PVC/static-server root and URL are validated as one internally consistent old mapping, while candidate SyRF/server-config values validate the ARRNC child beneath the existing public root as a separate internally consistent target mapping. They are not required to match until after the initial copy; activating the target URL before content exists is forbidden.

8.2 Single-owner handover

At Gate B:

1. close only new-session admission in staging while accepted multipart sessions retain lifecycle endpoints, then prove those sessions quiescent;
2. while notifier publication and scheduled reconciliation continue, release and drain eligible held events until every accepted job is terminal; Uploaded or another known nonterminal state is not sufficient;
3. only after that terminal proof, set the reviewed Bulk PDF publication pause while durable capture continues and capture the hold-table S3-object/job/publication inventory, preserving any late capture;
4. drain the Kubernetes queue/claims/journal and create an initial manifest of PVC output, transferring it through a reviewed controlled path into the ARRNC staging child, and prove checksums, canonical paths, stored-link compatibility, and candidate public-static-root reads; no overwrite or unexplained difference is allowed;
5. switch the reviewed staging public-base/routing configuration to the ARRNC child while the old Kubernetes static server still runs; only now require exact target equality across API, Project Management, registrar, and agent, then prove old and new references resolve from ARRNC. A mixed mapping rolls back before consumer activation;
6. merge/sync a cluster-gitops drain change that scales only the staging PDF-agent Deployment to zero while retaining Application, ingress, PVC, and configuration;
7. with the old consumer fenced, perform a final delta copy, prove no post-fence PVC writes, and reconcile every S3 object/hold-table entry with terminal state or a preserved durable queue event;
8. unpause ARRNC staging, prove consumer count exactly one, clear the Bulk PDF publication pause, and prove the scheduled reconciler drains eligible held rows with stable-identity redelivery safety;
9. reopen staging new-session admission and run authenticated end-to-end tests.

No dual-consumer soak is permitted. If any proof is ambiguous, intake stays off and the sequence stops.

8.3 Authenticated proof

The proof bundle is redacted and contains:

• ARRNC desired record, reconciled digest, container limits, and health;
• exact child mount/source, sentinel, ACL denial to siblings, quota, and public-read mapping;
• normal authenticated two-file upload through UI/API, S3, notifier, RabbitMQ, agent, PM, SignalR, CSV, Study link, and existing-public-root read;
• EICAR terminal Infected result with no delivered bytes;
• duplicate/redelivery and restart replay with one final outcome and no duplicate writes;
• clamd loss, RabbitMQ loss, mount loss, process kill, full-disk threshold, and recovery;
• queue drain, consumer ownership, error transport, logs/metrics/alerts, and rollback rehearsal.

9. Kubernetes retirement

ARRNC acceptance begins a soak while the Kubernetes Deployment remains at zero and the PVC stays bound. The first retirement PR disables/removes ingress and runtime ownership but retains a restorable storage artifact. Before PVC removal:

• enumerate and checksum the PVC content;
• explain every difference from ARRNC staging output;
• create and restore-test a VolumeSnapshot or equivalent controlled archive;
• record retention owner, expiry, and restore command;
• prove queues, claims, and journals remain owned by ARRNC.

The final cluster-gitops cleanup removes the Application and obsolete PDF-agent service values, then the PVC only after the retention gate. Reverting Git plus restoring the tested snapshot must recreate the old shape without activating it. While ARRNC is fenced, all output written since snapshot capture is inventory-checked and copied back with checksum/collision proof before the old public route or consumer is enabled. Imperative kubectl delete and orphan-prune shortcuts are forbidden.

10. Preview lifecycle

Activation sequence:

1. approved same-repository PR opt-in and authorized-maintainer approval are bound to the exact current head SHA and verified digest;
2. atomically reserve CPU, memory, PIDs, I/O, runtime disk, scratch, storage quota/allocation, ports/routes, network, RabbitMQ, and scanner capacity from the fleet ledger, or queue without partial state;
3. provision the exact rootless container, broker scope, network/scanner, child mount, quota, sentinel, and derived record for that PR;
4. reconcile container paused;
5. prove isolation/readiness;
6. expose preview Bulk configuration with the derived existing-base-plus-child public URL and enable its consumer;
7. run a small authenticated clean/EICAR smoke and verify the clean public link stays beneath the exact child while EICAR produces no link.

Teardown sequence:

1. remove only new preview intake while accepted multipart sessions retain signing, acknowledgement, heartbeat, completion, cancellation, and abandonment paths;
2. quiesce every accepted session while notifier publication and scheduled reconciliation continue, and prove every accepted job terminal;
3. only after terminal proof, pause publication while durable capture continues, inventory every object/hold/job/publication state, then disable new S3 event invocation only after no accepted session can create another object;
4. fence the consumer and drain/quarantine its queue, claims, and journal;
5. stop/remove the exact agent and scanner, fencing all processes from the child;
6. unmount the ordinary agent's exact child;
7. give the root-controlled cleanup worker a one-use exact-child capability, prove its view is the remote previews/pr-N child rather than the local mount underlay, reject traversal, symlink/reparse-point, and sibling targets, then no-follow delete only that child or quarantine it, and remove the cleanup view;
8. retain the scheduled reclaimer and preview OAuth client until every active/retention-waiting hold reaches conditional exact-version deletion and authenticated cleanup acknowledgement. Atomically transition poison to QuarantinedHold with its full authority/debit; never evidence-copy or force- discard it. Prove zero active holds, zero quarantine-ledger rows, zero retirement outboxes, and zero active Project Management release records, unless matching PM-link, source-tombstone, active-destination, and PM-finalization receipts prove the durable handshake completed for every stable key/source event/duplicate fence, obligation, and debit;
9. only then disable normal EventBridge reclamation, revoke the preview OAuth client, remove capture, and switch the reconciler to credential-free tombstone-only purge. Retain the exact table, purge owner/schedule, and tombstone-only dynamic record until zero replay-fenced capture/transfer tombstones, then remove them and revoke exact broker and secret scopes;
10. only after those routes/functions can no longer need Project Management state, apply the snapshotted close-time database decision: without lock-db, remove every PR-head, agent, notifier, and user-facing access path, but retain the exact database, last-approved cleanup-only API/PM path, and protected synthetic-harness identity through all snapshotted P7D receipt windows. The server derives the exact teardown-owned synthetic Project set from the protected manifest and rejects caller-selected/user/persistent Projects. Invoke ordinary Project deletion after each receipt/hold/outbox fence passes, purge to zero receipts, revoke the last identity/path and wait token expiry, then delete/prove database absence. Failure retains the cleanup debit without an agent slot; with lock-db, preserve it unchanged as promised, remove every preview runtime/client access path, and record the existing preservation handoff. Never run the current early database decision while a hold or active release record remains;
11. remove the desired record, disposable rootless runtime state, per-preview network/routes/ports, and only image layers proven unused by another slot;
12. after the hold-row and retirement-outbox sets plus active release-record set are empty, notifier invocations are disabled, the client revoked, and both functions are removed, abort every incomplete multipart upload and enumerate/delete every current/noncurrent object version and delete marker beneath only the exact preview data prefix and notifier-package key;
13. repeat version/marker/MPU listings to zero, then prove process/runtime/network/storage/ credential absence, atomically release only the proven-absent parts of the fleet reservation, and publish durable absence proof. Current-key recursive deletion alone is not accepted. Any quarantined resource remains charged to the owning reservation, or an atomic quarantine debit against the same aggregate budget, until reclamation is proved.

Close, label/config removal, and reconciliation all call the same idempotent state machine. Failed cleanup is visible and retried; it never deletes another PR or a persistent environment.

11. Observability and rollback

Dashboards/alerts distinguish environment and PR without logging secrets or filenames. Minimum signals are desired/reconciled digest, container state/restarts, CPU/RAM/PIDs, scratch/share free space and quota, mount/sentinel, clamd readiness/signature age, RabbitMQ consumer/ready/unacked/ error depth, current claim age, processing duration/outcome, and preview teardown age.

Rollback gates:

• before Gate B: stop ARRNC staging; Kubernetes remains unchanged;
• after Kubernetes scale-to-zero but before retirement: close new admission while accepted session lifecycle paths remain available; quiesce them while notifier publication/reconciliation continue; drain held events until every accepted job is terminal; pause publication with durable capture and inventory late events; only then fence ARRNC and drain, reconcile ARRNC-created output back to the retained PVC with checksum and collision proof, scale Kubernetes back to one and prove its static server ready, restore the prior public-base/routing configuration, prove sole consumer and old/new link continuity, then resume publication and intake;
• after Application removal but before PVC expiry: repeat the same admission/session/notifier quiescence and final pause/inventory gate, then fence ARRNC, revert Git, restore/rebind the tested storage artifact, checksummed-copy all post-artifact ARRNC output into it, and prove current-link continuity before enabling the old route or consumer;
• image regression: repeat the same quiescence and final pause gate, drain/fence, pin the prior digest, prove sole consumer, resume publication, then restore intake;
• storage isolation/provenance failure: stop immediately; there is no permissive fallback.

12. Approval model

Gate A approves the architecture and numbered plan only. Gate B separately approves the live staging drain, ARRNC consumer activation, and later Kubernetes retirement after dry-run proof is reviewed. Production has an additional Gate C with a separate PR set and explicit approval.

Merging implementation code/config does not imply an environment apply. Applying staging does not imply Kubernetes deletion. Staging acceptance does not imply production promotion.