Skip to content

ADR-015: Host every Bulk PDF agent on ARRNC API

Status

In-Review. This record requires explicit architecture approval before it may become Approved or merge. It does not authorize implementation, an ARRNC apply, a Kubernetes change, or production promotion.

This record supersedes the hosting/final-storage deployment parts of ADR-012, especially Decision 2 and every statement that places staging or preview PDF agents in Kubernetes on PVCs. It also supersedes ADR-012's completed-object age expiry and direct notifier-to-agent presigned-URL pipeline in Decisions 1, 3, and 6: the notifier durably gates an immutable version through Project Management, and Project Management mints a fresh version-pinned URL only in an accepted agent-claim response. ADR-012 remains authoritative for multipart semantics, scanning/processing, normalization, and application-level storage authority where this record does not explicitly replace it.

ADR-010 remains authoritative for declarative chart/ACK ownership of preview notifier IAM roles and functions. This record narrows ADR-010's preview row: Wave 1C supplies the registrar-owned canonical exact-PR record to the chart, while Terraform owns only ACK/registrar bootstrap and guardrails and must not create the replacement per-PR roles.

Context

The merged 2026-08-11 design split one image across two hosting models: production as a Docker application on the ARRNC API host, but staging and PR previews as Kubernetes workloads using PVC-backed storage. A later explicit owner decision corrected that split: production, staging, and dynamic PR preview agents are separate Docker containers on ARRNC API, with preview creation and teardown tied to the PR lifecycle. That correction was acknowledged but not merged into the authoritative documents.

The stale documents were subsequently followed to recover staging. As of 2026-08-30, read-only inspection showed a healthy Argo CD-managed staging Deployment with PDF-agent, ClamAV, and static server containers, a bound 20 Gi PVC, Service, and ingress. Operational approval of that recovery did not explicitly reverse the later ARRNC decision.

Current repository evidence also establishes:

  • server-config already declares an exact, gated syrf-pdf-agent-production slot, a dedicated syrf-pdf-net, ClamAV, and a production-child CIFS mount;
  • the ARRNC config service reconciles an exact container-name manifest and has no bounded dynamic preview-family contract yet;
  • cluster-gitops enables the staging PDF agent and leaves production disabled;
  • SyRF already validates canonical application roots production, staging, and previews/pr-N;
  • the current ACK Lambda Function has no VPC configuration, so a security group or prose allowlist alone cannot constrain its AWS-managed internet egress;
  • the current Identity seeder has no notifier-only OAuth client or release-state scope, while the current preview workflow publishes a mutable PR-head Lambda package as soon as the preview label is present;
  • the preserved, uncommitted Juniper hosting rewrite contains the corrected environment partition and lifecycle intent, but is mixed with unrelated work and is evidence only.

The mismatch is therefore documentary and architectural, not permission to change a live environment during this planning phase.

Decision

1. One hosting model

No Bulk PDF agent runs in GKE or another Kubernetes cluster. Production, staging, and each enabled PR preview run as separately named Docker containers on ARRNC API. Ordinary application services may remain in Kubernetes; only the PDF processing agent and its environment-owned final-storage path move.

The persistent slots are:

Environment Container Storage child
Production syrf-pdf-agent-production production
Staging syrf-pdf-agent-staging staging
Preview PR N syrf-pdf-agent-pr-N previews/pr-N

Names and roots are derived from a canonical positive, non-zero-padded PR number. Callers cannot supply arbitrary container names, host paths, networks, commands, or repositories.

2. Exact child-scoped storage

The physical Camarades share is partitioned into production, staging, and previews/pr-N children. Each ordinary agent sees only its exact child at /app/data/output and a private scratch directory at /app/data/scratch. It never sees the share root or a sibling environment.

Each child has:

  • a distinct least-privilege SMB identity and server-side ACL;
  • a direct child mount on ARRNC API, owned by the agent's pinned non-root UID/GID;
  • an immutable storage-owner allocation record binding environment, canonical child, principal, ACL, quota, and remote share identity;
  • one ARRNC-operator-created readiness sentinel, written only after the mounted source and allocation record have been verified, whose content binds environment, canonical child, and mount identity;
  • fail-closed checks for mount source, sentinel, write capability, free space, and path identity;
  • no CIFS credential in container environment, image, GitHub Actions, Kubernetes, or PR code.

Direct child mounting and child ACLs are activation prerequisites, not assumptions. If the SMB export cannot mount the exact child or enforce distinct identities, work stops for a separately reviewed storage decision. Mounting the common root and relying only on container bind paths is not an approved fallback.

Production backup/retention is unchanged by the staging migration, but Gate C must obtain an external owner attestation for the remote share: ARRNC's local host backup explicitly excludes the mounted PDF tree and is not evidence that the share is backed up. Staging is non-production and not promoted, but a delivered file remains while any job, Study, report, or other durable application reference can generate or consume its URL. The 30-day policy applies only to proven unreferenced/orphan test data after coordinated application-state retirement; a blanket age-delete is forbidden. Preview data is not backed up; teardown removes only the exact preview child after fencing. An orphan reconciler must converge active runtime/session/job/hold/outbox/ release-record cleanup, runtime/public-child removal, or quarantine handoff within 24 hours. The intentional non-authorizing P7D receipt plus isolated-database cleanup debit is outside that clock and must purge/release within 24 hours after the latest snapshotted receipt expiry. Server-enforced quotas and ARRNC free-space alarms are required before enabling staging or previews. Preview concurrency is configurable from measured ARRNC CPU, memory, PID, I/O, disk, storage-quota, route/port, RabbitMQ, and scanner capacity; it is never hard-coded to one. Before each activation, one fleet-wide admission transaction reserves every required resource or queues the request without creating partial runtime, storage, broker, network, or route state. Before a queued request can reserve resources, the registrar atomically revalidates repository identity, open/enabled PR state, current head SHA, verified digest, and current authorized-maintainer approval. A stale or ineligible request is discarded without provisioning.

Dynamic preview storage has an implementation dependency on an audited university-storage-owner operation or an equivalently isolated pre-provisioned allocation pool. In one atomic operation it creates or allocates the exact previews/pr-N child, distinct principal, ACL, quota, server-side connection/bandwidth/IOPS limits, and immutable allocation record without exposing that authority to GitHub or the preview container. It does not create the readiness sentinel. It rejects traversal, symlink/reparse-point, sibling, and local mount-underlay targets. The registrar reserves the allocation atomically with the other fleet resources, verifies the allocation record, mounts and proves the exact source/ACL/quota, and only then creates the sole readiness sentinel. Missing automation is work to deliver, not evidence that approved previews require a different hosting model; broad credentials and unbounded shared-SMB I/O remain prohibited fallbacks.

ARRNC root mounts the exact child and bind-mounts only that child into its disposable rootless preview container. SMB credentials, sibling mounts, and the common root never enter the container. Provisioning and teardown test source identity, no-follow behavior, sibling denial, mount loss, ordered container stop/unmount, and exact allocation retirement.

3. Declarative persistent slots and bounded dynamic previews

server-config is authoritative for host users, mounts, directories, networks, ClamAV, secrets scope, persistent slots, verification, and the ARRNC config-service policy.

Production and staging are explicit persistent manifest entries. Preview entries are controlled by a dedicated bounded family registrar built into the declarative config-service model. It accepts a canonical PR number and an immutable SyRF PDF-agent image digest with verified repository, commit, and PR provenance, then derives every other field. It may create, pause, reconcile, and retire only syrf-pdf-agent-pr-N records.

The registrar must reject:

  • arbitrary image repositories, tags without resolved digests, commands, paths, ports, networks, environment names, and secret scopes;
  • any pull request whose head repository is not exactly the SyRF base repository, including every fork; only explicitly approved same-repository SyRF heads may use this path;
  • a request whose declared per-slot profile or requested concurrency exceeds a configured hard limit; transient exhaustion within valid limits queues instead of rejecting the request;
  • a request whose PR is not open and explicitly enabled;
  • any request that could expose production or staging storage or broker credentials.

An approved same-repository SyRF PR is a trusted preview workload, not an arbitrary hostile-code source. Its build job may publish only an image digest and attestation; it receives no ARRNC, storage, broker, or registrar credential. A protected, base-branch-controlled activation workflow performs the privileged request after re-verifying the open PR, fleet capacity, and an explicit authorized-maintainer approval record bound to the exact current head SHA and verified digest. Every synchronize event invalidates the record on a protected non-PR-controlled generation boundary. Before the new head may deploy, that boundary closes new Bulk session admission and fences new-head replacement while the last-approved event handler, reconciler, publication, and client remain active so accepted sessions can complete, cancel, abandon, or policy-expire and every Uploaded job can terminalize. Only then does it pause publication, inventory/reconcile late holds, and disable S3 invocation after no accepted session can create another object. It next provisions and proves a publication-disabled cleanup-only client generation, injects it only into the retained last-approved reconciler, and switches reconciliation to it. Only then does it deny the old client at API admission, revoke it at the protected issuer, drain calls, and wait maximum token lifetime. The cleanup-only generation remains through zero holds/outboxes/active records. A fresh exact-head approval then replaces it before calls reopen, so unapproved synchronized code receives no old secret or bearer token. If no fresh approval arrives, teardown retains the last-approved runtime and already-issued cleanup generation; it never targets the unapproved head. The binding is rechecked immediately before initial registration and again immediately before dequeue-time reservation and provisioning. Activation uses a short-lived, preview-family-only identity. No job that checks out or executes branch code may hold that identity, and PR changes cannot replace the activation workflow or its pinned dependencies.

Wave 1B0 is staging-only. Its Terraform change may extend the existing SyRFS3NotifierLambdaBoundary for the exact staging hold table and indexes, staging failure queue, and staging Bulk PDF bucket/key prefix only. It adds no preview-family Bulk PDF ARN, wildcard pr-* table, preview queue, preview bucket/prefix, role, session, or production authority. It must not remove or narrow the boundary's legacy preview log-group and syrfapp-uploads-* read caps while any shared/current preview role remains. Wave 1C retires those legacy caps only after every admitted preview has moved to its exact-N boundary and zero legacy preview consumers are proved. Before that retirement, Wave 1B0 must add permanent exact staging replacements: logs:CreateLogGroup on only arn:aws:logs:eu-west-1:318789018510:log-group:/aws/lambda/syrfAppUploadS3Notifier-staging, logs:CreateLogStream and logs:PutLogEvents on only that log group's :* streams, and the version-specific metadata reads required by the existing ReferenceUpload and ReferenceUpdate dispatch paths on only arn:aws:s3:::syrfapp-uploads-staging/Projects/*. Policy simulation must prove those exact staging grants still work after the shared wildcards are removed.

Wave 1C owns every preview IAM grant as part of the protected registrar lifecycle. The registrar derives one canonical positive, non-zero-padded PR number from independently verified provider, base-repository, PR, head-SHA, digest, and approval state; caller-supplied PR fields or session tags are never provenance. The same canonical number must bind the exact table syrf-bulk-pdf-holds-pr-N and /index/*, exact failure queue, exact syrfapp-uploads-pr-N bucket, and singular object prefix preview/pr-N/Projects/_bulk-staging/*, including the matching ListBucketVersions prefix condition, exact package object arn:aws:s3:::camarades-terraform-state-aws/lambda-packages/pr-N.zip, the distinct handler and reconciler roles/functions, the exact-N Project Management receipt-reconciler grant, and the exact-N teardown cleanup role. The handler and reconciler boundaries and inline policies also name only their corresponding exact-N CloudWatch log groups and allow logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents. If policy variables cannot provably bind that one number across every resource, the registrar renders and validates an exact per-preview permissions boundary and inline role policy instead; a wildcard family escape is not an approved fallback.

The existing Terraform preview_prs set, preview commit/version maps, shared preview Lambda role, and wildcard per-PR role-management patterns are migration inventory, not an identity source or the replacement owner. Wave 1C must retire or constrain them so only the registrar's independently verified immutable record can render exact-N chart values. ACK Role/Policy and Function resources remain the lifecycle owner; Terraform is limited to ACK/registrar bootstrap and guardrails and a Terraform caller cannot make N authoritative.

Only the protected registrar or its protected identity broker may mint the short-lived registrar/deployment control-plane role session and stamp a canonical PR session tag where one is used. The registrar record drives protected chart/ACK reconciliation. The ACK Lambda controller's bootstrap principal may narrowly iam:PassRole only the exact-N handler execution role to the exact-N handler function and the distinct exact-N reconciler execution role to the exact-N reconciler function. It cannot cross-pass or assume either execution role. Each per-PR execution role separately trusts only the AWS Lambda service principal (lambda.amazonaws.com) so its function can obtain runtime credentials. The exact-N receipt-reconciler role trusts only the independently identified Project Management runtime principal and grants strongly consistent exact-key GetItem; the exact-N teardown role trusts only the protected registrar/teardown principal and grants the already-defined exact-prefix cleanup actions. PR jobs and preview code can assume none of these roles. The trust and boundary policy deny missing, duplicate, malformed, non-positive, or zero-padded values; direct caller sts:TagSession, iam:TagRole, iam:UntagRole, arbitrary sts:AssumeRole, and iam:PassRole are denied. PR jobs cannot change the role trust policy, inline/attached policy, permissions boundary, tags, session duration, or transitive-tag controls. Create, retry, reconcile, and retire all re-fetch or independently verify provenance, validate the effective exact-N policy before activation, and revoke the exact preview identities in lifecycle order: retain the receipt role through drain proof and the teardown role through version/marker/MPU absence proof, then retire the complete exact-N role/policy set. Active/cutover preview Role/Policy CRs retain iam.deletionPolicy: retain. After those proofs, the registrar records a one-way exact-N deletion authorization, the preview-only values transition that exact set to iam.deletionPolicy: delete, ACK deletes it, and AWS absence is proved before the dynamic record is removed. Staging and production remain retain. None of this changes the trusted approved-preview workload decision or adds protected PDF publication, IIS mutation, VM isolation, or production authority.

For previews, this decision supersedes the earlier ACK documents' shared-boundary and setup-job instructions. The registrar-rendered exact per-preview boundary is the role's only boundary; SyRFS3NotifierLambdaBoundary is not attached. Preview values set setupJob.enabled: false, so serviceaccount.yaml, permission-job.yaml, and env-vars-job.yaml render no preview setup identity or hook. The shared syrf-ack-setup-job role loses system:serviceaccount:pr-*:ack-setup-job trust and all preview-family Lambda mutation resources. Only the protected registrar may apply exact same-N preview function environment and S3 invoke permission after re-verifying provenance. PR-head jobs and preview code receive no lambda:UpdateFunctionConfiguration, lambda:AddPermission, or lambda:RemovePermission authority; cross-N simulation must deny sibling function/bucket pairs.

Before a preview digest reaches the host, protected activation must verify the registry manifest's compressed total and a signed build/verifier attestation of unpacked size against reviewed ceilings, then reserve Docker data-root headroom for pull, unpack, and rollback margin. Teardown removes preview-only image references and layers once no live or persistent slot uses them; reconciliation detects leaked layers and low headroom. If ownership cannot be proven, cleanup quarantines and alerts rather than pruning shared images.

The imperative arrnc-api-deploy gatekeeper remains a transport/execution boundary where needed, but it cannot become a second source of desired state. Any changes there must preserve the config-service's authorization, provenance, audit, and exact-record invariants.

4. Runtime parity

Release staging and production use the same SyRF PDF-agent image by immutable digest. A preview uses the same Dockerfile and runtime contract at the PR's own verified immutable digest. Environment differences are declarative values only.

Every slot has equivalent:

  • ClamAV 1.4 contract, scan limits, readiness, signature freshness, and no externally published clamd port;
  • environment-specific RabbitMQ virtual host, queue, user, and least-privilege permissions;
  • CPU, memory, PID, restart, scratch, and disk limits;
  • non-root execution, dropped capabilities, no privileged mode, no Docker socket, no host network, and a read-only root filesystem where the image permits it;
  • mount/sentinel, clamd, broker, queue-owner, and free-space health checks;
  • stop, drain, replay/idempotency, journal recovery, and compatible previous-digest rollback procedures. The first staging slot rolls back to stopped/absent until a prior digest is independently proven to honor paused startup without subscribing.

Each preview receives its own RabbitMQ identity and broker-enforced connection/channel, message-size, publish-rate, queue-length/bytes, and storage limits, plus shared-broker free-space alarms. Admission reserves broker and scanner capacity across all active previews before creating a slot. If aggregate limits cannot preserve persistent environments, requests queue until capacity is available or use a separately reviewed dedicated broker.

Each approved preview gets a disposable rootless Docker container, exact storage allocation, network namespace/policy, scanner boundary, resource reservation, and audit record. It never joins production or staging networks and receives no production or staging mount, secret, broker account, static-serving authority, Docker socket, privileged mode, host namespace, device, or host-administration capability. Production and staging remain separate persistent host containers.

ARRNC root enforces per-preview and aggregate CPU, memory, PID, block-I/O, disk, storage-quota, port/route, network, broker, and scanner ceilings. Container code cannot raise them. Bounded write/discard/fsync and simultaneous-preview load tests prove every reservation and acceptable production/staging latency and health. Failure to establish an exact slot leaves that request queued or failed closed without affecting other previews.

A separate Docker bridge is necessary but not sufficient. Root-controlled host egress policy denies preview traffic by default, blocks the host gateway, metadata endpoints, ARRNC/UoE internal CIDRs and persistent-environment endpoints, and permits only the exact preview scanner plus the minimum controlled DNS, presigned-S3 HTTPS, and preview RabbitMQ destinations. PR code cannot alter this policy. Activation tests both the allowlist and explicit denials. The root-controlled per-preview policy enforces ingress and egress rate and burst ceilings, including allowed presigned-S3 traffic. Simultaneous bounded-load tests prove aggregate reservations protect the ARRNC uplink and production/staging health.

The trusted ClamAV updater is outside the PR agent's network identity. It alone receives a separately constrained route to a pinned internal signature mirror, publishes a verified read-only signature set to the preview scanner, and proves refresh/failure behavior without opening agent egress.

The existing public static root already serves child directories and is the only approved serving boundary. Production uses its production child, staging uses the stable staging child, and an approved trusted preview uses its lifecycle-owned previews/pr-N child. Every runtime receives an exact canonical HTTP(S) PublicBaseUrl ending in its application root, so links are the existing public base URL plus production/, staging/, or previews/pr-N/. No per-preview IIS binding, virtual directory, hostname, certificate, or other IIS mutation is required or authorized.

A deployment-owned TrustedExistingPublicBase authority is independent of PublicBaseUrl and caller input. It fixes the exact scheme, host, effective port, and already-served root path. The API, Project Management, and preview registrar receive separately rendered copies from protected configuration, derive only the canonical child suffix, and require exact agreement before initiation, persistence, registration, or link generation. A wrong or missing authority fails closed; PR code cannot supply or override it.

Public preview PDF links are intentional for explicitly approved trusted same-repository previews. That explicit approval is the code-trust decision: the preview image is treated as trusted application code, while exact-child isolation and the reviewed scan/type/atomic-copy path contain accidental defects rather than claiming a hostile-code publication boundary. The normal agent scan, type validation, and atomic copy contracts apply before a PDF reaches the served child. The protected-preview mode and host-side output-inspection capability are not part of this architecture: they duplicated an already-served storage boundary and contradicted the required public-link outcome. Storage principals, mounts, and application link builders cannot list, traverse, or derive another preview's child, but a client that already knows a valid delivered public URL may fetch it; this design claims no caller-environment authentication boundary. The protected base-branch harness still restricts preview input to synthetic, non-sensitive fixtures and keeps ARRNC/storage credentials outside PR code.

Preview Bulk processing accepts only synthetic, non-sensitive fixtures supplied by the protected base-branch harness through a dedicated preview object prefix and broker path. That harness uses a protected synthetic identity/API path to create the matching Project, create and attach its systematic search, seed bounded synthetic Study/PDF-path rows, verify their authorization and cleanup ownership, obtain the resulting GetStudyPdfPaths fingerprint, initiate the upload session, upload and acknowledge every multipart fixture part, complete the session, and read only that synthetic scope's bounded history. Only then may notifier publication proceed. Because object completion can race application persistence, the notifier conditionally persists the validated immutable release envelope, including bucket, key, and S3 version ID but no presigned URL, in an environment-scoped DynamoDB hold table before handler success. A bounded scheduled reconciler leases due rows and publishes the stable-identity object-ready command to Project Management only after a minimal authenticated read proves the matching environment/job/object key is committed as Uploaded; lease recovery and downstream idempotency tolerate crashes/redelivery without assuming another S3 event or transport exactly-once. That read uses a dedicated endpoint on the existing authenticated SyRF API boundary, backed by the Project Management job store. It does not add or require an object-version field in Project BSON during the split API/PM rollout. Project Management instead validates the stable job binding and performs an exact version-specific S3 HEAD for the command's bucket, key, and immutable version before accepting release; absence, replacement, or metadata/size/hash mismatch fails closed. Before acknowledging the object-ready command, Project Management idempotently persists a separate BulkPdfUploadReleaseRecord keyed by environment/job/stable-message identity with the immutable bucket/key/version/size/hash/metadata binding and release state. It is not embedded in Project BSON and contains no URL. Object-ready registration is a reversible pre-claim handoff: the matching Project job remains cancellable until Project Management atomically accepts and persists the agent's processing claim token. If cancellation wins, a later claim emits no capability and idempotently converts the release to CleanupOnly while terminalizing the job; if the claim wins, later cancellation reports processing_already_started. Only after a durable accepted claim does Project Management repeat the exact version-specific HEAD and mint a fresh version-pinned URL. Redelivery after claim commit reuses the stable claim and remints a fresh URL without double-processing; no command, process memory, or expiring capability is the durable handoff. Cooperative cancellation after an accepted claim is outside this rollout. For a hold that is terminal before publication, the reconciler emits no agent work and instead sends a stable IBulkPdfUploadCleanupRegistrationCommand with the same immutable binding. Project Management validates the live terminal job and exact version-specific HEAD, then idempotently persists the record as non-claimable CleanupOnly before acknowledging registration. Reclamation cannot begin without a durable processing-capable or cleanup-only record.

The target Project Management execution principal therefore receives exact-environment/prefix s3:GetObjectVersion for that HEAD and the version-pinned claim capability. Its role, permissions boundary, and bucket policy deny list/write/delete, sibling prefixes, and other environments. The same traced principal receives strongly consistent dynamodb:GetItem only on the exact environment hold/outbox table for receipt-drain proof, with no Query, Scan, transaction, write, sibling/cross-environment, or production-table access. These Project Management grants are never attached to notifier principals; their separate exact handler/reconciler grants are enumerated below. The implementation traces the current aws-s3 secret reference to its non-secret principal ARN and declarative policy owner; if that owner cannot be established, activation stops rather than adding an ad-hoc console grant.

A notifier-only machine identity may read only the minimum bound job state: committed Uploaded for release, or terminal/nonterminal disposition plus active/retriable-claim presence for any held cleanup candidate, including one that becomes terminal before publication. The route returns no S3 version or URL. The selected issuer for each environment must own a dedicated confidential client with client-credentials grant and exactly two notifier scopes: release-state read and cleanup-acknowledgement write. The write scope authorizes only idempotent quarantine-fence registration/transfer, cleanup acknowledgement, and the subsequent hold-retired confirmation for the same environment/object binding and immutable digest. The read scope admits only the job-bound state route and a no-job-identity storage-binding resolver limited to the mapped environment plus bucket/key/version and a bounded zero/one/multiple result. Neither grants enumeration or general Project/job mutation. The API owns an explicit fail-closed mapping from the authenticated client identity (client_id/azp, and subject where the issuer supplies one) to exactly one environment, and rejects a valid notifier token on every other environment's route. For OpenIddict, SyRF Identity registers and tests that client and scope; for an environment still using Auth0, the named Auth0 tenant administrator creates the equivalent machine-to-machine client and API permission. The environment secret owner creates, stores, rotates, and revokes the credential. Rotation creates a next-generation client with the same two scopes, installs its ID/secret into the reconciler only, proves both read routes and all three cleanup-write operations, and only then revokes and removes the old client. Preview teardown keeps its client admitted and usable by the reconciler until zero active holds, zero quarantine-ledger rows, zero retirement outboxes, and zero active release records are proved. Only then does it deny the client ID at the API, revoke token entries where supported, disable/delete the client, and retain that deny fence until the maximum already-issued token lifetime elapses before claiming credential absence. No staging or production client is reused by a preview, and production identity remains unchanged before Gate C.

For OpenIddict, issuing the scopes is insufficient by itself: Identity maps both notifier scopes to the exact API/introspection audience. The API bearer selector admits the read scope only to the dedicated release-state and storage-binding-resolver routes and the write scope only to the quarantine-fence, cleanup-acknowledgement, and hold-retired-confirmation routes. The resolver rejects a job ID or caller-selected environment. All other API routes retain the existing syrf_api requirement. Tests prove each route rejects the other scope, the notifier token cannot authenticate broader controllers, and neither narrow scope can be exchanged for syrf_api. The API overwrites environment from the authenticated client mapping and sends stable messages to named Project Management storage-binding, quarantine-fence, and hold-retired consumers; those consumers alone own server-side lookup, durable deletion fencing, or conditional receipt update. Neither API nor notifier gains direct Mongo access.

Both the S3 event handler and separately named scheduled reconciler receive their own complete, different environment configuration from protected setup jobs. The handler receives only event, hold-table/object-prefix, publication-pause, and existing broker settings; it receives no API/auth endpoint, audience, client ID, or secret. Only the reconciler receives the release-state, storage-binding-resolver, quarantine-fence, cleanup-acknowledgement, and hold-retired API endpoints, issuer, token endpoint, audience, client ID, and secret-backed client secret. They retain the repository's existing supported Lambda network shape. The Lambda client accepts only protected absolute token and release-state URLs, compares scheme, host, effective port, and exact path, disables redirects, and fails closed on variants; direct Mongo access and broad AWS credentials are not granted. A dedicated notifier VPC, private subnets, VPC endpoints, DNS/stateful egress firewall, or ENI-policy programme is optional future hardening only and is not an activation gate. The existing SyRFS3NotifierLambdaBoundary remains mandatory for staging, so its Terraform owner at camarades-infrastructure/terraform/lambda/ack-iam.tf must first add only the staging hold-table action sets, exact staging failure-queue actions, and staging Bulk PDF DeleteObjectVersion namespace cap, reconciler exact-table PutItem/UpdateItem constituents for the TransactWriteItems call, plus separate DeleteItem for acknowledged outbox removal, replay-fenced capture/transfer tombstone removal, and full-binding conditional RejectedPreHold retirement, and exact-prefix-conditioned ListBucketVersions. IAM has no dynamodb:TransactWriteItems action; ConditionCheckItem is added only for a separately implemented condition-check item, and the event handler remains denied DeleteItem. Each staging Lambda role's inline policy remains narrower at the one rendered staging table ARN/indexes and exact S3 prefix. This staging-only IAM dependency adds no preview, production, or notifier VPC/network authority. Wave 1C separately provisions each preview's exact-N effective policy under the registrar controls in Decision 3; preview activation cannot consume Wave 1B0 authority. Pending, leased, published, retention-waiting, and receipt-retirement-pending active holds, plus QuarantinedHold items, have no DynamoDB TTL. Successful publication starts terminal-state observation but does not assign TTL. After proved exact-version deletion and a successful cleanup acknowledgement, the reconciler marks the same row ReceiptRetirementPending. One conditional DynamoDB transaction atomically creates an immutable HoldRetirementOutbox item and converts the same stable-event capture item in place to CaptureTombstone, so retirement cannot be reported while an active hold exists. Exact-binding duplicate capture is success against the tombstone; mismatch poisons. The outbox calls the dedicated hold-retired-confirmation route under the same environment-bound cleanup-write scope, persists the returned immutable acknowledgement ID/time, and is deleted only afterward. A crash or uncertain response resumes from the pending row or outbox. The tombstone retains binding/delete digest through its snapshotted replay deadline and is deleted only after that deadline, an empty visible/in-flight failure queue with a newer oldest-message watermark, and authoritative inventory checkpoint; later redrive is forbidden. None has TTL. DynamoDB TTL is never teardown or absence proof. Missing, mismatched, unauthenticated, or unavailable state fails closed and remains held without ageing out.

Terminal payload retention is deployment-owned and mandatory for every active non-production environment: protected cluster-gitops values set reconciler.terminalPayloadRetention to PT24H for staging and PT1H for previews. Schema/render validation rejects missing, zero, negative, malformed, or lower values; production remains unset and inactive until Gate C chooses its policy. At the first authenticated terminal/no-active-or-retriable-claim observation, the reconciler conditionally snapshots the policy value/version, terminal-observed time, and retention-eligible time into the hold row. Later configuration changes apply only to rows first observed afterward; they never recompute or accelerate an existing row.

The exact S3 object version referenced by a nonterminal hold cannot age out either. Staging and preview completed Bulk PDF objects have no age-based lifecycle expiration; only incomplete multipart cleanup remains a lifecycle safety net. Before expiry is removed, successful exact-version metadata reads for reserved Bulk PDF keys with malformed, missing, or unrecognized immutable metadata must create non-publishable RejectedPreHold rows. Those rows contain no caller-derived job authority and snapshot PT24H staging/PT1H preview invalid-object retention. Before deletion, the reconciler uses the dedicated authenticated resolver under the mapped environment's release-state read scope to ask Project Management for only the exact server-owned environment/bucket/key/version storage binding, with no job identity. The deletion-independent BulkPdfUploadStorageBinding is persisted at completed-object transition (without adding a version field to Project BSON) and blocks ordinary pruning/Project deletion before this read returns. It is retired only with objectless absence proof or the cleanup acknowledgement transaction, so the later quarantine-fence write has no deletion race. Zero matches permits unowned exact-version cleanup without fabricated authority: after retention the reconciler conditionally records the resolver-decision digest, delete result, and later authoritative inventory absence checkpoint, then retires the row by full immutable-binding conditional delete. Crashes resume from that row; no acknowledgement or retirement outbox is invented. One matching Uploaded or authoritative terminal completed-object job repeats version-specific HEAD and idempotently persists/reuses CleanupOnly without agent work. Uploaded terminalizes failed-cleanup-pending and releases capacity in the same transaction that persists CleanupOnly; an already-terminal binding preserves its disposition/history. Objectless contradiction, active/retriable claim, or ambiguity first uses the narrow write route: Project Management re-resolves server-owned candidates and atomically persists a no-TTL BulkPdfUploadQuarantineFence for each candidate plus an immutable fence-set receipt without exposing identities. Only after that receipt does the stable capture item become QuarantinedHold; unavailable state remains a retryable active hold. The quarantine retains the full binding/evidence/delete state, retry authority, duplicate suppression, cleanup-client requirement, source event identity, fence-set receipt, and aggregate debit without TTL. It leaves cleanup custody only by normal reclamation or a durable idempotent cross-store handshake; no Mongo/DynamoDB cross-service transaction is assumed. PM first records TransferPending against a stable transfer ID while retaining every candidate fence. The destination then conditionally stores a dormant prepared hold with the same capture key/source event, binding/evidence, retry and duplicate obligations, cleanup-client requirement, and stable debit ID. PM durably links that exact prepared-destination receipt without releasing the original fences. Only then may the source become QuarantineTransferTombstone, the destination activate against that tombstone receipt, and PM finalize after both receipts. Every step is queryable/idempotent; conflicting input fails closed, crashes or lost responses resume, and the shared debit ID is counted once. Only the linked destination may acknowledge exact cleanup, after which PM clears the fences transactionally. The source tombstone suppresses duplicates and retains the debit until that acknowledgement. Purge requires the ordinary CaptureTombstone replay/queue/inventory fence plus matching destination exact-version cleanup acknowledgement, PM's durable fence-set-clearance/final disposition, and authoritative shared-debit release. Pending/failed destination cleanup retains the tombstone, table, purge schedule, and dynamic record; evidence-only copying cannot clear teardown. Read/config failures retry, and an encrypted exact-environment SQS on-failure destination captures exhausted invocations. The scheduled reconciler independently checkpoints exact-prefix ListObjectVersions, reconstructing any missing normal/rejected hold via the trusted storage resolver after DLQ loss/expiry or destination failure. Non-Bulk objects keep existing skip behavior, and expiry remains until destination/redrive/inventory/lag proof is green. The bounded EventBridge-scheduled reconciler is the application-owned terminal reclaimer. Its durable row supplies the immutable environment/job/bucket/key/version and cleanup ledger; it revalidates authenticated terminal job state and active/retriable-claim absence for any held row. A row that terminalizes before publication is never published; after its snapshotted retention, either that explicit disposition or a published terminal job conditionally leases deletion of only that exact version. The separate BulkPdfUploadReleaseRecord is also the deletion-independent cleanup authority. Before bounded Project history may prune a terminal job, a processing-capable or cleanup-only record must already hold the immutable binding; the same Mongo session transaction records terminal disposition and claim state there and only then removes the embedded history. A terminal job with a completed-object cleanup obligation that is awaiting cleanup registration is skipped by pruning and alerted. The obligation exists after multipart completion or whenever a matching notifier hold/release record exists. Objectless cancellation/abandonment instead atomically persists the sweeper's MPU-absent and object-absent proof with terminal state, creates no fictional record, and needs no receipt. Whole-Project deletion fails closed while any nonterminal Bulk PDF job/session, completed-object terminal job without a completed cleanup receipt, objectless terminal job without its persisted absence proof, active processing/cleanup record, unresolved BulkPdfUploadStorageBinding, or BulkPdfUploadQuarantineFence exists. Bounded history pruning applies the same fence. Cleanup acknowledgement conditionally verifies the exact delete/ack digest and current cleanup owner, converts the active record to its receipt, marks the storage binding cleanup-complete, and retires every member of an optional fence set with one immutable final-disposition receipt in the same PM Mongo transaction. Missing/extra/mismatched/partially updated members fail closed and replay returns the same receipt. It does not remove the matching embedded terminal history entry. The job and report remain visible until ordinary bounded-history pruning, while the receipt satisfies the completed-object Project-deletion fence. The authenticated route reads the live job or this record, so neither history pruning nor aggregate deletion can strand reclamation. After exact-version deletion, the acknowledgement route conditionally converts the active record to a non-authorizing, bounded audit receipt and returns idempotently for the same binding. The reconciler then persists ReceiptRetirementPending and its immutable delete/ack result in the hold row, atomically creates the immutable retirement outbox and converts the stable capture key to CaptureTombstone, and only the outbox obtains the idempotent Project Management retirement confirmation through that dedicated authenticated route. The outbox is deleted only after conditionally persisting the acknowledgement. Notifier absence proof requires zero active release records, zero active holds, zero QuarantinedHold items, and zero retirement outboxes. Capture tombstones and audit receipts are not cleanup authority or container capacity; transfer tombstones retain only the shared debit/replay evidence until their stricter purge fence clears. The exact table, credential-free protected tombstone-purge owner, and tombstone-only dynamic record nevertheless remain until every capture/transfer tombstone conditionally clears its replay/queue/inventory fence; a transfer tombstone also needs matching destination cleanup acknowledgement, PM fence-set final clearance, and authoritative shared-debit release; only then is final notifier/table/record absence true. An unlocked retained database carries a bounded cleanup debit. Protected values set receipt retention to P90D for staging and P7D for previews and snapshot ReceiptCreatedAt, policy version, and ReceiptExpiresAt; production remains unset before Gate C and later configuration cannot accelerate an existing receipt. The Project Management purge owner removes a receipt only after expiry, acknowledged retirement, OutboxDrainedAt from a later strongly consistent exact-key DynamoDB absence proof, zero active processing/cleanup record, and ordinary pruning of its embedded job or Project deletion. Duplicate acknowledgement within retention returns the stored result; after safe purge it fails closed with 410 CleanupReceiptRetired, and a missing hold-retired confirmation or drain proof prevents purge and alerts. The Project Management receipt reconciler derives the exact outbox key from the immutable receipt and receives strongly consistent dynamodb:GetItem only on the mapped environment table; absence conditionally records OutboxDrainedAt. Query, Scan, transaction, write, sibling/cross-environment, and production-table access stay denied, so commit-plus-lost-response remains replayable beyond receipt expiry. The environment reconciler role alone receives exact-prefix GetObjectVersion/DeleteObjectVersion plus bucket-level ListBucketVersions constrained by the same exact prefix through its permissions boundary and bucket policy; unversioned delete, unscoped/sibling list, write, sibling/cross-environment, event-handler, and target Project Management delete are denied. To make that target executable, BulkPdfMultipartSweepProcessor and BulkPdfUploadController.CancelUpload replace their completed- object unversioned DeleteObjectAsync paths for cancellation/abandonment races with a cleanup- pending obligation that keeps active capacity charged. Durable notifier capture supplies the exact version; Project Management atomically persists/reuses CleanupOnly, terminalizes the job, and releases capacity. Only that cleanup-authority commit permits release; the exact-version reclaimer later finishes cleanup without a delete marker, false absence proof, or agent release. Deploy and prove that replacement while the legacy narrow PM delete grant remains, then remove the grant declaratively; rollback preserves the behavior/permission pair. Missing proof fails closed. Preview-wide IAM selects the singular S3 upload prefix preview/pr-{n}/, not the plural public child previews/pr-N/; neither namespace is derived from the other. Version/marker/MPU deletion remains part of fenced preview teardown. The shared event-handler role retains GetObjectVersion for each environment bucket/prefix already handled by ReferenceUpload and ReferenceUpdate, adding the Bulk PDF prefix where required, because metadata validation occurs before upload-kind dispatch. Every handler delete remains denied.

Before durable hold capture is enabled in staging or a preview, close new admission briefly, quiesce accepted sessions, inventory legacy versions, suspend completed-version expiry, and prove the rendered rule absent. Then prove the route, ledger, paused scheduled worker, exact policies, and synthetic terminal-version reclamation before unpausing publication. Failure leaves capture disabled and never restores expiry over a captured row. Production lifecycle is unchanged before Gate C.

Neither notifier function nor the durable hold/object-ready command creates, stores, or publishes a presigned download URL. Project Management mints a fresh version-pinned URL only in every accepted or idempotently accepted agent-claim response. The agent receives no AWS credentials, and a retry after the previous capability lifetime therefore receives a new URL only after the same job and version-specific HEAD checks pass. Before new-format commands are emitted, deploy a compatibility agent to the retained Kubernetes consumer that accepts both legacy commands carrying ZipDownloadUrl and the new claim response, preferring the latter. Prove that compatibility agent against the legacy producer and then against the new claim-response producer, plus queued replay. The old agent/new producer direction is forbidden because the new command deliberately carries no URL. ARRNC uses the same immutable compatibility image, and rollback may not go below that floor while new-format commands remain queued. An audited environment pause applies only to held Bulk PDF publication. Durable Bulk PDF capture/inventory continues, and the shared notifier's existing ReferenceUpload and ReferenceUpdate flows retain their current success/retry behavior. Human/user upload initiation is disabled for preview environments by a control outside PR code. No clinical, participant, production, or otherwise sensitive PDF may enter a PR-controlled agent or its preview child. If that enforcement cannot be proven, the preview Bulk capability stays disabled.

5. Exactly one queue owner

Two agents must never consume from the same environment queue during coexistence. The ARRNC staging container is first reconciled with consumption paused. Cutover requires:

  1. close only new-session admission while allowing already accepted multipart sessions to sign, acknowledge, heartbeat, complete, cancel, or abandon;
  2. prove active upload sessions quiescent while notifier publication and scheduled reconciliation continue, then release and drain eligible held events until every accepted job is terminal;
  3. only after that terminal proof, pause held Bulk PDF publication while durable capture continues and inventory every S3-object/hold-table/job/publication state, preserving any late capture;
  4. before the freeze validate the current Kubernetes/PVC public mapping and the candidate ARRNC mapping separately; drain the Kubernetes consumer, perform the initial copy, then switch public reads and require exact target equality across API, Project Management, registrar, and agent before scaling it to zero declaratively while retaining its PVC and rollback shape;
  5. after that fence, perform and verify a final delta copy and prove no further PVC writes;
  6. reconcile the notifier inventory and durable queue, enable only the ARRNC staging consumer, then resume held Bulk PDF publication;
  7. reopen new-session admission and run authenticated end-to-end proof.

Rollback mirrors the forward quiescence boundary: close only new admission; let accepted sessions complete, cancel, abandon, or policy-expire while publication/reconciliation continue; drain held events until every accepted job is terminal; then pause publication while capture continues and inventory any late event. Only after that proof may ARRNC be fenced, its messages/claims drained or made safely recoverable, and the inverse storage transfer begin. The Kubernetes consumer remains off until the reconciled PVC and sole ownership are proved. Queue depth alone is insufficient evidence.

6. Staging before retirement

ARRNC staging is implemented, applied, and proven before any Kubernetes resource is removed. Proof includes authenticated upload through the normal UI/API path, S3 notification, exact queue ownership, ClamAV clean and EICAR outcomes, child-scoped write and public read, CSV and Study effects, retry/idempotency, process and mount failure recovery, queue drain, observability, and stopped/absent plus retained-Kubernetes rollback. Previous-digest rollback is required only when a prior ARRNC digest has independently proven the paused, zero-subscription contract.

Only after that proof and explicit operational activation approval may cluster-gitops:

  1. keep the old Deployment at zero and PVC retained through a soak;
  2. capture a checksummed data manifest and a tested snapshot or equivalent restorable archive;
  3. remove the Application, ingress, Service, Deployment, and chart ownership declaratively;
  4. remove the PVC only after the retention evidence and rollback deadline pass.

Any rollback to a retained or restored Kubernetes claim first fences ARRNC, inventories all output written since the artifact was captured, and performs a collision-safe checksummed inverse transfer into that claim. The old static route or consumer is not enabled until every current stored reference resolves against the reconciled claim.

The live pod, Application, ingress, or PVC is never deleted imperatively or abruptly.

7. Preview lifecycle

An explicitly enabled, authorized same-repository PR creates its slot from a verified digest through the protected activation workflow. Readiness must prove the exact root, sentinel, network, ClamAV, RabbitMQ identity, and resource limits before any SyRF preview receives Bulk PDF configuration.

The ordinary PR workflow may build and attest a notifier package, but its mutable lambda-packages/pr-N.zip object is never a credentialed deployment source. A protected base-branch activation workflow binds an immutable S3 object version and package SHA-256 to the approved repository, PR number, exact head SHA, and maintainer approval. It revalidates that record immediately before deploying both the event-handler and reconciler functions and before injecting the preview-only secret into the reconciler alone. A synchronize event invokes the protected generation fence above before the new head may deploy; unapproved new code cannot use either running function, client secret, or issued bearer token. Teardown disables new S3 invocation only after accepted sessions quiesce, but retains the scheduled reclaimer and preview client until every hold has reached exact deletion plus cleanup acknowledgement. Only then does it disable the schedule, revoke the client, remove both functions, and retire the immutable package reference through the exact-key version/delete-marker cleanup below. This is an approval boundary for credential delivery, not protected preview serving or a claim that approved code is hostile.

Closure, disablement, or staleness runs idempotent teardown:

  1. disable new-session admission while accepted multipart sessions retain their lifecycle paths;
  2. complete, cancel, abandon, or policy-expire every accepted session while durable notifier capture, publication, and scheduled reconciliation continue until every accepted job is terminal;
  3. pause preview publication while capture continues, inventory every late notification, then disable new S3 event invocation only after no accepted session can create another object and fence the consumer;
  4. drain or quarantine the preview's queue and prove no active claims;
  5. stop and remove the exact container and scanner, fencing every process from the child;
  6. unmount the ordinary agent's exact child so PR code no longer has access;
  7. give a root-controlled cleanup worker a one-use capability for only previews/pr-N, prove that its cleanup view is the expected remote child rather than a local mount underlay, reject traversal, symlink/reparse-point, or sibling targets, then delete only that child with no-follow validation or quarantine it and remove the cleanup view;
  8. keep the scheduled reclaimer and preview OAuth client active until every active/retention-waiting hold reaches conditional exact-version deletion and authenticated cleanup acknowledgement. Poison atomically becomes QuarantinedHold with full authority/debit; evidence-only copy and force-discard are forbidden. Prove zero active holds, zero quarantine-ledger rows, zero retirement outboxes, and zero active Project Management release records, unless matching PM-link, source-tombstone, active-destination, and PM-finalization receipts prove the durable handshake completed for each stable key/source event/duplicate fence, obligation, and debit;
  9. only then disable normal EventBridge reclamation, revoke the preview OAuth client, remove capture, and switch the reconciler to credential-free tombstone-only purge. Retain the exact table, purge schedule/owner, and tombstone-only dynamic record until zero replay-fenced capture/transfer tombstones, then remove them and revoke the exact RabbitMQ account/vhost and its secret;
  10. only after those routes/functions can no longer require Project Management state, apply the snapshotted close-time database decision: without lock-db, remove PR-head, agent, notifier, and user-facing access, but retain the exact database, last-approved cleanup-only API/PM path, and protected synthetic-harness identity through every snapshotted P7D receipt window. The server derives the teardown-owned synthetic Project set from the protected manifest and rejects caller-selected/user/persistent Projects. After receipt/hold/outbox fences pass, invoke ordinary Project deletion, purge receipts to zero, revoke the last identity/path and wait token expiry, then delete/prove database absence. Failure retains the cleanup debit without an agent slot; with lock-db, preserve it unchanged as promised, remove every preview runtime/client access path, and record the existing preservation handoff. Never retain the current early database decision while a hold or active release record exists;
  11. remove the disposable rootless runtime state, per-preview network, routes, and ports; remove the dynamic manifest record only after its tombstone-only phase reaches zero tombstones;
  12. remove the exact preview image reference and only layers proven unused by any other live or persistent slot;
  13. after the hold-row and retirement-outbox sets plus active release-record set are empty, notifier invocation is disabled, the preview client revoked, and both functions removed, abort every incomplete multipart upload and enumerate/delete every current/noncurrent version and delete marker beneath only the exact preview data prefix and notifier-package key;
  14. prove empty version/marker/MPU listings plus process/runtime/network/storage/credential/ notifier-hold absence, release only the corresponding proven-absent parts of the atomic fleet reservation, and publish durable absence evidence. A recursive current-key delete is not proof.

Capacity for any quarantined resource remains charged to its owning fleet reservation, or moves atomically to an explicit quarantine debit that counts against the same aggregate budget, until reclamation is proved. Containers, writable runtime state, identities, networks, and storage allocations are never reused across PRs without the reviewed reset, fencing, and identity-rotation contract.

If an exact-child cleanup capability or remote-mount proof is unavailable, quarantine the child and alert. Cleanup must never fall back to a common-share credential or delete the local directory under an unmounted path.

An hourly reconciler compares open, enabled PRs with registered slots. Active runtime/session/job/ hold/outbox/release-record cleanup, runtime/public-child removal, or quarantine handoff must converge within 24 hours. The non-authorizing retained-receipt/database tail is governed by its snapshotted P7D window and must purge/release within 24 hours after the latest receipt expiry. Failures alert and fail closed rather than broadening deletion authority.

8. Production is a separate promotion

This decision and the staging migration do not apply production configuration, deploy a production container, change production credentials, drain a production queue, enable the production feature flag, or migrate existing PDFs. Production stays untouched until:

  • staging and preview evidence is accepted;
  • a production promotion plan and diff are separately reviewed;
  • rollback and operator coverage are current;
  • the user explicitly approves production promotion.

9. Documentation authority

This ADR and its companion correction design and plan are authoritative for PDF-agent hosting. Older Kubernetes/PVC hosting statements are historical only and must carry a supersession notice. After the migration, documentation and chart/GitOps cleanup remove obsolete deployment guidance so it cannot be followed as a recovery runbook.

Approval gates

  • Gate A — architecture approval: approve this planning PR after its checks and reviews are clean. No implementation begins before this approval.
  • Gate B — staging activation approval: after implementation PRs are reviewed, dry-run evidence is attached, and ARRNC staging is ready with consumption paused, explicitly approve the drain/cutover and later Kubernetes retirement sequence.
  • Gate C — production promotion approval: a distinct future approval after staging and preview acceptance. Gate A or B never implies Gate C.

Consequences

Positive:

  • one deployment model and one storage shape across environments;
  • staging and previews validate the same on-prem constraints as production;
  • preview lifecycle and credentials become explicit, bounded, and auditable;
  • Kubernetes retirement removes a known source of architectural drift.

Costs and risks:

  • ARRNC needs measured multi-preview capacity, child-share ACLs, dynamic registration, cleanup, and atomic fleet admission;
  • approved preview images run on a shared on-prem kernel, so dependency or account compromise and accidental destructive code remain residual risks bounded by rootless least privilege, environment isolation, immutable provenance, resource limits, audit, and teardown;
  • staging cutover needs a short controlled upload freeze to preserve single-consumer ownership;
  • restoring a retired Kubernetes agent is slower than restarting a container, so snapshot and inverse-copy rollback proof are load-bearing; a compatible previous ARRNC digest becomes an additional rollback option only after independent proof.

Rejected alternatives

  • Keep staging/previews in Kubernetes: contradicts the corrected owner decision and preserves the hosting variation this record removes.
  • Run staging on ARRNC but previews in Kubernetes: still leaves preview-only deployment and storage behavior untested for production.
  • Mount the common share root and bind children: exposes more authority than an ordinary agent requires and turns a host-path convention into the primary isolation control.
  • Allow arbitrary dynamic gatekeeper entries: broadens a deployment mechanism into an on-prem remote-code execution surface.
  • Start ARRNC and Kubernetes consumers together for a soak: competing consumers make ownership, drain, retry, and rollback evidence ambiguous.
  • Remove the Kubernetes Application/PVC as soon as ARRNC starts: makes recovery destructive before end-to-end proof and restorable data evidence exist.

References