Skip to content

Bulk PDF Upload v2 — Implementation Plan

Hosting correction (2026-08-30): The ARRNC hosting migration plan supersedes PR-0c completed-object expiration, PR-1's completed-object delete/proof branch, PR-2 notifier scope, PR-5, PR-6, Wave 3, rollout, and rollback instructions wherever they place a PDF agent in Kubernetes/PVC or limit ARRNC hosting to production. Those entries describe the historical rollout only and must not be executed. It also supersedes every production apply, production smoke, production flag-enable, launch, announcement, and monitoring instruction in this document, including the full production portions of rollout steps 3 and 4 below. Production requires a separate plan, review, and explicit Gate C approval; Gate A or B cannot authorize it. It also supersedes the flag-off-first rollback lever: rollback closes only new-session admission while accepted lifecycle operations quiesce under the corrected notifier drain/pause sequence. PR-0b's production Terraform handover is historical and is not a dependency of the ARRNC hosting correction. The corrected staging/preview critical path requires no camarades-infrastructure notifier-VPC, private-subnet, or egress-firewall change; such work is optional future hardening. It does require ADR-015's separate IAM-only update to the existing notifier permissions boundary. The pre-agent staging routing rehearsal in rollout step 2 is also retired: Bulk PDF publication remains paused until the dual-read compatibility consumer is deployed and proved, and no diagnostic queue may discard a URL-free agent command. Preview input remains a protected synthetic harness, but a clean delivered PDF is intentionally public at the existing base URL plus its lifecycle-owned previews/pr-N/ child. No per-preview IIS mutation is authorized.

Design: 2026-08-11-bulk-pdf-upload-v2-design.md. This plan is executable PR-by-PR; each PR section lists repo, scope, key files, tests, and acceptance criteria. Waves order the PRs by dependency; PRs within a wave are parallel.

Wave map

Wave PR Repo Summary
0 PR-0a camaradesuk/syrf s3-notifier chart: template notification filter + lifecycle rules
0 PR-0b camaradesuk/camarades-infrastructure prod role ignore_changes handover block
0 PR-0c camaradesuk/cluster-gitops per-env filter/lifecycle values + Function CR adoption fix + verified no-op prod sync
1 PR-1 camaradesuk/syrf Domain + API: server-owned multipart session, sweeper, processing claim, PM consumers, normalization contract
1 PR-5 camaradesuk/server-config clamd role, syrf-pdf-net, CIFS mount, syrf runner, agent slot
2 PR-2 camaradesuk/syrf s3-notifier code: dispatch switch, BulkPdfUpload kind, version-pinned object-ready release
3 PR-3 camaradesuk/syrf Agent service + chart + image CI + arrnc-api deploy workflow
3 PR-6 camaradesuk/cluster-gitops agent service wiring for staging + previews (after PR-3)
4 PR-4 camaradesuk/syrf Frontend: folder upload, multipart/reconciliation client, truthful results UX, user guide

Merge order = wave order. PR-5 production apply (manual dispatch) happens with Wave 3. Exception to "PRs within a wave are parallel": PR-6 targets the chart/image PR-3 publishes (the ApplicationSet would otherwise generate an Application for a chart revision that doesn't exist yet), so within Wave 3, PR-3 must merge and its image/chart tag land in GHCR/cluster-gitops before PR-6 merges.

Wave-3 prerequisite (runner-boundary review): before PR-3 merges, the self-hosted deploy runner on arrnc-api must be approved and CLAUDE.md's runner-boundary section updated in PR-3 to define the deploy-runner category (server-config-managed host, gatekeeper-restricted sudo, no build/test workloads, repo-scoped) — OR PR-3 switches to the fallback: a GitHub-hosted job calling container-web-deploy over SSH with a restricted key. Settle this before PR-3 is written, not during its review.

Wave 0 — ADR-010 minimal unblock (prerequisite)

Why bundled: the notifier change (PR-2) cannot reach production through the chart while the Phase-5 cutover is stuck, and the chart's first successful prod sync as-written would drop the Projects/ notification filter and the live multipart-cleanup lifecycle rule (design §7).

PR-0a — chart templating (syrf)

  • src/services/s3-notifier/.chart/templates/bucket.yaml: render spec.notification.lambdaFunctionConfigurations[].filter.key.filterRules from .Values.bucket.notificationFilterPrefix (omit when empty), and spec.lifecycle.rules from .Values.bucket.lifecycleRules (omit when empty).
  • src/services/s3-notifier/.chart/values.yaml: bucket.notificationFilterPrefix: "", bucket.lifecycleRules: [] defaults.
  • Tests: helm template golden-file assertions for empty/filtered/lifecycle variants.
  • Acceptance: chart renders byte-identical manifests for existing envs when new values unset.

PR-0b — Terraform role parity (camarades-infrastructure)

  • terraform/lambda/main.tf: add to aws_iam_role.production_lambda_role the same lifecycle { ignore_changes = [tags, tags_all, description, assume_role_policy] } block the staging role gained in the ADR-010 Phase-4 handover (main.tf:189-204).
  • Acceptance: terraform plan shows no proposed changes to the production role.

PR-0c — values + adoption fix + verification (cluster-gitops)

  • syrf/environments/production/s3-notifier/values.yaml: bucket.notificationFilterPrefix: "Projects/"; production lifecycle remains unchanged before the separately approved Gate C plan.
  • The former seven-day expire-bulk-staging rule, including noncurrent-version and delete-marker expiration, is retired and non-executable for staging and previews. Their rendered Bucket CRs retain only the distinct incomplete-MPU lifecycle safety net for Bulk PDF keys. Completed object versions are reclaimed only by the 2026-08-30 plan's named, bounded-schedule reconciler phase: its durable row supplies the exact version, authenticated state proves terminal/no active or retriable claim, and it snapshots deployment-owned retention (PT24H staging, PT1H preview) before a conditional lease gates exact-prefix DeleteObjectVersion. A terminal-before- publication hold first uses the stable cleanup-registration command to persist a non-claimable cleanup-only Project Management release record and never emits agent work; cleanup acknowledgement conditionally converts the deletion-independent Project Management release record to a non-authorizing audit receipt. The reconciler persists ReceiptRetirementPending plus the immutable delete/ack result, then one transaction atomically creates HoldRetirementOutbox and converts the stable capture key to replay-fenced CaptureTombstone. Only that outbox obtains and durably records Project Management retirement confirmation before deleting itself; receipt purge separately requires OutboxDrainedAt from an exact-key absence proof. TTL is never absence proof. No lifecycle rule substitutes for that implementation.
  • Resolve the terminal Function CR: delete the conflicting AdoptedResource/Function pair and re-adopt in order on chart v1.17.1 (nameOrID fix), per the chart's own forward-path comment (adopted-resource.yaml:40-45).
  • Gates (in order, STOP on failure): (1) re-authenticate AWS CLI and independently confirm live bucket notification + lifecycle + Lambda config and current bucket CORS before touching anything; (2) after sync, s3-notifier-production Synced/Healthy; (3) aws s3api get-bucket-notification-configuration and get-bucket-lifecycle-configuration are canonically identical to the captured production baseline — i.e. the cutover is a no-op for production runtime behaviour — while rendered staging/preview Bucket CRs omit completed-version expiry and retain only incomplete-MPU cleanup; (4) an operator harness creates a staging MPU and presigns one browser-origin UploadPart whose signature binds exact content length and SHA-256; a real browser PUT of an exact known-length Blob succeeds with user-agent-owned Content-Length and JavaScript-set checksum header, proving CORS does not require JavaScript to set the forbidden header; (5) a staging search-import smoke test passes (the P-3 verification ADR-010 never ran).
  • Acceptance: production app Synced/Healthy with notification and lifecycle verified unchanged; staging/preview rendering proves completed-version expiry absent and incomplete-MPU cleanup retained; chartTag pinned ≥ the PR-0a chart version.

Wave 1

PR-1 — Domain + API (syrf)

Scope (design §4.1–§4.3): BulkPdfUploadJob embedded entity + guards + counts + bounded history (last 20/search, pruned on new-job creation); server-owned S3 multipart session bound to exact upload ID/object key/size/whole-object SHA-256/part geometry/virtual host; configurable 20-minute renewable lease, server-advertised 60-second heartbeat, completed-part renewal, and exact-content-length + checksum-bound presigned UploadPart URLs with 5-minute request-start validity; configurable API-advertised 600-second maximum client part-request duration; endpoints to initiate/status/sign part/ acknowledge part/heartbeat/complete/abandon/cancel/fresh retry; backend-only CompleteMultipartUpload; atomic monotonic completion-vs-abort transitions; typed terminal rejection carrying authoritative state; storage reconciliation for ambiguous completion; one durable completion-operation lease shared by the original API completion and PM recovery (provisional 180 seconds around a 120-second server-owned storage timeout), with live-owner exclusion, atomic expired takeover, 60-second legacy missing-claim grace, exact ownership revalidation before every storage side effect/proof write, and a final claimed HEAD before Aborting; API persistence of each latest signed-part URL expiry before returning the signature; a hosted sweeper that promptly issues AbortMultipartUpload and derives cleanup eligibility as max(AbortStartedAt, LatestSignedPartUrlExpiresAt) + MaximumUploadPartRequestDuration, and then repeatedly aborts and verifies ListParts/HEAD. The objectless branch requires two joint empty/NoSuchUpload MPU and object-absence observations at least one 30-second sweep interval apart, resets proof on any nonempty MPU or present object, and atomically persists that absence proof before recording Abandoned/Cancelled and releasing active capacity. A present completed object is never deleted by this sweeper: it records cleanup-pending while capacity remains charged, then durable notifier capture supplies the exact version and Project Management atomically persists/reuses CleanupOnly, terminalizes the job, and releases capacity. The exact-version reclaimer as the sole deletion path. Direct absence proof applies only to jobs that truly have no MPU or completed object; the S3 incomplete-MPU lifecycle rule remains only a safety net; legacy BSON compatibility without read-time inferred expiry; Study.BulkPdfDeliveredAt and BulkPdfDeliveredPath; centralized bulk-aware PDF-URL builder routed through ALL five existing call sites (Study.GetLinkToPdf, StudyROBDto, StudyListItemDto, StatsWithIncompleteDto/StudyBaseDto, StudyDto, PdfConverterService) with per-segment percent-encoding; BulkPdfUploadJobDto + ProjectDetailsDto.BulkPdfUploadJobs + AutoMapper mapping (SignalR wiring); search/project deletion guard while a job is non-terminal; BulkPdfUpload policy; feature flag; history/report/pdfPaths endpoints; PM receive endpoint with processing-claim + progress + finalize consumers (single writer, concurrency 1, optimistic-concurrency retry policy); the agent claim must be persisted before download/scratch/sentinel/scan/copy and its opaque token must bind every progress/finalize message; delivered-outcome-qualified study marking; CSV report generation (RFC 4180 + formula neutralization); message contracts (path ≤ 512 B / detail ≤ 256 B caps); normalization helper + vector file + portability/collision/root-relative contracts; BsonClassMap registrations.

Key files

  • src/libs/project-management/SyRF.ProjectManagement.Core/Model/ProjectAggregate/BulkPdfUploadJob.cs (new)
  • .../Model/StudyAggregate/Study.cs (BulkPdfDeliveredAt, link generation)
  • src/libs/project-management/SyRF.ProjectManagement.Messages/Commands/ — IProcessBulkPdfUploadCommand, IReportBulkPdfUploadProgressCommand, IFinalizeBulkPdfUploadCommand, BulkPdfFileOutcome
  • src/libs/kernel/SyRF.SharedKernel/PdfPathNormalization/ — PdfPathNormalizer.cs, normalization-vectors.json
  • src/libs/kernel/SyRF.SharedKernel/Enums/UploadKind.cs — add BulkPdfUpload
  • src/services/api/SyRF.API.Endpoint/Controllers/ — BulkPdfUploadController + multipart DTO/options validation
  • src/libs/appservices/SyRF.AppServices/FileServices/BulkPdfUpload/ — backend multipart storage abstraction/implementation
  • src/services/project-management/SyRF.ProjectManagement.Endpoint/Consumers/ — claim, progress, and finalize consumers + endpoint registration
  • src/services/project-management/SyRF.ProjectManagement.Endpoint/Services/ — due-session sweeper/processor + validated options
  • src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/Repositories/ProjectRepository.cs — class maps
  • src/charts/syrf-common/env-mapping.yaml + pnpm run generate:flags output
  • appsettings — PdfBaseUrl env-supplied (no hardcoded default)

Tests: domain guard/idempotency/monotonicity suite (port the cases from #2373's PdfUploadJobTests/PdfUploadJobIdempotencyTests, fresh code); normalizer vs vector file; multipart controller/storage tests for exact bindings, signed content length + checksum, positive request size, API-advertised request timeout distinct from URL validity, part visibility/checksum/size, lease expiry, typed terminal responses, stale heartbeat non-resurrection, late successful part acknowledgement after abort, completion-vs-abort winners and ambiguous completion reconciliation; real-Mongo sweeper query/fairness tests; cleanup retention through the derived latest-URL quiescence boundary and first-empty observation, nonempty proof reset, second-empty interval enforcement, legacy/completed-object direct absence, transient storage errors, and repeated abort/ListParts; durable cross-replica abort-sweep claim acquisition, active-claim exclusion, expired takeover, exact-token/unexpired proof writes, bounded storage passes, prompt normal release, expiry takeover after process/host cancellation, and zero storage work by claim losers; separate durable completion-operation claim tests for API and PM ownership, legacy grace, active exclusion, expired takeover, per-side-effect revalidation, server timeout, final HEAD, and completion-vs-abort convergence; claim-before-side-effects token contracts; finalize consumer matching/dedup/Missing-Unmatched taxonomy (port UpdateStudyPdfStatusConsumerTests cases); controller authz + flag gating; CSV generation.

Acceptance: all endpoints 404 behind the flag; no behaviour change flag-off — the only production-visible code path PR-1 touches is link generation, which branches on Study.BulkPdfDeliveredAt, a field no existing document has (null → legacy URL) and which only the finalize consumer ever sets, so flag-off behaviour is byte-identical and no MongoDB migration is needed; swagger + generated TS client regenerated. Server state changes are available through ProjectDetailsDto/SignalR and status/heartbeat endpoints so PR-4 can reconcile missed real-time delivery. Storage IDs, keys, and claim tokens are never exposed to the browser. Do not enable the feature flag until the timing gate below passes.

Infrastructure boundary (not PR-1): before the first staging multipart rehearsal, verify the API runtime AWS identity and bucket policy allow the exact backend/presign operations under Projects/_bulk-staging/ (PutObject, GetObject, DeleteObject, AbortMultipartUpload, ListMultipartUploadParts) and verify bucket CORS accepts browser PUT with the JavaScript-set checksum header from every SyRF origin while the user agent supplies the SigV4-bound exact Content-Length for a known-length Blob. Content-Length must not appear in the API's JavaScript-settable RequiredHeaders. PR-0c preserves the incomplete-MPU lifecycle rule, which is cleanup defence in depth. If IAM, policy, CORS, or the real browser PUT proof is missing, add the narrow environment-repository change in the appropriate later wave; do not expand PR-1 or infer that the lifecycle rule makes the protocol safe.

PR-5 — Hosting (server-config)

Scope (design §6): docker_networks + syrf-pdf-net; roles/clamav (Ansible-managed, restart_policy: unless-stopped, host volume for signature DB, no published ports, and a templated clamd.conf drop-in raising StreamMaxLength/MaxScanSize/MaxFileSize to cover the 500 MB entry cap) + structural ansible_managed: true projects entry; cifs_mounts item for the ecrf1 share → /srv/data/syrf-pdf/production/output (mount_user = agent uid, smb_version per share); new cifs/syrf-pdfs age scope; github_runner block for camaradesuk/syrf (os_user: runner-syrf); gatekeeper project entry syrf-pdf-agent-production (host_port: 8089 nominal, volume_path: /srv/data/syrf-pdf/production, container_volume_path: /app/data, skip_web_acl: true) + mirror entry in deployments/arrnc-api.yml (managed: false, gated: true); non-secret env in portal-env-overrides.yaml; sentinel-file creation task (output/.syrf-storage-ok) guarded on the mount being active.

Tests/acceptance: repo CI (deployment-manifest exhaustiveness, secrets validation, lifecycle vocabulary) green; rehearsal-host apply clean; production apply deferred to Wave 3 (manual dispatch). Needs the share name + credentials (design §10) before the production apply, not before merge.

Wave 2

PR-2 — Notifier code (syrf)

Superseded scope: this PR-2 section is historical and is not an executable notifier plan. Step 4 of the 2026-08-30 ARRNC correction plan is the mandatory scope and dependency: durable hold table, event handler plus scheduled reconciler, authenticated API/PM state route, notifier identity and resource mapping, both-function configuration/network/IAM, version-specific PM proof, lifecycle retention, and tests must land together behind its gates.

Scope (design §4.4): dispatch Enum.TryParse + switch with logged no-op default; BulkPdfUpload case (metadata validation including expected completed-object size/SHA-256, immutable S3-version validation, durable stable-identity object-ready command to Project Management with no URL, exact version-specific HEAD before release, and fresh version-pinned presigned GET only in each accepted/idempotent PM agent-claim response); update the stale sections of docs/architecture/systematic-search-upload-flow.md this touches (uploadkind/virtualhost, flattened contract).

Key files: src/services/s3-notifier/SyRF.S3FileSavedNotifier.Endpoint/S3FileReceivedFunction.cs; tests in the endpoint test project (dispatch table: ReferenceUpload / ReferenceUpdate / BulkPdfUpload / missing-metadata / unknown-kind ⇒ no-throw).

Acceptance: unknown-kind and missing-metadata events log and return without throwing (kills the current single-study-PDF error/retry noise). Note for the PR-2 description: this is a deliberate semantics change — today an unrecognised object falls into the search-import branch and throws, triggering S3 retries; after this PR unknown kinds are a logged no-op with no retry. Reviewers should see that stated, not discover it. LocalStack integration tests cover the new kind end-to-end through the PM release gate: an exact version-specific HEAD, no URL in notifier/hold/durable-command state, and a freshly minted claim response URL that actually dereferences. They advance beyond the previous capability lifetime and prove an idempotent retry returns a new working URL (the test URL host is localhost:4566, so the Wave-3 agent fixtures must use the same LocalStack endpoint). Ships as s3-notifier-v{next} through the standard promotion (staging auto, production manual — now unblocked by Wave 0).

Wave 3

PR-3 — Agent service (syrf)

Scope (design §4.5): src/services/pdf-agent/ — SyRF.PdfAgent worker + consumer, mandatory PM processing claim as the first operation before sentinel/scratch/download or any other side effect; claim-token-bound progress/finalize; clamd INSTREAM client (limits sized per design §6.2), download byte-cap and exact whole-object SHA-256 enforcement, extraction guards + portability/collision contracts + .pdf-extension enforcement, conflict policies incl. SHA-256, sentinel check, crash-safe outcome journal (scratch/{uploadId}/journal.jsonl), lock/age-guarded scratch sweep, progress/finalize reporting; Dockerfile (dedicated non-root uid); pdf-agent.slnf; GitVersion; syrf.sln entries; .chart/ (deployment with Recreate strategy + PVC + sentinel init container + clamd sidecar with raised limits + optional static file server for staging/preview); ci-cd.yml build/push/tag jobs; full PR-level CI wiring — the agent must be added to pr-tests.yml change predicates, detect-service-changes.sh, and pr-preview.yml's detection/version/image matrix + preview deployment values, or agent PRs get no tests and previews can't run the PR's own agent image; .github/workflows/deploy-pdf-agent.yml (workflow_dispatch, runs-on: [self-hosted, linux, live], container-web-deploy with GHCR GITHUB_TOKEN auth and -e secrets — respecting the shared-runner boundary rules in CLAUDE.md); docker-compose.dev.yml gains a clamd container; LocalStack init extended.

Tests: consumer unit tests prove no filesystem/network-processing side effect precedes a successful PM claim, rejected/mismatched claims do nothing, and redelivery uses the persisted claim; fixture ZIPs (clean / infected via EICAR / zip-slip / zip-bomb / macOS junk / non-PDF / duplicate paths / unicode names); conflict-policy matrix incl. hash-compare; sentinel fail-closed; integration test against dockerised clamd + LocalStack.

Acceptance: image builds + pushes in CI with pdf-agent-v{semver} tags; staging E2E (below) passes; production deploy dispatch works after the PR-5 production apply.

PR-6 — Agent env wiring (cluster-gitops)

Sequencing: merges after PR-3 (not parallel — see wave-map note above); targets the chart/image tag PR-3 publishes.

Scope: syrf/services/pdf-agent/{config,values}.yaml; staging + preview environment values (PVC size, PdfBaseUrl, static server enabled, RabbitMQ vhost); ApplicationSet picks the service up like any other. Preview flag remains off by default; enabled per-PR via #preview-config when testing.

Acceptance: staging agent Synced/Healthy consuming the staging vhost; a preview PR with the flag enabled gets a working in-cluster pipeline.

Retired operational step: The former Wave-3 production apply/direct-command smoke sequence is non-executable. Production remains untouched before ADR-015 Gate C. Any separately approved Gate C proof must use the durable version-bound notifier-to-PM release gate and PM claim-time capability; it may not inject IProcessBulkPdfUploadCommand or a presigned URL directly.

Wave 4

PR-4 — Frontend (syrf)

Scope (design §4.7): folder selection (drag-drop + picker), client validation, pre-upload match preview (shared normalization, GET .../pdfPaths), streaming ZIP web worker (STORE) and chunked whole-object SHA-256; server-owned multipart client (initiate, signed part URLs, exact known-length Blob.slice for every part, browser-owned Content-Length, JavaScript-set checksum RequiredHeaders, UploadPart, server-verified part acknowledgement, heartbeat, backend completion); enforce the API-advertised configurable 600-second request timeout with the upload transport's timeout/abort support (XMLHttpRequest.timeout/abort(), or AbortController where supported), separately from the 300-second presigned request-start validity; bounded automatic failed-part retry while the same lease remains recoverable; honest local states (Uploading only during active part request/retry, Connection lost — retrying, Upload interrupted, Cancelling, confirming interruption with server); AbortController shutdown of local work on terminal state; idempotent abandon command durably queued client-side while offline and resent on reconnect/reload; authoritative session query before rendering restored state; SignalR-driven state with heartbeat/status and bounded polling reconciliation fallback; server terminal state stops progress/retry and discards queued completion; fresh independent Retry; result summary panel, history table, CSV download; TS normalization mirror + vector-file test; user-guide page (help.syrf.org.uk) for the new workflow.

Tests: store/component specs for every state (validating, preview, uploading, retrying, interrupted/confirming/cancelling, scanning, copying, complete, failed, infected); worker ZIP and chunked-hash tests; network drop with successful automatic part retry; terminal drop then reconnect; reload with queued abandonment; lost client that never returns; server abort while client thinks active; missed SignalR recovered by heartbeat/poll; late successful part after abort; stale heartbeat cannot resurrect; exact part slicing and request-timeout cancellation; assert Content-Length is never treated as a JavaScript-settable required header; completion-vs-abort winner convergence; honest accessible status text; normalization parity spec loading normalization-vectors.json; systematic-searches integration spec for flag gating.

Acceptance: flag-off = zero UI change; the UI never displays Uploading/In progress when it knows there is no active transfer; server terminal state always wins reconciliation; a local terminal attempt never looks resumable and Retry always creates a fresh upload. Preview input is not browser- or human-driven: the protected synthetic harness runs the multipart lifecycle with an exact PublicBaseUrl derived from the independent deployment-owned TrustedExistingPublicBase, with API/Project Management/registrar equality proved, then proves the clean delivered PDF is publicly readable only at the existing base plus its previews/pr-N/ child while infected, malformed, partial, and quarantined output produces no link or read, and sibling/cross-preview storage, mount, traversal, or link derivation is denied. A client that already knows another preview's valid delivered public URL may fetch it; no authenticated serving boundary is claimed. Staging timing proof must show every 16 MiB part of a slow 1 GB upload completes within the advertised 600-second request timeout, including the background/throttled cases.

Rollout & verification sequence — retired

This historical sequence is non-executable in full. Follow only the dependency-ordered staging and preview waves in the 2026-08-30 ARRNC hosting migration plan. In particular, do not enable staging publication, create a diagnostic queue, or complete an upload before the dual-read compatibility agent is deployed to the retained consumer and proved against the legacy producer. The corrected sequence keeps publication paused through capture/inventory proof, deploys the compatibility floor, then proves it against the new URL-free claim-response producer before ARRNC staging cutover. No production apply, smoke, flag, or lifecycle mutation is authorized before Gate C.

Rollback levers

  • New-session admission off while every already accepted session retains signing, acknowledgement, heartbeat, completion, cancellation, and abandonment until it quiesces. Drain eligible held events to terminal jobs, then set only the Bulk PDF publication pause and inventory late capture before fencing a consumer. The broad feature flag may be turned off only after no accepted lifecycle operation depends on it; it is not the first rollback action.
  • Agent: stop/redeploy previous image via the gatekeeper (--skip-pull env-only redeploys supported); messages queue durably while it's down.
  • Notifier: standard zip-version rollback via GitHub Release + config pin — with a floor: once any bulk upload has occurred, PR-2's version is the minimum safe notifier (pre-PR-2 code throws + S3-retries on any non-ReferenceUpdate kind, so an in-flight BulkPdfUpload event would loop). Rolling back below it requires the flag off and bulk events drained first.
  • Wave 0 is deliberately verified as behaviour-neutral before anything rides on it.

Risks

Risk Mitigation
ADR-010 Function CR fix goes sideways PR-0c hard gates + AWS-side pre-verification; production runtime stays on the TF Lambda until Synced/Healthy is proven
CIFS mount flaps under the container Sentinel fail-closed; _netdev,nofail fstab opts; runbook: restart container after remount
Normalization drift between TS and C# Single vector file consumed by both test suites; adding a vector on one side breaks the other until implemented
1 GB browser zipping on low-end machines STORE mode + streaming worker; documented guidance; cap enforced client- and agent-side
Slow links/background throttling outlive a live session Configurable 20-minute lease + 60-second heartbeat + completed-part renewal; server-advertised 600-second request timeout distinct from 300-second URL request-start validity; staging must prove every 16 MiB part fits before the narrow timing STOP gate clears
Browser cannot satisfy the signed part contract Sign exact content length + checksum; upload an untransformed known-length Blob.slice; browser owns forbidden Content-Length, JavaScript sets only returned checksum headers; PR-0c real-browser CORS/PUT proof
Client disappears or an in-flight part/completion races abort Durable idempotent abandon command when the client returns; monotonic Aborting; persist latest signed URL expiry, prompt abort, then delay proof until that expiry plus the maximum part-request duration; repeated abort/list/HEAD with two joint MPU-empty/object-absent observations ≥30 seconds apart for objectless work; one durable 180-second completion claim around a 120-second storage timeout fences API/PM completion while a separate abort claim owns cleanup. A present completed object enters cleanup-pending with capacity charged until Project Management atomically commits CleanupOnly and terminalizes/releases it; neither controller nor sweeper performs an unversioned delete, creates a delete marker, or records false absence. Lifecycle remains a backstop
RabbitMQ plaintext exposure (existing posture) Separate AMQPS hardening PR; agent flips URI when it lands
clamd signature staleness freshclam auto-updates; health check surfaces staleness in logs; scan failure ⇒ job Failed (never silently unscanned)