Recovered Statistics Design Reconciliation

Purpose and evidence

This is the decision ledger between the March-April 2026 statistics design and the current materialized-project-statistics programme. It prevents the older work from being reduced to a broad idea or the newer work from silently replacing decisions that were already supported by code-level analysis.

Primary recovered evidence:

• docs/planning/mongodb-stats-precalculation-plan.md from pull request #2534;
• docs/planning/stats-precalculation-master-plan.md;
• implementation plans M008 through M011, including every slice plan;
• the authoritative StudyStats aggregation on current main;
• current mutation, membership, question, import, reservation, API and SignalR paths; and
• the preserved screening candidate from pull request #2985.

The old branch is evidence only. File and handler names in it are not assumed current until Phase 0 reconfirms them on main.

Disposition vocabulary

• Restore earlier: the earlier decision is the preferred current design.
• Combine: retain the proven earlier mechanism and add a newer safeguard or broader contract.
• Retain newer: the newer plan closes a real gap without discarding valid earlier evidence.
• Reject earlier: the earlier decision is invalid or unsupported and must not return.

Recovered decision register

The master plan numbered thirteen decisions. Their current dispositions are:

Earlier decision	Disposition	Current decision
D01: separate pmProjectStats collection, not fields embedded on Project	Combine	Preserve a separate disposable read model, but use natural scoped-current, delta, checkpoint and control collections rather than one payload row.
D02: Study mutation methods return DDD delta value objects	Restore earlier	Before/after transition value objects remain the preferred ordinary path; repository mapping owns physical field paths.
D03: Study and statistics delta commit in one transaction	Restore earlier	Add the immutable unique delta/provenance record to the same transaction.
D04: store settings-independent tally dictionaries	Restore earlier	Preserve raw tallies where query parity proves them; derive configuration-dependent totals instead of duplicating them.
D05: initial project/pairwise kappa scope	Reject earlier	The stale implementation's cross-stage cohort is invalid. Raw decision profiles remain; any kappa needs a separate same-stage specification.
D06: Legacy to Backfilling to Live with a short write freeze	Combine	Use Missing/Stale/Rebuilding/Fresh, scoped fences and authoritative fallback. A measured short gate is last resort when snapshot boundary plus delta replay cannot be proven.
D07: tracking/sentinel document in the stats collection	Reject earlier	Use typed control, immutable delta provenance and checkpoint-run evidence; no magic date sentinel or growing mutable failures array.
D08: two-phase import and deletion	Combine	Retain PendingImportJobId/PendingDeletion and signed bulk deltas; make import visibility one Project-level hidden token rather than per-row clearing, and add stable operation/fence/progress plus an authoritative-rebuild escape hatch.
D09: three-source audit and repair	Combine	Keep materialized versus legacy facets versus independent streamed classification at one boundary; repair only through fenced evidence-bearing rebuild.
D10: sparse daily snapshots with tracking metadata	Combine	Keep sparse real observations and explicit gaps; add fixed-size checkpoint roots over bounded pages that may reference prior unchanged immutable observations, revision/provenance and bounded retention/history.
D11: state-aware current, history and realtime reads	Combine	Use one authorized catalogue query, coherent requested-bundle fallback and revision/scope SignalR invalidation/refetch.
D12: delete statistics on Project deletion	Combine	Deny/tombstone immediately and purge current/history under the Project's approved retention policy.
D13: question deletion resets affected annotation sessions/statistics	Combine	Preserve exact completed-to-incomplete bucket moves and definition identity; fence/rebuild only when the rewrite is unbounded or non-additive.

M008-M011 implementation-plan crosswalk

Recovered slice	Disposition	Required successor evidence
M008 S01 screening/annotation classification value objects	Restore earlier	Truth tables and real query fixtures must match all applicable current facets, including exact boundaries.
M008 S02 ScreeningDelta/AnnotationDelta, rescreening old-state capture and inverse property	Restore earlier	Signed old-bucket decrement/new-bucket increment, zero-delta and inverse tests remain mandatory.
M008 S03 pmProjectStats repository and ReviewDomainService transaction	Combine	Retain transactional orchestration, replace one large current document with natural scopes and add unique immutable delta history/retry safety.
M009 S01 staged two-phase import	Combine	Retain invisible staging and bounded atomic reveal/bulk delta; add durable operation fencing, restart progress and measured transaction limits.
M009 S02 pending two-phase deletion	Combine	Retain batched classify + delete + negative delta transactions; hold fence until final parity/reconciliation.
M009 S03 remaining session/reconciliation/membership/question mutation coverage	Combine	Re-audit current main for renamed/new owners, including durable SlotReservation effects, then use signed deltas where supported.
M009 S04 settings reclassification from tallies	Restore earlier	Recompute from approved settings-independent tallies when bounded; rebuild only if required source facts are absent.
M009 bootstrap slice	Combine	Prefer source boundary plus committed-delta replay and authoritative fallback over a mandatory review freeze.
M010 S01 replace aggregation read and remove sync-over-async	Combine	Cut over one authorized consumer at a time after parity; keep compatible DTO adapters until all requested families exist.
M010 S02 three-source audit	Combine	Preserve exact legacy-query and streamed-classifier sources; add scoped leases, immutable mismatch evidence and non-blind repair.
M010 S03 stats change-stream full push	Retain newer replacement	Emit client-invalidation revision/family invalidation after commit with source/projection provenance and refetch through authorization; do not push stored supersets.
M011 S01 sparse daily snapshot	Combine	Keep truthful sparse observations, key checkpoints by the three-component identity (source revision, projection revision, mode epoch) so any kind of change can append, and add bounded retention.
M011 S02 history endpoint and failure metadata	Combine	Use the shared current/history catalogue contract, typed unavailable/410 results and current authorization.
M011 S03 area-chart activation and fill-forward	Retain newer phasing	Backend history is required; visible UI is a later independently flagged consumer and may render explicit gaps rather than silently masking failures.

Mutation ownership recovered from the earlier plan

These are evidence anchors, not assumed-current implementation names. Phase 0 must re-resolve every row against current main before assigning file ownership. Every source-changing row also commits the bounded all-mode source-operation receipt and client-invalidation outbox entry; the table below names its family-specific projection action:

Mutation	Earlier/current evidence anchor	Required materialized action
Submit/correct/rescreen	ReviewController submission paths and ReviewSubmissionService.AddScreening	Transactional source + project/reviewer signed screening moves + delta record
Candidate annotation save/complete	ReviewController.SubmitSession and ReviewSubmissionService.AddSessionData	Transactional stage/reviewer/question signed moves where catalogue-supported
Domain reconciliation session lifecycle	Session save/complete/delete paths catalogued in M009	Transactional stage/reviewer reconciliation moves; this is not parity repair
Threshold/agreement change	ProjectAgreementThresholdUpdatedHandler	Reclassify from screening tallies when bounded, otherwise fence/rebuild
Stage/session settings change	Project/stage aggregate replacement handlers	Reclassify from session tallies when exact; otherwise fence affected scopes
Systematic-search import	Earlier SearchImportJobConsumer, now requiring current-owner resolution	Two-phase staged invisibility plus bounded bulk delta or fenced rebuild
Systematic-search deletion	ProjectManagementService.DeleteSearchStudies	PendingDeletion, delete + negative delta batches, final parity before fence clear
Question deletion	ProjectManagementService.DeleteQuestionAsync and ResetSessionsForStagesAsync	Preserve completed-to-incomplete reset semantics and question/stage invalidation
Membership add/disable/remove	Project membership lifecycle, including later DisableMembership behavior	Current authorization immediately; bounded reviewer-scope delta/invalidation or fenced non-additive rebuild
Durable reviewer reservation lifecycle	Later SlotReservation claim/release/expiry repositories and consumers	Include only if a current approved availability/capacity formula uses durable reservations
Project deletion	Project deletion owner resolved in Phase 0	Deny/tombstone immediately, then retention-policy purge
Legacy manual question tally refresh	Existing Project tally refresh path	Remains authoritative until the question-family cutover, then is superseded

Authoritative formula and facet baseline

The earlier documents describe an 11-facet query, but current main has 15 facets. The catalogue must name and fixture every current facet before replacing it:

Current facet	Meaning retained for parity
Investigator	Investigator-level screening facts used by membership profiles
All	All available studies in the project population
UnexcludedInvestigatorStats	Per-investigator annotation values for the unexcluded population
ExcludedInvestigatorStats	Per-investigator annotation values for the excluded population
AllUnexcluded	Project/stage annotation values for the unexcluded population
AllExcluded	Project/stage annotation values for the excluded population
UnexcludedSessioned	Unexcluded studies with stage session state
ExcludedSessioned	Excluded studies with stage session state
UnexcludedCandidateSessionsStartedGroupedCount	Candidate-session started tally for unexcluded studies
UnexcludedStartedReconciliationCount	Reconciliation-started tally for unexcluded studies
UnexcludedCompletedReconciliationCount	Reconciliation-completed tally for unexcluded studies
ExcludedCandidateSessionsStartedGroupedCount	Candidate-session started tally for excluded studies
ExcludedStartedReconciliationCount	Reconciliation-started tally for excluded studies
ExcludedCompletedReconciliationCount	Reconciliation-completed tally for excluded studies
ScreeningGroupedCount	Count by number of screenings and number of include decisions

The consumer-visible per-stage AnnotationSessionStats does not take its reconciliation counts from the standalone reconciliation facets above: CreateAnnotationSessionStats maps each stage's started and completed reconciliation values from that stage's Sessioned.CompletedAndStartedReconciliation and Sessioned.CompletedAndCompletedReconciliation. Phase 3 must fixture a multi-stage project with unequal per-stage reconciliation counts, so an implementation cannot assign project-wide reconciliation totals to every stage while still matching the standalone facets.

The current screening equations to preserve exactly until a separately approved semantic correction are:

• number-screened sufficiency: NumberScreened is greater than or equal to the configured NumberScreened threshold;
• agreement sufficiency: AbsoluteAgreementRatio is strictly greater than the configured ratio, or strictly greater than -1 when the threshold is absent;
• sufficiently included: both sufficiency predicates and Inclusion strictly greater than 0.5;
• sufficiently excluded: both sufficiency predicates and Inclusion strictly less than 0.5;
• Inclusion exactly 0.5 is in neither included nor excluded under the current query;
• started: NumberScreened strictly greater than zero;
• overscreened: NumberScreened strictly greater than the current hard-coded lower bound of 3;
• overscreened-yet-insufficiently-screened: NumberScreened strictly greater than 3 and AbsoluteAgreementRatio strictly less than the configured ratio (or less than -1 when absent). This is not the logical intersection of Overscreened and InsufficientlyScreened: equality with the agreement threshold increments InsufficientlyScreened but not this composite facet, and this facet does not test the configured NumberScreened sufficiency threshold;
• overscreened-and-sufficiently-included/excluded: NumberScreened strictly greater than 3, AbsoluteAgreementRatio strictly greater than the configured ratio (or greater than -1 when absent), and Inclusion respectively strictly greater than or strictly less than 0.5. These composite facets do not test NumberScreened greater than or equal to the configured sufficiency threshold, so a Study above 3 screenings but below a higher configured threshold can increment one of them while remaining InsufficientlyScreened; and
• ScreeningGroupedCount key: (NumberOfScreenings, IncludedCount), retaining valid mixed include and exclude decisions from different reviewers.

Phase 2 project-level query parity fixtures must lock these legacy asymmetries explicitly: agreement ratio exactly equal to its threshold, and NumberScreened above 3 but below a configured threshold greater than 4. Phase 4 applies the equivalent fixtures to membership/reviewer screening facets when that family is introduced. They remain compatibility facts, not approval of the labels or a semantic correction.

The current annotation query uses a hard-coded minimum of two candidate sessions (minNumberSessions = 2, StudyStats.cs) in exactly ten predicates even though a stage has SessionCountTarget. SessionedGroupStage applies it to Available, Full, Completed, CompletedAndNotStartedReconciliation, CompletedAndStartedReconciliation, CompletedAndInProgressReconciliation and CompletedAndCompletedReconciliation; InvestigatorSessionGroupStage applies it to NumberOfStudiesSessionAvailable, NumberOfStudiesSessionFull and NumberOfStudiesReconcileSessionAvailable. That is a parity fact and an unresolved product semantic, not permission to substitute the target during materialization. The catalogue must first decide whether to preserve or separately correct it, and the Phase 3 parity fixtures must assert each of these ten fields independently on stages whose SessionCountTarget is 1 and 3, so an implementation cannot use the configured target for some of them while passing the generic lifecycle cases.

The current membership/reviewer mapper uses project-wide allInfo.All values, rather than the investigator facet, for SufficientlyIncluded, SufficientlyExcluded, SufficientlyScreened, InsufficientlyScreened, Overscreened, OverscreenedYetInsufficientlyScreened, OverscreenedAndSufficientlyIncluded and OverscreenedAndSufficientlyExcluded. It also maps the project-wide denominator itself: TotalNumberStudies is project-wide allInfo.All.Count, and every membership screening percentage (PercentIncluded, PercentExcluded, PercentScreened, PercentAvailable, PercentUnavailable) and the NumberVerification balance predicate divide or balance against that project-wide total, never a per-reviewer count. Each percentage is truncated, not rounded: the current value is Math.Truncate(10000 * numerator / TotalNumberStudies) / 100, so a non-terminating ratio keeps its two-decimal floor (one of six is 16.66, never 16.67) and a raw decimal or ordinarily rounded result would break current-response parity. It also passes the investigator facet's SufficientlyExcludedAndAgree value into both the SufficientlyIncludedAndAgree and SufficientlyExcludedAndAgree output fields. The remaining membership screening outputs — Screened, Included, Excluded, SufficientlyIncludedAndDisagree and SufficientlyExcludedAndDisagree — map directly from the investigator facet, and Phase 4 fixtures must include unequal project/investigator values for these direct mappings too, so a replacement can neither source them from project-wide totals nor recompute them differently while passing the exceptional-case fixtures. Its availability outputs are subtraction formulas rather than direct facet aliases:

• Available = allInfo.All.InsufficientlyScreened - investigator.InsufficientlyScreened; and
• Unavailable = allInfo.All.SufficientlyScreened - investigator.SufficientlyScreened.

Phase 4 must preserve and fixture all of these mappings with unequal project/investigator, included-agree/excluded-agree, and availability-subtraction source values — including fixtures where the project-wide allInfo.All.Count total differs from the reviewer's own screened/available counts, so that a per-reviewer denominator would visibly change TotalNumberStudies and every derived percentage, and a non-terminating-ratio fixture for each derived percentage that would fail under rounding or untruncated output — until a separately approved semantic correction changes the live contract; intuitively substituting reviewer-specific, direct or one-to-one values would not have current-response parity.

The following stored, settings-independent tallies from the earlier plans remain the preferred raw inputs where current-query parity confirms them:

• screeningTallyCounts[numberOfScreenings][includedCount];
• candidateSessionsCountLookup[stage][candidateSessionCount][completedSessionCount], preserving the authoritative per-Study joint distribution rather than collapsing Studies with equal candidate counts. The grouped facet contains only Studies with at least one candidate session; the current mapper explicitly appends the stage (0 candidate, 0 completed) bucket as totalCount - sessioned.Count, separately for the unexcluded and the excluded population, each with its own total and sessioned facet. Phase 3 must record this subtraction as its own mapping and fixture it for both populations with non-zero sessioned counts, rather than deriving the no-session bucket from the grouped tally or from the wrong population;
• candidate and domain-reconciliation started/in-progress/completed transitions by stage and inclusion class; and
• membership-scoped equivalents needed to derive currently authorized available, unavailable, completed and agreement views.

The current CreateMembershipAnnotationStats mapper applies the following formulas independently to the unexcluded and excluded populations, with that population's allCount, aggregate sessioned facet and membership/stage investigator facet:

• Available = allCount - sessioned.Count + sessioned.Available - investigator.NumberOfStudiesSessionAvailable;
• InProgress = investigator.NumberOfStudiesSessionInProgress;
• Completed = investigator.NumberOfStudiesSessionCompleted;
• Unavailable = sessioned.Full - investigator.NumberOfStudiesSessionFull;
• Total = allCount and CandidateAnnotationSessionsGloballyCompleted = sessioned.Completed;
• ReconcileAvailable = sessioned.CompletedAndNotStartedReconciliation - (AllowSelfReconciliation ? 0 : investigator.NumberOfStudiesReconcileSessionAvailable);
• ReconcileInProgress and ReconcileCompleted are the matching investigator reconciliation counts; and
• ReconcileUnavailable = AllowSelfReconciliation ? 0 : investigator.NumberOfStudiesReconcileSessionAvailable, with SelfReconciliationEnabled = AllowSelfReconciliation.

The outer membership-stage envelope additionally maps two fields of its own: Total = allInfo.All.Count — the project-wide population, not either population-specific allCount used by the inner formulas above — and IncompleteThreshold = stage.MaxInProgress, passed through even when unset. Phase 3 must fixture a configured MaxInProgress and a project-wide total that differs from both the unexcluded and excluded population counts, so a replacement can neither derive the outer total from the wrong population nor drop the threshold while passing the inner parity cases.

The stage capacity flags are derived from these population stats by the Stage aggregate, not stored: HasReachedMaxInProgress is false when MaxInProgress is null or non-positive, and otherwise true exactly when the counted in-progress value is greater than or equal to MaxInProgress, where the counted value is the unexcluded in-progress count alone when HideExcludedStudiesFromReviewers is true and the sum of the unexcluded and excluded in-progress counts otherwise. HasReachedMaxInProgressReconciliation applies the same boundary and the same conditional population to the reconciliation in-progress counts.

Missing facets contribute zero exactly as the nullable current mapper does. Phase 3 must fixture unequal project/session/investigator values, both AllowSelfReconciliation states, and — for both capacity flags — both HideExcludedStudiesFromReviewers states with in-progress counts one below, exactly at and one above the MaxInProgress boundary plus the null no-limit case; replacing these formulas with intuitive direct per-reviewer availability or reconciliation counts is a semantic change outside this programme.

The consumer-visible MergedMembershipAnnotationStats then combines the unexcluded (U) and excluded (E) results according to ExcludedSessionStatsGrouping (G) with another non-intuitive compatibility mapping:

• Available = U.Available;
• InProgress = U.InProgress + (G == WithUnexcluded ? E.InProgress : 0) and Completed follows the equivalent formula;
• Unavailable = U.Unavailable + E.Unavailable + E.Available + (G == Unavailable ? E.InProgress + E.Completed : 0);
• Total = U.Total + E.Total, and candidate sessions globally completed sum both populations;
• ReconcileAvailable = U.ReconcileAvailable;
• ReconcileInProgress and ReconcileCompleted add their excluded value only for WithUnexcluded;
• ReconcileUnavailable = U.ReconcileUnavailable + E.ReconcileUnavailable + E.ReconcileAvailable + (G == Unavailable ? E.ReconcileInProgress + E.ReconcileCompleted : 0); and
• Separate additionally exposes the excluded candidate and reconciliation in-progress/completed values in its separate block instead of folding them into the merged progress/completion fields.

Phase 3 must fixture all three WithUnexcluded, Unavailable and Separate modes with non-zero, unequal excluded values. Excluded availability contributes to merged Unavailable in every mode; changing that behavior requires a separately approved live-contract correction.

The same value object additionally derives consumer-visible fields at read time rather than storing them: InsufficientlyCompletedCandidateSessions = Total - CandidateAnnotationSessionsGloballyCompleted, and eight percentage outputs (PercentAvailable, PercentInProgress, PercentCompleted, PercentUnavailable and their four Reconcile equivalents) computed as Total == 0 ? 0 : Math.Truncate(10000 * value / Total) / 100 — truncation to two decimal places, never rounding, with a zero total mapping to zero. In the Separate grouping, the separate excluded block's percentages (PercentExcludedInProgress, PercentExcludedCompleted and their two reconciliation equivalents) divide the excluded-only counts by the merged overall Total = U.Total + E.Total, not the excluded population's own total. Phase 3 fixtures must include non-terminating ratios (for example one of six yielding 16.66, not 16.67) and unequal unexcluded/excluded totals proving the merged-total denominator for every separate-block percentage and the derived insufficiently-completed subtraction. These are compatibility facts, not approval of the denominators or the truncation.

Derived values should be calculated from those stable tallies at query time when that avoids duplicate counters. A derived field may be stored only after measurement demonstrates that derivation itself is materially expensive.

Decision-ready material differences

Area	Recovered earlier decision	Current-plan divergence	Disposition and reason
Primary update path	Ordinary screening and annotation writes return signed before/after deltas and apply targeted MongoDB increment/decrement operations in the same transaction as Study	Every mutation emits an outbox event and waits for an asynchronous projector; pending events force live aggregation	Restore earlier. Same-database point mutations should atomically update source, affected scoped statistics and immutable delta history. This removes routine projection lag and fallback load.
Domain ownership	Study mutation methods produce ScreeningDelta or AnnotationDelta because the aggregate knows the exact transition	Domain facts are translated later by a reporting projector	Combine. Keep domain transition classifiers/value objects, but make the persisted delta contract versioned and catalogue-owned so domain behavior is not coupled to a wire DTO.
Delta semantics	Signed integer moves, including decrement old tally bucket and increment new bucket; inverse property is mandatory	Generic event replay and immutable replacement blocks	Restore earlier. Targeted signed bucket moves are smaller, naturally idempotent with an operation ledger and directly testable against the legacy query.
Transaction boundary	Study plus current statistics in one MongoDB transaction	Source transaction and later projection transaction	Restore earlier. Add the bounded source-operation receipt and immutable uniquely keyed delta/history record to the same source transaction. Source, receipt, current projection and Delta all commit or none do.
Driver retry behavior	Transactional atomicity was specified, but callback replay and unknown commit details were incomplete	Stable event IDs and applied-event checks cover redelivery	Combine. Allocate OperationId outside the callback, reread/guard the authoritative revision, generate deterministic deltas, use the all-mode source-operation receipt plus an applicable unique Delta, resolve unknown commits from those records and source revision, and never perform irreversible external effects inside a retryable callback.
Source-only idempotency	Earlier operation tracking assumed materialized writes remained available	Kill switch or ledger pressure can commit source without a Delta	Retain newer fallback, add a bounded safety receipt. Every source change commits a minimal unique operation receipt even when projection writes are disabled. Retain it through retry/redelivery/audit floors; reject old or over-capacity operations before changing source rather than risking a duplicate.
Physical current model	One current document per project in pmProjectStats	Manifest, immutable blocks, generation partitions, parent chains and CAS publication	Combine, simplified. Keep one logical ProjectStatisticsProjection but store scoped rows by natural family/scope/publication generation plus a small control/summary. A family guard bounds visibility; each scope resolves its newest row at/below that generation, preserving unchanged scopes without payload copies or parent chains.
Natural partitions	One project payload with project, stage and membership dictionaries	Hash-partitioned immutable block index	Restore the earlier named domains, revised physically. Use project/stage, reviewer or reviewer-stage, question and search scopes. Natural keys make authorization, repair and delta writes explicit; unit and outcome scopes are excluded.
Shared summary contention	Not treated as a blocker; expected reviewer counts are small	No source write may touch any shared project statistic	Reject the newer prohibition. A small project summary is acceptable. Benchmark 1, 2, 5 and 10 concurrent reviewers and retry rates. Striped counters are a measured fallback only.
Document-size strategy	Estimated dictionaries and pair matrices fit below MongoDB's 16 MB limit	Assume one project document will eventually become unsafe	Combine. Natural partitions remove most structural risk, but the sparse screeningTallyCounts[(NumberScreened, IncludedCount)] dictionary remains data-dependent. Every physical document has measured dynamic-key and encoded-BSON admission ceilings no greater than 50,000 keys/8 MiB. A new key that would cross either limit is rejected before its MongoDB update; source commits through the source-only receipt/invalidation path, the family becomes Stale/capacity-ineligible and reads fall back authoritatively. Backfill proves exact size before publication. Do not use the old estimate or a family-wide ceiling to justify an unbounded document.
Superseded current rows	One document overwritten in place	Token-addressed candidates coexist with current rows	Retain bounded candidates, add reclamation. After a guard flip, keep each scope's newest row at/below VisibleGeneration and active candidate/lease/build references; delete other rows in bounded batches after the oldest snapshot-reader grace boundary. Pressure disables materialized writes rather than growing without bound.
Immutable history	Sparse daily copies of current plus a tracking sentinel	Immutable versioned generations and reference-aware retention	Combine. Every accepted mutation has an immutable delta/provenance record; user-facing checkpoints remain sparse, truthful and bounded. No full snapshot per mutation and no mutable sentinel failure array.
Current/history atomicity	Current changes immediately; daily history is a later job and can fail independently	A history generation can reference the same atomic publication	Combine. Source, current scoped changes and immutable delta history commit together. Periodic checkpoint creation atomically publishes a fixed-size root over bounded pages/new observations and may reference unchanged immutable observations from the same coherent logical bundle.
Checkpoint manifest size	One project snapshot document	Complete logical bundle references	Retain newer semantics, revise physically. Publish a fixed-size root with an ordered-page Merkle root over BSON-bounded reference pages and immutable observations. References use the same deterministic full key as the technical plan—(MetricFamily, MetricKey, ScopeType, canonical ScopeTuple, ObservationId)—so page boundaries and range lookup remain stable when families share a typed scope. Unpublished pages are invisible; the root is the atomic visibility boundary and page proofs keep scoped reads bounded.
Checkpoint publication safety	Daily job copies current state	Sparse immutable checkpoint publication	Combine. Root names the exact BuildToken. A materialized build requires a Fresh compatible snapshot with zero active fences and CASes global/project/family guards. An authoritative build may publish during source-only, kill-switch or fenced projection intervals; it CASes its captured source revision and authoritative visibility boundaries and rejects concurrent source/visibility changes without requiring projection Freshness.
Checkpoint identity	Daily snapshots were primarily date-addressed	Source and projection revisions can advance independently	Retain newer revision provenance, use both clocks plus the mode epoch. Key a checkpoint by the ordered three-component identity (source revision, projection revision, fleet-wide mode epoch) so source-only changes, projection-only repairs and reviewer-mode transitions can each append without collision; expose an opaque stable cursor/ID over the complete identity.
Provenance	Snapshot date, settings snapshot, first/last run and failures	Schema/source/catalogue versions, source watermark, reason and operation IDs	Retain newer. These fields are required for interpretability, retry dedupe and safe schema upgrades.
Retention	Changed daily observations indefinitely, fill-forward in UI	90-day daily, weekly to one year, monthly to five years, maximum 256	Retain newer provisionally. Bounded storage is mandatory; Phase 0 must capacity-test and approve the exact numbers. Compaction selects real observations only.
Delta-ledger retention	Operation tracking was not bounded independently	Immutable operation history and rebuild replay	Combine with a hard bound. Keep exact identities and replayable moves only above measured floors; compact safe ranges into at most 64 mergeable seals. Below-floor redelivery fails closed. Stop materialized writes if safety prevents compaction before the ceiling.
Bootstrap	Legacy to Backfilling to Live with a short review-write freeze	Snapshot/replay generation build while ordinary writes continue	Combine. Use scoped fencing and authoritative fallback; capture a source boundary and replay committed deltas. Permit a short project-scoped write gate only if the repository cannot prove a replay boundary and a measured operational limit is approved.
Authoritative aggregation role	Initial baseline, three-way audit and repair	Routine fallback while every point event is pending	Restore earlier. Full aggregation is for backfill, scoped recovery, formula/schema upgrades and periodic reconciliation, plus exceptional fallback—not every normal write.
Rebuild concurrency	Read-only snapshot transaction and later correction write	Lease, watermark, replay and publication fencing	Retain newer. Scoped leases, source boundary and stale/rebuilding states close races the earlier correction write did not fully address.
Repair policy	Three-way audit could overwrite current from a selected source	Repair creates a new generation and evidence	Combine. Compare materialized state, legacy facets and independent streamed classifier at one provable boundary. Never blind-overwrite; fence, record mismatch evidence and publish a scoped rebuild with provenance.
Ordinary fallback	Legacy or live state selects one read model	Whole requested bundle falls back if any block is stale/pending	Combine. Keep coherent requested-bundle fallback, but a successful transactional point mutation has no pending interval. Missing, stale, rebuilding, incompatible, killed or fenced scopes still use authoritative reads.
Read consistency	State-aware reads, but no complete control/scope snapshot predicate	Control plus naturally partitioned scope reads	Retain newer fallback, strengthen the read. Require one MongoDB snapshot-read transaction over global/project control, scopes and fences. Separate majority reads are prohibited because an older scope can look unchanged; snapshot failure falls back authoritatively.
Bulk import	Insert with PendingImportJobId invisibly, calculate bulk delta, then atomically reveal studies and update stats	Generic fence, batched mutation, authoritative rebuild and replay	Combine. Retain PendingImportJobId, but make visibility one bounded Project.ActiveHiddenImportJobId gate read by authoritative queries. A separate ActiveImportOperationId blocks overlap until every family publishes; per-Study cleanup never reveals partial batches.
Bulk deletion	Mark PendingDeletion, then classify and delete batches with matching negative deltas	Generic fence and rebuild	Combine. Preserve per-batch delete plus negative delta transactions and restartable two-phase progress. Keep the affected scope fenced until every batch and final reconciliation prove completeness.
Bulk final publication	Final update assumed bounded enough for one transaction	Publish rebuilt scopes and clear all fences atomically	Combine, keep every transaction bounded. Stage token-addressed candidate scopes by stable batch ID behind a per-family marker/guard, catch up ordered deltas, clear fences in batches while the guard forces fallback, then flip one constant-size guard transaction with a new public revision and replacement-provenance record.
Search add/remove	Search population changes included in import/delete mechanics	Search is only a generic population invalidation	Restore earlier specificity. Search scope is a natural partition and every systematic-search create/import/rollback/delete path must own exact population deltas or a fenced rebuild.
Threshold/settings changes	Recompute derived counters from settings-independent tally dictionaries without Study queries	Always fence and rebuild	Restore earlier when bounded. Reclassify from raw tallies transactionally. Use fenced authoritative rebuild only when the formula needs source facts not present in approved tallies or exceeds measured bounds.
Membership changes	Active-membership totals and reviewer dictionaries adjusted; deactivated historical facts retained	Generic family fence plus authorization reevaluation	Combine. Apply bounded membership-scope deltas/invalidation synchronously; retain immutable historical facts, always apply current authorization, and rebuild only when membership changes alter a non-additive denominator.
Annotation question changes	Question deletion resets affected completed sessions to incomplete and requires bulk annotation delta	Version-aware question family with generic fence	Combine. Retain the exact reset transition and negative/positive bucket moves; add immutable question definition/version identity, scoped fence and rebuild for unbounded schema/content changes.
Slot reservations	Predates durable active-reviewer reservation state	Current plan treats durable reservation transitions as statistical inputs	Retain newer conditionally. Catalogue only the current authoritative availability/capacity formulas affected by durable reservation changes. Live connection presence remains operational, not statistics.
Annotation unit scope	Older plans sometimes implied reusable unit dimensions	Explicitly excluded	Retain newer. A unit is per Study/reviewer annotation; no cross-study aggregate or measured need exists. Re-entry requires a separate approved feature.
Outcome scope	Older broad plan and partition vocabulary allowed question/outcome scope	User decision removes outcome materialization	Reject earlier. Outcome is not a shared typed scope or metric family in this programme. Like unit scope, it may return only through a separately approved feature with a concrete authoritative aggregate and consumer.
Kappa/agreement	Project Fleiss from screening tallies and pairwise Cohen matrices, including cross-stage assumptions in the stale implementation	All agreement/kappa excluded	Reject earlier implementation. Do not revive cross-stage pairwise calculations or broad kappa UI. Existing raw decision-profile tallies remain; any derived agreement statistic needs a separately approved same-stage cohort and denominator.
Query shape	FullStats-compatible current document, history endpoint, REST/SignalR permission filtering	Domain-neutral current/history metric-key queries and typed unavailability	Combine. Retain current wire compatibility through adapters, but expose one authorized catalogue query contract for current and history with repeatable metric + canonical project/stage/membership/membership-stage/stage-question-version/search scope pairs. Heterogeneous bundles retain one-snapshot fallback without an ambiguous cross-product. A transport/streaming raw-body and raw-item cap applies before parsing/canonicalization (duplicates count), followed by hard deduplicated selection-count/encoded-key limits before a snapshot, so repeated input cannot consume unbounded parser work and pages cannot invent stores or fetch unbounded families; exact missing history returns typed unavailability/410.
Authorization	Explicit project/stage graph and leaderboard permissions; own reviewer row remains visible where currently allowed	Current authorization reapplied to stored superset	Combine. Preserve every earlier permission rule and own/peer distinction, apply current authorization on every current/history request, and never serialize persistence documents directly.
SignalR	Change stream pushes the whole current stats document with per-user filtering	Publish client-invalidation revision/family invalidation and refetch authorized data	Retain newer, make durable. Insert the invalidation into a dedicated MongoDB outbox in the source/receipt/current/delta or projection-only publication transaction. A leased at-least-once dispatcher publishes after commit; clients deduplicate by family/client revision and refetch. Retryable callbacks never push externally.
DTO read path	Remove sync-over-async aggregation from Project DTO mapping and use one current lookup	Defer all page cutover until shared projection phases	Combine. Remove hot aggregation only consumer-by-consumer after parity. Preserve compatible DTO adapters during migration and independently flag each cutover.
Parity source	Legacy query, stored stats and independent streamed classification	Generic shadow parity	Restore earlier specificity. Keep the three-source model at a single provable source boundary, with separate two-source checks only for metrics lacking a proven legacy facet.
Parity fixtures	Real production patterns, local Mongo backup, empty/new/rescreen/overscreen/annotation lifecycle, inverse delta	Representative corpus without named legacy cases	Combine. Make the recovered cases mandatory and add mixed reviewer decisions, current 15-facet fixtures, retry/redelivery, membership/question/search mutations and authorization views.
Query-level tests	Classifier results compared directly with legacy MongoDB aggregate	Mostly domain/repository/API parity	Restore earlier. A query-level fixture must execute the real authoritative pipeline and compare every catalogue bucket, not only compare two C# classifiers.
Performance proof	1,000/10,000/50,000-study estimates and Atlas transaction limits	20% read-p95, 80% aggregation reduction, under 10% mutation regression	Combine. Keep acceptance thresholds, add fixed datasets and realistic reviewer concurrency. Measure transaction p50/p95, retries, write conflicts, oplog/60-second limits, per-scope bytes, facet count and aggregation frequency.
Rollout state	Legacy, Backfilling, Live; new projects go directly Live	Global/family/consumer flags, allowlist, shadow mode and Fresh/Stale/Rebuilding/Incompatible	Combine. Use newer flags and lifecycle. Do not label a project Live until each requested family passes; independently roll back consumers.
Distributed write disable	No rolling-deployment flag barrier	Cached runtime write kill switch	Revise newer. Use a durable monotonic WriteEpoch and Enabled/Disabling/Disabled control read inside every source transaction and serving snapshot. Cached flags only request transitions; re-enable requires compatible writers and epoch-matched rebuilt scopes.
External side effects	Change stream and jobs were described after database updates but callback constraints were implicit	Event/outbox architecture separates publication	Retain the safety rule, make invalidation durable. Email, broker publish, SignalR, object storage and external APIs are forbidden inside retryable MongoDB callbacks. Required revision/scope invalidations use a dedicated durable transactional outbox; it is not the routine statistics update path.
Project deletion	Delete current and historical statistics	Tombstone, deny, then policy-driven purge	Combine. Deny immediately and purge under the Project retention policy; the projection creates no independent retention right.
Frontend history	Daily sparse snapshots, fill-forward, failure markers and area-chart wiring	Backend history first; visible UI optional	Retain newer phasing, preserve earlier contract evidence. Durable API history is required; UI activation is a later independently reviewed consumer using real checkpoints and explicit gaps.
PR #2985	Did not exist	Screening-specific current/history candidate	Retain as evidence, revise before reuse. Reuse classifiers, lifecycle, flags, lease, telemetry and tests where they fit scoped transactional documents; supersede screening-only schemas and per-mutation full-payload snapshots.

Preferred physical model

One logical ProjectStatisticsProjection is stored in naturally partitioned documents:

Document	Natural key	Role
ProjectStatisticsGlobalControl	singleton	Durable Enabled/Disabling/Disabled mode, monotonic WriteEpoch and GlobalClientInvalidationRevision checked inside every source transaction and materialized-read snapshot
ProjectStatisticsControl	project	Small versions, flags, lifecycle summaries, scalar source/projection/client-invalidation clocks, delta replay/identity and source-operation idempotency floors, last reconciled revision and deletion state
ProjectStatisticsSummary	project + bounded family + publication generation/token	Small approved project-wide totals/profile buckets selected like other scopes; splitting/striping still requires measurement
ProjectStatisticsScope	project + family + typed scope key + schema version + publication generation/token	Mutable visible/candidate counters and deletion tombstones; each scope first selects the exact visible generation/token row when present or otherwise the greatest older published row, then checks compatibility; an incompatible newest selected row rejects the materialized bundle rather than falling through, while unchanged older rows remain visible without exposing abandoned candidates
ProjectStatisticsSourceOperationReceipt	project + stable namespaced operation ID	Minimal bounded all-mode idempotency receipt for source changes, including source-only fallback; retained through retry/redelivery/audit floors and capacity-fail-closed
ProjectStatisticsRevisionRecord	project + unique namespaced record ID; ordered index and cursor by project + committed revision + stable RecordId	Delta uses stable operation/batch identity and carries replayable signed moves; all bounded sibling records at an equal revision must be consumed before that revision is complete; each bounded ReplacementPublication uses deterministic operation + family PublicationId, references a paged manifest and forms a replay barrier; PublicationAbort/SourceImportAbort preserve failure provenance without claiming publication; uniqueness spans upgrades
ProjectStatisticsRevisionRangeSeal	project + compacted revision range	Bounded mergeable audit digest below the delta replay/identity floors; never used to replay an individual operation
ProjectStatisticsCheckpointSet	project + three-component checkpoint identity (source revision, projection revision, fleet-wide mode epoch)	Fixed-size immutable publication root with selected BuildToken, page/reference counts and Merkle root for one complete logical checkpoint; source-only changes and projection-only repairs both append without collision
ProjectStatisticsCheckpointReferencePage	project + build token + ordinal	Immutable BSON-bounded, typed-scope-key-ordered page with content digest, key range and Merkle proof, allowing authenticated scoped reads; unpublished pages are invisible without the root
ProjectStatisticsCheckpointObservation	project + metric/family/typed scope + first-observed three-component checkpoint identity (source revision, projection revision, mode epoch) + content digest	Immutable real scope occurrence; an unchanged observation may be referenced by later checkpoint sets, while A→B→A creates a distinct later A occurrence rather than reusing pre-B provenance
ProjectStatisticsCheckpointBuild	project + BuildToken, with one active marker per CheckpointId	Bounded durable Building/Published/Abandoned/Cleaning/Cleaned marker, monotonic renewable lease, captured boundaries, counts/digest and restartable orphan-cleanup cursor; root publication CASes the marker and expired owners cannot publish or delete
ProjectStatisticsCheckpointBuildAdmission	project	Singleton admission and byte/page reservation for one unpublished build token across all CheckpointIds; newer requests coalesce without allocating artifacts, and an expired/abandoned build must be cleaned before successor admission
ProjectStatisticsGlobalNotificationOutbox	singleton + notification kind	One bounded coalescing global slot updated atomically with GlobalClientInvalidationRevision and broadcast to every statistics subscription without enumerating projects
ProjectStatisticsNotificationOutbox	project + metric family + notification kind	One bounded coalescing slot with monotonic target/delivered ClientInvalidationRevision and leased at-least-once delivery; source and projection-only changes both invalidate, and a reserved project-wide visibility slot reaches every family subscription
ProjectStatisticsOperationFence	project + operation + family/scope	Restartable bulk/import/delete/configuration boundary and progress
ProjectStatisticsRebuildLease	project + family/scope	Fenced lease for authoritative rebuild and publication
ProjectStatisticsPublicationGuard	project + bounded metric family	Independent monotonic GuardVersion plus constant-size VisibleGeneration/VisibleToken, one candidate generation/token, lifecycle and active-fence count used by serving/checkpoint CAS
ProjectStatisticsPublicationOperation	project + operation	Bounded operation-wide coordinator with fixed affected-family set, captured project/guard revisions and Preparing/SourceComplete/AllFamiliesAdmitted/Completed/Aborting/Aborted lifecycle (forward batches only in Preparing; the SourceComplete seal revokes the loader generation and is the state from which admission proceeds); the same owner may CAS AllFamiliesAdmitted back to SourceComplete, never to Preparing, after a lost final-publication race so it can catch up and re-admit; recovery authority across the admission-to-final-commit crash boundary, with measured per-project row/byte admission ceilings
ProjectStatisticsPublicationMarker	project + operation + family	Bounded idempotent ready/fence/page counts, manifest reference/digest and candidate token for constant-size final bulk publication
ProjectStatisticsPublicationManifestPage	project + operation + family + build token + ordinal	Immutable bounded affected-scope/tombstone manifest page, never embedded in the final revision transaction
ProjectStatisticsCurrentCleanup	project + family + superseded generation	Restartable reachability cursor/grace watermark for bounded deletion of non-selected old current rows
ProjectStatisticsPublicationManifestCleanup	project + family + manifest	Restartable floor/reference-aware reclamation of published, abandoned and aborted manifest pages
ProjectStatisticsPublicationOperationCleanup	project + operation	Restartable floor/reference-aware reclamation of terminal coordinators; non-terminal coordinators are never TTL-deleted
ProjectStatisticsOperationFenceCleanup	project + family + operation	Restartable floor/reference/snapshot-safe reclamation of terminal fence rows; active fences are never TTL-deleted
ProjectStatisticsSourceImportRollbackCleanup	project + import operation	Restartable floor/reference/source-postcondition/snapshot-safe reclamation of terminal identity/before-image rollback manifests; active or retryable manifests are never TTL-deleted

The authoritative Project additionally carries at most one ActiveImportOperationId and matching ActiveHiddenImportJobId. Imported Studies keep PendingImportJobId through bounded loading. Authoritative queries exclude exactly the hidden token in one source snapshot. Every affected-family manifest must pass bounded admission first; one bounded all-family transaction flips every guard, clears both Project gates, makes every imported Study visible together and advances all family plus project-wide notifications. There is no partial-family publication, and later per-Study cleanup has no visibility semantics. A terminal source-loading error instead uses a paged rollback manifest to delete the still-hidden prefix and atomically releases both gates only after zero matching rows and candidate artifacts are proven; it never reveals a partial import or leaves the next import blocked. Successful and aborted rollback manifests are reclaimed only after terminal operation, idempotency, audit, redelivery, reconciliation, reference and snapshot-reader floors prove they cannot be needed again.

An ordinary supported mutation performs one transaction:

1. load or guard the source aggregate and source revision;
2. calculate deterministic before/after classifications and signed bucket moves;
3. insert the bounded all-mode source-operation receipt or prove its identical retry;
4. write the authoritative source mutation;
5. apply targeted increments/decrements to every affected natural scope;
6. insert an immutable uniquely keyed (project, revision, stable RecordId)-ordered Delta revision record and advance the project committed-revision clock; and
7. advance source/client invalidation revisions and enqueue required family invalidations in the durable transactional notification outbox; dispatch occurs only after confirmed commit.

A duplicate operation with the same source revisions, schema/source versions and digest is a successful no-op. The same operation ID with different revisions, versions or digest is a conflict that marks affected scopes stale and requires investigation. MongoDB callback retries must never generate new IDs or repeat irreversible external side effects.

Exact delta identities are retained only above measured replay floors. The separate minimal source-operation receipt remains the source-mutation idempotency authority in materialized and source-only modes; receipts compact only after the redelivery/audit floor, and older operations fail closed. Safe contiguous delta ranges compact into at most 64 mergeable range seals. If delta compaction cannot keep within its configured ceiling, materialized writes stop and authoritative fallback takes over; if the bounded receipt cannot be admitted, the authoritative source mutation itself fails before changing data.

Recovered validation contract

Phase 0 must turn the following into named fixtures and commands:

• query-level equality against all 15 current facets;
• the consumer-visible FullStats identity envelope, not only its metric blocks: MembershipIds (every membership for an administrator request; only the selected investigator's membership for a restricted request) and the derived StageIds, with an administrator-view and a restricted-view fixture for both lists so a replacement adapter cannot pass every metric fixture while omitting or mis-scoping them;
• zero studies; one screening; exactly-threshold; below/above agreement threshold; Inclusion exactly 0.5; valid mixed include/exclude reviewer decisions; rescreening; and more than three screenings;
• candidate and reconciliation session start, incomplete, completion, deletion and question-reset transitions for excluded and unexcluded populations;
• membership-stage mapper parity with unequal all/sessioned/investigator values for candidate availability, unavailability, completion and every reconciliation output under both AllowSelfReconciliation states;
• outer merged membership-annotation parity with non-zero unequal excluded values under each ExcludedSessionStatsGrouping mode, including the Separate excluded block;
• membership own/peer views, inactive membership history, project/stage permission changes and project-wide invalidation of every affected family subscription;
• unequal project/investigator sufficiency and overscreening values proving membership output preserves the current project-wide mappings, plus unequal investigator SufficientlyIncludedAndAgree/SufficientlyExcludedAndAgree inputs proving the mapper duplicates the excluded-agree value into both output fields, and unequal project/investigator sufficiently screened and insufficiently screened values proving the exact Available/Unavailable subtraction formulas;
• search import, spreadsheet import with screenings, rollback, deletion and redelivery;
• delta inverse property and old-bucket decrement/new-bucket increment assertions;
• callback replay, transient transaction failure, unknown commit, duplicate operation and conflicting duplicate;
• concurrent screening of the same Study and different Studies at 1, 2, 5 and 10 reviewers;
• three-source parity at one snapshot boundary: materialized projection, current StudyStats facets and independent streamed classifier;
• sparse checkpoint ordering, truthful gaps, retention selection and rebuild provenance; and
• source-only or compacted scope-proof gaps forcing a new real observation even when the digest repeats;
• cold/warm 1,000, 10,000 and 50,000-study reads plus mutation latency, retry/conflict rates, oplog size, transaction duration and storage growth.

Decisions still requiring Phase 0 evidence

1. Whether Inclusion equal to 0.5, the strict agreement comparison, overscreening above 3 and the annotation minimum of 2 are intended product semantics or legacy defects. Materialization mirrors them until a separate correction is approved.
2. Which current import consumer and repository methods own staged visibility after the later import redesign; the old SearchImportJobConsumer name is not authoritative.
3. Which question-answer transitions are safely incremental and which schema/suppression changes require a fenced rebuild.
4. The maximum bounded setting-change recomputation and import reveal transaction before the design switches to fenced batching/rebuild.
5. The measured contention threshold that would justify splitting the small project summary or using striped counters.
6. The exact checkpoint retention/capacity values and whether a short bootstrap write gate is ever operationally preferable to boundary capture plus delta replay.

These are evidence gates, not reasons to revert ordinary point mutations to asynchronous projection.