Skip to content

Materialized Project Statistics

SyRF will provide fast, authorized current and historical project statistics without repeatedly running the complete authoritative MongoDB aggregate on ordinary reads. Materialized values remain disposable, versioned read projections; Project and Study remain the source of truth and every current consumer can fall back safely to the approved live calculation.

The detailed architecture and phased delivery gates are defined in the technical plan. The recovered-design reconciliation records every material difference from the March-April 2026 plans and M008-M011, with an explicit disposition.

User outcomes

  • Project users can load approved screening and annotation statistics without the latency of a full project-wide recalculation on every ordinary read.
  • Users can understand meaningful distributions of Study workflow state, rather than seeing only simple totals.
  • Authorized reviewers and project managers can request equivalent membership/reviewer breakdowns without exposing peer data they cannot currently access.
  • Users can request durable historical checkpoints with clear calculation time, source provenance and schema version.
  • A stale, rebuilding, missing or incompatible current projection falls back to the current authoritative calculation; historical APIs never fabricate an old point from current data.
  • Operators can disable writes or serving globally, per family, per consumer or per pilot project without deleting projection data.

State-profile requirement

The primary product statistic is a bounded, catalogue-defined distribution over meaningful current Study states. It is not an arbitrary analytics cross-product.

Approved profile families will cover:

  • project screening combinations such as reviewer include/exclude decisions, screening-decision number/status, sufficiency, started and overscreening classifications;
  • authorized membership/reviewer screening profiles;
  • project/stage annotation-session combinations such as no session, incomplete/in-progress and completed; and
  • authorized membership/reviewer-stage annotation profiles, including availability and reconciliation state where the current live calculation defines them.

A Study may validly contain include and exclude decisions from different reviewers. Project profiles must preserve and tally these conflicting reviewer-decision combinations. A profile is excluded as impossible only when an enforced application invariant proves it cannot occur. Every profile key, formula, permitted overlap, ordering rule and authorization class must mirror an approved current live calculation and be versioned in the metric catalogue.

MVP boundary

The minimum independently useful runtime delivery is:

  1. a dark shared projection/history foundation with transactional signed deltas, immutable operation history and fail-safe current fallback;
  2. project screening state-profile materialization with parity and performance evidence; and
  3. one screening-only API/Project Overview consumer behind an independently reversible flag.

The first consumer must not request broad FullStats, because that bundle also requires annotation and membership families and would correctly fall back until they exist.

Requirements

Correctness and safety

  • Materialized values never become domain truth and can always be rebuilt from Project and Study.
  • A current response is evaluated against one project committed revision or wholly from the authoritative calculation. An unchanged Fresh scope may retain an older last-changed revision.
  • Supported same-MongoDB point mutations update authoritative source, affected materialized scopes and an immutable unique delta/history record in one transaction.
  • Family guards select a visible generation; each scope selects its newest eligible published row at/below that generation without compatibility filtering, and evaluates compatibility only after selection, so an incompatible newest selected row forces whole-bundle authoritative fallback and is never skipped in favour of a superseded older row. Bounded bulk candidates remain isolated while unchanged older scopes stay visible. A constant-size guard flip assigns a distinct public revision and replacement provenance.
  • Bulk, cross-system, non-additive and unsupported mutations fence affected scopes until a rebuild or bounded bulk-delta publication completes.
  • Large imports remain wholly hidden from authoritative queries behind one Project-level visibility token, while a separate admission lock blocks overlap until every affected family publishes or a verified terminal abort releases the gates. A publication-capacity abort marks the affected families Stale and deliberately reveals already committed source only through authoritative fallback; a source- loading abort instead proves and removes the hidden prefix before release. Bounded Study cleanup never becomes the visibility mechanism.
  • Historical checkpoint-set roots publish a complete compatible bundle through BSON-bounded reference pages of immutable real scope observations. Unchanged scopes may reuse an earlier observation reference; missing or incompatible history is explicitly unavailable and is never reconstructed from current state.
  • Current and historical reads apply current authorization, including own-row and peer-row rules.
  • Source capture, retries, redelivery, concurrent writers, rebuild leases and publication are idempotent and fail closed.

History and provenance

  • Checkpoints record project, metric/scope keys, catalogue/schema/source versions, source watermark, observed/calculated time, trigger and operation/event provenance.
  • Current/delta publication and its durable revision/scope notification outbox row commit together; a fixed-size checkpoint root names one BuildToken and atomically publishes BSON-bounded immutable reference pages/observations only after epoch, lifecycle and fence guards pass.
  • Retention is bounded so compaction cannot remove delta provenance required by surviving checkpoints, retries, rebuilds or audits.
  • Exact delta identities and moves have measured replay/idempotency floors and a hard ceiling; safe old ranges compact into bounded seals, while unsafe pressure disables materialized writes and falls back.
  • Bootstrap records one truthful observation at enablement and never invents earlier history.

Rollout and evidence

  • All serving starts disabled and is controlled by durable epoch-aware global/project safety gates plus family, consumer and project-pilot flags; cached process flags are not correctness authorities.
  • Shadow parity compares materialized and authoritative calculations before any consumer cutover.
  • Each family proves exact integer parity, safe fallback, mutation/invalidation coverage, authorization, history behavior and bounded storage.
  • Each consumer demonstrates a material read-p95 improvement and authoritative aggregation reduction against named reproducible datasets before activation.

Explicit exclusions

  • Unit-level statistics/materialization: an annotation unit is scoped to one Study and reviewer annotation; no useful cross-study aggregate or performance need has been demonstrated.
  • Outcome-level statistics/materialization: no stable authoritative cross-study aggregate, authorized consumer or measured performance need has been demonstrated, so outcomes are not represented in the programme's scope key or metric catalogue.
  • Arbitrary profile dimensions: clients cannot construct unbounded cross-products or invent formulas.
  • Broad agreement/kappa: PR #2534's cross-stage calculation is invalid; any future measure requires a separately approved same-stage cohort and denominator.
  • Operational progress/presence: import/export job state, Bulk PDF progress and live connections remain in their operational models.
  • Runtime implementation or rollout: this planning PR creates no product code, migration, staging or production activation.

Unit or outcome materialization may return only through a future separately approved feature with a concrete useful aggregate, authoritative definition, consumer and measured performance need.

Delivery and approval gates

This brief and its technical plan approve architecture review only. The next deliverable is a separate documentation/read-only Phase 0 completion PR containing the exhaustive metric/consumer catalogue, method-level mutation ownership, profile formulas, fixed benchmark datasets and executable commands.

Phase 1 implementation requires explicit user approval after that Phase 0 completion review. Every later phase, staging proof, production pilot, wider rollout and legacy retirement has its own gate in the technical plan.

Success criteria

  • Approved profile distributions exactly match authoritative live calculations, including valid mixed include/exclude reviewer decisions.
  • No stale, rebuilding, incompatible or pending-event scope is served as Fresh.
  • Historical unavailability is explicit and never substituted with current data.
  • The first screening-only consumer executes at least 80% fewer authoritative statistics aggregations and improves response p95 by at least 20% on the approved benchmark.
  • Supported source-mutation p95 regresses by less than 10%; realistic ½/5/10-reviewer benchmarks show acceptable retry and write-conflict rates. A small shared summary is allowed unless measurements justify splitting or striped counters.
  • Rollback to authoritative reads is immediate and does not require data deletion.
  • Epic #1831 — pre-calculated statistics programme
  • PR #2534 — original broad design evidence; stale implementation is not resumed
  • PR #2985 — screening-only implementation evidence to reconcile after shared contracts are approved
  • FEAT-006 — domain reconciliation compatibility input
  • FEAT-009 — screening annotations compatibility input
  • FEAT-013 — export/report consumers enabled by the shared query contract