Skip to content

Proportional Study Allocation

Outcome

Project administrators can divide the new annotation work in a stage between active project members. A configured percentage is a share of review slots, not a share of studies. For example, in a two-review stage, a 50% reviewer is eligible for one review slot on every study.

This implementation replaces the abandoned approach in PR #2450. It deliberately delivers the smallest usable workflow and does not retain that branch's separate mapping collection, reserve pool, reviewer statuses, rebalance operations, dashboard, or OR-Tools dependency.

Administrator workflow

  1. Create an annotation-only stage and set the number of reviews required per study.
  2. Keep the stage disabled while configuring it.
  3. Open Stage administration → Proportional workload allocation.
  4. Enable allocation and assign percentages to the active project members. Use 0% for members who should receive no new work; the remaining total must be exactly 100%.
  5. Save the allocation, then enable the stage for review.

Allocation can be changed or disabled only before any reviewer has started work in the stage. "Started" means a saved review session or an outstanding slot reservation: with active reviewer tracking a reviewer annotates against a reservation that graduates into a session only on their first save, so counting sessions alone would let an administrator republish the plan underneath live work. If the team needs different proportions after work starts, create a new annotation stage. This protects completed and in-progress work from being silently reassigned.

Assignment semantics

  • The feature is off by default behind proportionalStudyAllocation.
  • An unconfigured stage follows the existing assignment path byte-for-byte.
  • Configured stages use a fixed 10,000-bucket domain derived deterministically from the study ID.
  • Every bucket contains exactly Stage.SessionCountTarget distinct reviewers.
  • Reviewer bucket counts match their basis-point share of all review slots to within one basis point. A reviewer's exact target is basisPoints * reviewsPerStudy bucket slots, and those targets always sum to 10,000 * reviewsPerStudy. A reviewer holds at most one slot per bucket, so a target above 10,000 is unreachable; that happens only when the review count does not divide 10,000 (an equal three-way split on a three-review stage is 3,334/3,333/3,333, whose first target is 10,002). The plan clamps such targets to the bucket count and redistributes the few surplus slots to the reviewers with the most headroom, deterministically. The per-reviewer configuration cap is therefore ceil(10,000 / reviewsPerStudy), not the truncated value: the truncated cap made an equal split of exactly reviewsPerStudy reviewers impossible to save.
  • Saved-session resumption is never filtered; allocation applies only to new annotation work.
  • New annotation work is assigned through the atomic capacity-claim path introduced by PR #2467: the eligible buckets are applied to the random candidate query inside AtomicAssignRandomStudyForAnnotationAsync, and the study is then claimed with a SlotReservation under the stage's EnforceAnnotationTarget policy. An empty eligible set makes that query return no candidate, so the claim loop takes its ordinary "no study available" exit without retrying. An existing reservation is a resumption, not new work, and is returned before any bucket is computed.
  • The bucket plan assumes each study is filled by exactly Stage.SessionCountTarget distinct reviewers. A stage with EnforceAnnotationTarget off claims slots without the capacity guard, so concurrent claims can push a study past that target; eligibility stays correct (the candidate query still requires an insufficiently-allocated study) but the realised split can drift from the configured percentages. Stage.UpdateStage locks reviews-per-study while shares are enabled; it does not lock target enforcement.
  • The stage's study selection mode is locked to Annotation alongside its review mode. StudyAssignmentPolicy keys the fetch strategy off the selection mode alone, so an annotation-only stage left on ScreeningAndAnnotation would take the NewForScreeningOrAnnotation path, which receives no buckets.
  • When a reviewer's own buckets are exhausted but the unfiltered pool still reports new work, the assignment is re-resolved with the new-annotation pool treated as empty, so a reviewer holding unfinished saved sessions resumes them instead of being told the stage is complete.
  • Reconciliation pools are deliberately not filtered by workload shares. Shares steer who annotates new work, not who reconciles it.
  • A reviewer omitted from the configuration receives no new studies.
  • Missing, invalid, or stale reviewer membership fails closed for new work: when the configuration names a member who is no longer active, or no longer validates against the stage's current reviews-per-study, reviewers receive no new studies rather than an error. Resuming a saved session and reconciliation are unaffected in that state.

The domain type is StageWorkloadShareConfiguration, held by Stage.WorkloadShares; the derived plan is StageWorkloadSharePlan and each study carries a WorkloadShareBucket. Administration is served by api/projects/{projectId}/stages/{stageId}/workload-shares. "Allocation" remains the product-facing name of the feature and of its proportionalStudyAllocation flag.

Because the bucket plan is derived from the versioned Stage configuration, there is no second mapping record that can become stale or fail to publish atomically.

Existing studies and indexes

New studies receive their stable allocation bucket when they are constructed, and Study.EndInit recomputes it on load whenever a document carries no bucket or an out-of-range one. The bucket is therefore a pure function of the study id for every in-memory instance, so a whole-document writer that loaded a legacy study before the backfill ran cannot write the sentinel back over it. Before enabling a configuration, the API additionally backfills missing or invalid buckets for every existing study in the project, so the bucket query does not depend on documents being read first. The configuration is saved only after that preparation succeeds.

MongoDB maintains the compound index ProjectId_1_WorkloadShareBucket_1. Rollback is safe: turning the feature flag off restores the existing selection path; bucket values and the index are inert and may remain in place. Rollback is also reversible: the flag gates only the enable operation on PUT .../workload-shares, so reading and disabling an already-configured stage keep working with the flag off. Gating the whole controller would strand such a stage, because its persisted configuration keeps Stage.UpdateStage refusing review-mode, selection-mode and reviews-per-study edits while the endpoint that lifts that lock has disappeared.

Direct annotation endpoint enforcement

With the runtime flag and stage allocation enabled, GET .../studies/{studyId}/review and ordinary annotation session submission use the same StageWorkloadShareEligibility bucket policy as next-study selection. Out-of-share or unconfigured reviewers receive 404 before usage writes, reservations, or submission. Stale/invalid configurations deny new work without throwing. Existing authorization and capacity checks still apply.

Saved ordinary sessions remain accessible for the same reviewer and stage, including completed work that needs correction. Submission must identify that exact saved session to receive this exemption. A different reviewer's session, another stage, a reconciliation session, or a bare slot reservation does not grant an exemption. Flagged ordinary submissions must have matching route/body stage and session IDs; mismatches return 400 before mapping or saving.

Reconciliation endpoints and submissions remain unfiltered. Submission mode follows the body used by the existing save path, not the unused query parameter. With the runtime flag off, or without enabled stage allocation, legacy eligibility behaviour is preserved.

Delivery status

The original MVP is merged but dormant. Direct enforcement is in PR #3211. This follow-up adds the stage-overview reviewer allocation page and administrator counts/progress, backed by authorized read-only APIs. These changes remain pending review/merge and do not enable any environment flags. See STATUS.md for the authoritative remaining-work order and the delivery continuation for the acceptance criteria and unresolved gates.

The reviewer page uses GET .../workload-shares/my-studies?offset=0&limit=25; it shows distinct assigned studies, own saved status, remaining and currently available work, with bounded direct-study links. The administrator view uses GET .../workload-shares/progress and the existing stage-design policy. Both APIs return 404 with the runtime flag off. Counts are calculated in MongoDB without loading complete project studies into the API or browser. No extra collection or live event stream is introduced.

Assigned studies include preserved own ordinary saved work; remaining is assigned minus completed. Startable new work also respects the existing capacity query and is reported separately. Stale configurations show a paused state with no new assigned buckets, while keeping saved work visible. Administrators see inactive configured reviewers and recovery guidance, but membership mutations and automatic redistribution are not introduced.

MVP limits and follow-up order

The MVP supports annotation-only stages and active project members.

Known limitation: stage-review authorization is not validated. Eligible reviewers are the project's active members. A stage with a restricted StageReviewPolicy can therefore be given a positive share for a member who cannot open that stage, and those bucket slots are never filled. The check is not made here because it cannot yet be made faithfully: StagePermission.IsStageAuthorized needs the reviewer's own application-group claims. Identity stores groups, but there is no canonical out-of-request resolver that preserves the current cross-provider token and stored-role semantics. It also resolves its defaults through ResourceSecurity.Instance, a write-once static assigned only by the API composition root and absent from the project-management host and the domain unit tests. An approximate check evaluated with no claims would wrongly reject a member authorized through an application role. This remains an explicit dependency in STATUS.md.

The following are progressive enhancements, in order:

  1. Review and merge direct endpoint enforcement, then deliver reviewer assigned-studies UI, remaining-share counts and an allocation-active indicator, followed by administrator assigned/completed counts. See STATUS.md for the complete delivery sequence.
  2. Add explicit reviewer lifecycle handling for membership changes after activation.
  3. Validate stage-review authorization when configuring shares and when listing eligible reviewers, once reviewer authorization can be evaluated outside a request principal.
  4. Memoise the derived plan by stage id, configuration version and reviews-per-study. StageWorkloadSharePlan.Create walks all 10,000 buckets on every "next study" request and there is no reviewer-count limit, so a large team adds measurable CPU per review request.
  5. Measure bucket-query performance at production scale before considering materialized plans.
  6. Add history-aware rebalancing for stages that already contain completed work.
  7. Add reserve/onboarding policy only after its review-slot semantics are specified.
  8. Add co-allocation optimisation and a progress dashboard if real usage demonstrates value.
  9. Refresh the allocation editor's reviews-per-study constraint when the stage entity changes without a stage switch (a Review Settings save that changes the target). The server still validates and fails closed with a 400, so this only affects client-side pre-validation.
  10. Coordinate a combined navigation-dialog SAVE that both enables allocation and changes a locked stage setting (review mode, study-selection mode, reviews-per-study). The allocation publishes first and the domain then rejects the stage PATCH; surface that rejection or validate the stage settings first. Tracked with item 9 in the review follow-up issue.