Repository navigation
Review fixtures: three samples trip sibling articles, so recall and clean rate 1.0 are out of reach for a faithful run #201
Description
Where: evaluation/review-fixtures.json cases built from three samples (main at 07e324d), scored by tools/Test-ReviewFixtures.ps1 at minimumExpectedRecall 1.0 and minimumCleanRate 1.0.
Run: BCQuality main at 07e324d, Claude Code 2.1.283, model claude-opus-5-5. All 108 cases were run, each reviewed only by its domain's leaf skill, as the per-case request files prepared by Test-ReviewFixtures.ps1 -PrepareDirectory describe.
The run scored recall 53/54 and clean rate 52/54. All three misses are findings that the leaf skills and articles themselves call for:
data-modeling/check-blocked-in-referencing-code-not-in-master.bad.al(positive case). Only that article was expected. The leaf also cited:owning-table-must-delete-dependents-in-ondelete.md: "Loyalty Point Entry"."Member No." has aTableRelationto "Loyalty Member", and "Loyalty Member" has noOnDelete. That is the article's own detection signal: a table declares aTableRelationto table X, and table X has noOnDeletetrigger.master-table-no-from-number-series-in-oninsert.md(minor): a custom master with aNo.Code[20] key, noNo. Seriesfield and noOnInsertnumber assignment.al-data-modeling-reviewnames this cue: a custom master table changes its primary key,No./No. Seriesfields orOnInsertwithout assigning a blankNo.from setup through a number series.
data-modeling/check-blocked-in-referencing-code-not-in-master.good.al(clean control): the same two articles, for the same reasons.upgrade/initvalue-does-not-update-existing-rows.good.al(clean control): it citesregister-upgrade-tags-with-subscribers.md. The upgrade step is guarded byHasUpgradeTagand set withSetUpgradeTag, but the tag is never added toOnGetPerCompanyUpgradeTags, which is that article's anti-pattern.
In all three cases, the per-case request's candidateArticles include the sibling article. The scorer counts any finding id outside expected, so a leaf that follows its skill cannot score 1.0 on these cases.
Suggested fix: make the samples comply with their sibling articles. For example, give "Loyalty Member" an OnDelete for its point entries (or a DeleteAllowed/refusal the article accepts) plus a No. Series assignment, and register the upgrade tag in the initvalue sample. The alternative is to let a case declare "also acceptable" ids for sibling concerns that its sample does not isolate.
Related open pull requests that also edit evaluation/review-fixtures.json: #181, #196, #198.
Activity
Nikolaj1407 commentedon Oct 13, 2026
Follow-up from the same evaluation on main at 86d809b (252 cases, model
claude-opus-5-5, each case reviewed only by its domain's leaf skill): the same kind of conflict, now in more domains, and one case family that is the testing domain.testing:
test-feature-scenario-tagsfires on 13 cases whose sample has no tags.al-testing-review.mdselects that article for every new or changed[Test]procedure "whether or not it already carries" the tags. These samples are[Test]codeunits written for another article and carry no[FEATURE]/[SCENARIO]/[GIVEN]/[WHEN]/[THEN]tags, so the leaf cites the tags article and the scorer counts it as unexpected (bad cases) or as a finding on a clean control. The 13 are the bad and good samples ofui-handlers-in-tests,reset-per-test-state-before-the-isinitialized-guard,commit-shared-test-fixture-inside-lazy-initialize,transactionmodel-attribute-governs-test-transactions,ui-test-codeunit-naminganduse-assert-isfalse-not-asserterror-for-boolean-checks, plus the bad sample ofasserterror-needs-expectederror-and-code.Other domains show the pattern of #201 as well, each with the sibling article's own detection signal met by the sample:
style:tooltip-required-on-page-fieldson the page samples that declare fields without a ToolTip.ui:bound-page-field-inherits-source-field-tooltipon page samples with no tooltip.performance:use-setloadfields-for-partial-recordson samples that read one or two fields from a full record, andapply-filters-before-iterating/calcsums-instead-of-calcfields-in-loopon thedesign-covering-keys-from-read-patternsamples (both the bad and the good sample contain the same filterable loop).web-services:check-http-status-before-consuming-response-bodyon thehandle-httpclient-platform-failuresamples, andexpose-systemid-as-the-api-keyon theapi-page-least-privilege-write-accesssamples.upgrade:register-upgrade-tags-with-subscriberson samples whose tag is local and not registered.The same two suggested fixes apply: make the samples comply with their sibling articles, or let a case declare "also acceptable" ids for the sibling concerns it does not isolate. For the testing family the smallest fix is to add the tags to the 13 samples.
Separately, the comment-stripping in case preparation affects two articles whose rule is about comments; that is PR #227.