Skip to content

fix: CMTK0003 reports assertions lost in nested combinators and through catches that cannot keep them - #172

Merged
CaffeinatedCoder merged 2 commits into
release/2.3.0from
fix/cmtk0003-nested-and-rethrow
Oct 29, 2026
Merged

CaffeinatedCoder merged 2 commits into
release/2.3.0from
fix/cmtk0003-nested-and-rethrow

Conversation

@CaffeinatedCoder

@CaffeinatedCoder CaffeinatedCoder commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Closes #165
Closes #166

Problem

CMTK0003 missed two ways an assertion of CodoMetis.TypeKit.Testing can fail without failing the test. Both were reproduced on 9878a6c.

  • CMTK0003 misses assertion failures hidden in nested task combinators #165: a Task.WhenAny inside a Task.WhenAll or a Task.Run finishes with the failed assertion's task as its value. So await Task.WhenAll(Task.WhenAny(a.ShouldBeNone(), b.ShouldBeNone())) and await Task.Run(() => Task.WhenAny(...)) pass while both assertions fail. The same loss through .Wait() and through an implicit plain-Task return was silent too. The rule saw that the outer combinator faults with the tasks it awaits, and stopped there.
  • CMTK0003 misses async assertions inside rethrowing catches #166: any catch around an awaited assertion in an async lambda counted as observing the failure. That included catch { throw; }, catch (OperationCanceledException) { } and catch (Exception) when (false) { }, none of which keep it.

Change

  • IgnoredOutcomeAnalyzer.HandsOn now returns a depth, not a bool. The depth is 0 where the root's task faults with the failure (WhenAll, Run). It goes up by one for each task that completes with the failure as a value instead (WhenAny, WhenEach, an async lambda that returns the task). It goes down by one for each await that unwraps such a task. DroppedAssertion counts an awaited, waited-for or converted combinator as observing a failure only at depth 0. This is one shared path, with no special case, so it holds at any depth. await await Task.WhenAny(...) stays silent.
  • Task.WaitAll(Task.WhenAny(...)) was already reported once, where the WhenAny converts to a plain Task, so AnalyzeWait is unchanged.
  • EnclosingAsyncFunction stops only at a catch that keeps the failure (Keeps), which takes all of:
    • the await is in the catch's try block, not in a handler or a finally;
    • the catch type is TypeKitAssertionException or one of its bases. The type is resolved in the Testing assembly by its metadata name (TypeKitSymbols.AssertionException), so a look-alike of the same name elsewhere doesn't count;
    • the handler can complete. Roslyn control flow decides: either its end point is reachable, or a return or break leaves it.
  • Decision on filters: a constant false filter leaves the handler unreachable, so control flow already reads it as never completing. A filter that isn't a constant, such as when (Expected(e)), counts as keeping the failure, because an Error reports only a failure that is lost whatever the filter answers. This is written down in the Keeps doc comment, in design/analyzers.md and in the analyzer README.
  • Scope rule: every form here can only be written with 2.3.0's new Testing API, so decision 0118 covers them and no 2.2.0 build reports more. The nested forms are assertion-only on purpose. Result tasks go through DroppedOutcome, which this PR does not touch.
  • Docs:
    • the analyzer README's CMTK0003 section has a new paragraph, quoted by claims T03-a-nested-when-any-is-reported, T03-a-catch-that-cannot-keep-the-failure-is-reported and T03-an-unknown-filter-counts-as-catching;
    • the CMTK0003 row in design/analyzers.md is updated, with the case count going from 830 to 1837;
    • the CMTK0003 Testing line in CHANGELOG.md is widened.

Tests

  • Catalogue: Variants/DroppedTaskShapes has 42 new forms, all assertion-only:

    • Nesting: WhenAll, WhenAny and Run, each inside each other, three and four deep. Arrays and Selects of WhenAny. Async lambdas that return a WhenAny or await one once, twice or three times.
    • Catch kinds that report: try/finally, rethrow, log and rethrow, throw another exception, an unrelated type, a false filter, nested rethrowing tries, and an await in a handler or in a finally.
    • Silent controls: catch all, catch TypeKitAssertionException, a true filter, a filter that isn't a constant, a conditional rethrow, a return from the catch, a later clause that keeps it, and nested tries that keep it.
    • Wait forms: Task.WaitAll and Task.WaitAny of nested combinators.

    The existing silent controls (_ =, held, returned, awaited forms that observe, a catch that swallows) are unchanged and still silent.

  • IgnoredOutcomeAnalyzerTests holds both issues' snippets, await await Task.WhenAny(...), and a look-alike TypeKitAssertionException of another type.

  • 11 guards seeded, each caught by DroppedTaskVariantTests (number of failing cases):

    Seed Failing cases
    The producer's depth check 192
    WhenAny's depth 168
    An async lambda's return 48
    await unwrapping 48
    A catch's try block only 48
    The exception type 48
    The rethrow check 144
    Its exit points 24
    The exception's metadata name 48
    The base chain 240
    An unreachable handler read as completing (turns the when (false) rows red) 24

    One more seed was not caught: an explicit constant-false-filter check passed the suite with the check removed. It was redundant with the control-flow check, so it is removed, and the when (false) rows stay as what the unreachable-handler seed shows held. Every seed was reverted by restoring the file, checked identical by diff, and touched.

  • Local runs: Analyzers 3689 passed, Conventions 2002 passed, Oracles 51 passed, docs-query check clean.

  • weaving-review (§3, §8):

    • The one new name is the Testing package's own exception, tied to the real assembly by the catch (TypeKitAssertionException) silent rows and the look-alike test, and shown by the metadata-name seed.
    • The neighbours checked for false reports are: Environment.FailFast and Assert.Fail in a handler (end point reachable, so silent), throw new Wrapped(e) (reported, since the task still faults) and an opaque filter (silent).
    • Not reached, all silent rather than false: more than one await at the statement's root (await await Task.WhenAny(Task.WhenAny(...))), a local function between the await and the lambda, Task.Factory.StartNew and ContinueWith, and a type-pattern filter, which counts as catching.

🤖 Generated with Claude Code

CaffeinatedCoder and others added 2 commits October 29, 2026 20:55
…gh catches that cannot keep them

A Task.WhenAny inside a Task.WhenAll or a Task.Run completes with the faulted
assertion as its value, so awaiting the outer task saw no failure and the
rule said nothing (#165). HandsOn now returns how deep the root holds the
failure instead of a bool: 0 where its task faults with it (WhenAll, Run),
one more for each task that completes with it as a value (WhenAny, WhenEach,
an async lambda that returns it), one fewer for each await that unwraps one.
DroppedAssertion treats an awaited, waited-for or converted combinator as
observing only a failure at depth 0, so the same path reports
await Task.WhenAll(Task.WhenAny(...)), await Task.Run(() => Task.WhenAny(...)),
their Wait() and plain-Task forms, at any depth, and keeps
await await Task.WhenAny(...) silent. Task.WaitAll of a WhenAny was already
reported once, at the WhenAny's conversion to a plain Task.

A catch around an awaited assertion in an async lambda counted as observing
the failure whatever it did (#166). EnclosingAsyncFunction now stops only at
a catch that keeps it: the await is in its try block, its type is or is a
base of TypeKitAssertionException (resolved in the Testing assembly by its
metadata name), and its handler can complete (control flow: endpoint
reachable, or a return/break leaves it). A constant false filter leaves the
handler unreachable, so control flow already reads it as never completing:
an explicit filter check, seeded, was caught by nothing, and is not kept.
catch { throw; }, a log and throw;, throw new ..., catch
(OperationCanceledException) and when (false) now report. A filter that is
no constant counts as keeping the failure: an Error reports only a failure
lost whatever the filter answers. Both forms are written only with 2.3.0's
Testing API, so decision 0118 covers them; no 2.2.0 build reports more.

Input class: AIS, an assertion's task lost by a statement.
Variants/DroppedTaskShapes gains 42 forms, all assertion-only: every
combinator inside each other, three and four deep, arrays, Selects and async
lambdas of WhenAny that return it, await it once, twice and three times;
every catch kind (try/finally, rethrow, log and rethrow, throw another,
unrelated type, false filter, nested tries, await in a handler or a finally)
against silent controls (catch all, catch the assertion's exception, true
filter, opaque filter, conditional rethrow, return, a later clause, nested
tries that keep it); and Task.WaitAll/WaitAny of nested combinators. 1837
cases, from 830. IgnoredOutcomeAnalyzerTests holds the issues' snippets and a
look-alike TypeKitAssertionException of another type. Seeded 11 ways, each
caught by DroppedTaskVariantTests (failing cases): the producer's depth (192),
WhenAny's depth (168), an async lambda's return (48), await unwrapping (48),
a catch's try block only (48), the exception type (48), the rethrow check
(144), its exit points (24), the exception's metadata name (48), the base
chain (240), and an unreachable handler read as completing, which turns the
when (false) rows red (24).

README (CMTK0003) says so; claims T03-a-nested-when-any-is-reported,
T03-a-catch-that-cannot-keep-the-failure-is-reported and
T03-an-unknown-filter-counts-as-catching quote it.

Closes #165
Closes #166

Defect-Class: AIS
Found-By: OR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CaffeinatedCoder CaffeinatedCoder added this to the 2.3.0 milestone Oct 29, 2026
@CaffeinatedCoder CaffeinatedCoder added bug Something isn't working silent Compiles or answers and does less than it appears to severity/medium Inconsistency or failure in a narrow scenario package/analyzers Affects CodoMetis.TypeKit.Analyzers package/testing Affects CodoMetis.TypeKit.Testing area/option-result Option, Result and their extensions labels Oct 29, 2026
@CaffeinatedCoder
CaffeinatedCoder merged commit 49be869 into release/2.3.0 Oct 29, 2026
18 checks passed
@CaffeinatedCoder
CaffeinatedCoder deleted the fix/cmtk0003-nested-and-rethrow branch October 29, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/option-result Option, Result and their extensions bug Something isn't working package/analyzers Affects CodoMetis.TypeKit.Analyzers package/testing Affects CodoMetis.TypeKit.Testing severity/medium Inconsistency or failure in a narrow scenario silent Compiles or answers and does less than it appears to

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant