Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ section is the GitHub release's notes.
- Generators: `IValueObject<TSelf, T>`, `IValueObjectMaterializer<TSelf, T>` or `IPlainValueObject<TSelf, T>` declared through an interface of yours, such as an `IEntityId<TSelf> : IValueObject<TSelf, Guid>`, or by a base record now fails with CMTK1011 naming that interface, where it failed with a Metalama error (LAMA0512) that named nothing you wrote. To constrain generic code, use `IValueObject<TSelf, T>` itself ([Write your own members](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/blob/main/src/CodoMetis.TypeKit.Generators/README.md#write-your-own-members)).
- EF Core: `Min`, `Max`, `Order`, `OrderDescending` or `OrderBy(x => x)` over a value object with its own `CompareTo`, cast to `object` first, as in `Select(o => (object)o.Priority).Max()`, or read back through `??`, a member of a projection, a grouping's key, a join or a subquery's `First()`, as in `Select(o => new { Key = (object)o.Priority }).OrderBy(a => a.Key)`, now throws as the same query without the cast does. EF dropped the cast and answered the column's order, the opposite of memory's for a descending `CompareTo` ([Queries](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/blob/main/src/CodoMetis.TypeKit.EntityFrameworkCore/README.md#queries)).
- Analyzers: CMTK0006 now warns when a constructor sets a member only on another object it creates, `new Order(...) { Id = id }` or `other with { Id = id }`, which it took as setting its own. In a build that treats warnings as errors, that can report code that built before ([CMTK0006](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/blob/main/src/CodoMetis.TypeKit.Analyzers/README.md#cmtk0006)).
- EF Core: a query filter that orders values your context holds without a lambda, as in `HasQueryFilter(o => o.Priority == Limits.Max())`, builds again; EF works it out in memory by your `CompareTo`. With a lambda, EF orders those values in the database, and the filter still fails ([Queries](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/blob/main/src/CodoMetis.TypeKit.EntityFrameworkCore/README.md#queries)).

### Fixed

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
verified: 2026-10-31
pins:
- Microsoft.EntityFrameworkCore.Sqlite 10.0.12
source: measured with plain EF Core on SQLite (`ToQueryString`), and with a value object whose `CompareTo` is descending in `HandWrittenOrderingTests`
turned-on: the EF satellite's refusal of an ordering in a query filter (`HandWrittenOrdering.OrderingRunsInMemory`, issue 225), which lets an ordering over the context's own values through only when it takes no lambda
test: test/CodoMetis.TypeKit.Facts/KB058.cs
---

# KB-058. EF orders captured values in SQL only when the chain takes a lambda

A LINQ chain over a collection a query captures, `values.Max()`, `values.Min()`, `values.Order().First()` or
`values.First()`, is evaluated in memory: the SQL holds one parameter with the answer. With a lambda,
`values.OrderBy(v => v).First()` or `values.OrderByDescending(v => v).First()`, EF sends the collection's elements and
orders them in SQL: for an `int[]`, one parameter per element in a `VALUES` list (EF 10's default), `ORDER BY "v"."Value"
DESC LIMIT 1`. `values.Max(v => v)` over an `int[]` throws a `NullReferenceException` inside EF. A query filter that
reads the context's own values, `HasQueryFilter(r => r.P == Values.Max())`, is keyed on the same distinction.

For a value object whose `CompareTo` is its own, the first kind orders by `CompareTo` and the second by the stored
value: over 1, 2, 3 with a descending `CompareTo`, `Values.Max()` is 1 and `Values.OrderByDescending(v => v).First()`
answered 3. `MaxBy(v => v)` over the parameter is not translated at all. One chain with a lambda,
`values.Select(v => (object)v).Max()`, was evaluated in memory too; the satellite still refuses it, since the
distinction it can rely on is the lambda.
1 change: 1 addition & 0 deletions design/knowledge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,4 @@ test and later for the harness's `Baseline.Fact`.
| [KB-053](KB-053-greatest-and-least-need-compatibility-level-160.md) | EF translates `Greatest` and `Least` on SQL Server only from compatibility level 160 | EF SqlServer 10.0.12 | `KB053` |
| [KB-054](KB-054-datetime2-keeps-every-tick-and-forgets-the-kind.md) | `datetime2` keeps every tick of a `DateTime` and reads it back with `Kind` Unspecified | SQL Server 2022, EF SqlServer 10.0.12 | `KB054`, with Docker |
| [KB-057](KB-057-a-nullable-contract-has-an-element-type-and-reads-null-as-null.md) | A `Nullable<T>` contract has `T` as its element type, and a JSON null read into it is null whatever `T`'s converter does | OpenApi 10.0.12, SDK 10.0.4xx | `KB057` |
| [KB-058](KB-058-ef-orders-captured-values-in-sql-only-with-a-lambda.md) | EF orders captured values in SQL only when the chain takes a lambda: `values.Max()` runs in memory, `values.Max(v => v)` in SQL | EF Sqlite 10.0.12 | `KB058` |
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ protected override Expression VisitMethodCall(MethodCallExpression node)
foreach (var argument in node.Arguments) Refuse(argument.Type, $"compared with {method.Name}");
}
}
else if (IsLinq(method) && OrderedType(method) is { } ordered)
else if (IsLinq(method) && OrderedType(method) is { } ordered && !OrderingRunsInMemory(node))
{
Refuse(ordered, $"ordered by {method.Name}");

Expand Down Expand Up @@ -384,6 +384,15 @@ private static Expression Unconverted(Expression expression) =>
/// </summary>
private bool RunsInMemory(Expression comparison) => skipsWhatRunsInMemory && !DatabaseReads.In(comparison);

/// <summary>
/// Whether EF evaluates an ordering before translating, in a filter: it reads nothing of the database and takes
/// no lambda. Over the context's own values EF evaluates <c>Limits.Max()</c> and <c>Limits.Order().First()</c> in
/// memory, by <c>CompareTo</c>, but sends <c>Limits</c> to the database as a parameter and orders it there for
/// <c>Limits.Max(l =&gt; l)</c> or <c>Limits.OrderByDescending(l =&gt; l).First()</c>, by the column's order
/// (measured with EF Core 10 on SQLite, #225). An ordering with a lambda that EF happens to evaluate stays refused.
/// </summary>
private bool OrderingRunsInMemory(Expression ordering) => RunsInMemory(ordering) && !LambdaFinder.In(ordering);

/// <summary>The type a LINQ operator orders by: the key, the element for <c>Order</c>, the result for <c>Min</c> and <c>Max</c>.</summary>
private static Type? OrderedType(MethodInfo method) => method.Name switch
{
Expand Down Expand Up @@ -418,6 +427,29 @@ private void Refuse(Type type, string how)
}

/// <summary>Whether an expression reads a parameter it does not declare itself, or a query.</summary>
/// <summary>Whether an expression holds a lambda, quoted or not.</summary>
private sealed class LambdaFinder : ExpressionVisitor
{
private bool _found;

public static bool In(Expression expression)
{
var finder = new LambdaFinder();
finder.Visit(expression);

return finder._found;
}

public override Expression? Visit(Expression? node) => _found || node is null ? node : base.Visit(node);

protected override Expression VisitLambda<T>(Expression<T> node)
{
_found = true;

return node;
}
}

private sealed class DatabaseReads : ExpressionVisitor
{
private readonly HashSet<ParameterExpression> _declared = [];
Expand Down
5 changes: 5 additions & 0 deletions src/CodoMetis.TypeKit.EntityFrameworkCore/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,11 @@ instead, which works everywhere and may add a cast:
and stays allowed. A query filter (`HasQueryFilter`, named or not) that does the same fails when
the model is built, naming the entity type, since it applies to every query of that type.

One exception: a filter may order values your context holds, without a lambda, as in
`HasQueryFilter(o => o.Priority == Limits.Max())`. EF works that out in memory, by your `CompareTo`. With a
lambda, as in `Limits.OrderByDescending(l => l).First()`, EF orders those values in the database, so that
filter fails too.

### On SQL Server

A value object behaves on SQL Server as the type inside it does there. That isn't always how it
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,10 @@ public void A_comparison_EF_evaluates_in_memory_is_not_refused()
[InlineData(typeof(FilteredByGreatestDb), "compared by Greatest", "the query filter of FilteredRank")]
[InlineData(typeof(FilteredByBoxedChildRankDb), "ordered by Max", "the query filter of RankParent")]
[InlineData(typeof(FilteredByProjectedChildRankDb), "ordered by Max", "the query filter of RankParent")]
[InlineData(typeof(FilteredByTableMaxDb), "ordered by Max", "the query filter of FilteredRank")]
[InlineData(typeof(CapturedOrderByDescendingFirstDb), "ordered by OrderByDescending", "the query filter of FilteredRank")]
[InlineData(typeof(CapturedMaxByDb), "ordered by MaxBy", "the query filter of FilteredRank")]
[InlineData(typeof(CapturedBoxedMaxDb), "ordered by Max", "the query filter of FilteredRank")]
public void A_query_filter_that_orders_by_its_own_CompareTo_is_refused(Type context, string how, string filter)
{
_connection.Open();
Expand Down Expand Up @@ -250,6 +254,29 @@ public void A_query_filter_by_Value_or_in_memory_is_not_refused()
db.Set<FilteredRank>().Select(r => r.Rank.Value).Order().ToList().ShouldBe([2, 3]);
}

/// <summary>
/// An ordering over the context's own values that takes no lambda reads nothing of the database, and EF evaluates
/// it in memory, by <c>CompareTo</c>: <c>Limits.Max()</c> over 1, 2, 3 is 1 for the descending order, so the filter
/// keeps the row ranked 1. Each was refused when the model was built (#225). With a lambda, EF orders the values
/// in the database: <c>OrderByDescending(l =&gt; l).First()</c> kept 3 where memory keeps 1, so those stay refused.
/// </summary>
[Theory]
[InlineData(typeof(CapturedMaxDb), 1)]
[InlineData(typeof(CapturedMinDb), 3)]
[InlineData(typeof(CapturedOrderFirstDb), 3)]
[InlineData(typeof(CapturedOrderDescendingFirstDb), 1)]
public void A_query_filter_that_orders_the_contexts_own_values_runs_in_memory(Type context, int kept)
{
_connection.Open();
using var db = (DbContext)Activator.CreateInstance(context, new DbContextOptionsBuilder().UseSqlite(_connection).UseTypeKit().Options)!;
db.Database.EnsureCreated();
foreach (var value in new[] { 1, 2, 3 }) db.Add(new FilteredRank { Rank = ProbeDescending.From(value) });
db.SaveChanges();
db.ChangeTracker.Clear();

db.Set<FilteredRank>().Select(r => r.Rank.Value).ToList().ShouldBe([kept]);
}

public void Dispose() => _connection.Dispose();

private RankedDb Seeded()
Expand Down Expand Up @@ -362,6 +389,63 @@ protected override void OnModelCreating(ModelBuilder modelBuilder)
}
}

/// <summary>A query filter by the greatest rank of the table itself, which the database orders by the column.</summary>
public sealed class FilteredByTableMaxDb(DbContextOptions options) : DbContext(options)
{
protected override void OnModelCreating(ModelBuilder modelBuilder) =>
modelBuilder.Entity<FilteredRank>().HasQueryFilter(r => r.Rank == Set<FilteredRank>().Max(x => x.Rank));
}

/// <summary>Contexts whose query filter orders values the context holds: in memory without a lambda, in the database with one.</summary>
public abstract class CapturedLimitsDb(DbContextOptions options) : DbContext(options)
{
public ProbeDescending[] Limits { get; } = [ProbeDescending.From(1), ProbeDescending.From(2), ProbeDescending.From(3)];

public List<ProbeDescending> LimitList { get; } = [ProbeDescending.From(1), ProbeDescending.From(2), ProbeDescending.From(3)];
}

public sealed class CapturedMaxDb(DbContextOptions options) : CapturedLimitsDb(options)
{
protected override void OnModelCreating(ModelBuilder modelBuilder) =>
modelBuilder.Entity<FilteredRank>().HasQueryFilter(r => r.Rank == Limits.Max());
}

public sealed class CapturedMinDb(DbContextOptions options) : CapturedLimitsDb(options)
{
protected override void OnModelCreating(ModelBuilder modelBuilder) =>
modelBuilder.Entity<FilteredRank>().HasQueryFilter(r => r.Rank == LimitList.Min());
}

public sealed class CapturedOrderFirstDb(DbContextOptions options) : CapturedLimitsDb(options)
{
protected override void OnModelCreating(ModelBuilder modelBuilder) =>
modelBuilder.Entity<FilteredRank>().HasQueryFilter(r => r.Rank == Limits.Order().First());
}

public sealed class CapturedOrderDescendingFirstDb(DbContextOptions options) : CapturedLimitsDb(options)
{
protected override void OnModelCreating(ModelBuilder modelBuilder) =>
modelBuilder.Entity<FilteredRank>().HasQueryFilter(r => r.Rank == LimitList.OrderDescending().First());
}

public sealed class CapturedOrderByDescendingFirstDb(DbContextOptions options) : CapturedLimitsDb(options)
{
protected override void OnModelCreating(ModelBuilder modelBuilder) =>
modelBuilder.Entity<FilteredRank>().HasQueryFilter(r => r.Rank == LimitList.OrderByDescending(l => l).First());
}

public sealed class CapturedMaxByDb(DbContextOptions options) : CapturedLimitsDb(options)
{
protected override void OnModelCreating(ModelBuilder modelBuilder) =>
modelBuilder.Entity<FilteredRank>().HasQueryFilter(r => r.Rank == Limits.MaxBy(l => l));
}

public sealed class CapturedBoxedMaxDb(DbContextOptions options) : CapturedLimitsDb(options)
{
protected override void OnModelCreating(ModelBuilder modelBuilder) =>
modelBuilder.Entity<FilteredRank>().HasQueryFilter(r => r.Rank == (ProbeDescending)Limits.Select(l => (object)l).Max()!);
}

/// <summary>A projection that holds a rank boxed.</summary>
public sealed class BoxedRank
{
Expand Down
96 changes: 96 additions & 0 deletions test/CodoMetis.TypeKit.Facts/KB058.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
using System.Linq.Expressions;
using Microsoft.Data.Sqlite;
using Microsoft.EntityFrameworkCore;

namespace CodoMetis.TypeKit.Facts;

/// <summary>
/// design/knowledge/KB-058: EF Core evaluates a LINQ chain over a collection the query captures in memory when the chain
/// takes no lambda, <c>values.Max()</c>, and sends the result as one parameter; with a lambda,
/// <c>values.OrderByDescending(v =&gt; v).First()</c>, it sends the collection's elements and orders them in SQL, and
/// <c>values.Max(v =&gt; v)</c> throws inside EF. A query filter over the context's own values does the same.
/// </summary>
public sealed class KB058
{
public sealed class Row
{
public int Id { get; set; }

public int P { get; set; }
}

private sealed class Db(DbContextOptions options) : DbContext(options)
{
public int[] Values { get; } = [1, 2, 3];

protected override void OnModelCreating(ModelBuilder modelBuilder) => modelBuilder.Entity<Row>();
}

private sealed class MaxFilterDb(DbContextOptions options) : DbContext(options)
{
public int[] Values { get; } = [1, 2, 3];

protected override void OnModelCreating(ModelBuilder modelBuilder) => modelBuilder.Entity<Row>().HasQueryFilter(r => r.P == Values.Max());
}

private sealed class KeyedMaxFilterDb(DbContextOptions options) : DbContext(options)
{
public int[] Values { get; } = [1, 2, 3];

protected override void OnModelCreating(ModelBuilder modelBuilder) => modelBuilder.Entity<Row>().HasQueryFilter(r => r.P == Values.OrderByDescending(v => v).First());
}

public static TheoryData<string, bool> Chains => new()
{
{ "Max()", false },
{ "Min()", false },
{ "Order().First()", false },
{ "First()", false },
{ "OrderBy(v => v).First()", true },
{ "OrderByDescending(v => v).First()", true },
};

[Theory, MemberData(nameof(Chains))]
public void A_chain_over_captured_values_is_ordered_in_SQL_only_with_a_lambda(string chain, bool inSql)
{
int[] values = [1, 2, 3];
Expression<Func<Row, bool>> filter = chain switch
{
"Max()" => r => r.P == values.Max(),
"Min()" => r => r.P == values.Min(),
"Order().First()" => r => r.P == values.Order().First(),
"First()" => r => r.P == values.First(),
"OrderBy(v => v).First()" => r => r.P == values.OrderBy(v => v).First(),
_ => r => r.P == values.OrderByDescending(v => v).First(),
};
using var connection = new SqliteConnection("Data Source=:memory:");
using var db = new Db(new DbContextOptionsBuilder().UseSqlite(connection).Options);

HoldsTheCollection(db.Set<Row>().Where(filter).ToQueryString()).ShouldBe(inSql);
}

[Fact]
public void A_keyed_Max_over_captured_values_throws_inside_EF()
{
int[] values = [1, 2, 3];
using var connection = new SqliteConnection("Data Source=:memory:");
using var db = new Db(new DbContextOptionsBuilder().UseSqlite(connection).Options);

Should.Throw<NullReferenceException>(() => db.Set<Row>().Where(r => r.P == values.Max(v => v)).ToQueryString());
}

[Fact]
public void A_query_filter_over_the_contexts_values_is_ordered_in_SQL_only_with_a_lambda()
{
using var connection = new SqliteConnection("Data Source=:memory:");

using (var db = new MaxFilterDb(new DbContextOptionsBuilder().UseSqlite(connection).Options))
HoldsTheCollection(db.Set<Row>().ToQueryString()).ShouldBeFalse();

using (var db = new KeyedMaxFilterDb(new DbContextOptionsBuilder().UseSqlite(connection).Options))
HoldsTheCollection(db.Set<Row>().ToQueryString()).ShouldBeTrue();
}

/// <summary>The collection's elements in the SQL: one parameter each in a <c>VALUES</c> list (EF 10's default), or as JSON.</summary>
private static bool HoldsTheCollection(string sql) => sql.Contains("VALUES") || sql.Contains("json_each");
}
13 changes: 13 additions & 0 deletions test/oracles/claims.json
Original file line number Diff line number Diff line change
Expand Up @@ -2016,6 +2016,19 @@
"HandWrittenOrderingTests"
]
},
{
"id": "E05-query-filter-own-values",
"source": "src/CodoMetis.TypeKit.EntityFrameworkCore/README.md#queries",
"quote": "One exception: a filter may order values your context holds, without a lambda, as in `HasQueryFilter(o => o.Priority == Limits.Max())`. EF works that out in memory, by your `CompareTo`. With a lambda, as in `Limits.OrderByDescending(l => l).First()`, EF orders those values in the database, so that filter fails too.",
"claim": "One exception: a filter may order values your context holds, without a lambda, as in `HasQueryFilter(o => o.Priority == Limits.Max())`. EF works that out in memory, by your `CompareTo`. With a lambda, as in `Limits.OrderByDescending(l => l).First()`, EF orders those values in the database, so that filter fails too.",
"oracle": "exception",
"surfaces": [
"HandWrittenOrderingAttribute.*"
],
"heldBy": [
"HandWrittenOrderingTests"
]
},
{
"id": "E05-value-order-allowed",
"source": "src/CodoMetis.TypeKit.EntityFrameworkCore/README.md#queries",
Expand Down
Loading