diff --git a/CHANGELOG.md b/CHANGELOG.md index a5efcf8..a29a56e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ section is the GitHub release's notes. - Generators: `IValueObject`, `IValueObjectMaterializer` or `IPlainValueObject` declared through an interface of yours, such as an `IEntityId : IValueObject`, or by a base record now fails with CMTK1011 naming that interface, where it failed with a Metalama error (LAMA0512) that named nothing you wrote. To constrain generic code, use `IValueObject` itself ([Write your own members](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/blob/main/src/CodoMetis.TypeKit.Generators/README.md#write-your-own-members)). - EF Core: `Min`, `Max`, `Order`, `OrderDescending` or `OrderBy(x => x)` over a value object with its own `CompareTo`, cast to `object` first, as in `Select(o => (object)o.Priority).Max()`, or read back through `??`, a member of a projection, a grouping's key, a join or a subquery's `First()`, as in `Select(o => new { Key = (object)o.Priority }).OrderBy(a => a.Key)`, now throws as the same query without the cast does. EF dropped the cast and answered the column's order, the opposite of memory's for a descending `CompareTo` ([Queries](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/blob/main/src/CodoMetis.TypeKit.EntityFrameworkCore/README.md#queries)). - Analyzers: CMTK0006 now warns when a constructor sets a member only on another object it creates, `new Order(...) { Id = id }` or `other with { Id = id }`, which it took as setting its own. In a build that treats warnings as errors, that can report code that built before ([CMTK0006](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/blob/main/src/CodoMetis.TypeKit.Analyzers/README.md#cmtk0006)). +- EF Core: a query filter that orders values your context holds without a lambda, as in `HasQueryFilter(o => o.Priority == Limits.Max())`, builds again; EF works it out in memory by your `CompareTo`. With a lambda, EF orders those values in the database, and the filter still fails ([Queries](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/blob/main/src/CodoMetis.TypeKit.EntityFrameworkCore/README.md#queries)). ### Fixed diff --git a/design/knowledge/KB-058-ef-orders-captured-values-in-sql-only-with-a-lambda.md b/design/knowledge/KB-058-ef-orders-captured-values-in-sql-only-with-a-lambda.md new file mode 100644 index 0000000..e3ede51 --- /dev/null +++ b/design/knowledge/KB-058-ef-orders-captured-values-in-sql-only-with-a-lambda.md @@ -0,0 +1,23 @@ +--- +verified: 2026-10-31 +pins: + - Microsoft.EntityFrameworkCore.Sqlite 10.0.12 +source: measured with plain EF Core on SQLite (`ToQueryString`), and with a value object whose `CompareTo` is descending in `HandWrittenOrderingTests` +turned-on: the EF satellite's refusal of an ordering in a query filter (`HandWrittenOrdering.OrderingRunsInMemory`, issue 225), which lets an ordering over the context's own values through only when it takes no lambda +test: test/CodoMetis.TypeKit.Facts/KB058.cs +--- + +# KB-058. EF orders captured values in SQL only when the chain takes a lambda + +A LINQ chain over a collection a query captures, `values.Max()`, `values.Min()`, `values.Order().First()` or +`values.First()`, is evaluated in memory: the SQL holds one parameter with the answer. With a lambda, +`values.OrderBy(v => v).First()` or `values.OrderByDescending(v => v).First()`, EF sends the collection's elements and +orders them in SQL: for an `int[]`, one parameter per element in a `VALUES` list (EF 10's default), `ORDER BY "v"."Value" +DESC LIMIT 1`. `values.Max(v => v)` over an `int[]` throws a `NullReferenceException` inside EF. A query filter that +reads the context's own values, `HasQueryFilter(r => r.P == Values.Max())`, is keyed on the same distinction. + +For a value object whose `CompareTo` is its own, the first kind orders by `CompareTo` and the second by the stored +value: over 1, 2, 3 with a descending `CompareTo`, `Values.Max()` is 1 and `Values.OrderByDescending(v => v).First()` +answered 3. `MaxBy(v => v)` over the parameter is not translated at all. One chain with a lambda, +`values.Select(v => (object)v).Max()`, was evaluated in memory too; the satellite still refuses it, since the +distinction it can rely on is the lambda. diff --git a/design/knowledge/README.md b/design/knowledge/README.md index 9177e43..1ca47cc 100644 --- a/design/knowledge/README.md +++ b/design/knowledge/README.md @@ -86,3 +86,4 @@ test and later for the harness's `Baseline.Fact`. | [KB-053](KB-053-greatest-and-least-need-compatibility-level-160.md) | EF translates `Greatest` and `Least` on SQL Server only from compatibility level 160 | EF SqlServer 10.0.12 | `KB053` | | [KB-054](KB-054-datetime2-keeps-every-tick-and-forgets-the-kind.md) | `datetime2` keeps every tick of a `DateTime` and reads it back with `Kind` Unspecified | SQL Server 2022, EF SqlServer 10.0.12 | `KB054`, with Docker | | [KB-057](KB-057-a-nullable-contract-has-an-element-type-and-reads-null-as-null.md) | A `Nullable` contract has `T` as its element type, and a JSON null read into it is null whatever `T`'s converter does | OpenApi 10.0.12, SDK 10.0.4xx | `KB057` | +| [KB-058](KB-058-ef-orders-captured-values-in-sql-only-with-a-lambda.md) | EF orders captured values in SQL only when the chain takes a lambda: `values.Max()` runs in memory, `values.Max(v => v)` in SQL | EF Sqlite 10.0.12 | `KB058` | diff --git a/src/CodoMetis.TypeKit.EntityFrameworkCore/HandWrittenOrderingInterceptor.cs b/src/CodoMetis.TypeKit.EntityFrameworkCore/HandWrittenOrderingInterceptor.cs index 3b922fd..d20033d 100644 --- a/src/CodoMetis.TypeKit.EntityFrameworkCore/HandWrittenOrderingInterceptor.cs +++ b/src/CodoMetis.TypeKit.EntityFrameworkCore/HandWrittenOrderingInterceptor.cs @@ -111,7 +111,7 @@ protected override Expression VisitMethodCall(MethodCallExpression node) foreach (var argument in node.Arguments) Refuse(argument.Type, $"compared with {method.Name}"); } } - else if (IsLinq(method) && OrderedType(method) is { } ordered) + else if (IsLinq(method) && OrderedType(method) is { } ordered && !OrderingRunsInMemory(node)) { Refuse(ordered, $"ordered by {method.Name}"); @@ -384,6 +384,15 @@ private static Expression Unconverted(Expression expression) => /// private bool RunsInMemory(Expression comparison) => skipsWhatRunsInMemory && !DatabaseReads.In(comparison); + /// + /// Whether EF evaluates an ordering before translating, in a filter: it reads nothing of the database and takes + /// no lambda. Over the context's own values EF evaluates Limits.Max() and Limits.Order().First() in + /// memory, by CompareTo, but sends Limits to the database as a parameter and orders it there for + /// Limits.Max(l => l) or Limits.OrderByDescending(l => l).First(), by the column's order + /// (measured with EF Core 10 on SQLite, #225). An ordering with a lambda that EF happens to evaluate stays refused. + /// + private bool OrderingRunsInMemory(Expression ordering) => RunsInMemory(ordering) && !LambdaFinder.In(ordering); + /// The type a LINQ operator orders by: the key, the element for Order, the result for Min and Max. private static Type? OrderedType(MethodInfo method) => method.Name switch { @@ -418,6 +427,29 @@ private void Refuse(Type type, string how) } /// Whether an expression reads a parameter it does not declare itself, or a query. + /// Whether an expression holds a lambda, quoted or not. + private sealed class LambdaFinder : ExpressionVisitor + { + private bool _found; + + public static bool In(Expression expression) + { + var finder = new LambdaFinder(); + finder.Visit(expression); + + return finder._found; + } + + public override Expression? Visit(Expression? node) => _found || node is null ? node : base.Visit(node); + + protected override Expression VisitLambda(Expression node) + { + _found = true; + + return node; + } + } + private sealed class DatabaseReads : ExpressionVisitor { private readonly HashSet _declared = []; diff --git a/src/CodoMetis.TypeKit.EntityFrameworkCore/README.md b/src/CodoMetis.TypeKit.EntityFrameworkCore/README.md index e9d01ac..f33da9e 100644 --- a/src/CodoMetis.TypeKit.EntityFrameworkCore/README.md +++ b/src/CodoMetis.TypeKit.EntityFrameworkCore/README.md @@ -222,6 +222,11 @@ instead, which works everywhere and may add a cast: and stays allowed. A query filter (`HasQueryFilter`, named or not) that does the same fails when the model is built, naming the entity type, since it applies to every query of that type. + One exception: a filter may order values your context holds, without a lambda, as in + `HasQueryFilter(o => o.Priority == Limits.Max())`. EF works that out in memory, by your `CompareTo`. With a + lambda, as in `Limits.OrderByDescending(l => l).First()`, EF orders those values in the database, so that + filter fails too. + ### On SQL Server A value object behaves on SQL Server as the type inside it does there. That isn't always how it diff --git a/test/CodoMetis.TypeKit.EntityFrameworkCore.Tests/HandWrittenOrderingTests.cs b/test/CodoMetis.TypeKit.EntityFrameworkCore.Tests/HandWrittenOrderingTests.cs index 941f1b3..63dabf0 100644 --- a/test/CodoMetis.TypeKit.EntityFrameworkCore.Tests/HandWrittenOrderingTests.cs +++ b/test/CodoMetis.TypeKit.EntityFrameworkCore.Tests/HandWrittenOrderingTests.cs @@ -204,6 +204,10 @@ public void A_comparison_EF_evaluates_in_memory_is_not_refused() [InlineData(typeof(FilteredByGreatestDb), "compared by Greatest", "the query filter of FilteredRank")] [InlineData(typeof(FilteredByBoxedChildRankDb), "ordered by Max", "the query filter of RankParent")] [InlineData(typeof(FilteredByProjectedChildRankDb), "ordered by Max", "the query filter of RankParent")] + [InlineData(typeof(FilteredByTableMaxDb), "ordered by Max", "the query filter of FilteredRank")] + [InlineData(typeof(CapturedOrderByDescendingFirstDb), "ordered by OrderByDescending", "the query filter of FilteredRank")] + [InlineData(typeof(CapturedMaxByDb), "ordered by MaxBy", "the query filter of FilteredRank")] + [InlineData(typeof(CapturedBoxedMaxDb), "ordered by Max", "the query filter of FilteredRank")] public void A_query_filter_that_orders_by_its_own_CompareTo_is_refused(Type context, string how, string filter) { _connection.Open(); @@ -250,6 +254,29 @@ public void A_query_filter_by_Value_or_in_memory_is_not_refused() db.Set().Select(r => r.Rank.Value).Order().ToList().ShouldBe([2, 3]); } + /// + /// An ordering over the context's own values that takes no lambda reads nothing of the database, and EF evaluates + /// it in memory, by CompareTo: Limits.Max() over 1, 2, 3 is 1 for the descending order, so the filter + /// keeps the row ranked 1. Each was refused when the model was built (#225). With a lambda, EF orders the values + /// in the database: OrderByDescending(l => l).First() kept 3 where memory keeps 1, so those stay refused. + /// + [Theory] + [InlineData(typeof(CapturedMaxDb), 1)] + [InlineData(typeof(CapturedMinDb), 3)] + [InlineData(typeof(CapturedOrderFirstDb), 3)] + [InlineData(typeof(CapturedOrderDescendingFirstDb), 1)] + public void A_query_filter_that_orders_the_contexts_own_values_runs_in_memory(Type context, int kept) + { + _connection.Open(); + using var db = (DbContext)Activator.CreateInstance(context, new DbContextOptionsBuilder().UseSqlite(_connection).UseTypeKit().Options)!; + db.Database.EnsureCreated(); + foreach (var value in new[] { 1, 2, 3 }) db.Add(new FilteredRank { Rank = ProbeDescending.From(value) }); + db.SaveChanges(); + db.ChangeTracker.Clear(); + + db.Set().Select(r => r.Rank.Value).ToList().ShouldBe([kept]); + } + public void Dispose() => _connection.Dispose(); private RankedDb Seeded() @@ -362,6 +389,63 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) } } +/// A query filter by the greatest rank of the table itself, which the database orders by the column. +public sealed class FilteredByTableMaxDb(DbContextOptions options) : DbContext(options) +{ + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity().HasQueryFilter(r => r.Rank == Set().Max(x => x.Rank)); +} + +/// Contexts whose query filter orders values the context holds: in memory without a lambda, in the database with one. +public abstract class CapturedLimitsDb(DbContextOptions options) : DbContext(options) +{ + public ProbeDescending[] Limits { get; } = [ProbeDescending.From(1), ProbeDescending.From(2), ProbeDescending.From(3)]; + + public List LimitList { get; } = [ProbeDescending.From(1), ProbeDescending.From(2), ProbeDescending.From(3)]; +} + +public sealed class CapturedMaxDb(DbContextOptions options) : CapturedLimitsDb(options) +{ + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity().HasQueryFilter(r => r.Rank == Limits.Max()); +} + +public sealed class CapturedMinDb(DbContextOptions options) : CapturedLimitsDb(options) +{ + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity().HasQueryFilter(r => r.Rank == LimitList.Min()); +} + +public sealed class CapturedOrderFirstDb(DbContextOptions options) : CapturedLimitsDb(options) +{ + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity().HasQueryFilter(r => r.Rank == Limits.Order().First()); +} + +public sealed class CapturedOrderDescendingFirstDb(DbContextOptions options) : CapturedLimitsDb(options) +{ + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity().HasQueryFilter(r => r.Rank == LimitList.OrderDescending().First()); +} + +public sealed class CapturedOrderByDescendingFirstDb(DbContextOptions options) : CapturedLimitsDb(options) +{ + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity().HasQueryFilter(r => r.Rank == LimitList.OrderByDescending(l => l).First()); +} + +public sealed class CapturedMaxByDb(DbContextOptions options) : CapturedLimitsDb(options) +{ + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity().HasQueryFilter(r => r.Rank == Limits.MaxBy(l => l)); +} + +public sealed class CapturedBoxedMaxDb(DbContextOptions options) : CapturedLimitsDb(options) +{ + protected override void OnModelCreating(ModelBuilder modelBuilder) => + modelBuilder.Entity().HasQueryFilter(r => r.Rank == (ProbeDescending)Limits.Select(l => (object)l).Max()!); +} + /// A projection that holds a rank boxed. public sealed class BoxedRank { diff --git a/test/CodoMetis.TypeKit.Facts/KB058.cs b/test/CodoMetis.TypeKit.Facts/KB058.cs new file mode 100644 index 0000000..21d9208 --- /dev/null +++ b/test/CodoMetis.TypeKit.Facts/KB058.cs @@ -0,0 +1,96 @@ +using System.Linq.Expressions; +using Microsoft.Data.Sqlite; +using Microsoft.EntityFrameworkCore; + +namespace CodoMetis.TypeKit.Facts; + +/// +/// design/knowledge/KB-058: EF Core evaluates a LINQ chain over a collection the query captures in memory when the chain +/// takes no lambda, values.Max(), and sends the result as one parameter; with a lambda, +/// values.OrderByDescending(v => v).First(), it sends the collection's elements and orders them in SQL, and +/// values.Max(v => v) throws inside EF. A query filter over the context's own values does the same. +/// +public sealed class KB058 +{ + public sealed class Row + { + public int Id { get; set; } + + public int P { get; set; } + } + + private sealed class Db(DbContextOptions options) : DbContext(options) + { + public int[] Values { get; } = [1, 2, 3]; + + protected override void OnModelCreating(ModelBuilder modelBuilder) => modelBuilder.Entity(); + } + + private sealed class MaxFilterDb(DbContextOptions options) : DbContext(options) + { + public int[] Values { get; } = [1, 2, 3]; + + protected override void OnModelCreating(ModelBuilder modelBuilder) => modelBuilder.Entity().HasQueryFilter(r => r.P == Values.Max()); + } + + private sealed class KeyedMaxFilterDb(DbContextOptions options) : DbContext(options) + { + public int[] Values { get; } = [1, 2, 3]; + + protected override void OnModelCreating(ModelBuilder modelBuilder) => modelBuilder.Entity().HasQueryFilter(r => r.P == Values.OrderByDescending(v => v).First()); + } + + public static TheoryData Chains => new() + { + { "Max()", false }, + { "Min()", false }, + { "Order().First()", false }, + { "First()", false }, + { "OrderBy(v => v).First()", true }, + { "OrderByDescending(v => v).First()", true }, + }; + + [Theory, MemberData(nameof(Chains))] + public void A_chain_over_captured_values_is_ordered_in_SQL_only_with_a_lambda(string chain, bool inSql) + { + int[] values = [1, 2, 3]; + Expression> filter = chain switch + { + "Max()" => r => r.P == values.Max(), + "Min()" => r => r.P == values.Min(), + "Order().First()" => r => r.P == values.Order().First(), + "First()" => r => r.P == values.First(), + "OrderBy(v => v).First()" => r => r.P == values.OrderBy(v => v).First(), + _ => r => r.P == values.OrderByDescending(v => v).First(), + }; + using var connection = new SqliteConnection("Data Source=:memory:"); + using var db = new Db(new DbContextOptionsBuilder().UseSqlite(connection).Options); + + HoldsTheCollection(db.Set().Where(filter).ToQueryString()).ShouldBe(inSql); + } + + [Fact] + public void A_keyed_Max_over_captured_values_throws_inside_EF() + { + int[] values = [1, 2, 3]; + using var connection = new SqliteConnection("Data Source=:memory:"); + using var db = new Db(new DbContextOptionsBuilder().UseSqlite(connection).Options); + + Should.Throw(() => db.Set().Where(r => r.P == values.Max(v => v)).ToQueryString()); + } + + [Fact] + public void A_query_filter_over_the_contexts_values_is_ordered_in_SQL_only_with_a_lambda() + { + using var connection = new SqliteConnection("Data Source=:memory:"); + + using (var db = new MaxFilterDb(new DbContextOptionsBuilder().UseSqlite(connection).Options)) + HoldsTheCollection(db.Set().ToQueryString()).ShouldBeFalse(); + + using (var db = new KeyedMaxFilterDb(new DbContextOptionsBuilder().UseSqlite(connection).Options)) + HoldsTheCollection(db.Set().ToQueryString()).ShouldBeTrue(); + } + + /// The collection's elements in the SQL: one parameter each in a VALUES list (EF 10's default), or as JSON. + private static bool HoldsTheCollection(string sql) => sql.Contains("VALUES") || sql.Contains("json_each"); +} diff --git a/test/oracles/claims.json b/test/oracles/claims.json index 8cc215c..2a4f013 100644 --- a/test/oracles/claims.json +++ b/test/oracles/claims.json @@ -2016,6 +2016,19 @@ "HandWrittenOrderingTests" ] }, + { + "id": "E05-query-filter-own-values", + "source": "src/CodoMetis.TypeKit.EntityFrameworkCore/README.md#queries", + "quote": "One exception: a filter may order values your context holds, without a lambda, as in `HasQueryFilter(o => o.Priority == Limits.Max())`. EF works that out in memory, by your `CompareTo`. With a lambda, as in `Limits.OrderByDescending(l => l).First()`, EF orders those values in the database, so that filter fails too.", + "claim": "One exception: a filter may order values your context holds, without a lambda, as in `HasQueryFilter(o => o.Priority == Limits.Max())`. EF works that out in memory, by your `CompareTo`. With a lambda, as in `Limits.OrderByDescending(l => l).First()`, EF orders those values in the database, so that filter fails too.", + "oracle": "exception", + "surfaces": [ + "HandWrittenOrderingAttribute.*" + ], + "heldBy": [ + "HandWrittenOrderingTests" + ] + }, { "id": "E05-value-order-allowed", "source": "src/CodoMetis.TypeKit.EntityFrameworkCore/README.md#queries",