A NaN with the sign bit clear sorts above +Infinity in an index on a double #257

IndexKey(double) gives a NaN with the sign bit clear a key above +Infinity, and a NaN with the sign bit set (the constant double.NaN) a key below -Infinity. On Arm64 every NaN the program computes has the sign bit clear, so index-served orderings put those documents last and index-served counts with a lower bound include them. On x64, where computed NaNs carry the sign bit, the same documents sort first: the order of a query depends on the CPU that wrote the value.

Found by the B-tree model test (tests/BLite.Tests/Oracles/BTreeModelTests.cs, #259), which carries the affected rows as a known candidate until this is fixed.

Sibling of #249 (-0.0 and +0.0 get different keys): the same encoding, whose text names NaN as a separate question; this is that question. #252 leaves NaN out of scope. If #254 is fixed by keying dynamic doubles with IndexKey(double), DynamicCollection inherits this too.

Repro

double zero = double.Parse("0");
double computed = zero / zero;             // 0x7FF8000000000000 on Arm64; double.NaN is 0xFFF8000000000000
IndexKey.Create(computed) > IndexKey.Create(double.PositiveInfinity);   // true
IndexKey.Create(computed) == IndexKey.Create(double.NaN);               // false

// DiffDoc has double Ratio with .HasIndex(x => x.Ratio).
await ctx.Docs.InsertBulkAsync([
    new DiffDoc { Name = "d1", Ratio = computed },
    new DiffDoc { Name = "d2", Ratio = 1.0 },
    new DiffDoc { Name = "d3", Ratio = double.NaN }]);
ctx.Docs.AsQueryable().OrderBy(x => x.Ratio).Take(3);              // expected d1, d3, d2; actual d3, d2, d1
ctx.Docs.AsQueryable().OrderByDescending(x => x.Ratio).Take(3);    // expected d2, d1, d3; actual d1, d2, d3
ctx.Docs.AsQueryable().Count(x => x.Ratio > 0.5);                  // expected 1;          actual 2
ctx.Docs.AsQueryable().Where(x => x.Ratio >= double.MaxValue);     // expected [];         actual [d1]

Expected: double.CompareTo, which LINQ-to-Objects and the unindexed collection use, orders every NaN below every number and equal to every other NaN; every comparison operator with a NaN operand is false in C#.

Actual: on Arm64, every computed NaN (0.0 / 0.0, Math.Sqrt(-1), ∞ - ∞ all give 0x7FF8000000000000) sorts above +Infinity, while double.NaN sorts below -Infinity.

Mechanism (verified on 0450b97)

• IndexKey(double) (src/BLite.Core/Indexing/IndexKey.cs:94-105) flips all bits of a value with the sign bit set and only the sign bit otherwise. A NaN's sign bit is arbitrary, so the two NaN halves land below -Infinity (sign set) and above +Infinity (sign clear), and every distinct payload is a distinct key.
• Where(x => x.Ratio > 0.5) itself returns the right answer because it is post-filtered with the BSON predicate; Count is answered from the index (CountRange) and is not.

Affected versions: since the sort-preserving double encoding in f83e0d5 (2026-03-17, first in v3.7.0). Reproduced on v5.2.3 / 0450b97 on Arm64; x64 not run.

Fix considerations

• Canonicalise in IndexKey(double) together with the -0.0 fix of Indexed double equality misses -0.0: IndexKey(double) gives the two zeros different keys #249: if (double.IsNaN(value)) value = double.NaN; before taking the bits gives every NaN the one key below -Infinity, which is where double.CompareTo puts it.
• Range bounds then exclude NaN from > x and >= x. Whether < x should include it (C#: false, CompareTo: NaN is smallest) is the same question the index already answers for -Infinity; the BSON post-filter decides for Where, not for Count.
• Existing indexes with a positive-NaN key need a rebuild.