Skip to content

Reduce initial full-index work on large codebases - #5390

Merged
Widthdom merged 10 commits into
mainfrom
perf/initial-full-index-throughput-20260921
Sep 21, 2026
Merged

Widthdom merged 10 commits into
mainfrom
perf/initial-full-index-throughput-20260921

Conversation

@Widthdom

Copy link
Copy Markdown
Owner

Summary

Initial full indexing spends substantial work repeating declaration probes, receiver-scope analysis and reference-source SQL for large repositories. This PR reduces that work across 10 commits:

  • Specialize fresh reference-source lookups when canonical names are sufficient, preserving display/legacy fallbacks. Bind fresh reference flags as SQL zero literals (12 parameters per row instead of 14; native batches increase from 36 to 42 rows) and skip ineligible known-zero mutual-recursion rows.
  • Prune impossible C# declaration, capture and enum probes; defer cumulative lookahead copies, scan appended semicolon state once, and cache only requested callable receiver scopes. C#, Razor, Blazor and CSHTML share these improvements.
  • Skip ordinary code spans during C#, Kotlin and Scala structural masking while preserving nested strings, interpolation, comments and source coordinates.

Names, extracted records, resolution rules, database layout, transaction boundaries and rollback behavior remain unchanged.

Validation

  • dotnet test CodeIndex.sln -c Release -p:UseSharedCompilation=false: .NET 8 12,733 passed / 9 skipped; .NET 9 12,239 passed / 435 skipped. Zero failures, errors, timeouts or aborted tests.
  • Full symbol/reference extractor suites: .NET 8 2,599 passed; .NET 9 2,570 passed / 29 skipped. Deterministic work/allocation budgets and complete-record comparisons cover the shared language paths.
  • Both final empty-database runs over the fixed 1,577-file snapshot match the saved pre-change output in logical files, chunks, 59,387 symbols, 562,615 references, reference lines, issues, candidates (including complete per-reference candidate sets), hotspot records, schema and stable metadata after normalizing generated IDs/indexing times. SQLite integrity checks pass. FTS posting payloads were not compared.
  • Changed-file formatting, changelog validation and diff checks pass. The canonical root index and both workspace members are healthy at the final HEAD.
  • Independent adversarial review of origin/main..HEAD: No blocking/actionable issues found.

Measurements

On the same C#-heavy source snapshot, the first two batches each had their own alternating Release .NET 8 before/after measurements: mean 43.6 → 37.7 seconds (~13.6%) and 37.40 → 34.51 seconds (~7.7%). These are separate comparisons, not a combined final-branch speedup.

The final batch has standalone Debug .NET 8 observations of 39.014 / 40.010 seconds, with no matched Release before/after timing; no percentage improvement is claimed for it. Details and allocation observations are in docs/initial-index-performance.md.

Documentation and changelog

Updated TESTING_GUIDE.md and the bilingual performance documentation. Added these bilingual fragments:

  • changelog.d/unreleased/+fresh-canonical-source-probes.changed.md
  • changelog.d/unreleased/+csharp-receiver-scope-reuse.changed.md
  • changelog.d/unreleased/+csharp-declaration-prefix-pruning.changed.md
  • changelog.d/unreleased/+fresh-reference-flag-work.changed.md
  • changelog.d/unreleased/+csharp-reference-probe-pruning.changed.md
  • changelog.d/unreleased/+csharp-header-probe-pruning.changed.md
  • changelog.d/unreleased/+csharp-lookahead-copy-reuse.changed.md
  • changelog.d/unreleased/+csharp-callable-receiver-cache.changed.md
  • changelog.d/unreleased/+structural-mask-code-spans.changed.md

Follow-up candidates

Collect matched Release timings for the final batch and extend measurements to large corpora with a broader language mix.

@Widthdom
Widthdom merged commit dc1792a into main Sep 21, 2026
11 checks passed
@Widthdom
Widthdom deleted the perf/initial-full-index-throughput-20260921 branch September 21, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant