Skip to content

Fix metadata collection and graph detail controls - #212

Open
YanivZalach wants to merge 4 commits into
masterfrom
fix/metadata-collection-and-graph-details
Open

YanivZalach wants to merge 4 commits into
masterfrom
fix/metadata-collection-and-graph-details

Conversation

@YanivZalach

@YanivZalach YanivZalach commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Summary

Historical metadata files now use explicit Spark schemas instead of JSON schema inference, reducing per-file schema-analysis requests while preserving v1/v2/v3 metadata values. Per-file analysis failures retain error records, and branch child ordering is deterministic. Metadata reads explicitly disable Spark's missing/corrupt-file skipping, so every metadata file yields a record or an error.

Restore FileTree's v0.20 behavior for bounded graph ranges by removing the comparison with full snapshot totals. Add expand/collapse controls to long manifest partition lists, reset their state when selecting another manifest, and offer each Graph navigation destination once when several links point to it. Update the FileTree Docs.

Validation

  • Backend: uv run ruff format . — 47 files unchanged.
  • Frontend, pnpm 11.24.0: format, lint, typecheck, and VITE_OUT_DIR=dist pnpm run build passed. Lint reports eight pre-existing warnings and no errors.
  • Full real-backend browser smoke test covered catalog selection, full and bounded ranges, exact snapshot IDs, Timeline, Metadata, FileTree, Graph, keyboard controls, deep links, URL cleanup, reload/history, cache restoration, Recompile, and Docs.
  • Metadata, FileTree, and Graph checks passed for format versions 1, 2, and 3.
  • Focused browser regression verified unique parent/child navigation choices with no console errors after the fix.
  • Isolated browser fixtures verified 25-partition expand/collapse, reset between manifests, field details, independent metadata/snapshot loading, and empty-table graph generation.
  • Read-only Spark comparisons confirmed the new historical metadata reader matches master for the latest metadata record of each v1/v2/v3 test table.
  • With explicit ignoreMissingFiles/ignoreCorruptFiles set to false, full metadata collection for v1/v2/v3 test tables matched master exactly and ran about 3x faster; a missing metadata path still fails only its own record.
  • Independent reviews found no remaining findings. git diff --check passed; temporary tests were removed.

Contributor License Agreement

The repository-owner exemption applies to this pull request.

Parse historical metadata with explicit Spark schemas and preserve error
records for failed or skipped reads. Keep branch child ordering stable.

Restore bounded FileTree warning behavior, add manifest partition
expand/collapse controls, and deduplicate Graph navigation destinations.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant