Integrate indexed remote reads and large-file staging

[forhappy]

Repository browsing and large-file publication performed work proportional to unrelated history. Native add and Git-clean also retained separate payload lifetimes. This PR adds authenticated selected reads, persistent Xet catalog updates and one streaming staging owner, while preserving visibility, rollback and retained-payload guarantees.

One PR targeting crab-v2. The two requested large-file audit reports and later user-owned advisor-plan edits are excluded. No merge into main is requested.

Behavior

• Path-state, commit graph, object locator, visibility and Xet catalog queries use authenticated pages. Captured refs authorize visibility; physical index presence alone cannot. Corruption, absence, incomplete authority, raw paths, operation budgets and reader cleanup remain distinct contracts.
• First-parent file history/blame follows relevant change points and reuses immutable comparisons. Multiple-parent blame, whole-file rename following and actual source-line attribution remain unfinished.
• Native add and Git-clean share streaming ingestion and staged payload ownership. Their Git framing/index transactions remain separate. Rollback, durable handoffs, complete historical recipes and flush-before-publication ordering remain required; publication retires only captured owners.
• Checkpoint format 9 uses persistent content-addressed catalog trees, reuses unchanged subtrees and verifies new pages before checkpoint/history/root visibility. File-binding precedence, immutable descriptor conflicts and reverse-placement order remain canonical. CLI/shared/protected publication and hydration use selected dependency closures.
• Capsule chunk hints use an authenticated global COW tree. Hints require current source membership and fresh origin identity. Missing pages can trigger recapture only after the advisory root changes; unchanged-root damage remains an error.
• Historical verification streams the pinned checkpoint catalog and resolves its file bindings in batches. Exact dependency counts and encoded-byte totals come from the completed proof; its authenticated checkpoint inventory is reused without a second physical walk. Empty durable descriptors remain counted. Source authentication, ref visibility, verified Git installation and whole-file/LFS hashing remain prerequisites.
• Selected recovery mapping checks use batches of 64 against one captured lookup, preserving duplicate occurrences, order, genuine misses and dependency errors. Complete post-fetch shard discovery shares that metadata capture and streams shard identities instead of materializing every recipe. Generation-only caching remains disabled for per-ref heads.
• Shard compaction authenticates complete stored shards, including optional bloom bytes, before handing native bodies to Xet’s EOF-based file loader. This fixes compaction of default native-push shards while retaining complete-object identity checks. Plain/bloom and tampered-trailer regression coverage uses the same existing parser as shard synchronization.
• Historical restore unions durable catalog roots through authenticated streaming columns and persistent edits. Newer file bindings retain precedence; file sizes, immutable descriptors and canonical reverse placements remain checked. Both input closures and every batch share encoded/logical admission. Nested recipe digests ignore valid page partitioning and are cached only after complete scans for this operation, so aliases do not repeat full comparisons. Prepared current pages are reused for publication. The final winning union recipes are byte-verified against the already authenticated historical Git refs before any checkpoint/root/ref transition; a corrupt current replacement cannot inherit the old recipe's proof.
• Full catalog inspection uses bounded private spill. GC and readiness share the verified descriptor cursor. Current fsck groups final file bindings by shard, checks every decoded xorb chunk, exact shard/xorb metadata, file sizes and reconstruction terms, then uses the canonical Git/whole-file/LFS byte proof. It retains one shard and bounded binding batches instead of a complete recipe map or a full file scan per shard. Constructor admission and the data phase may each inspect metadata; that cost remains explicit. Normal GC/readiness do not pay for grouping.
• GC follows current, historical and retained catalog closures. Admitted provider deletes drain before fences release. Auth-issued view-local hint cleanup verifies scope, acquires global admission before repository admission, preserves live/exact IDs and immutable-object grace, and fails on corrupt current pages. Shared-global hint maintenance retains bucket ownership.
• Publishers check lease cancellation before visibility and preserve acknowledged commits through late cancellation or cleanup failures. Native push no longer writes the unused advisory push-state hint that could falsely report a committed push as failed.

Ownership and delivery ledger: crab/docs/architecture/indexed-remote-reads.md. Packed-delta decoding and remaining catalog-rewrite callers still materialize logical catalogs; aggregate uncheckpointed-delta intake remains a gate. The durable restore union retains no complete recipe map. Its final byte proof adds administrative work rather than establishing a latency improvement. Published tags lack the capsule codec; retired unshipped formats through 8 are rejected without a fallback reader. Earlier SQLite WAL-safety and Cellule-pin work remains visible in the dependency diff; this restore slice changes no manifest, lockfile, dependency, configuration or remote format.

Validation

Latest frozen source 408e38fe06a8643eaa5eac657b69699a879de150b286e3fbe0c0da6ec88ab275, debug binary SHA256 1f5f212cd9d5ff403ef7d1606a1fd575a339541c38db6577bd43d03d3f74713d: 721 affected tests pass (metadata57, checkpoint catalog11, capture1, read237, shared checkpoint9, GC134, fsck70, mirror75, metadb35, shard sync28, recovery command34, compaction18, historical recovery12). Minimal-feature checks, strict metadata/read library Clippy, normal publication/CLI Clippy, final binary build and format pass. Normal CLI Clippy still reports81 warnings; it is not a strict pass. The related enlarged restore-future/underscore warnings were corrected. No source changes occurred during the passing qualification.

An earlier run timed out on two five-second cache-startup waits (235/237 passed). The isolated three tests, full default-concurrency237-test rerun, five repeated three-test runs, and final237-test run pass without deadline/test changes. The earlier failed receipt is retained; its cause remains unproven.

Fresh RustFS validation on that exact source/binary passes 224 commands /117 checks. Real native/Git uploads and checkpoints,64-entry selected recovery batches with duplicates/mismatches/misses, cold current/historical bytes, missing/corrupt catalog/shard/xorb rejection, deep metadata rebuild, exhaustive replica verification and cleanup all pass. Real shard compaction changes the winning bindings; historical preview preserves visibility, apply restores the original refs, a cold clone reconstructs original bytes, then a fresh Git-clean edit/commit/push in the restored epoch passes full fsck and an independent cold byte oracle. Source/binary identities remain unchanged and logs pass the credential-disclosure check. This16MiB fixture is Level3 component proof, not10GiB/release or hosted-latency qualification.

The first live run stopped at real compaction (201 commands/92 prior checks) because native-push bloom trailers reached Xet's EOF-based loader. The new regression reproduces that error before the fix; all18 compaction cases and the fresh live run pass after using the existing trailer parser following complete-object authentication. The failed receipt is retained.

Prior RustFS component evidence covers selected catalog lifecycle, native/Git staging and exact retirement, current/historical cold reconstruction, protected publication, scoped GC, cache-service behavior, SIGINT and acknowledged-publication fault handling. Deterministic fixed catalog edits at 100/10,000/100,000 entries stay within 256 KiB read/write bounds; fixed hint edits stay within 512 KiB. The distinct-recipe spill fixture reads 15.8 MB and enforces temporary-space admission. These are component/request bounds, not complete opening/history or hosted-latency acceptance.

The recreated format-9 two-file 10 GiB matrix is in progress on its frozen hint/catalog revision and debug binary. Entropy/native passes 75 commands / 57 checks, including all four versions' stage/push/ref/retirement checks, eight cold byte oracles and full remote fsck with zero errors/repairs. Entropy/Git is running; four repeated/zero cases remain planned. The prior cleanup-interrupted runs remain failed/incomplete. This does not qualify the final combined revision, release performance or hosted-S3 latency.

Remaining merge gates and priorities

1. Finish remaining maintenance conversion and bound opening, ref-count and uncompacted-history amplification.
2. Complete combined lifecycle, fresh-install, crash/restart, concurrent publication, retained-reader and final-source release-mode 10 GiB qualification. RustFS is authorized; hosted-S3 latency remains unproven.
3. Implement multiple-parent blame, whole-file rename following, correct source paths/lines and versioned whitespace/ignored-revision semantics, with native line-porcelain oracles.
4. Resolve CI. The tracked architecture checker still fails on exactly two pending inventory corrections: the qualified crab-storage test-support dev edge and rusqlite 0.39/bundled/fallible_uint versus the stale 0.34/bundled expectation. The tracked checker is unchanged; its unapplied two-line proposal passes all architecture checks on this source. Inventory changes still need explicit approval under repository rules. At3b79f76, the CI guardrail job reproduces exactly these two failures; the remaining new-head checks are in progress. At11d088, macOS/Windows protocol, RustFS GC race/crash, Server Cell recovery and other component gates passed; the Compose image exercise stopped after five ECR AWS CLI image pull attempts returned a data-limit/rate-limit error; that run did not complete the recovery gate. The earlier cluster node-loss/recovery failure also requires a successful new-head gate. New-head CI and appropriate full/cross-platform proof remain required. Extended CLI lib/test Clippy has a separately recorded pre-existing failure; it is not reported as passing.

Is this the best fix? Shared ownership, authenticated persistent pages and operation-scoped spill address the measured duplication/scanning without a second integrity policy or authoritative fallback. Historical accounting reuses the authenticated inspection ledger; recovery point checks and complete shard enumeration share the same capture owner. Durable restore union and final winning-recipe verification share these owners; remaining catalog rewriters, packed-delta intake and opening/history amplification still need qualification and conversion. Compaction removes the optional trailer at the native-loader boundary after authentication, using the existing format owner and sibling contract. The union removes map retention while preserving complete inspection and explicit resource limits; it does not claim constant-cost full maintenance. Final merge approval remains deferred until the open gates pass. Unrelated PRs #505 and #522 remain outside this initiative.