Repository navigation
feat(fs): select a mount write-durability tier and drain before unmount - #97
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
The change adds a well-scoped durability flag with clear normalization/validation and driver-specific constraints, and the update is consistently applied across CLI wiring, mount result reporting, documentation/spec, and unit plus e2e/live test coverage without any noted outstanding issues.
No open issues 🎉
What changed in this PR
Expose Drive9 write-durability tiers in ti fs mount-file-system via a new --durability flag (keeping --write-back-cache as a legacy alias), and report the effective tier in mount results and dry-run output.
Changes:
- Add
--durability auto|interactive|fsync|close-sync|write-sync(defaultinteractive) with normalization, validation, and conflict detection vs--write-back-cache. - Forward durability to the companion for effective FUSE mounts; enforce WebDAV and checkpoint constraints (WebDAV reports/uses
auto, checkpoint read-only forcesauto). - Replace the mount result’s legacy boolean with an explicit
durabilityfield and update docs/spec plus unit/e2e/live tests accordingly.
| File | Description |
|---|---|
| README.md | Updates user-facing documentation and examples to describe --durability, the legacy alias behavior, and the durability field in mount results. |
| internal/fs/vault.go | Adjusts vault mount result construction to use and report the new durability field (defaulting to auto). |
| internal/fs/mount.go | Plumbs durability through mount option parsing/normalization, enforces driver-specific rules, and reports effective durability in results and dry-run output. |
| internal/fs/mount_durability.go | Implements the durability tier model, validation/normalization, legacy alias reconciliation, and forwarding/rejection rules for FUSE/WebDAV/checkpoint mounts. |
| internal/fs/mount_durability_test.go | Adds unit tests covering tier parsing, alias conflicts, forwarding decisions, WebDAV/checkpoint constraints, and dry-run alignment. |
| internal/fs/drive9_mount_audit.go | Updates companion argv/audit logic to conditionally forward --durability and to return the effective tier in structured results. |
| internal/fs/drive9_companion.go | Updates companion interaction/result shaping to align with the new durability reporting expectations. |
| internal/cli/commands.go | Registers and wires the new --durability flag (and legacy alias reconciliation) into the CLI mount command. |
| e2e/live_test.go | Updates live E2E assertions to validate reported durability for FUSE mounts and auto behavior for WebDAV mounts. |
| e2e/cli_test.go | Extends CLI E2E coverage to assert forwarding and validation behavior across default/explicit tiers and WebDAV restrictions. |
| docs/spec/done/0038-fs-mount-durability.md | Adds the completed spec documenting the durability contract, validation rules, driver behavior, and verification steps. |
| AGENTS.md | Updates repository guidance and examples to reflect the durability-tier semantics and forwarding rules. |
Review effort: Balanced
Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.
126f0ec to
a61626f
Compare
a61626f to
39f4ce4
Compare
`ti fs mount-file-system` only exposed the boolean `--write-back-cache`, and only an explicit `--driver fuse` mount forwarded anything: true pinned the companion to `interactive` and false forwarded nothing, so the engine fell back to `auto`. The companion has long supported five tiers (`mount --durability auto|interactive|fsync|close-sync|write-sync`), so a caller that needed close-sync or write-sync had to mount around ti and drain by hand, leaving the client-side loss window unbounded. Add `--durability` and forward the tier to the companion for FUSE mounts. The default stays `interactive`, so an ordinary mount keeps its documented behavior instead of following the engine's `auto` default, including on `--driver auto` mounts that resolve to FUSE. - Keep `--write-back-cache` as a legacy alias (true = interactive, false = auto) and refuse an inconsistent explicit combination. - WebDAV mounts have no tiers: forward none, report `auto`, and reject every explicit tier except `auto` instead of silently downgrading. - Checkpoint mounts are read-only: force `auto` and reject any other explicit tier. - Replace the `write_back_cache` result boolean with the effective `durability` tier, because a boolean cannot express five tiers and a derived boolean invites disagreeing with the mount. - Report the same tier in `--dry-run`. Spec: docs/spec/done/0038-fs-mount-durability.md
Write-back uploads happen after flush, so detaching a mount is the last moment a client-side loss can still be prevented. Three holes made that loss invisible: `ti fs drain-file-system` discarded the companion's `mount drain --json` report, `ti fs unmount-file-system` never drained, and `--force` was parsed but never read. - Parse the drain report into the result `response`: `ti fs drain` now returns `ok`, `error_kind`, `failed_path`, the duration and the pending counters, and prints those counters in text output. - A drain that leaves write-side work fails with `fs.drain_pending_work` and names the counters plus the retry command. - Unmount drains first, blocks with the mount locator preserved while work is pending, and gates only ti fs FUSE mounts: WebDAV has no control socket, a `vault` locator is not a file-system mount (the engine drains only those), and a locator written before the driver was recorded is not gated so an upgrade cannot block an existing mount. The mount now records the driver the companion actually used in the non-secret locator. - `--force` becomes the explicit "accept the loss" escape hatch and is reported as `drain.status: "skipped"`; `--ignore-absent` may continue past a failed drain but still reports it. - `--dry-run` reports the gate without running the drain. Spec: docs/spec/done/0039-fs-unmount-drain-gate.md
39f4ce4 to
038ed45
Compare
mornyx
left a comment
There was a problem hiding this comment.
Review summary
This PR forwards an existing engine write-durability tier instead of inventing a new one and parses the engine's own drain report instead of reimplementing drain — the architecture is right, the specs are decision-complete, and the fail-closed bias (refuse tiers the driver cannot honor, treat an unparseable drain report as a failure, preserve the locator on a blocked unmount) is consistently applied. However, two merge-blocking problems were verified against the code (one by running the test suite on macOS): --ignore-absent unconditionally bypasses the pending-work drain gate, recreating the exact silent write-loss failure this PR exists to close, and the new e2e assertions hard-code the Linux outcome of --driver auto so make e2e now fails on macOS.
What looks good
- The gate decision (
mountDrainGate) is a pure function shared by unmount and both dry-runs, and the new logic lands in two focused files (mount_durability.go,drive9_mount_drain.go) instead of growing the shared companion path — easy to reason about and well exploited by the tests. - Consistently fail-closed choices: explicit strict tiers on WebDAV are usage errors rather than silent downgrades, a missing/undecodable drain report fails, a blocked unmount preserves the locator for retry, and locators written before the driver was recorded are never gated, so an upgrade cannot strand existing mounts.
- Honest migration handling: the legacy
--write-back-cacheboolean becomes a strict alias with conflict detection, and the engine's drain counters are surfaced (not just exit status), so a lost write is attributable instead of silent.
Blocking (please address before merge)
1. --ignore-absent bypasses the pending-work drain gate
internal/fs/drive9_companion.go:1389 — if drainErr != nil && !opts.IgnoreAbsent continues past any drain failure, including fs.drain_pending_work, then unmounts, removes the locator, and exits 0 (the only trace is drain.status: "failed" plus a checks[] entry). The README added in this PR scopes the flag to "a drain that fails because the mount is already gone", and spec 0039 says the same — but the code cannot distinguish the two cases, and --ignore-absent is the natural idempotency flag for cleanup scripts, so any script passing it routinely has the entire safety gate disabled exactly when writes are pending. This is precisely the "unmount returned success while the upload never happened" failure the PR claims to fix (TestUnmountIgnoreAbsentReportsDrainFailureStill pins the lossy behavior via the pending-work fixture).
Fix: the taxonomy already supports a local fix — fs.drain_pending_work can only come from a live engine that answered with pending counters, while a gone mount surfaces fs.drain_failed — so block on fs.drain_pending_work even under --ignore-absent (loss acceptance belongs to --force alone), keep tolerating fs.drain_failed, and mention --ignore-absent in the blocked-drain error text so a genuinely dead mount has an accurate remedy.
2. The new e2e assertions break make e2e on macOS
e2e/cli_test.go:1230 — the default mount (no --driver, i.e. auto) asserts "durability": "interactive", but on darwin effectiveMountMode("auto") resolves to WebDAV, so ti forces durability: "auto" and forwards nothing. Reproduced on this machine with a make build-equivalent binary: the test fails exactly there (stdout should contain "\"durability\": \"interactive\"", actual "durability": "auto"). The strictMount case is worse on darwin: --durability close-sync without --driver fuse exits 2 with fs.webdav_durability_unsupported, so wantExitCode(0) fails too, and the follow-on unmount-drained assertions fail for the same reason. CI stays green only because it is ubuntu-only, but macOS is a first-class platform for this repo and make e2e is the mandatory gate that was platform-neutral before this PR.
Fix: make the expectations deterministic — pin --driver fuse on the mounts that must forward/report a tier (the fake companion never mounts real FUSE, so this costs no coverage), and either assert the default mount's platform-appropriate value (branch on runtime.GOOS like the existing unit tests do) or drop the default-mount durability assertion.
Should fix (non-blocking, strong)
3. Two divergent "auto driver" resolutions now live next to each other
internal/fs/mount_durability.go:124 — the new GOOS-only effectiveMountMode is a second resolution of --driver auto in the same package that already owns a prerequisite-aware one: mountdriver.Resolve (used meters away in normalizeLayerMountOptions and the dry-run mount_driver check) prefers FUSE whenever it is installable, including darwin with macFUSE. On such hosts the two rules disagree (fuse vs webdav), so the prerequisite check validates a mount as FUSE while the durability gate treats it as WebDAV — rejecting a valid --durability close-sync and skipping the documented interactive default for a mount ti itself calls FUSE. Related fallout: the mount result sets Durability: opts.Durability regardless of the driver parsed from companion stderr, so whenever ti's mirror and the companion disagree the result claims a tier that is not in effect. Consolidate on one canonical auto resolution (delegate to mountdriver, or reconcile the reported durability with the companion-reported driver).
4. The published English docs were not updated for these user-visible changes
The --durability flag, the legacy-alias semantics, the removed write_back_cache result field, the drain report, and the new --force/--ignore-absent behavior all change published CLI surface, and AGENTS.md requires the matching English pages under docs/pingcap-docs/docs/ to be updated in the same change. The gap is understandable (the submodule is not checked out in this tree), but the published mount page now contradicts the binary. Land the doc updates before the next release cut, or open a tracked follow-up referenced from both specs so it cannot be dropped silently.
Nits
- Dry-run/execution mismatch in drain support:
DryRunDrainFileSystemreports "not applicable" for WebDAV/vault mounts, but executingti fs drain-file-systemstill runsmount drainunconditionally and fails with a raw companion error;drainBeforeUnmountalso inlines thetimeout <= 0default that the existingdrainTimeouthelper owns, and the pending counters are rendered twice with different formats (drainPendingSummaryvs the inlineDrainResult.Humanlist, which drops the byte counters). - An explicitly empty
--durability ""is silently treated as the interactive default while still marked explicit, so--durability "" --write-back-cache=falsefails as a "conflict" — the repo's own convention for--regionis to reject an explicit empty value. drive9MountLocatorProfileis now a wrapper with zero callers after this PR converted every caller tomountLocatorProfile; delete it.Locator.WithDrivertook overWithServiceAddress's doc comment (the stacked comment now mis-attributes the first line), leavingWithServiceAddressundocumented.MountResult.Human()does not print the effective durability tier, so a--output textcaller cannot verify what the mount got — the very thing the new field exists to report — while the drain and unmount text renderers did gain their new fields.
Recommendation
Request changes. The --ignore-absent loss-acceptance leak and the macOS-broken e2e gate are verified merge-stoppers; everything else can land as follow-ups.
…-neutral Addresses the review on #97. - `--ignore-absent` is the cleanup idempotency flag, so it must not accept undrained writes: `fs.drain_pending_work` now blocks the unmount even with it, the blocked-drain error says so, and only a drain that cannot reach the mount may continue (still reported as `drain.status: "failed"`). `--force` remains the only loss acceptance. - `ti fs drain-file-system` refuses mounts the engine cannot drain (WebDAV and any non-`fs` locator kind) with `fs.drain_not_applicable`, matching what the dry-run already reported instead of forwarding a request that fails raw. - The e2e mount expectations pin `--driver fuse` wherever a forwarded tier is asserted and branch the automatic-mode case on the host platform, so `make e2e` is green on macOS/Windows too. Both fake companions now mirror drive9's `--mode auto` resolution so the fixtures cannot disagree with the companion. - Consolidate the automatic-mode rule in `companionMountMode` with the reason it cannot use the prerequisite-aware `mountdriver.Resolve`, and reconcile the reported `durability` with the driver the companion reported. - Nits: reject an explicitly empty `--durability`, share one pending-counter formatter between the drain and unmount renderers, print the effective tier in the mount text output, use `drainTimeout`, drop the unused `drive9MountLocatorProfile` wrapper, and restore the locator comments. The published English pages are tracked in issue #98, referenced from both specs, because the documentation submodule is not checked out here.
|
Thanks for the detailed review — both blockers were real and are fixed in Blocking1. 2. macOS e2e breakage — fixed. Should fix3. Two divergent auto resolutions — consolidated where it can be. 4. Published English docs — tracked instead of silently dropped. Nits
Verification
I have not merged anything and will not — merging is yours. Re-requesting your review. |
mornyx
left a comment
There was a problem hiding this comment.
Review summary
Thanks for the thorough point-by-point reply — nearly all of it checks out on 41dc94f. The macOS e2e breakage is genuinely fixed (the full e2e suite passes on a macOS host against a freshly built binary), the auto-driver rule now lives in exactly one place with an accurate rationale, the reported tier is reconciled with the driver the companion actually mounted, the docs debt is tracked as #98 with a release-cut gate, and every round-1 nit is resolved. One narrow hole remains in the fixed blocker: under --ignore-absent, the unmount still keys its go/no-go decision on the engine's error_kind string rather than the pending counters ti has already parsed, so a drain that fails with any other classification while writes are queued still detaches with exit 0. That is the last merge-blocker; the remaining items are hardening and polish.
What looks good
- The round-2 fixes are precise and each carries a dedicated pinning test (pending work blocks under
--ignore-absent, gone-mount tolerance, vault/WebDAV drain not-applicable, driver recorded at mount), and platform-neutrality was achieved by pinning--driver fusewhere a tier is asserted rather than by weakening the assertions. companionMountModeis a genuine consolidation: one mirror of the companion's GOOS-only auto rule with a written reason for not reusing the prerequisite-awaremountdriver.Resolve, pinned by a test, with both fake companions mirroring the same rule so fixtures cannot drift.- The gate keeps its ownership boundary clean: the companion owns the drain and its report, ti owns the decision, every escape is recorded in the result, and the locator is preserved on block — losses stay attributable instead of silent.
Resolved since last review
- macOS e2e breakage (blocking): fixed — tier mounts pin
--driver fuse, the auto case branches onruntime.GOOS, fakes mirror the companion's mode rule; full e2e re-run green on macOS. --ignore-absentpending-work bypass (blocking): fixed for thepending_work_remainingclassification — it now blocks regardless of the flag, the error names both flags, and both directions are pinned by tests and e2e. See Blocking 1 below for the remaining hole.- Divergent auto-driver resolution (should fix): fixed via the single
companionMountModeand driver-reconciled durability reporting. - Missing published docs (should fix): tracked as #98, referenced from both specs as a release-cut blocker — acceptable as a follow-up.
- All seven round-1 nits: fixed (
fs.drain_not_applicablegate,drainTimeoutreuse, one counter formatter with byte counters, empty--durabilityrejected, wrapper deleted, doc comments restored,Human()durability line).
Blocking (please address before merge)
1. --ignore-absent tolerance trusts the engine's error classification, not the pending counters
internal/fs/drive9_companion.go:1397 — the unmount blocks only when apperr.CodeFor(drainErr) == "fs.drain_pending_work", and that code is derived solely from response.ErrorKind == "pending_work_remaining" (internal/fs/drive9_mount_drain.go:87). Under --ignore-absent, a drain that fails with any other classification while the parsed report shows queued work — the realistic case being a drain that hits its 30s timeout against a write-back backlog — proceeds to umount and exits 0, printing the very pending counters it is discarding. Verified against the compiled CLI with a companion reporting ok:false, error_kind:"deadline_exceeded" with commit_queue_pending:3, uploader_queued:4: --ignore-absent unmounted successfully. This contradicts the README's "--ignore-absent continues only past a drain that cannot reach the mount, never past pending work" and the spec's "must never report success while write-back work is still local" — and --ignore-absent is exactly the flag cleanup scripts pass routinely.
Fix: block whenever the parsed report shows pending work, regardless of classification — in the drainErr != nil branch, also return when drain.Response != nil and its pending counters are non-zero (a small hasPendingWork helper next to drainPendingSummary, which already computes non-zero-ness). The intended carve-out — a mount that is already gone — is distinguishable because it produces no report at all, so it stays continuable and both existing TestUnmountIgnoreAbsent* tests survive unchanged. If the engine today only ever emits pending_work_remaining with non-zero counters, the added check is a no-op — strictly safer either way.
Should fix (non-blocking, strong)
2. The blocked-drain error mis-advises operators for non-pending failures
internal/fs/drive9_mount_drain.go:178 — the remedy text always appends "(--ignore-absent does not skip this check)", but that is only true for fs.drain_pending_work; for an fs.drain_failed classification an --ignore-absent unmount does continue, so the message tells the operator their correct cleanup flag will not work and steers them to --force — documented loss acceptance — for a mount that may simply be gone. The no-report path (fs.drain_report_missing, i.e. a dead mount or older companion) is worse: it returns the raw run error with no unmount-escape guidance at all. Build the remedy tail from the classified code (pending → --force is the only way past; non-pending → --ignore-absent continues if the mount is gone, --force accepts the loss) and give the no-report path the same structured message.
3. The drain --dry-run predicts neither the failure nor the plan
internal/fs/mount.go:356 — drain_support is hard-coded to Status: "passed" with the message from the unmount gate (mountDrainGate) instead of the drain command's own rule (drainApplicability). Two verified divergences: for a WebDAV mount, ti fs drain-file-system --dry-run reports drain_support: passed while normal execution deterministically fails with fs.drain_not_applicable (exit 2) on the same locally-known locator fact; for a legacy locator with no recorded driver, the dry-run says "not applicable … records no driver" while the real command drains it successfully — spec 0039 explicitly keeps unrecorded-driver mounts drainable for the drain command. Compare DryRunMountFileSystem, which reports Status: "failed" for unmet local prerequisites; --dry-run's contract is to validate local state and predict the planned action. Base the check on drainApplicability(locator) and emit failed when not applicable.
Nits
UnmountResult.PIDandDrainResult.PIDare populated by nothing (before and after this PR) and theHuman()renderers no longer print them — delete the dead fields while the types are being reshaped.mountDrainSkipReasonand the WebDAV branch ofdrainApplicabilityhard-code the same "no mount control socket to drain" sentence; share it.- The tolerated-failure
drainruntime check is appended on theunmountedpath but not on theabsentpath (drive9_companion.go:1421), so the two success shapes differ for the same condition. - Coverage gaps around the new gate: the WebDAV branch of
drainApplicabilityhas no direct unit row, and the absent-with-failed-drain unmount path never runs in tests because both fake companions'umountalways succeeds.
Recommendation
Request changes. Only Blocking 1 stands between this and an approve; the counter-based gate is a small, strictly-safer change to code you already have the data for.
…cation Addresses the second review on #97. - `--ignore-absent` no longer decides from the engine's `error_kind`: the unmount blocks whenever the parsed drain report shows undrained work, so a drain that times out against a write-back backlog (non-pending classification, queued counters) can no longer detach with exit 0. Only a drain that produced no report at all stays continuable, which is exactly the already-gone-mount carve-out. - The drain error tail is built from the classification: pending work points at `--force` and says `--ignore-absent` will not help, while a non-pending or report-less failure points at `--ignore-absent` for a gone mount and at `--force` for accepting the loss. A drain with no report at all is now a structured `fs.drain_failed`/`fs.drain_report_invalid` instead of a raw companion error. - The drain `--dry-run` predicts the drain command's own rule (`drainApplicability`), so a WebDAV mount reports `drain_support: failed` while an unrecorded-driver locator stays passed. - The tolerated drain failure is recorded on both the `unmounted` and `absent` outcomes, and the shared "no mount control socket" sentence is reused. - Drop the never-populated `PID` fields from `UnmountResult`/`DrainResult`. - New pins: parsed-pending block under `--ignore-absent` (unit and e2e), non-pending remedy text, absent-path drain check, WebDAV and unrecorded-driver `drainApplicability`, and the drain dry-run check in both directions.
|
Round-2 review addressed in Blocking1. The gate keyed on the engine's classification, not the parsed counters — fixed. pending := drain.Response != nil && hasPendingWork(drain.Response.Pending)
if pending || apperr.CodeFor(drainErr) == "fs.drain_pending_work" || !opts.IgnoreAbsent {
return UnmountResult{}, drainErr
}
Should fix2. Remedy text mis-advised operators — fixed. 3. Drain func drainSupportCheck(locator mountlocator.Locator) dryrun.Check {
if drainable, reason := drainApplicability(locator); !drainable {
return dryrun.Check{Name: "drain_support", Status: "failed", Message: reason}
}
...
}A WebDAV mount now predicts Nits
Verification
|
qiffang
left a comment
There was a problem hiding this comment.
Re-reviewed f1b3afc against 41dc94f and the previous formal review. The remaining blocking counter-based bypass is fixed: drainErrorCode treats nonzero parsed pending counters as fs.drain_pending_work independently of error_kind, and unmount also checks the retained report before allowing --ignore-absent. The regression covers deadline_exceeded with queued commits/uploads and asserts no umount call occurs.
The other previous feedback is also addressed: classification-specific remedy text, structured missing-report failures, dry-run using drainApplicability, dead PID fields removed, shared WebDAV explanation, and failed-drain reporting on the absent outcome. I inspected the changed tests and companion fixtures; the new unit fixture deliberately exercises a zero-exit response with ok:false, while the E2E fixture also exercises a nonzero exit. Current test and CLA checks pass. No substantive blocker or submitted sensitive credential found. Source/call-chain and existing CI review only; no local execution or live mounts.
Advisory contract wording: the new comments/spec say only a failure with no report may be tolerated, but the actual --ignore-absent gate also tolerates a parsed failed report whose counters are all zero and whose error_kind is not pending_work_remaining. The pending-work blocker is closed; keep that narrower behavior and the documentation aligned.
qiffang's approval noted that the comment and specs claimed only a report-less drain failure may be tolerated, while the code also tolerates a parsed report with zero counters and a non-pending classification. The behavior is the narrower one we want, so only the wording changes: the comment, both specs, README, and AGENTS now say that --ignore-absent tolerates a drain failure that reports no pending work (no report at all, or zero counters with a non-pending classification) and that the block condition stays the parsed counters.
|
Thanks for the approval — and for the precise advisory note. You are right that the wording was narrower than the code:
No behavior change in this commit — docs and comments only. The pending-work blocker keeps its regression coverage ( |
qiffang
left a comment
There was a problem hiding this comment.
Reviewed all changes since f1b3afc. This increment changes documentation and code comments only: AGENTS.md, README, spec 0039 and the unmount gate comment now consistently describe the existing tolerance for a missing report or a zero-counter report with a non-pending classification. That resolves my previous advisory wording mismatch. The executable counter-based gate is unchanged, so the previous blocking fix remains intact. No new substantive blocker or sensitive submission found. Current test check passes on this exact head. Static review only; no local execution.
mornyx
left a comment
There was a problem hiding this comment.
Review summary
f1b3afc lands the round-2 blocker the right way: the unmount gate now blocks on the parsed pending counters (drain.Response != nil && hasPendingWork(...)) independent of the engine's error classification, the drain error code folds the same counters in, and the remedy text is built from the classification — so what the error prints, what the gate blocks on, and what --ignore-absent tolerates can no longer disagree. Both should-fixes (remedy text, dry-run drain_support) are fixed via the structurally correct routes (drainRemedy, drainSupportCheck → drainApplicability), all four round-2 nits turned out to be addressed as well, and 33823dad is verified docs/comments-only with wording that exactly matches the code's tolerated set. Unit tests and the full black-box e2e suite pass on macOS. Nothing blocks merge from my side any more; what remains is one hardening-grade structural point about the data-loss predicate and a few polish nits.
What looks good
- The gate fix targets the real invariant instead of patching another string special case: one shared zero-check feeds the gate, the error code, and the operator summary, and the tolerated carve-out is exactly the narrow set the docs now describe (no report at all, or zero counters with a non-pending classification) — each shape pinned by unit and e2e tests.
drainRemedykeyed on the classification and the dry-run reusing the execution path's own rule (drainSupportCheck→drainApplicability) both eliminate predict-execution-by-duplicating-logic hazards rather than patching messages.33823dadis a model response to a maintainer wording objection: behavior verified as intended, then only the words moved, with the diff cleanly separable for review.
Resolved since last review
- Gate keyed on
error_kindinstead of parsed counters (blocking): fixed —drive9_companion.go:1402-1405blocks onhasPendingWork(drain.Response.Pending)regardless of classification;drainErrorCodefolds the counters in too;TestUnmountIgnoreAbsentStillBlocksParsedPendingWorkpins thedeadline_exceeded+ queued-work shape and the e2e twin asserts the block under--ignore-absent. The gone-mount carve-out (no report) stays continuable. - Remedy text mis-advising operators (should fix): fixed — pending work gets "
--forceis the only way past this (--ignore-absentdoes not skip this check)"; non-pending/report-less failures point at--ignore-absentfor a gone mount and--forcefor accepting loss; the no-report path returns structuredfs.drain_failed/fs.drain_report_invalidwith the same tail instead of a raw companion error. - Drain
--dry-runcontradicting execution (should fix): fixed —drain_supportderives fromdrainApplicabilityand reportsfailedwhen execution would refuse; WebDAV e2e asserts the failed dry-run check and the real exit 2. - All four round-2 nits: fixed (dead
PIDfields dropped, skip-reason sentence deduplicated throughmountDrainSkipReason, absent path records the failed-drain check with a pinning test, applicability/absent coverage gaps closed) — the commit addressed more than the reply claimed, which is the right direction.
Should fix (non-blocking, strong)
1. The write-loss predicate is derived from the operator-facing rendering, and its counter semantics are an unconfirmed engine contract
internal/fs/drive9_mount_drain.go:91 — hasPendingWork is drainPendingSummary(pending) != "none": the authoritative data-loss predicate parses its own display output, so any future cosmetic change to the renderer (rewording "none", dropping a counter from display) silently changes both the --ignore-absent carve-out and the fs.drain_pending_work classification. Beyond the inversion, the predicate treats every field of DrainPending as losable work — including uploader_cached/uploader_cached_bytes and commit_queue_conflicts, whose engine semantics (warm cache of already-uploaded data? cumulative event count rather than queue depth?) cannot be verified from this repository. Today the failure direction is fail-closed (over-block), which is the right default, but if any counter can be non-zero with nothing losable, an --ignore-absent unmount over-blocks while the remedy says "--force is the only way past this" — loss-acceptance language for a no-loss situation.
Fix: invert the sharing — make the predicate the explicit enumeration over the counters (a method on DrainPending in mountcontrol reads naturally), have drainPendingSummary return "none" when the predicate is false, and confirm with the Drive9 contract that the pending block only ever contains undrained work, recording that invariant in spec 0039 next to the counter list. Optionally split loss-carrying from informational counters once the contract is confirmed; keep the conservative default until then.
Nits
- The gate condition carries two overlapping signals of the same fact (
pending || code == "fs.drain_pending_work"— the code check already folds the counters in); pick one authority so the terms cannot drift apart. fs.drain_report_missingis defined but never propagates:drainMount's decode-failure branch re-codes everything exceptfs.drain_report_invalidto plainfs.drain_failed. Either propagate it like the invalid case (and name it in spec 0039) or drop the third code.drainSupportCheck's passed message says "drain control socket" even for driver-less legacy locators, where ti explicitly cannot confirm drain support — use a neutral message ("file-system mount; the drain command will ask the companion") or branch on the recorded driver.- Spec 0039's verification wording overstates coverage: no fixture produces the newly documented tolerated shape (ok:false, non-pending kind, zero counters), and the unrecorded-driver dry-run direction is pinned only at the helper level. Add a fixture for the zero-counter tolerated shape (or trim the claim), and one
--dry-runcase against a driver-less locator.
Recommendation
Comment. The blocking history of this PR is genuinely cleared on 33823dad; the predicate/printer inversion and the counter-contract confirmation are worth a small follow-up before or shortly after merge, but nothing here should hold it.
…authority Addresses mornyx's remaining hardening note and nits. - `DrainPending.HasPendingWork()` is now the single data-safety predicate, an explicit enumeration of the counters instead of a parse of the operator-facing summary, so rewording the text output can no longer change a safety decision. `drainPendingSummary` returns "none" only when the predicate is false. Spec 0039 records the engine contract behind it: the drain-failure exit record counts dirty_handles + commit_queue_pending + uploader_queued as pending files and commit_queue_bytes + uploader_cached_bytes as pending bytes, and ti keeps the conservative over-block for the informational counters until Drive9 publishes a loss-carrying split. - The unmount gate has one authority: `fs.drain_pending_work` already folds in the parsed counters and the engine's classification, so the second `hasPendingWork(...)` term is gone and the terms cannot drift. - `fs.drain_report_missing` now propagates like `fs.drain_report_invalid` instead of being collapsed into `fs.drain_failed`, and both are named in the spec. - `drain_support`'s passed message is neutral, so it no longer claims a control socket for driver-less legacy locators. - New pins: the predicate table independent of rendering, the tolerated zero-counter failure shape (unit and e2e fixtures), and a drain dry-run against a driver-less locator.
|
Thanks — good catch on the predicate/printer inversion. Should fixPredicate derived from the rendering — fixed by inverting the sharing. func (p DrainPending) HasPendingWork() bool {
return p.OpenHandles != 0 || p.DirtyHandles != 0 || p.CommitQueuePending != 0 ||
p.CommitQueueBytes != 0 || p.CommitQueueInFlight != 0 || p.CommitQueueDelayed != 0 ||
p.CommitQueueConflicts != 0 || p.UploaderQueued != 0 || p.UploaderInFlight != 0 ||
p.UploaderCached != 0 || p.UploaderCachedBytes != 0
}On the counter contract: Drive9's own drain-failure exit record counts exactly Nits
Verification
|
mornyx
left a comment
There was a problem hiding this comment.
Review summary
dcf8d97 resolves the round-3 hardening note in the right shape: the data-safety predicate now lives on the type (DrainPending.HasPendingWork, an explicit 11-field enumeration with the engine contract and the deliberate over-block rationale documented beside it), the operator summary is derived from it, and the unmount gate collapsed to a single fs.drain_pending_work code check. The collapse was verified rather than assumed: every error path drainMount can return was walked — the report-less branch returns codes that cannot carry parsed counters, and the classified branch always passes through drainErrorCode, which folds any non-zero counter into fs.drain_pending_work — so there is no path where the report shows queued work but the unmount proceeds. All four round-3 nits are genuinely fixed with real fixtures, and unit tests plus the full e2e suite pass. One coverage gap remains between the code and its claim, plus small polish; nothing blocks merge.
What looks good
- Putting the predicate on
DrainPendinginmountcontrol— the type that owns the counters — with rendering derived from it, is the correct inversion: a cosmetic reword of the summary can no longer flip a data-safety decision, and every future drift errs toward understating pending work in the message while the gate stays strict. - The gate reduction to the code check alone was earned, not assumed: the surviving single authority provably folds the parsed counters in on every branch that can carry them, and the comments explain the invariant rather than the mechanics.
- The new fixtures are complementary by design: the unit fake fails with
ok=false/exit 0 and the e2e fake with exit 1, so both distinct failure paths throughdrainMountare each pinned by a fixture that actually exercises them, and the zero-counter tolerated shape is asserted end to end (unmounted+drain.status: "failed").
Resolved since last review
- Predicate derived from the rendering (should fix): fixed —
HasPendingWorkis the explicit enumeration;drainPendingSummaryreturns "none" only when the predicate is false; no safety consumer derives pending-ness from the display string. The spec records the counter invariant (undrained-work counters by construction, informational counters non-zero only while work remains) and the "tighten when Drive9 publishes a split" note. The cited engine-siderecordDrainFailurecannot be verified from this repository, but the ti-side conservative over-block is safe regardless of its truth. - Gate carried two overlapping signals: fixed — single code authority; verified path by path that no non-zero-counter report escapes
fs.drain_pending_work. fs.drain_report_missingnever propagated: fixed — propagates likefs.drain_report_invalid, both named in the spec taxonomy.drain_supportpassed message claimed a control socket for driver-less locators: fixed — neutral message, pinned byTestDryRunDrainSupportForDriverlessLocator.- Spec coverage claims exceeded fixtures: fixed —
FAKE_DRIVE9_DRAIN_ZERO_KINDfixtures produce the tolerated zero-counter shape in unit and e2e; the driver-less dry-run direction is now pinned at the service level.
Should fix (non-blocking, strong)
1. The predicate "pin" test covers only 8 of the 11 counters it claims to pin
internal/fs/drive9_mount_drain_test.go:468 — TestHasPendingWorkIsIndependentOfRendering has no rows for CommitQueueInFlight, CommitQueueDelayed, or UploaderInFlight, yet the commit message and the PR reply both state it "pins every counter". Today, deleting any of those three != 0 terms from HasPendingWork passes the whole suite silently, and the gate under-blocks exactly the in-flight/delayed work the conservative over-block exists to catch. This matters more than a usual coverage gap because spec 0039 explicitly schedules a future tightening edit of this exact predicate, and this table is the guardrail for that edit.
Fix: add the three missing table rows (one non-zero value each, want: true) — three lines, no other changes.
Nits
drainPendingSummarykeeps a vestigiallen(rendered) == 0 ||term next to!pending.HasPendingWork()— the two conditions cover the same 11 fields, and dead weight in a commit whose stated goal is one authority. Drop the first term.- Spec 0039 line 64 still says "any other drain failure uses
fs.drain_failed", contradicting the new taxonomy paragraph (report-less failures usefs.drain_report_missing/fs.drain_report_invalid). Reword to match. - The new "### The pending predicate" heading was spliced into the middle of the unmount command's bullet list in spec 0039, orphaning the last two bullets. Move the section below the completed list.
- The decode-failure message says "produced no report" for both sub-cases; for
fs.drain_report_invalidstdout existed — "produced no usable report", or branch the phrasing on the code. hasPendingWorkis now a one-line delegation used once in non-test code; callresponse.Pending.HasPendingWork()at thedrainErrorCodesite and keep the doc comment on the method.
Recommendation
Comment. The blocking history stays cleared and the refactor is verified safe; the three-line pin-test completion should land with or before merge so the file matches its own claim.
Addresses mornyx's round-4 review. - `TestHasPendingWorkIsIndependentOfRendering` now has a row for all eleven counters, including the three it silently skipped (commit_queue_in_flight, commit_queue_delayed, uploader_in_flight). Deleting any term from the predicate now fails the suite, which matters because spec 0039 schedules a future tightening edit of exactly this predicate. - Drop the vestigial `len(rendered) == 0` term from `drainPendingSummary`; the summary now derives "none" from the predicate alone. - Drop the one-line `hasPendingWork` delegation and call `response.Pending.HasPendingWork()` at the single consuming site, keeping the doc comment on the method. - Decode-failure message says "produced no usable report", which is also true when stdout existed but was unparseable. - Spec 0039: the drain-command taxonomy lists `fs.drain_report_missing` and `fs.drain_report_invalid` alongside `fs.drain_failed`, and the pending predicate section sits after the completed unmount bullet list instead of splitting it.
|
All five points from round 4 are in Should fixPin test covered 8 of 11 counters — fixed. Nits
Verification
|
qiffang
left a comment
There was a problem hiding this comment.
Re-reviewed exact head 64fa530 against 33823da. No new substantive blocker found.
DrainPending.HasPendingWork now enumerates the same eleven counters directly, so the data-safety decision no longer depends on rendered text. drainErrorCode still folds both pending_work_remaining and nonzero counters into fs.drain_pending_work, and unmount consumes that classification. Removing the duplicate counter check at unmount therefore preserves the previous blocking behavior. The per-counter regression, pending deadline failure, zero-counter tolerated failure, missing-report path and driverless dry-run cases cover the changed contract.
Missing and invalid reports retain their specific structured error codes and actionable guidance. The spec and comments match the existing conservative policy. The prior blocker and documentation advisory remain closed. Current test and CLA checks pass; no submitted sensitive credential identified. Static source/call-chain and existing regression/CI inspection only; no local target execution or live mount operations.
mornyx
left a comment
There was a problem hiding this comment.
Review summary
64fa5305 is exactly what it claims — three files, +24/−27, no behavior change on any reachable input — and it closes every remaining round-4 item. The pin test now has one want: true row per DrainPending counter (11:11 against the struct, including the three previously skipped commit_queue_in_flight, commit_queue_delayed, uploader_in_flight), so deleting any predicate term fails the suite; that guard was verified both by construction and empirically against mutated copies of the predicate. The cleanup deleted indirection rather than polishing it (the vestigial summary mask and the one-line wrapper are gone, leaving the method as the single authority with two direct call sites), the error wording is accurate for both no-report sub-cases, and the spec's taxonomy, --ignore-absent contract, and code now agree on all three drain error codes with the predicate section back below the completed bullet list. Unit tests, vet, and gofmt pass on HEAD, and the full history of this PR's blockers is resolved and regression-pinned. Nothing blocking, important, or nit-worthy remains.
What looks good
- The pin test is now a true mutation guard — one row per enumerated counter, each independently failing the suite — which is precisely the protection the spec's planned future predicate tightening needs; the row renames also remove the ambiguity that let three counters be skipped silently in the first place.
- Dropping the
len(rendered) == 0mask rather than keeping it as a belt was the right call: the render list and the predicate are now forced to stay in sync, because divergence surfaces as a visibly empty summary instead of a reassuring "none" on a report that actually has pending work. - Across the whole review history this PR converged the way a safety-critical change should: every round's finding landed with its own pinning test, the data-loss decision ended up as one explicit predicate on the type that owns the data, and the docs were kept in literal agreement with the code at each step.
Recommendation
Approve. Every finding from all review rounds is verifiably fixed on 64fa5305, the commit introduces no regressions, and the merge gate this PR built is now exactly as strict as its documentation claims.
Solves two related mount-safety gaps in one PR: the write-durability tier was not selectable, and an unmount could silently drop pending local writes.
1.
--durabilitypassthroughBefore: ti only had the
--write-back-cacheboolean, and only an explicit--driver fusemount forwarded anything —truepinnedinteractive,falseforwarded nothing so the engine fell back toauto. The companion already supportedauto|interactive|fsync|close-sync|write-sync, so a caller that neededclose-sync/write-synchad to mount around ti and drain by hand.--durabilityaccepts the five tiers, defaults tointeractive(previous behavior), and is validated before the companion runs (fs.invalid_mount_durability).--driver automounts on platforms whose automatic choice is FUSE, so an ordinary mount never silently follows the engine'sautodefault.--write-back-cachestays as a legacy alias (true= interactive,false= auto); an inconsistent explicit combination fails withfs.conflicting_mount_durability.auto(fs.webdav_durability_unsupported) instead of silently downgrading. Checkpoint mounts forceautoand reject other explicit tiers.durabilityand no longer reports thewrite_back_cacheboolean (a boolean cannot express five tiers; the field was never a documented contract).2. Drain before unmount
Before:
ti fs drainranmount drain --jsonand threw the report away,ti fs unmountnever drained, and--forcewas parsed but never read. Since the engine's SIGTERM path discards its own drain errors, an unmount could return success while writes never left the client, with nothing to attribute it to afterwards.ti fs drainnow returns the engine's report asresponse(ok,error_kind,failed_path, duration, pending counters) and prints the counters in text output. Undrained work fails withfs.drain_pending_workplus the counters and the retry command.ti fs unmountdrains first, blocks with the mount locator preserved while writes are pending, and only skips the gate where it cannot apply: WebDAV mounts (no control socket) and locators written before the driver was recorded. The mount records the driver the companion actually used in the non-secret locator.--forceis now the explicit "accept the loss" escape hatch and is reported asdrain.status: "skipped";--ignore-absentmay continue past a failed drain but still reportsdrain.status: "failed".--dry-runreports the gate (pre_unmount_drain) without running it.Files
internal/cli/commands.go: mount--durabilityflag, unmount--forcesemantics.internal/fs/mount_durability.go,internal/fs/mount.go,internal/fs/drive9_mount_audit.go: tier resolution, argv, result, dry-run.internal/fs/drive9_mount_drain.go,internal/fs/drive9_companion.go: drain report parsing, unmount gate, locator/profile resolution.internal/fs/mountlocator/locator.go: non-secretdriver.docs/spec/done/0038-fs-mount-durability.md,docs/spec/done/0039-fs-unmount-drain-gate.md; README + AGENTS updated.internal/fs/mount_durability_test.go,internal/fs/drive9_mount_drain_test.go, fake companion argv/drain modes,e2e/cli_test.go,e2e/live_test.go.Verification
go build ./...,go vet ./...,gofmt -lclean.go test ./internal/...passes: tier table, legacy alias, conflict, WebDAV/checkpoint rejections,--mode automirror, forwarded argv for all five tiers, interactive default, dry-run body; drain report + counters, pending-drain error, gate blocking with locator preserved,--force, WebDAV and legacy-locator skips,--ignore-absent, driver recorded at mount.make build+TI_E2E_BIN=bin/ti go test ./e2epasses. Fake-companion e2e asserts--durability interactive/close-syncforwarding, no forwarding for WebDAV, both rejection paths, drain counts in JSON and text, a pending drain blocking both drain and unmount, and--forcereportingskipped.make live-e2e-*was not run here (no live credentials in this environment). Live coverage added: fs-git mount with--durability close-sync, WebDAV mount reportingauto, and drain assertingresponse.pending.Not in this PR
The English PingCAP pages live in the
docs/pingcap-docs/docssubmodule, which is not checked out here, so the mount page still needs the--durabilityand drain/unmount sections.The drive9-side fixes —
drive9 umountgating on pending work itself, and the SIGTERM handler surfacingflushOpenHandles/chain.Drainerrors — live in the Drive9 repository, which is outside this workspace. They are written up with exact locations indocs/spec/done/0039-fs-unmount-drain-gate.mdand need their own PR there; this PR is the ti-side mitigation.