Repository navigation
AKO: Prevent transfer SSH channel saturation from starving runtime observation #116
Description
Evidence
Montage58 real V5 has two consumers preparing five whole-workspace routes each. Read-only console diagnostics on both destinations expired with wait for SSH channel: context deadline exceeded; the workflow itself remained running without a current failure. The shared SSH session manager has a fixed limit of four channels per multiplex master, whereas workspace parallelism permits eight routes per consumer. Runtime commands and bulk transfer commands share that limiter.
Required investigation and solution
Measure channel queue wait separately from remote execution. Prevent long bulk transfers from starving runtime observation and health diagnostics, respecting the actual SSH server MaxSessions capacity. Keep conservative HPC defaults, strict host trust, exact instance identity, full workspace validation and execution barriers. Do not simply raise global concurrency or create uncontrolled extra TCP connections. Test cancellation, release, mixed bulk/control contention and per-connection isolation. Apply only at a safe execution point; no performance gain is established yet.
Workflow
AKO-N branch, tests and local commit merged into local main; no push. Preserve live Montage outputs and avoid duplicate execution.
Activity
ovvesley commentedon Oct 8, 2026
Implemented locally in AKO-116/ssh-control-channel-reservation and merged into local main, without push. Bulk transfer SSH calls now acquire a three-channel reservation inside the existing four-channel total limit, leaving capacity for runtime inspection/control. No extra transport, host-trust bypass, HPC MaxSessions increase, workspace filtering or barrier change. Tests cover bulk/control contention, per-connection independence, idempotent release, cancellation while the total transport is full and reservation reuse; provider, transfer and execution controlplane suites passed. Not deployed to the live healthy Montage run; real validation and performance impact remain pending. This addresses control starvation, not a claim of higher bulk throughput.
ovvesley commentedon Oct 14, 2026
SSH bulk/control channel capacity is separated and reserved, preserving bounded cancellation and control access; implementation is integrated. No universal speedup claim. Located through AKO issue-number commits ed966c3 and verified integrated in published main 4aecc4c via PR #175. Full Go suite was rerun on main successfully for this audit; PR Go/docs CI is green. Closure covers this implementation/investigation scope only; remaining scientific forecast gates are tracked separately.