Skip to content

Storage overflow: log why a host has to wait - #2108

Merged
fog-workflows[bot] merged 2 commits into
dev-branchfrom
fix/overflow-says-why
Nov 18, 2026
Merged

fog-workflows[bot] merged 2 commits into
dev-branchfrom
fix/overflow-says-why

Conversation

@mastacontrola

Copy link
Copy Markdown
Member

Problem

Forums 18310: on 1.6.0.181, 12 hosts deployed to a 3-node group with Max Clients 10. Ten ran on the master. Two waited in the queue instead of moving to the two free nodes. The poster checked the node settings, the location settings, and that the main server reaches every node on 22, 443 and 80. My lab, on the same code, spreads the hosts every time.

StorageGroup::getOverflowNode() picks the node at PXE boot. When no node qualifies, the host stays and waits, and nothing records why. A full group, a disabled node, an unreachable node and a node that failed the host all look the same.

Change

When a host is over its node's Max Clients and no other node can take it, getOverflowNode() writes one line to the web server's error log:

FOG storage node overflow: node id 2 has 12 tasks, over its Max Clients 10, so host id 412 stays and waits. Group testgroup (id 2) has no node to take it: node id 3 is full (10 tasks, Max Clients 10).
  • The line names each node passed over and the reason: disabled or Max Clients 0, full (with its count), no answer on its SSH or web port, or failed this host recently.
  • enablednodes already leaves out disabled nodes and nodes with Max Clients 0, so the line lists those from allnodes.
  • A host that moves, or fits, writes nothing.
  • The method is called only at PXE boot, so a waiting host writes one line per boot, not one per check-in poll.
  • It uses error_log() unconditionally, like _fallbackNode(), so it does not depend on a logging setting.
  • Node selection is unchanged.

Tests

  • tests/optimal-node-balances-load.test.php gains five cases: each skip reason, Max Clients 0, a full node, a one-node group, and no line for a host that moves or fits. The stub now builds enablednodes the way loadEnablednodes() does. The new cases fail on dev-branch.
  • Both phpstan passes are clean.
  • Live on a 1.6 lab, with the changed file over the deployed tree: 22 deploys on a node with Max Clients 10, in a group with a second node with Max Clients 10. Ten hosts moved to the second node. Each of the other 12 wrote the line above. There were no fatals.

🤖 Generated with Claude Code

mastacontrola and others added 2 commits November 18, 2026 09:54
When a host PXE boots onto a node over its Max Clients and no other node
in the group can take it, the host stays and waits in the queue. That
left no trace of why: a full group, a disabled node, an unreachable node
and a node that failed the host all looked the same (forums 18310).

getOverflowNode() now writes one line to the web server's error log in
that case. It names the node, its task count and Max Clients, the host
and the group, and each node passed over with the reason. A host that
moves, or fits, writes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
loadEnablednodes() drops a disabled node and one with Max Clients 0, so
the loop never saw them and the log could not name them. Those are the
settings most likely to be wrong, so list every node in allnodes that is
missing from enablednodes. The test stub now builds enablednodes the same
way, so it can no longer prove the reason on input production never has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fog-workflows
fog-workflows Bot enabled auto-merge November 18, 2026 15:57
@fog-workflows
fog-workflows Bot added this pull request to the merge queue Nov 18, 2026
Merged via the queue into dev-branch with commit 4d48102 Nov 18, 2026
12 checks passed
@fog-workflows
fog-workflows Bot deleted the fix/overflow-says-why branch November 18, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant