Repository navigation
Mock devices never report healthy — Allocatable never shows registered GPU resources #21
Description
Environment:
Kind cluster (kindest/node:v1.34.0), single control-plane node
Docker Desktop on Windows, WSL2 backend
HAMi installed via Helm (hami-charts/hami), hami-scheduler healthy (2/2 Running)
hami-device-plugin in CrashLoopBackOff as expected (no real GPU/NVIDIA Container Toolkit present) — not the subject of this report
mock-device-plugin installed per README (k8s-mock-rbac.yaml + k8s-mock-plugin.yaml)
Steps to reproduce:
Create a fresh Kind cluster
Label the node gpu=on
Install HAMi via Helm
Apply k8s-mock-rbac.yaml and k8s-mock-plugin.yaml from this repo
Wait for hami-mock-device-plugin-daemonset to reach 1/1 Running
Check kubectl logs on the mock plugin pod
Check kubectl describe node under Allocatable
Expected behavior:
Per the README's documented example output, Allocatable should show entries like nvidia.com/gpu, nvidia.com/gpucores, nvidia.com/gpumem, etc.
Actual behavior:
The plugin starts and logs that it's "Running mocking dp" for each device type (NVIDIA, DCU, all Ascend variants), but immediately and continuously logs every device as unhealthy, repeating every ~30 seconds indefinitely:
Full log output (click to expand)
$ kubectl logs hami-mock-device-plugin-daemonset-kt484 -n kube-system --tail=30 I0729 10:33:52.238648 1 device.go:103] device DCU is unhealthy on this node I0729 10:33:52.239578 1 device.go:141] device Ascend910B3 is unhealthy on this node I0729 10:33:52.241173 1 device.go:141] device Ascend910B4 is unhealthy on this node I0729 10:34:22.618828 1 device.go:141] device Ascend910B3 is unhealthy on this node I0729 10:34:22.626068 1 device.go:103] device DCU is unhealthy on this node I0729 10:34:22.626584 1 device.go:141] device Ascend910B4 is unhealthy on this node I0729 10:34:22.627345 1 device.go:141] device Ascend910A is unhealthy on this node I0729 10:34:22.630906 1 device.go:141] device Ascend310P is unhealthy on this node I0729 10:34:22.654231 1 device.go:141] device Ascend910C is unhealthy on this node I0729 10:34:22.654761 1 device.go:174] device NVIDIA is unhealthy on this node I0729 10:34:22.655559 1 device.go:141] device Ascend910B4-1 is unhealthy on this node I0729 10:34:22.655634 1 device.go:141] device Ascend910B2 is unhealthy on this node I0729 10:34:53.127300 1 device.go:141] device Ascend310P is unhealthy on this node I0729 10:34:53.950226 1 device.go:141] device Ascend910B4 is unhealthy on this node I0729 10:34:53.959983 1 device.go:141] device Ascend910B3 is unhealthy on this node I0729 10:34:53.961199 1 device.go:141] device Ascend910C is unhealthy on this node I0729 10:34:53.965928 1 device.go:174] device NVIDIA is unhealthy on this node I0729 10:34:53.966334 1 device.go:141] device Ascend910B4-1 is unhealthy on this node I0729 10:34:53.976223 1 device.go:141] device Ascend910B2 is unhealthy on this node I0729 10:34:53.977093 1 device.go:103] device DCU is unhealthy on this node I0729 10:34:53.977399 1 device.go:141] device Ascend910A is unhealthy on this node I0729 10:35:23.132977 1 device.go:141] device Ascend310P is unhealthy on this node I0729 10:35:23.957007 1 device.go:141] device Ascend910B4 is unhealthy on this node I0729 10:35:23.978996 1 device.go:174] device NVIDIA is unhealthy on this node I0729 10:35:23.979900 1 device.go:141] device Ascend910B3 is unhealthy on this node I0729 10:35:23.980437 1 device.go:141] device Ascend910C is unhealthy on this node I0729 10:35:23.980952 1 device.go:141] device Ascend910B2 is unhealthy on this node I0729 10:35:23.981167 1 device.go:141] device Ascend910B4-1 is unhealthy on this node I0729 10:35:23.981978 1 device.go:103] device DCU is unhealthy on this node I0729 10:35:23.982028 1 device.go:141] device Ascend910A is unhealthy on this nodeAs a result, kubectl describe node never shows any of the mock GPU resources under Allocatable — only standard cpu/memory/pods/ephemeral-storage.
What I've ruled out:
Suspected this might be a clock-sync/handshake timeout issue (HAMi's health check is timestamp-based, and WSL2 is known for clock drift). Tested by fully restarting WSL2 (wsl --shutdown from Windows host, not from inside WSL) and rebuilding the cluster from scratch — the "unhealthy" cycling persisted identically on the fresh cluster/clock, so this doesn't appear to be a timing/environment issue specific to my setup.
Is there a required configuration step (e.g., in the hami-scheduler-device ConfigMap, or a plugin flag) needed to make mock devices report healthy that isn't currently documented in the README? Happy to submit a docs PR once I understand the correct setup.
Investigated with Claude Code assistance to narrow down the clock-drift hypothesis; reproduction steps and findings are my own testing.
Activity
coderabbitai commentedon Jul 29, 2026
🔗 Related PRs
#18 - feat(ascend,hygon): register AI-core/dcucores and support new and legacy vnpus config [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
hollarhollar commentedon Jul 29, 2026
Update: this does look like the same issue as #16 (same Kind + no-real-GPU environment, same symptom). I tried the MOCK_DEVICE_FORCE_REGISTER=true env var patch mentioned there:
kubectl patch daemonset hami-mock-device-plugin-daemonset -n kube-system
--type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"MOCK_DEVICE_FORCE_REGISTER","value":"true"}}]'
Confirmed the env var landed in the running pod, but it made no difference — the "unhealthy" cycling continues identically for every device type. So that specific workaround doesn't appear to resolve it, at least not on Kind.
Combined with @DSFans2014's explanation on #14 (that this log fires when a device is configured but has no corresponding node.status.capacity resource, and that the real device plugin + matching hardware is required) — it seems like mock-device-plugin may not currently work standalone on GPU-less environments like Kind, despite the README's framing suggesting it should. Happy to help narrow this down further if there's a specific config or setup step I'm missing, or to help document the actual constraint once it's clear.
Investigated with Claude Code assistance; reproduction steps and findings are my own testing.
DSFans2014 commentedon Jul 30, 2026
mock-device-pluginsimply parses the annotation and then register the resources. If you want to simulate GPUs on a machine without GPU devices, please use other components such as fake-gpu.hollarhollar commentedon Jul 30, 2026
Thanks @DSFans2014, that clears it up. Makes sense, I'll follow up with a small PR to the README clarifying that mock-device-plugin requires an existing GPU-annotated node rather than working standalone on hardware-free environments, and pointing to fake-gpu for that use case instead. Appreciate the quick response!
moezdil commentedon Sep 22, 2026
Old PRs and issues are being closed because they are no longer relevant. Or it was closed because of th
inactivity or PR/issues owners.
If you believe the issue is still relevant, please submit feedback.