Skip to content

Mock devices never report healthy — Allocatable never shows registered GPU resources #21

Closed

Description

@hollarhollar

Environment:

Kind cluster (kindest/node:v1.34.0), single control-plane node
Docker Desktop on Windows, WSL2 backend
HAMi installed via Helm (hami-charts/hami), hami-scheduler healthy (2/2 Running)
hami-device-plugin in CrashLoopBackOff as expected (no real GPU/NVIDIA Container Toolkit present) — not the subject of this report
mock-device-plugin installed per README (k8s-mock-rbac.yaml + k8s-mock-plugin.yaml)

Steps to reproduce:

Create a fresh Kind cluster
Label the node gpu=on
Install HAMi via Helm
Apply k8s-mock-rbac.yaml and k8s-mock-plugin.yaml from this repo
Wait for hami-mock-device-plugin-daemonset to reach 1/1 Running
Check kubectl logs on the mock plugin pod
Check kubectl describe node under Allocatable

Expected behavior:
Per the README's documented example output, Allocatable should show entries like nvidia.com/gpu, nvidia.com/gpucores, nvidia.com/gpumem, etc.

Actual behavior:
The plugin starts and logs that it's "Running mocking dp" for each device type (NVIDIA, DCU, all Ascend variants), but immediately and continuously logs every device as unhealthy, repeating every ~30 seconds indefinitely:

Full log output (click to expand) $ kubectl logs hami-mock-device-plugin-daemonset-kt484 -n kube-system --tail=30 I0729 10:33:52.238648 1 device.go:103] device DCU is unhealthy on this node I0729 10:33:52.239578 1 device.go:141] device Ascend910B3 is unhealthy on this node I0729 10:33:52.241173 1 device.go:141] device Ascend910B4 is unhealthy on this node I0729 10:34:22.618828 1 device.go:141] device Ascend910B3 is unhealthy on this node I0729 10:34:22.626068 1 device.go:103] device DCU is unhealthy on this node I0729 10:34:22.626584 1 device.go:141] device Ascend910B4 is unhealthy on this node I0729 10:34:22.627345 1 device.go:141] device Ascend910A is unhealthy on this node I0729 10:34:22.630906 1 device.go:141] device Ascend310P is unhealthy on this node I0729 10:34:22.654231 1 device.go:141] device Ascend910C is unhealthy on this node I0729 10:34:22.654761 1 device.go:174] device NVIDIA is unhealthy on this node I0729 10:34:22.655559 1 device.go:141] device Ascend910B4-1 is unhealthy on this node I0729 10:34:22.655634 1 device.go:141] device Ascend910B2 is unhealthy on this node I0729 10:34:53.127300 1 device.go:141] device Ascend310P is unhealthy on this node I0729 10:34:53.950226 1 device.go:141] device Ascend910B4 is unhealthy on this node I0729 10:34:53.959983 1 device.go:141] device Ascend910B3 is unhealthy on this node I0729 10:34:53.961199 1 device.go:141] device Ascend910C is unhealthy on this node I0729 10:34:53.965928 1 device.go:174] device NVIDIA is unhealthy on this node I0729 10:34:53.966334 1 device.go:141] device Ascend910B4-1 is unhealthy on this node I0729 10:34:53.976223 1 device.go:141] device Ascend910B2 is unhealthy on this node I0729 10:34:53.977093 1 device.go:103] device DCU is unhealthy on this node I0729 10:34:53.977399 1 device.go:141] device Ascend910A is unhealthy on this node I0729 10:35:23.132977 1 device.go:141] device Ascend310P is unhealthy on this node I0729 10:35:23.957007 1 device.go:141] device Ascend910B4 is unhealthy on this node I0729 10:35:23.978996 1 device.go:174] device NVIDIA is unhealthy on this node I0729 10:35:23.979900 1 device.go:141] device Ascend910B3 is unhealthy on this node I0729 10:35:23.980437 1 device.go:141] device Ascend910C is unhealthy on this node I0729 10:35:23.980952 1 device.go:141] device Ascend910B2 is unhealthy on this node I0729 10:35:23.981167 1 device.go:141] device Ascend910B4-1 is unhealthy on this node I0729 10:35:23.981978 1 device.go:103] device DCU is unhealthy on this node I0729 10:35:23.982028 1 device.go:141] device Ascend910A is unhealthy on this node

As a result, kubectl describe node never shows any of the mock GPU resources under Allocatable — only standard cpu/memory/pods/ephemeral-storage.

What I've ruled out:
Suspected this might be a clock-sync/handshake timeout issue (HAMi's health check is timestamp-based, and WSL2 is known for clock drift). Tested by fully restarting WSL2 (wsl --shutdown from Windows host, not from inside WSL) and rebuilding the cluster from scratch — the "unhealthy" cycling persisted identically on the fresh cluster/clock, so this doesn't appear to be a timing/environment issue specific to my setup.

Is there a required configuration step (e.g., in the hami-scheduler-device ConfigMap, or a plugin flag) needed to make mock devices report healthy that isn't currently documented in the README? Happy to submit a docs PR once I understand the correct setup.

Investigated with Claude Code assistance to narrow down the clock-drift hypothesis; reproduction steps and findings are my own testing.

Activity

  1. coderabbitai commented on Jul 29, 2026

    @coderabbitai
    ⚠️ Possible Duplicate Issue(s)
    🔗 Related PRs

    #18 - feat(ascend,hygon): register AI-core/dcucores and support new and legacy vnpus config [merged]


    📝 Issue Planner

    Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.

    • Create Plan

    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

  2. hollarhollar commented on Jul 29, 2026

    @hollarhollar
    Author

    Update: this does look like the same issue as #16 (same Kind + no-real-GPU environment, same symptom). I tried the MOCK_DEVICE_FORCE_REGISTER=true env var patch mentioned there:

    kubectl patch daemonset hami-mock-device-plugin-daemonset -n kube-system
    --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"MOCK_DEVICE_FORCE_REGISTER","value":"true"}}]'

    Confirmed the env var landed in the running pod, but it made no difference — the "unhealthy" cycling continues identically for every device type. So that specific workaround doesn't appear to resolve it, at least not on Kind.

    Combined with @DSFans2014's explanation on #14 (that this log fires when a device is configured but has no corresponding node.status.capacity resource, and that the real device plugin + matching hardware is required) — it seems like mock-device-plugin may not currently work standalone on GPU-less environments like Kind, despite the README's framing suggesting it should. Happy to help narrow this down further if there's a specific config or setup step I'm missing, or to help document the actual constraint once it's clear.

    Investigated with Claude Code assistance; reproduction steps and findings are my own testing.

  3. DSFans2014 commented on Jul 30, 2026

    @DSFans2014
    Member

    mock-device-plugin simply parses the annotation and then register the resources. If you want to simulate GPUs on a machine without GPU devices, please use other components such as fake-gpu.

  4. hollarhollar commented on Jul 30, 2026

    @hollarhollar
    Author

    Thanks @DSFans2014, that clears it up. Makes sense, I'll follow up with a small PR to the README clarifying that mock-device-plugin requires an existing GPU-annotated node rather than working standalone on hardware-free environments, and pointing to fake-gpu for that use case instead. Appreciate the quick response!

  5. moezdil commented on Sep 22, 2026

    @moezdil

    Old PRs and issues are being closed because they are no longer relevant. Or it was closed because of th
    inactivity or PR/issues owners.
    If you believe the issue is still relevant, please submit feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions