Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 40 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,20 @@ The mock plugin **does not detect hardware**. It does the following every ~30s:

```text
node annotation: hami.io/node-<vendor>-register = [ {devmem, devcore, ...}, ... ] (1)
node capacity: <count resource> (e.g. nvidia.com/gpu) > 0 (2) <- health gate
node capacity: <count resource> (e.g. nvidia.com/gpu) > 0 (2) <- health gate for supplemental resources
| mock reads
v
registers into allocatable: <vendor>/...mem , <vendor>/...cores , ...
```

On a **real** cluster, (1) and (2) are produced by the real device plugin. In a **mock-only** (no hardware) environment you provide them yourself:
On a **real** cluster, each family's required inputs are produced by the real device plugin. In a **mock-only** (no hardware) environment you provide them yourself:

- **(1) the `node-<vendor>-register` annotation** describing the fake cards -- `kubectl annotate`.
- **(2) the count extended resource** (e.g. `nvidia.com/gpu`) -- patched onto the node `status`.
- **(2) the count extended resource** (e.g. `nvidia.com/gpu`) -- patched onto the node `status` for NVIDIA, Hygon and Ascend.

> There is **no auto-labeller** in this repo, so (1) and (2) are manual today. Forgetting them is the usual cause of `device xxx is unhealthy` / `no allocation` -- see issues #14 / #16.
Kunlun vXPU uses only its `hami.io/node-register-xpu` annotation. The mock derives both `kunlunxin.com/vxpu` and `kunlunxin.com/vxpu-memory` from healthy annotation entries, so it does not need a separate count-resource status patch.

> There is **no auto-labeller** in this repo, so the required inputs are manual today. Forgetting them is the usual cause of `device xxx is unhealthy` / `no allocation` -- see issues #14 / #16.

## Prerequisites

Expand All @@ -47,7 +49,8 @@ The mock **derives the registered resources from the annotation, not from the co
- **Number of fake cards = the number of entries in the annotation array** (not the count value).
- Registered `...-memory` = **sum of `devmem`** over all entries.
- Registered `...-cores` / `...-core` = **sum of `devcore`** over all entries.
- The **count extended resource is only a health gate**: its value just needs to be `> 0`. It does **not** affect the registered memory/cores. By convention it is set to `cards x splits-per-card` (e.g. Ascend `2 x VDeviceCount(4) = 8`), but `1` would work equally well for the memory/cores to appear.
- For NVIDIA, Hygon and Ascend, the **count extended resource is only a health gate**: its value just needs to be `> 0`. It does **not** affect the registered memory/cores. By convention it is set to `cards x splits-per-card` (e.g. Ascend `2 x VDeviceCount(4) = 8`), but `1` would work equally well for the memory/cores to appear.
- Kunlun vXPU is fully annotation-derived: `vxpu = sum(count)` and `vxpu-memory = sum(devmem)` across entries whose `health` is `true`.

Annotation entry fields:

Expand All @@ -56,17 +59,17 @@ Annotation entry fields:
| `id` | unique device UUID (any string) |
| `devmem` | per-card memory in MB -- **summed** into `...-memory` |
| `devcore` | per-card cores. **NVIDIA/Hygon:** summed into `...-cores` (NVIDIA: percentage, 100 = a whole card). **Ascend:** ignored -- `huawei.com/<chip>-core` is percentage-based, registered as **100 per card**. |
| `count` | per-card split count (informational for the mock) |
| `count` | per-card split count. **Kunlun:** summed into `vxpu`; informational for the other mocks. |
| `type` | device model string |
| `health` | must be `true` to be counted |
| `health` | scheduler-visible health; only healthy Kunlun entries contribute to its mock resources |
| `index` | card index `0,1,2,...` (`0` may be omitted) |
| `numa`, `mode` | optional |

> **Worked example (Ascend, below):** the annotation has **2 entries**, each `devmem=32768`. So `...-memory = 2x32768 = 65536` and `...-core = 2x100 = 200` (Ascend `...-core` is **percentage-based**: a whole card is always **100**). The count resource `=8` is a separate health-gate value and is unrelated to these numbers.

## Usage by vendor

To mock one card you always need the **three pieces**: the vendor **config block** (in the `hami-scheduler-device` ConfigMap, gives the resource names), the **count extended resource** (passes the health gate), and the **node-register annotation** (describes the fake cards). Replace `<node>` below with your target node. Resource names follow the ConfigMap (HAMi defaults shown). See [Understanding the values](#understanding-the-values) for how the numbers are derived.
NVIDIA, Hygon and Ascend need the vendor **config block**, the **count extended resource** health gate and the **node-register annotation**. Kunlun vXPU needs its config block and node-register annotation, but not a separate status patch. Replace `<node>` below with your target node. Resource names follow the ConfigMap (HAMi defaults shown). See [Understanding the values](#understanding-the-values) for how the numbers are derived.

### NVIDIA (e.g. A100-80GB)

Expand Down Expand Up @@ -127,6 +130,33 @@ kubectl get node <node> -o json | jq '.status.allocatable|with_entries(select(.k
# expect: hygon.com/dcumem=32768, hygon.com/dcucores=200 (2 cards x devcore 100)
```

### Kunlun vXPU

- config block: `kunlun:` | annotation: `hami.io/node-register-xpu` (JSON)
- mock registers: `kunlunxin.com/vxpu`, `kunlunxin.com/vxpu-memory`
- no separate count-resource status patch is required

The default HAMi config is:

```yaml
kunlun:
resourceCountName: kunlunxin.com/xpu
resourceVCountName: kunlunxin.com/vxpu
resourceVMemoryName: kunlunxin.com/vxpu-memory
```

To model one healthy P800-OAM card with four 24 GiB vXPU shares:

```bash
kubectl annotate node <node> \
'hami.io/node-register-xpu=[{"id":"XPU-MOCK-0","count":4,"devmem":98304,"devcore":100,"type":"XPU","health":true}]'
# verify (~30s later)
kubectl get node <node> -o json | jq '.status.allocatable|with_entries(select(.key|test("kunlunxin.com")))'
# expect: kunlunxin.com/vxpu=4, kunlunxin.com/vxpu-memory=98304
```

Each healthy entry contributes its `count` value to `vxpu` and its `devmem` value in MiB to `vxpu-memory`. The mock also acknowledges HAMi Scheduler's `hami.io/node-handshake-xpu` probe, matching the real vXPU device-plugin startup protocol. Unhealthy entries remain visible to the HAMi scheduler through the annotation but do not contribute kubelet capacity, matching HAMi's `FitVXPU` health check.

## Ascend config compatibility (new vs legacy)

The Ascend `vnpus` config has two layouts and the plugin accepts **both**:
Expand Down Expand Up @@ -155,8 +185,9 @@ The new nested format is tried first; if that fails it falls back to the legacy
| Nvidia GPU | `nvidia.com/gpumem`, `nvidia.com/gpumem-percentage`, `nvidia.com/gpucores` |
| Hygon DCU | `hygon.com/dcumem`, `hygon.com/dcucores` (when `resourceCoreName` is set) |
| Ascend | `huawei.com/Ascend{chip}-memory`, `huawei.com/Ascend{chip}-core` (when `resourceCoreName` is set **and** the node is in `hami-vnpu-core` mode) |
| Kunlun vXPU | `kunlunxin.com/vxpu`, `kunlunxin.com/vxpu-memory` |

**Note:** If the counted memory is too large (e.g. > 120GB) it may display as 0. Set `memoryFactor` in the `hami-scheduler-device` ConfigMap (default 1).
**Note:** For device configs that expose `memoryFactor`, set it in the `hami-scheduler-device` ConfigMap if counted memory is too large (e.g. > 120GB) and displays as 0. Kunlun vXPU does not use `memoryFactor`; its `vxpu-memory` value is the direct sum of healthy entries' `devmem` values.

## Build

Expand Down
42 changes: 41 additions & 1 deletion internal/pkg/api/device/device.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ import (
"github.com/ccoveille/go-safecast"
corev1 "k8s.io/api/core/v1"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/kubernetes"
"k8s.io/klog/v2"
)

Expand Down Expand Up @@ -77,6 +79,12 @@ type Devices interface {
RunManager()
}

// HandshakeReporter is implemented by device families whose real device
// plugin acknowledges scheduler health probes through a Node annotation.
type HandshakeReporter interface {
HandshakeAnnotation() string
}

type ResourceNames struct {
ResourceCountName string
ResourceMemoryName string
Expand Down Expand Up @@ -114,18 +122,50 @@ func RunManagers() error {

func Register(l *mock.MockLister, dev Devices) {
nodeName := os.Getenv("NODE_NAME")
kubeClient := client.GetClient()
for {
node, err := client.GetClient().CoreV1().Nodes().Get(context.Background(), nodeName, v1.GetOptions{})
node, err := kubeClient.CoreV1().Nodes().Get(context.Background(), nodeName, v1.GetOptions{})
if err != nil {
klog.Error("Get node error", err.Error())
} else {
resourceMap := dev.GetResource(node)
l.SetResource(resourceMap)
if reporter, ok := dev.(HandshakeReporter); ok {
if err := reportHandshake(context.Background(), kubeClient, node, reporter.HandshakeAnnotation()); err != nil {
klog.ErrorS(err, "Failed to report device handshake", "node", nodeName, "device", dev.CommonWord())
}
}
}
time.Sleep(time.Second * 30)
}
}

func reportHandshake(ctx context.Context, kubeClient kubernetes.Interface, node *corev1.Node, annotation string) error {
patch, ok, err := handshakePatch(node, annotation)
if err != nil || !ok {
return err
}
_, err = kubeClient.CoreV1().Nodes().Patch(ctx, node.Name, types.JSONPatchType, patch, v1.PatchOptions{})
return err
}

func handshakePatch(node *corev1.Node, annotation string) ([]byte, bool, error) {
if node == nil || annotation == "" || node.Annotations == nil {
return nil, false, nil
}
request, ok := node.Annotations[annotation]
if !ok || !strings.HasPrefix(request, "Requesting_") {
return nil, false, nil
}

annotationPath := strings.NewReplacer("~", "~0", "/", "~1").Replace(annotation)
patch, err := json.Marshal([]map[string]string{
{"op": "test", "path": "/metadata/annotations/" + annotationPath, "value": request},
{"op": "replace", "path": "/metadata/annotations/" + annotationPath, "value": "Reported_" + strings.TrimPrefix(request, "Requesting_")},
})
return patch, true, err
}

func GetResourceName(name string) string {
if _, after, found := strings.Cut(name, "/"); found {
return after
Expand Down
44 changes: 43 additions & 1 deletion internal/pkg/api/device/device_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,13 @@ limitations under the License.
package device

import (
"encoding/json"
"errors"
"testing"

"gotest.tools/v3/assert"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func Test_DecodeNodeDevices(t *testing.T) {
Expand Down Expand Up @@ -100,4 +103,43 @@ func Test_DecodeNodeDevices(t *testing.T) {
}
})
}
}
}

func TestHandshakePatch(t *testing.T) {
const annotation = "hami.io/node-handshake-xpu"
node := &corev1.Node{ObjectMeta: metav1.ObjectMeta{
Name: "mock-node",
Annotations: map[string]string{annotation: "Requesting_2026-08-26 03:00:00"},
}}

patch, ok, err := handshakePatch(node, annotation)
assert.NilError(t, err)
assert.Assert(t, ok)
var operations []map[string]string
err = json.Unmarshal(patch, &operations)
assert.NilError(t, err)
assert.Equal(t, len(operations), 2)
assert.DeepEqual(t, operations[0], map[string]string{
"op": "test",
"path": "/metadata/annotations/hami.io~1node-handshake-xpu",
"value": "Requesting_2026-08-26 03:00:00",
})
assert.DeepEqual(t, operations[1], map[string]string{
"op": "replace",
"path": "/metadata/annotations/hami.io~1node-handshake-xpu",
"value": "Reported_2026-08-26 03:00:00",
})
}

func TestHandshakePatchIgnoresNonRequest(t *testing.T) {
const annotation = "hami.io/node-handshake-xpu"
node := &corev1.Node{ObjectMeta: metav1.ObjectMeta{
Name: "mock-node",
Annotations: map[string]string{annotation: "Reported_2026-08-26 03:00:00"},
}}

patch, ok, err := handshakePatch(node, annotation)
assert.NilError(t, err)
assert.Assert(t, !ok)
assert.Assert(t, patch == nil)
}
51 changes: 36 additions & 15 deletions internal/pkg/api/device/kunlun/device.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,15 @@ package kunlun
import (
"errors"
"fmt"
"strings"

"github.com/HAMi/mock-device-plugin/internal/pkg/api/device"
"github.com/HAMi/mock-device-plugin/internal/pkg/mock"
"github.com/kubevirt/device-plugin-manager/pkg/dpm"

//"github.com/kubevirt/device-plugin-manager/pkg/dpm"
corev1 "k8s.io/api/core/v1"
utilvalidation "k8s.io/apimachinery/pkg/util/validation"
"k8s.io/klog/v2"
)

Expand All @@ -36,30 +38,40 @@ const (
HandshakeAnnos = "hami.io/node-handshake-xpu"
)

var (
KunlunResourceVCount string
KunlunResourceVMemory string
)

type KunlunConfig struct {
ResourceCountName string `yaml:"resourceCountName"`
ResourceVCountName string `yaml:"resourceVCountName"`
ResourceVMemoryName string `yaml:"resourceVMemoryName"`
}

type KunlunVDevices struct {
config KunlunConfig
}

func InitKunlunVDevice(config KunlunConfig) *KunlunVDevices {
KunlunResourceVCount = config.ResourceVCountName
KunlunResourceVMemory = config.ResourceVMemoryName
return &KunlunVDevices{}
vCountVendor, vCountResource, vCountQualified := strings.Cut(config.ResourceVCountName, "/")
vMemoryVendor, vMemoryResource, vMemoryQualified := strings.Cut(config.ResourceVMemoryName, "/")
if config.ResourceVCountName == "" || config.ResourceVMemoryName == "" ||
config.ResourceVCountName == config.ResourceVMemoryName ||
!vCountQualified || !vMemoryQualified ||
vCountVendor == "" || vMemoryVendor == "" ||
vCountResource == "" || vMemoryResource == "" ||
len(utilvalidation.IsQualifiedName(config.ResourceVCountName)) > 0 ||
len(utilvalidation.IsQualifiedName(config.ResourceVMemoryName)) > 0 ||
vCountVendor != vMemoryVendor {
return nil
}
return &KunlunVDevices{config: config}
}

func (dev *KunlunVDevices) CommonWord() string {
return XPUDevice
}

func (dev *KunlunVDevices) HandshakeAnnotation() string {
return HandshakeAnnos
}

func (dev *KunlunVDevices) GetNodeDevices(n *corev1.Node) ([]*device.DeviceInfo, error) {
anno, ok := n.Annotations[RegisterAnnos]
if !ok {
Expand All @@ -70,8 +82,14 @@ func (dev *KunlunVDevices) GetNodeDevices(n *corev1.Node) ([]*device.DeviceInfo,
klog.ErrorS(err, "failed to unmarshal node devices", "node", n.Name, "device annotation", anno)
return []*device.DeviceInfo{}, err
}
for idx := range nodeDevices {
nodeDevices[idx].DeviceVendor = dev.CommonWord()
for idx, nodeDevice := range nodeDevices {
if nodeDevice == nil {
return []*device.DeviceInfo{}, fmt.Errorf("device %d in %s is null", idx, RegisterAnnos)
}
if nodeDevice.Count < 0 || nodeDevice.Devmem < 0 || nodeDevice.Devcore < 0 {
return []*device.DeviceInfo{}, fmt.Errorf("device %d in %s has negative capacity", idx, RegisterAnnos)
}
nodeDevice.DeviceVendor = dev.CommonWord()
}
if len(nodeDevices) == 0 {
klog.InfoS("no gpu device found", "node", n.Name, "device annotation", anno)
Expand All @@ -81,8 +99,8 @@ func (dev *KunlunVDevices) GetNodeDevices(n *corev1.Node) ([]*device.DeviceInfo,
}

func (dev *KunlunVDevices) GetResource(n *corev1.Node) map[string]int {
memoryResourceName := device.GetResourceName(KunlunResourceVMemory)
vCountResourceName := device.GetResourceName(KunlunResourceVCount)
memoryResourceName := device.GetResourceName(dev.config.ResourceVMemoryName)
vCountResourceName := device.GetResourceName(dev.config.ResourceVCountName)
resourceMap := map[string]int{
memoryResourceName: 0,
vCountResourceName: 0,
Expand All @@ -93,16 +111,19 @@ func (dev *KunlunVDevices) GetResource(n *corev1.Node) map[string]int {
return resourceMap
}
for _, val := range devInfos {
resourceMap[vCountResourceName] += int(val.Devcore)
if !val.Health {
continue
}
resourceMap[vCountResourceName] += int(val.Count)
resourceMap[memoryResourceName] += int(val.Devmem)
}
klog.InfoS("Add resource", vCountResourceName, resourceMap[vCountResourceName], memoryResourceName, resourceMap[memoryResourceName])
return resourceMap
}

func (dev *KunlunVDevices) RunManager() {
lmock := mock.NewMockLister(device.GetVendorName(KunlunResourceVCount))
device.Register(lmock, dev)
lmock := mock.NewMockLister(device.GetVendorName(dev.config.ResourceVCountName))
go device.Register(lmock, dev)
mockmanager := dpm.NewManager(lmock)
klog.Infof("Running mocking dp: %s", dev.CommonWord())
mockmanager.Run()
Expand Down
Loading