diff --git a/README.md b/README.md
index 8d8490d..67e7ae0 100644
--- a/README.md
+++ b/README.md
@@ -263,6 +263,7 @@ Beyond the Vacation Planner samples, the repository includes standalone tutorial
| [keda/queue-storage](tutorials/keda/queue-storage/) | Scales a consumer on an [Azure Storage](https://learn.microsoft.com/en-us/azure/storage/queues/storage-queues-introduction) queue with the [`azure-queue`](https://keda.sh/docs/2.20/scalers/azure-storage-queue/) scaler. The only tutorial that uses workload identity end to end: the scaler, the producer and the consumer all authenticate as the shared identity, so no data-plane secret exists anywhere. |
| [keda/event-hubs](tutorials/keda/event-hubs/) | Scales a consumer on an [Azure Event Hubs](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about) hub with the [`azure-eventhub`](https://keda.sh/docs/2.20/scalers/azure-event-hub/) scaler, whose backlog is not a queue depth but the distance between the last enqueued event and the consumer group's blob checkpoints. |
| [key-vault-csi-driver](tutorials/key-vault-csi-driver/) | Mounts secrets from [Azure Key Vault](https://learn.microsoft.com/en-us/azure/key-vault/general/overview) into a pod with the [Azure Key Vault provider for Secrets Store CSI Driver](https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver), demonstrating both the Microsoft Entra Workload ID and the user-assigned managed identity access modes. |
+| [gpu-node-pools](tutorials/gpu-node-pools/) | Runs GPU training jobs on a [Spot](https://learn.microsoft.com/en-us/azure/aks/spot-node-pool) NVIDIA GPU node pool that scales from zero: [KEDA](https://keda.sh/docs/2.20/concepts/scaling-jobs/) starts a job for each training request on an Azure Service Bus queue, the [cluster autoscaler](https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler-overview) adds a GPU node for its pod, the [NVIDIA device plugin](https://learn.microsoft.com/en-us/azure/aks/use-nvidia-gpu) advertises the GPU, and the job trains a small PyTorch model, authenticating with Microsoft Entra Workload ID. A second run simulates the eviction of the Spot node with `az vmss simulate-eviction`, and the job completes on a new node. On the emulator, GPU nodes advertise their GPUs without a GPU behind them, so the same manifests run on a machine without one. |
| [terraform/tags-labels-taints](tutorials/terraform/tags-labels-taints/README.md) | Deploys a modular, feature-rich AKS stack with [Terraform](https://developer.hashicorp.com/terraform), focused on classifying and steering workloads across agent pools with [Azure resource tags](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/tag-resources), [Kubernetes node labels](https://learn.microsoft.com/en-us/azure/aks/use-labels), and [node taints](https://learn.microsoft.com/en-us/azure/aks/use-node-taints). Unlike the other tutorials, it enables a wide set of cluster features (OIDC issuer, Microsoft Entra Workload ID, Azure RBAC, Key Vault Secrets Provider, Vertical Pod Autoscaler, Managed Gateway API, Container Insights) and deploys additional Azure resources (Log Analytics workspace, virtual network, user-assigned managed identity, Container Registry, Key Vault), then validates the tags, labels, and taints end-to-end through both the ARM and Kubernetes APIs. |
| [bicep/tags-labels-taints](tutorials/bicep/tags-labels-taints/README.md) | The same modular, feature-rich AKS stack built with [Bicep](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview): agent-pool tags, node labels, and taints, the same broad set of cluster features, and the same additional Azure resources, with parameters and outputs that mirror the Terraform tutorial one-to-one. |
diff --git a/tutorials/gpu-node-pools/README.md b/tutorials/gpu-node-pools/README.md
new file mode 100644
index 0000000..3ccce61
--- /dev/null
+++ b/tutorials/gpu-node-pools/README.md
@@ -0,0 +1,127 @@
+## Run GPU training jobs on a Spot GPU node pool that scales from zero
+
+[Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/what-is-aks) runs GPU workloads on node pools of [NVIDIA GPU VM sizes](https://learn.microsoft.com/en-us/azure/aks/use-nvidia-gpu). A pool of [Spot VMs](https://learn.microsoft.com/en-us/azure/aks/spot-node-pool) makes them much cheaper, at the price of evictions whenever Azure needs the capacity back, and the [cluster autoscaler](https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler-overview) lets such a pool start with no node at all and grow only when a pod asks for a GPU.
+
+This tutorial adds a Spot GPU node pool that scales from zero to a cluster, and runs training jobs on it: [KEDA](https://keda.sh/) starts a Kubernetes Job for each training request on an [Azure Service Bus](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview) queue, the job's pod asks for one GPU, the cluster autoscaler adds a GPU node for it, and the pod trains a small [PyTorch](https://pytorch.org/) model. A second run simulates the eviction of the Spot node while a job trains, and the job completes on the node that replaces it. It runs unchanged against Azure in the cloud and against the [LocalStack for Azure](https://docs.localstack.cloud/azure/) emulator.
+
+On Azure the job trains on the node's NVIDIA GPU. On the emulator, GPU nodes advertise the GPUs of their VM size without a GPU behind them: the pods schedule exactly as on AKS, and PyTorch finds no CUDA device, so the job trains on the CPU. That is enough to develop and test the scheduling, the scaling and the eviction handling on a machine without a GPU. See [Run GPU workloads](https://docs.localstack.cloud/azure/services/aks/#run-gpu-workloads) in the emulator's documentation.
+
+> **Running on LocalStack?** Install the [lstk CLI](https://docs.localstack.cloud/aws/developer-tools/running-localstack/lstk/) and run `lstk az start-interception` to route Azure CLI calls to the emulator. See [Run against LocalStack](../../README.md#run-against-localstack) for the full setup.
+
+## Architecture
+
+The GPU node pool joins the cluster's system and user pools with no node. KEDA watches the queue and starts a job for each message; the job's pod is the only workload that tolerates the GPU pool's taints and asks for a GPU, so the cluster autoscaler adds a node to that pool, and only to that pool, to run it.
+
+```mermaid
+%%{init: {'themeVariables': {'clusterBkg': 'transparent', 'clusterBorder': '#8c8c8c'}}}%%
+flowchart LR
+ subgraph aks["Azure Kubernetes Service cluster"]
+ subgraph kubesystem["kube-system"]
+ operator["keda-operator"]
+ autoscaler["cluster autoscaler"]
+ end
+ subgraph gpupool["gpupool: Spot Standard_NV18ads_A10_v5, 0 to 1 node"]
+ plugin["nvidia-device-plugin
DaemonSet pod"]
+ pod["training job pod
nvidia.com/gpu: 1"]
+ end
+ subgraph trainingns["namespace gpu-training"]
+ scaledjob["gpu-training
ScaledJob"]
+ job["gpu-training-xxxxx
Job"]
+ end
+ end
+
+ subgraph azure["Azure"]
+ subgraph sbns["Service Bus namespace"]
+ queue(["training-requests
queue"])
+ end
+ end
+
+ sender["05-run-training-job.sh
06-simulate-eviction.sh"] -->|"sends a training request"| queue
+ operator -->|"reads the message count
with workload identity"| queue
+ scaledjob -.->|"read by"| operator
+ operator -->|"starts one job per message"| job
+ job -->|"creates"| pod
+ autoscaler -->|"adds a node for the pending pod"| gpupool
+ plugin -->|"advertises the GPU"| pod
+ pod -->|"receives the message
with workload identity"| queue
+```
+
+## Prerequisites
+
+- An AKS cluster reachable through `kubectl`, created with [scripts/01-user-assigned-managed-identity.sh](../../scripts/01-user-assigned-managed-identity.sh) (or the system-assigned variant), which enables the KEDA add-on, the OIDC issuer and workload identity. The values in [scripts/00-variables.sh](scripts/00-variables.sh) (cluster `local-aks-test`, resource group `local-rg`, location `ItalyNorth`) must match the cluster those scripts create; edit them if you changed their `prefix`, `suffix` or `location`.
+- On Azure, Spot capacity and quota for one `Standard_NV18ads_A10_v5` node: 18 vCPUs of the region's Spot quota, which every VM family shares. Check it with `az vm list-usage --location italynorth --query "[?name.value=='lowPriorityCores']" --output table`. When Azure has no Spot capacity for the size, the job's pod stays pending, and `05-run-training-job.sh` prints the cluster autoscaler's status when it gives up waiting.
+- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) (`az`), authenticated with `az login`. Running against LocalStack? Point the CLI at the emulator first, then run the same scripts unchanged.
+- [kubectl](https://kubernetes.io/docs/tasks/tools/), [yq](https://github.com/mikefarah/yq), used to fill the manifests in with the values from `00-variables.sh`, and [jq](https://jqlang.org/), `curl` and `openssl`, used to send the training requests and to print the GPU node.
+
+## How to run it
+
+Run the numbered scripts in order. Each one sources [00-variables.sh](scripts/00-variables.sh), and the setup scripts (`01` to `04`) are idempotent, so they can be re-run safely.
+
+| Script | What it does | Expected result |
+| --- | --- | --- |
+| `00-variables.sh` | Variables for this tutorial (the GPU node pool, the device plugin, Service Bus, the managed identity and the training job). Sourced by every script, never run directly. | Variables set. |
+| `functions.sh` | The functions the scripts share: reading the Service Bus endpoints, sending a training request, and following the jobs. Sourced by `04` to `06`. | Functions defined. |
+| `01-add-gpu-node-pool.sh` | Verifies the cluster exists with its OIDC issuer, enables the KEDA add-on if needed, merges the cluster credentials into `kubeconfig` (run it first), and adds the Spot GPU node pool with no node and the cluster autoscaler from 0 to 1 node. | The pool exists with `count: 0`, `scaleSetPriority: Spot` and the Spot label and taint AKS adds, and no node. |
+| `02-deploy-device-plugin.sh` | Deploys the NVIDIA device plugin DaemonSet from Microsoft's guide, selecting the nodes AKS labels `accelerator=nvidia` and tolerating the GPU pool's taints. | The DaemonSet exists and schedules no pod until a GPU node joins. |
+| `03-create-resources.sh` | Creates (or reuses) the Service Bus namespace, the `training-requests` queue and the managed identity, grants the identity the **Azure Service Bus Data Owner** role on the namespace, federates it with the KEDA operator and with the training job's service account, and binds the operator to it. | The resources and the role assignment exist. The operator restart is skipped when the binding is already in place. |
+| `04-deploy-scaled-job.sh` | Creates the `gpu-training` namespace, the job's service account bound to the identity, the training script as a config map, the `TriggerAuthentication` and the `ScaledJob`, deriving the Service Bus endpoints at apply time. | The `ScaledJob` is `Ready`. |
+| `05-run-training-job.sh` | Sends a training request, waits for KEDA to start a job and for the cluster autoscaler to add a GPU node for its pod, prints the node's AKS labels, taints and GPU, and waits for the job to complete. | The job completes; its log says whether PyTorch found CUDA and how many steps it trained. |
+| `06-simulate-eviction.sh` | Sends a longer training request and, once its pod reports training progress, simulates the eviction of its Spot node with `az vmss simulate-eviction`, follows the node until AKS deletes it, and waits for the job to complete on a new node. | The node is tainted for the eviction and deleted, and the job completes on another node. |
+| `07-cleanup.sh` | Deletes the two namespaces, the GPU node pool, the Service Bus namespace with the identity's role on it, and the identity's federation with the training job. | Everything the tutorial created is gone; the identity and the operator binding stay for the KEDA tutorials, which share them. |
+
+```bash
+cd gpu-node-pools/scripts
+./01-add-gpu-node-pool.sh
+./02-deploy-device-plugin.sh
+./03-create-resources.sh
+./04-deploy-scaled-job.sh
+./05-run-training-job.sh
+./06-simulate-eviction.sh
+# ./07-cleanup.sh # when finished
+```
+
+### Kubernetes manifests
+
+Every manifest in [scripts/](scripts/) is valid, applyable YAML whose environment-specific fields are empty strings; the scripts fill them in with `yq`, which is why the same files work against the emulator and against Azure without edits.
+
+| Manifest | What it creates | Filled in at apply time |
+| --- | --- | --- |
+| `nvidia-device-plugin.yml` | The `nvidia-device-plugin-daemonset` DaemonSet, with `FAIL_ON_INIT_ERROR=false` so that the plugin waits instead of exiting on a node without a usable GPU. | The name, the namespace, the image and the tolerations. |
+| `serviceaccount.yml` | `gpu-training`, the job's service account, annotated with the managed identity's client id. | The name, the namespace and the client id. |
+| `triggerauthentication.yml` | `gpu-training-trigger-auth`, which tells KEDA to read the queue as the managed identity with workload identity. | The name, the namespace and the client id. |
+| `scaledjob.yml` | `gpu-training`, the `ScaledJob`: one job per message, a pod that asks for one GPU, selects the GPU pool's nodes and tolerates their taints. | The image, the Service Bus host and custom endpoint, the queue, the resources, the tolerations and the scaler's `endpointSuffix`. |
+
+### Application
+
+| Path | What it is |
+| --- | --- |
+| `src/train.py` | The training job. It receives one message with `DefaultAzureCredential`, which signs in with the pod's workload identity, reads `train_seconds` from it, and trains a small model with PyTorch on `cuda` when `torch.cuda.is_available()`, on `cpu` otherwise, printing its progress every 30 seconds. |
+
+The job runs the official PyTorch image with CUDA (`pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime`, about 3 GB, so the first pod on each new node spends a few minutes pulling it), and installs `azure-servicebus`, `azure-identity` and `websocket-client` into `/tmp` when it starts. The pod runs as a non-root user with a read-only root file system, so the libraries go to an `emptyDir` volume.
+
+## How it works
+
+**A pool that scales from zero.** The GPU pool is created with `--node-count 0 --min-count 0 --max-count 1 --enable-cluster-autoscaler`. While no pod needs it, it has no node and costs nothing. The job's pod requests `nvidia.com/gpu: 1`, selects `workload: gpu` and tolerates the pool's `sku=gpu:NoSchedule` taint and the Spot taint, so no existing node can run it. The cluster autoscaler checks which pools could, finds the GPU pool, and adds a node to it. Once the job is done, it removes the node again after the node has been unneeded for a while: 10 minutes by default on AKS (the `scale-down-unneeded-time` of the [autoscaler profile](https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler#use-the-cluster-autoscaler-profile)), 30 seconds on the emulator.
+
+**The labels and taints AKS sets.** AKS labels every node of an NVIDIA GPU size `accelerator=nvidia`, and every node of a Spot pool `kubernetes.azure.com/scalesetpriority=spot`, with the matching `NoSchedule` taint, whether or not the pool declares them. The device plugin DaemonSet selects the first, so it runs only on GPU nodes; the job tolerates the second, since only workloads that can handle an eviction belong on Spot nodes.
+
+**Why the device plugin.** AKS installs the NVIDIA driver on the nodes of a GPU size, but Kubernetes learns about their GPUs only from a device plugin. Until the plugin runs on a node, the node advertises no `nvidia.com/gpu`, and a pod that asks for one can't be scheduled there. This tutorial deploys the plugin's DaemonSet as Microsoft's guide does. AKS can also install it for you on [AKS-managed GPU node pools](https://learn.microsoft.com/en-us/azure/aks/aks-managed-gpu-nodes).
+
+**One job per message.** The `ScaledJob` declares an `azure-servicebus` trigger with `messageCount: "1"`: every polling interval, KEDA starts as many jobs as there are messages waiting, up to `maxReplicaCount`. Each job's pod receives one message and deletes it, so a request is processed once.
+
+**The eviction.** `az vmss simulate-eviction` evicts a Spot instance the way Azure does when it needs the capacity back. AKS taints the node `remediator.kubernetes.azure.com/unschedulable:NoSchedule`, evicts its pods and deletes the node with its scale set instance. The job's pod is gone, so the Job starts a replacement, which stays pending until the cluster autoscaler adds a new node, and completes there. The first pod had already received and deleted the message, so the replacement pod finds the queue empty and trains for the default time (`JOB_DEFAULT_TRAIN_SECONDS`). A production job would rather keep its message locked until it finishes, or checkpoint its training, so that a replacement pod picks up where the evicted one stopped.
+
+**How KEDA and the job authenticate.** The managed identity is federated with two service accounts: `keda-operator` in `kube-system`, so the operator can read the queue's message count, and the job's `gpu-training`, so the pod can receive a message. Both exchange their projected service account token for a Microsoft Entra token for the identity, and the **Azure Service Bus Data Owner** role on the namespace, granted on the identity's principal (object) id, authorizes them. No secret is involved, except the namespace key the scripts use on your machine to send the training requests over the HTTPS API.
+
+**Why the job may use a custom endpoint.** On Azure the job connects to `{namespace}.servicebus.windows.net` over AMQP on port 5671, and `SERVICE_BUS_CUSTOM_ENDPOINT` is empty. On the emulator, the namespace's AMQP endpoint has its own port, which [04-deploy-scaled-job.sh](scripts/04-deploy-scaled-job.sh) reads from the namespace's `serviceBusEndpoint` and passes as a custom endpoint address. With a custom endpoint, the Python Service Bus library uses AMQP over WebSockets, which needs the `websocket-client` package: that is why the job installs it. The KEDA scaler, for its part, reads the message count over the HTTPS API, served on the Azure Resource Manager port: 443 on Azure, the gateway port on the emulator. Hence the trigger's `cloud: Private` and its derived `endpointSuffix`, as in the [KEDA Service Bus tutorial](../keda/service-bus/).
+
+## Resources
+
+- [Use GPUs for compute-intensive workloads on AKS](https://learn.microsoft.com/en-us/azure/aks/use-nvidia-gpu)
+- [Add an Azure Spot node pool to an AKS cluster](https://learn.microsoft.com/en-us/azure/aks/spot-node-pool)
+- [Cluster autoscaling in AKS](https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler-overview)
+- [Simulate an eviction of an Azure Spot VM](https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/use-spot#simulate-an-eviction)
+- [KEDA ScaledJobs](https://keda.sh/docs/2.20/concepts/scaling-jobs/) and the [Azure Service Bus scaler](https://keda.sh/docs/2.20/scalers/azure-service-bus/)
+- [Securely scale your applications using the KEDA add-on and workload identity](https://learn.microsoft.com/en-us/azure/aks/keda-workload-identity)
+- [NVIDIA device plugin for Kubernetes](https://github.com/NVIDIA/k8s-device-plugin)
+- [LocalStack for Azure: Run GPU workloads on AKS](https://docs.localstack.cloud/azure/services/aks/#run-gpu-workloads)
diff --git a/tutorials/gpu-node-pools/scripts/00-variables.sh b/tutorials/gpu-node-pools/scripts/00-variables.sh
new file mode 100644
index 0000000..3804f67
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/00-variables.sh
@@ -0,0 +1,101 @@
+# Variables for the GPU node pools tutorial.
+#
+# Source this file from every script with: source ./00-variables.sh
+#
+# The AKS cluster is created by scripts/01-user-assigned-managed-identity.sh; these values MUST match that script
+# (prefix "local", suffix "test", location "ItalyNorth") so the tutorial adds its GPU node pool to the cluster it
+# creates instead of standing up its own.
+
+# Azure Kubernetes Service (AKS)
+PREFIX="local"
+SUFFIX="test"
+AKS_NAME="${PREFIX}-aks-${SUFFIX}"
+AKS_RESOURCE_GROUP_NAME="${PREFIX}-rg"
+LOCATION="ItalyNorth"
+
+# The Spot GPU node pool. Standard_NV18ads_A10_v5 runs half an NVIDIA A10 GPU, which Kubernetes sees as one
+# nvidia.com/gpu, with 18 vCPUs. Spot VMs count against the region's Spot vCPU quota (lowPriorityCores) whatever their
+# VM family, and Azure starts one only when it has Spot capacity. Check the quota with:
+# az vm list-usage --location italynorth --query "[?name.value=='lowPriorityCores']" --output table
+# The pool starts with no node: the cluster autoscaler adds one when a pod asks for a GPU and removes it when it is idle.
+GPU_NODE_POOL_NAME="gpupool"
+GPU_NODE_VM_SIZE="Standard_NV18ads_A10_v5"
+GPU_NODE_OS_SKU="AzureLinux"
+GPU_NODE_MIN_COUNT=0
+GPU_NODE_MAX_COUNT=1
+GPU_NODE_PRIORITY="Spot"
+GPU_NODE_EVICTION_POLICY="Delete"
+# -1 pays up to the on-demand price, so Azure evicts the node when it needs the capacity back, never for its price
+GPU_NODE_SPOT_MAX_PRICE=-1
+# The jobs select the pool's nodes by this label, and only pods that tolerate this taint run on them
+GPU_NODE_LABEL_KEY="workload"
+GPU_NODE_LABEL_VALUE="gpu"
+GPU_NODE_TAINT_KEY="sku"
+GPU_NODE_TAINT_VALUE="gpu"
+# AKS adds this label and this NoSchedule taint to every node of a Spot node pool
+SPOT_KEY="kubernetes.azure.com/scalesetpriority"
+SPOT_VALUE="spot"
+# AKS labels the nodes of NVIDIA GPU VM sizes with it, and the device plugin DaemonSet selects them by it
+GPU_NODE_ACCELERATOR_LABEL="accelerator=nvidia"
+
+# The NVIDIA device plugin DaemonSet, from Microsoft's guide to installing it by hand. AKS installs the NVIDIA driver
+# on the GPU nodes, and the plugin advertises their GPUs to Kubernetes as nvidia.com/gpu.
+DEVICE_PLUGIN_NAMESPACE="gpu-resources"
+DEVICE_PLUGIN_NAME="nvidia-device-plugin-daemonset"
+DEVICE_PLUGIN_IMAGE="nvcr.io/nvidia/k8s-device-plugin:v0.18.0"
+
+# Azure Service Bus. Namespace names are global on Azure, between 6 and 50 characters, and can't end in -sb or -mgmt.
+SERVICE_BUS_NAMESPACE_NAME="${PREFIX}-gpu-servicebus-${SUFFIX}"
+SERVICE_BUS_SKU="Standard"
+SERVICE_BUS_QUEUE_NAME="training-requests"
+SERVICE_BUS_AUTHORIZATION_RULE="RootManageSharedAccessKey"
+# The role the KEDA operator needs to read the queue's message count, and the job to receive a message, granted on the
+# namespace to the managed identity's principal id. https://learn.microsoft.com/en-us/azure/aks/keda-workload-identity
+ROLE="Azure Service Bus Data Owner"
+
+# The user-assigned managed identity the KEDA operator and the training job sign in with. It is the identity of the KEDA
+# tutorials: the keda-operator service account exists once, in kube-system, and is bound to one identity, so sharing it
+# lets this tutorial and the KEDA ones run on the same cluster. Whichever runs first creates it and binds the operator.
+MANAGED_IDENTITY_NAME="${PREFIX}-keda-uami-${SUFFIX}"
+FEDERATED_IDENTITY_AUDIENCE="api://AzureADTokenExchange"
+FEDERATED_IDENTITY_NAME_KEDA="keda-operator"
+FEDERATED_IDENTITY_NAME_JOB="gpu-training-job"
+KEDA_NAMESPACE="kube-system"
+KEDA_OPERATOR_DEPLOYMENT="keda-operator"
+KEDA_OPERATOR_SERVICE_ACCOUNT="keda-operator"
+KEDA_SCALED_JOB_CRD="scaledjobs.keda.sh"
+
+# The training job. KEDA's ScaledJob creates one Kubernetes Job for each message on the queue. The job's pod asks for
+# one GPU, receives one message with workload identity, reads how many seconds to train from it, and trains a small
+# PyTorch model on the GPU when PyTorch finds one, on the CPU otherwise. The image is the official PyTorch runtime with
+# CUDA, about 3 GB; the pod installs the Service Bus and identity libraries when it starts.
+NAMESPACE="gpu-training"
+SERVICE_ACCOUNT_NAME="gpu-training"
+SCRIPT_CONFIG_MAP_NAME="gpu-training-script"
+TRIGGER_AUTHENTICATION_NAME="gpu-training-trigger-auth"
+SCALED_JOB_NAME="gpu-training"
+JOB_IMAGE="pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime"
+# websocket-client: with a custom endpoint address, which the emulator needs (see functions.sh), the Python Service Bus
+# library uses AMQP over WebSockets, and that transport needs websocket-client. On Azure the job doesn't load it.
+JOB_PYTHON_PACKAGES="azure-servicebus==7.13.0 azure-identity==1.19.0 websocket-client==1.9.2"
+JOB_CPU_REQUEST="1"
+JOB_MEMORY_REQUEST="2Gi"
+JOB_MEMORY_LIMIT="4Gi"
+JOB_BACKOFF_LIMIT=3
+# Seconds the job trains when its message doesn't say
+JOB_DEFAULT_TRAIN_SECONDS=60
+SCALED_JOB_POLLING_INTERVAL=10
+SCALED_JOB_MAX_REPLICAS=2
+
+# The two runs of the tutorial: a short job, and a long one whose Spot node is evicted while it trains
+FIRST_JOB_TRAIN_SECONDS=60
+EVICTED_JOB_TRAIN_SECONDS=600
+
+# Waits poll every POLL_INTERVAL seconds and give up after their timeout, in seconds. On Azure a Spot GPU node takes a
+# few minutes to start and to install the NVIDIA driver, and the PyTorch image takes a few minutes to pull.
+POLL_INTERVAL=10
+JOB_START_TIMEOUT=1800
+# A running pod installs its libraries and receives its message before it trains, then logs its progress every 30 s
+TRAINING_PROGRESS_TIMEOUT=600
+JOB_TIMEOUT=2400
+NODE_DELETION_TIMEOUT=900
diff --git a/tutorials/gpu-node-pools/scripts/01-add-gpu-node-pool.sh b/tutorials/gpu-node-pools/scripts/01-add-gpu-node-pool.sh
new file mode 100755
index 0000000..dcf7b29
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/01-add-gpu-node-pool.sh
@@ -0,0 +1,115 @@
+#!/bin/bash
+
+# Step 1: make sure the cluster runs the KEDA add-on, merge its credentials into kubeconfig, and add a Spot GPU node
+# pool that starts with no node and that the cluster autoscaler grows to one node when a pod asks for a GPU.
+#
+# The cluster-creation scripts already pass --enable-keda, --enable-oidc-issuer and --enable-workload-identity, so on
+# such a cluster this script only adds the node pool. The pool's VM size and priority can't change after it is
+# created, so an existing pool with other values stops the script rather than being reused.
+# https://learn.microsoft.com/en-us/azure/aks/use-nvidia-gpu
+# https://learn.microsoft.com/en-us/azure/aks/spot-node-pool
+
+# Variables
+source ./00-variables.sh
+
+# Make sure the AKS cluster exists (it is created by scripts/01-user-assigned-managed-identity.sh)
+OIDC_ISSUER_ENABLED=$(az aks show \
+ --name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query oidcIssuerProfile.enabled \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ -z $OIDC_ISSUER_ENABLED ]]; then
+ echo "The [$AKS_NAME] AKS cluster doesn't exist in the [$AKS_RESOURCE_GROUP_NAME] resource group"
+ echo "Create the cluster first with scripts/01-user-assigned-managed-identity.sh"
+ exit 1
+fi
+if [[ $OIDC_ISSUER_ENABLED != "true" ]]; then
+ echo "The [$AKS_NAME] AKS cluster has no OIDC issuer, which workload identity needs"
+ echo "Create the cluster with scripts/01-user-assigned-managed-identity.sh, which enables it"
+ exit 1
+fi
+
+# Enable the KEDA add-on only if it is not enabled yet
+echo "Checking if the KEDA add-on is enabled in the [$AKS_NAME] AKS cluster..."
+KEDA_ENABLED=$(az aks show \
+ --name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query workloadAutoScalerProfile.keda.enabled \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ $KEDA_ENABLED == "true" ]]; then
+ echo "The KEDA add-on is already enabled in the [$AKS_NAME] AKS cluster"
+else
+ echo "Enabling the KEDA add-on in the [$AKS_NAME] AKS cluster..."
+ if ! az aks update \
+ --name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --enable-keda \
+ --only-show-errors 1>/dev/null; then
+ echo "Failed to enable the KEDA add-on in the [$AKS_NAME] AKS cluster"
+ exit 1
+ fi
+ echo "The KEDA add-on was successfully enabled in the [$AKS_NAME] AKS cluster"
+fi
+
+# Merge the cluster credentials into kubeconfig and set it as the current context
+echo "Merging credentials for the [$AKS_NAME] AKS cluster into kubeconfig..."
+if ! az aks get-credentials \
+ --name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --overwrite-existing \
+ --only-show-errors; then
+ echo "Failed to merge the credentials for the [$AKS_NAME] AKS cluster"
+ exit 1
+fi
+
+# Add the Spot GPU node pool unless it exists
+echo "Checking if the [$GPU_NODE_POOL_NAME] node pool exists in the [$AKS_NAME] AKS cluster..."
+EXISTING_POOL=$(az aks nodepool show \
+ --name "$GPU_NODE_POOL_NAME" \
+ --cluster-name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query "[vmSize, scaleSetPriority]" \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r' | tr '\t\n' ' ')
+if [[ -n $EXISTING_POOL ]]; then
+ if [[ $EXISTING_POOL != "$GPU_NODE_VM_SIZE $GPU_NODE_PRIORITY " ]]; then
+ echo "The [$GPU_NODE_POOL_NAME] node pool exists with VM size and priority [$EXISTING_POOL], not [$GPU_NODE_VM_SIZE $GPU_NODE_PRIORITY]"
+ echo "Delete it with 07-cleanup.sh, or change GPU_NODE_POOL_NAME in 00-variables.sh"
+ exit 1
+ fi
+ echo "The [$GPU_NODE_POOL_NAME] node pool already exists"
+else
+ echo "Adding the [$GPU_NODE_POOL_NAME] Spot node pool of [$GPU_NODE_VM_SIZE] VMs, with $GPU_NODE_MIN_COUNT to $GPU_NODE_MAX_COUNT nodes, to the [$AKS_NAME] AKS cluster..."
+ if ! az aks nodepool add \
+ --name "$GPU_NODE_POOL_NAME" \
+ --cluster-name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --mode User \
+ --node-vm-size "$GPU_NODE_VM_SIZE" \
+ --os-sku "$GPU_NODE_OS_SKU" \
+ --priority "$GPU_NODE_PRIORITY" \
+ --eviction-policy "$GPU_NODE_EVICTION_POLICY" \
+ --spot-max-price "$GPU_NODE_SPOT_MAX_PRICE" \
+ --enable-cluster-autoscaler \
+ --node-count "$GPU_NODE_MIN_COUNT" \
+ --min-count "$GPU_NODE_MIN_COUNT" \
+ --max-count "$GPU_NODE_MAX_COUNT" \
+ --labels "$GPU_NODE_LABEL_KEY=$GPU_NODE_LABEL_VALUE" \
+ --node-taints "$GPU_NODE_TAINT_KEY=$GPU_NODE_TAINT_VALUE:NoSchedule" \
+ --only-show-errors 1>/dev/null; then
+ echo "Failed to add the [$GPU_NODE_POOL_NAME] node pool to the [$AKS_NAME] AKS cluster"
+ exit 1
+ fi
+ echo "The [$GPU_NODE_POOL_NAME] node pool was successfully added to the [$AKS_NAME] AKS cluster"
+fi
+
+# Print the node pool as AKS stores it: AKS adds the Spot label and taint, and reports the GPU driver it installs
+az aks nodepool show \
+ --name "$GPU_NODE_POOL_NAME" \
+ --cluster-name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query "{name: name, vmSize: vmSize, count: count, minCount: minCount, maxCount: maxCount, scaleSetPriority: scaleSetPriority, scaleSetEvictionPolicy: scaleSetEvictionPolicy, spotMaxPrice: spotMaxPrice, gpuProfile: gpuProfile, nodeLabels: nodeLabels, nodeTaints: nodeTaints}" \
+ --output json \
+ --only-show-errors
diff --git a/tutorials/gpu-node-pools/scripts/02-deploy-device-plugin.sh b/tutorials/gpu-node-pools/scripts/02-deploy-device-plugin.sh
new file mode 100755
index 0000000..ca51ff3
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/02-deploy-device-plugin.sh
@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# Step 2: deploy the NVIDIA device plugin DaemonSet. AKS installs the NVIDIA driver on the nodes of the GPU pool, and the
+# plugin advertises their GPUs to Kubernetes as nvidia.com/gpu: until it runs on a node, the scheduler sees no GPU
+# there. The DaemonSet selects the nodes AKS labels accelerator=nvidia and tolerates the GPU pool's taints, so it
+# starts on each GPU node the cluster autoscaler adds. The GPU pool has no node yet, so it schedules no pod for now.
+# https://learn.microsoft.com/en-us/azure/aks/use-nvidia-gpu#manually-install-the-nvidia-device-plugin
+
+# Variables
+source ./00-variables.sh
+
+# Create the namespace if it does not already exist
+if kubectl get namespace "$DEVICE_PLUGIN_NAMESPACE" &>/dev/null; then
+ echo "The [$DEVICE_PLUGIN_NAMESPACE] namespace already exists"
+else
+ echo "Creating the [$DEVICE_PLUGIN_NAMESPACE] namespace..."
+ if ! kubectl create namespace "$DEVICE_PLUGIN_NAMESPACE"; then
+ echo "Failed to create the [$DEVICE_PLUGIN_NAMESPACE] namespace"
+ exit 1
+ fi
+fi
+
+# Deploy the DaemonSet
+echo "Deploying the [$DEVICE_PLUGIN_NAME] DaemonSet with image [$DEVICE_PLUGIN_IMAGE] to the [$DEVICE_PLUGIN_NAMESPACE] namespace..."
+cat nvidia-device-plugin.yml |
+ yq "(.metadata.name)|=\"$DEVICE_PLUGIN_NAME\"" |
+ yq "(.metadata.namespace)|=\"$DEVICE_PLUGIN_NAMESPACE\"" |
+ yq "(.spec.template.spec.containers[0].image)|=\"$DEVICE_PLUGIN_IMAGE\"" |
+ yq "(.spec.template.spec.tolerations[0].key)|=\"$GPU_NODE_TAINT_KEY\"" |
+ yq "(.spec.template.spec.tolerations[0].value)|=\"$GPU_NODE_TAINT_VALUE\"" |
+ yq "(.spec.template.spec.tolerations[1].key)|=\"$SPOT_KEY\"" |
+ yq "(.spec.template.spec.tolerations[1].value)|=\"$SPOT_VALUE\"" |
+ kubectl apply -f -
+if [[ $? -ne 0 ]]; then
+ echo "Failed to deploy the [$DEVICE_PLUGIN_NAME] DaemonSet"
+ exit 1
+fi
+
+kubectl get daemonset "$DEVICE_PLUGIN_NAME" --namespace "$DEVICE_PLUGIN_NAMESPACE"
diff --git a/tutorials/gpu-node-pools/scripts/03-create-resources.sh b/tutorials/gpu-node-pools/scripts/03-create-resources.sh
new file mode 100755
index 0000000..c46b26b
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/03-create-resources.sh
@@ -0,0 +1,194 @@
+#!/bin/bash
+
+# Step 3: create (or reuse) the Service Bus namespace and the queue the training requests arrive on, and the managed
+# identity the KEDA operator and the training job sign in with:
+# - the identity gets the Azure Service Bus Data Owner role on the namespace, on its principal (object) id;
+# - it is federated with the keda-operator service account, so the operator can read the queue's message count, and
+# with the training job's service account, so the job can receive a message;
+# - the keda-operator service account is annotated with the identity's client id, and the operator restarted so the
+# workload identity variables are injected. This step is skipped when the annotation is already in place, for
+# example because one of the KEDA tutorials, which share the identity, ran first.
+# https://learn.microsoft.com/en-us/azure/aks/keda-workload-identity
+
+# Variables
+source ./00-variables.sh
+
+# Get or create the Service Bus namespace
+echo "Checking if the [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace exists in the [$AKS_RESOURCE_GROUP_NAME] resource group..."
+if az servicebus namespace show \
+ --name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors &>/dev/null; then
+ echo "The [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace already exists"
+else
+ echo "Creating the [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace..."
+ if ! az servicebus namespace create \
+ --name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --location "$LOCATION" \
+ --sku "$SERVICE_BUS_SKU" \
+ --only-show-errors 1>/dev/null; then
+ echo "Failed to create the [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace"
+ exit 1
+ fi
+ echo "The [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace was successfully created"
+fi
+
+# Get or create the queue
+echo "Checking if the [$SERVICE_BUS_QUEUE_NAME] queue exists in the [$SERVICE_BUS_NAMESPACE_NAME] namespace..."
+if az servicebus queue show \
+ --name "$SERVICE_BUS_QUEUE_NAME" \
+ --namespace-name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors &>/dev/null; then
+ echo "The [$SERVICE_BUS_QUEUE_NAME] queue already exists"
+else
+ echo "Creating the [$SERVICE_BUS_QUEUE_NAME] queue..."
+ if ! az servicebus queue create \
+ --name "$SERVICE_BUS_QUEUE_NAME" \
+ --namespace-name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors 1>/dev/null; then
+ echo "Failed to create the [$SERVICE_BUS_QUEUE_NAME] queue"
+ exit 1
+ fi
+ echo "The [$SERVICE_BUS_QUEUE_NAME] queue was successfully created"
+fi
+
+# Get or create the managed identity
+echo "Checking if the [$MANAGED_IDENTITY_NAME] managed identity exists in the [$AKS_RESOURCE_GROUP_NAME] resource group..."
+if az identity show \
+ --name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors &>/dev/null; then
+ echo "The [$MANAGED_IDENTITY_NAME] managed identity already exists"
+else
+ echo "Creating the [$MANAGED_IDENTITY_NAME] managed identity..."
+ if ! az identity create \
+ --name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --location "$LOCATION" \
+ --only-show-errors 1>/dev/null; then
+ echo "Failed to create the [$MANAGED_IDENTITY_NAME] managed identity"
+ exit 1
+ fi
+ echo "The [$MANAGED_IDENTITY_NAME] managed identity was successfully created"
+fi
+
+# The client id identifies the identity when a workload signs in; the principal (object) id is what role assignments
+# are granted to. They are different values and are not interchangeable.
+MANAGED_IDENTITY_CLIENT_ID=$(az identity show \
+ --name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query clientId \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+MANAGED_IDENTITY_PRINCIPAL_ID=$(az identity show \
+ --name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query principalId \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ -z $MANAGED_IDENTITY_CLIENT_ID || -z $MANAGED_IDENTITY_PRINCIPAL_ID ]]; then
+ echo "Failed to retrieve the client id and the principal id of the [$MANAGED_IDENTITY_NAME] managed identity"
+ exit 1
+fi
+
+# Grant the role on the namespace unless the identity already has it
+SERVICE_BUS_NAMESPACE_ID=$(az servicebus namespace show \
+ --name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query id \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ -z $SERVICE_BUS_NAMESPACE_ID ]]; then
+ echo "Failed to retrieve the resource id of the [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace"
+ exit 1
+fi
+ROLE_ASSIGNMENT=$(az role assignment list \
+ --assignee "$MANAGED_IDENTITY_PRINCIPAL_ID" \
+ --scope "$SERVICE_BUS_NAMESPACE_ID" \
+ --query "[?roleDefinitionName=='$ROLE'].roleDefinitionName" \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ -n $ROLE_ASSIGNMENT ]]; then
+ echo "The [$MANAGED_IDENTITY_NAME] managed identity already has the [$ROLE] role on the [$SERVICE_BUS_NAMESPACE_NAME] namespace"
+else
+ echo "Granting the [$ROLE] role on the [$SERVICE_BUS_NAMESPACE_NAME] namespace to the [$MANAGED_IDENTITY_NAME] managed identity..."
+ if ! az role assignment create \
+ --assignee-object-id "$MANAGED_IDENTITY_PRINCIPAL_ID" \
+ --assignee-principal-type ServicePrincipal \
+ --role "$ROLE" \
+ --scope "$SERVICE_BUS_NAMESPACE_ID" \
+ --only-show-errors 1>/dev/null; then
+ echo "Failed to grant the [$ROLE] role to the [$MANAGED_IDENTITY_NAME] managed identity"
+ exit 1
+ fi
+ echo "The [$ROLE] role was successfully granted"
+fi
+
+# Federate the identity with the keda-operator service account and with the training job's service account
+AKS_OIDC_ISSUER_URL=$(az aks show \
+ --name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query oidcIssuerProfile.issuerUrl \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ -z $AKS_OIDC_ISSUER_URL ]]; then
+ echo "Failed to retrieve the OIDC issuer URL of the [$AKS_NAME] AKS cluster"
+ exit 1
+fi
+for FEDERATION in \
+ "$FEDERATED_IDENTITY_NAME_KEDA system:serviceaccount:$KEDA_NAMESPACE:$KEDA_OPERATOR_SERVICE_ACCOUNT" \
+ "$FEDERATED_IDENTITY_NAME_JOB system:serviceaccount:$NAMESPACE:$SERVICE_ACCOUNT_NAME"; do
+ FEDERATED_IDENTITY_NAME=${FEDERATION%% *}
+ SUBJECT=${FEDERATION#* }
+ if az identity federated-credential show \
+ --name "$FEDERATED_IDENTITY_NAME" \
+ --identity-name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors &>/dev/null; then
+ echo "The [$FEDERATED_IDENTITY_NAME] federated identity credential already exists"
+ continue
+ fi
+ echo "Federating the [$MANAGED_IDENTITY_NAME] managed identity with [$SUBJECT]..."
+ if ! az identity federated-credential create \
+ --name "$FEDERATED_IDENTITY_NAME" \
+ --identity-name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --issuer "$AKS_OIDC_ISSUER_URL" \
+ --subject "$SUBJECT" \
+ --audiences "$FEDERATED_IDENTITY_AUDIENCE" \
+ --only-show-errors 1>/dev/null; then
+ echo "Failed to create the [$FEDERATED_IDENTITY_NAME] federated identity credential"
+ exit 1
+ fi
+ echo "The [$FEDERATED_IDENTITY_NAME] federated identity credential was successfully created"
+done
+
+# Bind the KEDA operator to the identity, unless its service account already names it
+CURRENT_CLIENT_ID=$(kubectl get serviceaccount "$KEDA_OPERATOR_SERVICE_ACCOUNT" \
+ --namespace "$KEDA_NAMESPACE" \
+ --output jsonpath='{.metadata.annotations.azure\.workload\.identity/client-id}' 2>/dev/null)
+if [[ $CURRENT_CLIENT_ID == "$MANAGED_IDENTITY_CLIENT_ID" ]]; then
+ echo "The [$KEDA_OPERATOR_SERVICE_ACCOUNT] service account is already bound to the [$MANAGED_IDENTITY_NAME] managed identity"
+else
+ echo "Binding the [$KEDA_OPERATOR_SERVICE_ACCOUNT] service account to the [$MANAGED_IDENTITY_NAME] managed identity..."
+ if ! kubectl annotate serviceaccount "$KEDA_OPERATOR_SERVICE_ACCOUNT" \
+ --namespace "$KEDA_NAMESPACE" \
+ "azure.workload.identity/client-id=$MANAGED_IDENTITY_CLIENT_ID" \
+ --overwrite; then
+ echo "Failed to annotate the [$KEDA_OPERATOR_SERVICE_ACCOUNT] service account"
+ exit 1
+ fi
+ # The workload identity variables are injected only when the operator's pods start
+ echo "Restarting the [$KEDA_OPERATOR_DEPLOYMENT] deployment..."
+ if ! kubectl rollout restart deployment "$KEDA_OPERATOR_DEPLOYMENT" --namespace "$KEDA_NAMESPACE"; then
+ echo "Failed to restart the [$KEDA_OPERATOR_DEPLOYMENT] deployment"
+ exit 1
+ fi
+ if ! kubectl rollout status deployment "$KEDA_OPERATOR_DEPLOYMENT" --namespace "$KEDA_NAMESPACE" --timeout=300s; then
+ echo "The [$KEDA_OPERATOR_DEPLOYMENT] deployment didn't restart in time"
+ exit 1
+ fi
+fi
diff --git a/tutorials/gpu-node-pools/scripts/04-deploy-scaled-job.sh b/tutorials/gpu-node-pools/scripts/04-deploy-scaled-job.sh
new file mode 100755
index 0000000..1fb2991
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/04-deploy-scaled-job.sh
@@ -0,0 +1,128 @@
+#!/bin/bash
+
+# Step 4: deploy the training job: the namespace, the job's service account bound to the managed identity, the training
+# script as a config map, the TriggerAuthentication and the KEDA ScaledJob. Three values are derived at apply time rather
+# than hardcoded, so the same manifests work against the emulator and against Azure:
+# - the identity's client id;
+# - the namespace's host name and, on the emulator, the custom endpoint address of its AMQP listener (functions.sh);
+# - the scaler's endpointSuffix: the KEDA scaler reads the message count over the HTTPS API, served on the Azure
+# Resource Manager port.
+# https://keda.sh/docs/2.20/concepts/scaling-jobs/
+# https://keda.sh/docs/2.20/scalers/azure-service-bus/
+
+# Variables
+source ./00-variables.sh
+source ./functions.sh
+
+# Wait for the ScaledJob custom resource definition, without which the ScaledJob can't be applied
+echo "Waiting for the [$KEDA_SCALED_JOB_CRD] custom resource definition..."
+if ! kubectl wait "crd/$KEDA_SCALED_JOB_CRD" --for=condition=Established --timeout=300s; then
+ echo "The [$KEDA_SCALED_JOB_CRD] custom resource definition isn't established: is the KEDA add-on running?"
+ exit 1
+fi
+
+# Read the identity's client id and the Service Bus endpoints
+MANAGED_IDENTITY_CLIENT_ID=$(az identity show \
+ --name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query clientId \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ -z $MANAGED_IDENTITY_CLIENT_ID ]]; then
+ echo "Failed to retrieve the client id of the [$MANAGED_IDENTITY_NAME] managed identity: run 03-create-resources.sh first"
+ exit 1
+fi
+read_service_bus_endpoints || exit 1
+echo "Service Bus host [$SERVICE_BUS_HOST], custom endpoint [${SERVICE_BUS_CUSTOM_ENDPOINT:-none}], scaler endpoint suffix [$SERVICE_BUS_ENDPOINT_SUFFIX]"
+
+# Create the namespace if it does not already exist
+if kubectl get namespace "$NAMESPACE" &>/dev/null; then
+ echo "The [$NAMESPACE] namespace already exists"
+else
+ echo "Creating the [$NAMESPACE] namespace..."
+ if ! kubectl create namespace "$NAMESPACE"; then
+ echo "Failed to create the [$NAMESPACE] namespace"
+ exit 1
+ fi
+fi
+
+# The job's service account, bound to the managed identity
+echo "Creating the [$SERVICE_ACCOUNT_NAME] service account in the [$NAMESPACE] namespace..."
+cat serviceaccount.yml |
+ yq "(.metadata.name)|=\"$SERVICE_ACCOUNT_NAME\"" |
+ yq "(.metadata.namespace)|=\"$NAMESPACE\"" |
+ yq "(.metadata.annotations.\"azure.workload.identity/client-id\")|=\"$MANAGED_IDENTITY_CLIENT_ID\"" |
+ kubectl apply -f -
+if [[ $? -ne 0 ]]; then
+ echo "Failed to create the [$SERVICE_ACCOUNT_NAME] service account"
+ exit 1
+fi
+
+# The training script, mounted into the job's pod
+echo "Creating the [$SCRIPT_CONFIG_MAP_NAME] config map from src/train.py..."
+kubectl create configmap "$SCRIPT_CONFIG_MAP_NAME" \
+ --namespace "$NAMESPACE" \
+ --from-file=train.py=../src/train.py \
+ --dry-run=client \
+ --output yaml |
+ kubectl apply -f -
+if [[ $? -ne 0 ]]; then
+ echo "Failed to create the [$SCRIPT_CONFIG_MAP_NAME] config map"
+ exit 1
+fi
+
+# The TriggerAuthentication: the KEDA operator reads the queue as the managed identity
+echo "Creating the [$TRIGGER_AUTHENTICATION_NAME] TriggerAuthentication..."
+cat triggerauthentication.yml |
+ yq "(.metadata.name)|=\"$TRIGGER_AUTHENTICATION_NAME\"" |
+ yq "(.metadata.namespace)|=\"$NAMESPACE\"" |
+ yq "(.spec.podIdentity.identityId)|=\"$MANAGED_IDENTITY_CLIENT_ID\"" |
+ kubectl apply -f -
+if [[ $? -ne 0 ]]; then
+ echo "Failed to create the [$TRIGGER_AUTHENTICATION_NAME] TriggerAuthentication"
+ exit 1
+fi
+
+# The ScaledJob
+JOB_COMMAND="pip install --quiet --no-cache-dir --target /tmp/pydeps $JOB_PYTHON_PACKAGES && exec python /app/train.py"
+echo "Creating the [$SCALED_JOB_NAME] ScaledJob with image [$JOB_IMAGE]..."
+cat scaledjob.yml |
+ yq "(.metadata.name)|=\"$SCALED_JOB_NAME\"" |
+ yq "(.metadata.namespace)|=\"$NAMESPACE\"" |
+ yq "(.spec.jobTargetRef.backoffLimit)|=$JOB_BACKOFF_LIMIT" |
+ yq "(.spec.jobTargetRef.template.spec.serviceAccountName)|=\"$SERVICE_ACCOUNT_NAME\"" |
+ yq "(.spec.jobTargetRef.template.spec.nodeSelector)|={\"$GPU_NODE_LABEL_KEY\": \"$GPU_NODE_LABEL_VALUE\"}" |
+ yq "(.spec.jobTargetRef.template.spec.tolerations[0].key)|=\"$GPU_NODE_TAINT_KEY\"" |
+ yq "(.spec.jobTargetRef.template.spec.tolerations[0].value)|=\"$GPU_NODE_TAINT_VALUE\"" |
+ yq "(.spec.jobTargetRef.template.spec.tolerations[1].key)|=\"$SPOT_KEY\"" |
+ yq "(.spec.jobTargetRef.template.spec.tolerations[1].value)|=\"$SPOT_VALUE\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].image)|=\"$JOB_IMAGE\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].args[0])|=\"$JOB_COMMAND\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].env[0].value)|=\"$SERVICE_BUS_HOST\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].env[1].value)|=\"$SERVICE_BUS_CUSTOM_ENDPOINT\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].env[2].value)|=\"$SERVICE_BUS_QUEUE_NAME\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].env[3].value)|=\"$JOB_DEFAULT_TRAIN_SECONDS\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].resources.requests.cpu)|=\"$JOB_CPU_REQUEST\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].resources.requests.memory)|=\"$JOB_MEMORY_REQUEST\"" |
+ yq "(.spec.jobTargetRef.template.spec.containers[0].resources.limits.memory)|=\"$JOB_MEMORY_LIMIT\"" |
+ yq "(.spec.jobTargetRef.template.spec.volumes[1].configMap.name)|=\"$SCRIPT_CONFIG_MAP_NAME\"" |
+ yq "(.spec.pollingInterval)|=$SCALED_JOB_POLLING_INTERVAL" |
+ yq "(.spec.maxReplicaCount)|=$SCALED_JOB_MAX_REPLICAS" |
+ yq "(.spec.triggers[0].authenticationRef.name)|=\"$TRIGGER_AUTHENTICATION_NAME\"" |
+ yq "(.spec.triggers[0].metadata.queueName)|=\"$SERVICE_BUS_QUEUE_NAME\"" |
+ yq "(.spec.triggers[0].metadata.namespace)|=\"$SERVICE_BUS_NAMESPACE_NAME\"" |
+ yq "(.spec.triggers[0].metadata.endpointSuffix)|=\"$SERVICE_BUS_ENDPOINT_SUFFIX\"" |
+ kubectl apply -f -
+if [[ $? -ne 0 ]]; then
+ echo "Failed to create the [$SCALED_JOB_NAME] ScaledJob"
+ exit 1
+fi
+
+# KEDA reports the ScaledJob Ready once its trigger can read the queue
+echo "Waiting for the [$SCALED_JOB_NAME] ScaledJob to be ready..."
+if ! kubectl wait "scaledjob/$SCALED_JOB_NAME" --namespace "$NAMESPACE" --for=condition=Ready --timeout=300s; then
+ echo "The [$SCALED_JOB_NAME] ScaledJob isn't ready; the KEDA operator's log says why:"
+ kubectl logs "deployment/$KEDA_OPERATOR_DEPLOYMENT" --namespace "$KEDA_NAMESPACE" --tail=30
+ exit 1
+fi
+kubectl get scaledjob "$SCALED_JOB_NAME" --namespace "$NAMESPACE"
diff --git a/tutorials/gpu-node-pools/scripts/05-run-training-job.sh b/tutorials/gpu-node-pools/scripts/05-run-training-job.sh
new file mode 100755
index 0000000..f100fac
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/05-run-training-job.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+
+# Step 5: send a training request and follow the job KEDA starts for it. The job's pod asks for one GPU, which no node
+# has yet: it stays Pending until the cluster autoscaler adds a node to the Spot GPU pool, the only pool whose nodes
+# can run it. The script prints that node's AKS labels, taints and GPUs, waits for the job to complete and prints its
+# log, which says whether PyTorch found CUDA: on Azure it trains on the NVIDIA GPU, on the LocalStack emulator, whose
+# GPU nodes advertise a GPU without one behind it, on the CPU.
+
+# Variables
+source ./00-variables.sh
+source ./functions.sh
+
+KNOWN_JOBS=$(list_jobs)
+send_training_request "$FIRST_JOB_TRAIN_SECONDS" || exit 1
+
+echo "Waiting for KEDA to start a job for the message..."
+JOB=$(wait_for_new_job "$KNOWN_JOBS")
+if [[ -z $JOB ]]; then
+ echo "KEDA started no job within $JOB_START_TIMEOUT seconds; the ScaledJob and the operator's log say why:"
+ kubectl describe scaledjob "$SCALED_JOB_NAME" --namespace "$NAMESPACE"
+ kubectl logs "deployment/$KEDA_OPERATOR_DEPLOYMENT" --namespace "$KEDA_NAMESPACE" --tail=30
+ exit 1
+fi
+echo "KEDA started the [$JOB] job"
+
+echo "Waiting for the cluster autoscaler to add a node to the [$GPU_NODE_POOL_NAME] node pool and for the job's pod to run..."
+NODE=$(wait_for_running_pod "$JOB")
+if [[ -z $NODE ]]; then
+ echo "No pod of the [$JOB] job runs after $JOB_START_TIMEOUT seconds. Its events, and the cluster autoscaler's status"
+ echo "on Azure, say why: when Azure has no Spot capacity for [$GPU_NODE_VM_SIZE], the pool gets no node."
+ kubectl describe pods --namespace "$NAMESPACE" --selector "job-name=$JOB" | tail -20
+ kubectl get configmap cluster-autoscaler-status --namespace kube-system --output jsonpath='{.data.status}' 2>/dev/null
+ exit 1
+fi
+echo "The pod of the [$JOB] job runs on the [$NODE] node"
+print_gpu_node "$NODE"
+
+echo "Waiting for the [$JOB] job to complete..."
+if ! wait_for_job_completion "$JOB"; then
+ echo "The [$JOB] job didn't complete:"
+ print_job_logs "$JOB"
+ exit 1
+fi
+echo "The [$JOB] job completed. Its log:"
+print_job_logs "$JOB"
+echo "Once its node is idle, the cluster autoscaler removes it, and the GPU pool is back to zero nodes."
diff --git a/tutorials/gpu-node-pools/scripts/06-simulate-eviction.sh b/tutorials/gpu-node-pools/scripts/06-simulate-eviction.sh
new file mode 100755
index 0000000..977a6ac
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/06-simulate-eviction.sh
@@ -0,0 +1,114 @@
+#!/bin/bash
+
+# Step 6: send a longer training request and, once its job's pod reports training progress, simulate the eviction of
+# its Spot node with az vmss simulate-eviction. As when Azure needs the capacity back, AKS taints the node
+# remediator.kubernetes.azure.com/unschedulable:NoSchedule, evicts its pods and deletes the node with its scale set
+# instance. The job then starts a replacement pod, which waits for the next node the cluster autoscaler adds and
+# completes there. The message was deleted when the first pod received it, so the replacement pod finds the queue
+# empty and trains for the default time.
+# https://learn.microsoft.com/en-us/azure/virtual-machine-scale-sets/use-spot#simulate-an-eviction
+
+# Variables
+source ./00-variables.sh
+source ./functions.sh
+
+KNOWN_JOBS=$(list_jobs)
+send_training_request "$EVICTED_JOB_TRAIN_SECONDS" || exit 1
+
+echo "Waiting for KEDA to start a job for the message..."
+JOB=$(wait_for_new_job "$KNOWN_JOBS")
+if [[ -z $JOB ]]; then
+ echo "KEDA started no job within $JOB_START_TIMEOUT seconds"
+ exit 1
+fi
+echo "Waiting for the pod of the [$JOB] job to run..."
+RUNNING_NODE=$(wait_for_running_pod "$JOB")
+if [[ -z $RUNNING_NODE ]]; then
+ echo "No pod of the [$JOB] job runs after $JOB_START_TIMEOUT seconds"
+ exit 1
+fi
+echo "The pod of the [$JOB] job runs on the [$RUNNING_NODE] node"
+
+# A running pod may still be installing its libraries or receiving its message. Evict its node only once the pod
+# trains, so that the eviction interrupts the training and the replacement pod finds the message already gone.
+echo "Waiting for the pod of the [$JOB] job to report training progress..."
+EVICTED_NODE=$(wait_for_training_progress "$JOB")
+if [[ -z $EVICTED_NODE ]]; then
+ echo "The pod of the [$JOB] job reported no training progress within $TRAINING_PROGRESS_TIMEOUT seconds. Its log:"
+ print_job_logs "$JOB"
+ exit 1
+fi
+echo "The pod of the [$JOB] job trains on the [$EVICTED_NODE] node"
+
+# The node's provider id names its scale set instance:
+# azure:///subscriptions/.../resourceGroups//providers/Microsoft.Compute/virtualMachineScaleSets//virtualMachines/
+PROVIDER_ID=$(kubectl get node "$EVICTED_NODE" --output jsonpath='{.spec.providerID}')
+INSTANCE_ID=${PROVIDER_ID##*/}
+SCALE_SET_NAME=${PROVIDER_ID%/virtualMachines/*}
+SCALE_SET_NAME=${SCALE_SET_NAME##*/}
+NODE_RESOURCE_GROUP=$(az aks show \
+ --name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query nodeResourceGroup \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ -z $INSTANCE_ID || -z $SCALE_SET_NAME || -z $NODE_RESOURCE_GROUP ]]; then
+ echo "Failed to find the scale set instance behind the [$EVICTED_NODE] node (provider id [$PROVIDER_ID])"
+ exit 1
+fi
+
+echo "Simulating the eviction of instance [$INSTANCE_ID] of the [$SCALE_SET_NAME] scale set..."
+if ! az vmss simulate-eviction \
+ --resource-group "$NODE_RESOURCE_GROUP" \
+ --name "$SCALE_SET_NAME" \
+ --instance-id "$INSTANCE_ID" \
+ --only-show-errors; then
+ echo "Failed to simulate the eviction of instance [$INSTANCE_ID] of the [$SCALE_SET_NAME] scale set"
+ exit 1
+fi
+
+# Follow the evicted node until AKS deletes it, printing its taints once AKS adds the eviction taint. A failed kubectl
+# call is retried; --ignore-not-found prints nothing once the node is gone.
+TAINT_SEEN=""
+NODE_DELETED=""
+DEADLINE=$((SECONDS + NODE_DELETION_TIMEOUT))
+EVICTION_START=$SECONDS
+while ((SECONDS < DEADLINE)); do
+ if NODE_TAINTS=$(kubectl get node "$EVICTED_NODE" \
+ --ignore-not-found \
+ --output jsonpath='{.metadata.name}{" "}{range .spec.taints[*]}{.key}={.value}:{.effect}{" "}{end}'); then
+ if [[ -z $NODE_TAINTS ]]; then
+ echo "The [$EVICTED_NODE] node was deleted $((SECONDS - EVICTION_START)) seconds after the eviction request"
+ NODE_DELETED="yes"
+ break
+ fi
+ if [[ -z $TAINT_SEEN && $NODE_TAINTS == *remediator.kubernetes.azure.com/unschedulable* ]]; then
+ TAINT_SEEN="yes"
+ echo "The [$EVICTED_NODE] node is tainted for the eviction: ${NODE_TAINTS#* }"
+ fi
+ fi
+ sleep "$POLL_INTERVAL"
+done
+if [[ -z $NODE_DELETED ]]; then
+ echo "The [$EVICTED_NODE] node still exists $NODE_DELETION_TIMEOUT seconds after the eviction request"
+ exit 1
+fi
+
+echo "Waiting for the [$JOB] job to complete on a new node..."
+if ! wait_for_job_completion "$JOB"; then
+ echo "The [$JOB] job didn't complete:"
+ print_job_logs "$JOB"
+ exit 1
+fi
+COMPLETED_NODE=$(kubectl get pods \
+ --namespace "$NAMESPACE" \
+ --selector "job-name=$JOB" \
+ --field-selector status.phase=Succeeded \
+ --output jsonpath='{.items[0].spec.nodeName}')
+if [[ -z $COMPLETED_NODE || $COMPLETED_NODE == "$EVICTED_NODE" ]]; then
+ echo "The [$JOB] job completed, but not on a new node (node [$COMPLETED_NODE])"
+ exit 1
+fi
+echo "The [$JOB] job completed on the [$COMPLETED_NODE] node after the eviction of [$EVICTED_NODE]:"
+kubectl get pods --namespace "$NAMESPACE" --selector "job-name=$JOB" --output wide
+print_job_logs "$JOB"
diff --git a/tutorials/gpu-node-pools/scripts/07-cleanup.sh b/tutorials/gpu-node-pools/scripts/07-cleanup.sh
new file mode 100755
index 0000000..629cf50
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/07-cleanup.sh
@@ -0,0 +1,91 @@
+#!/bin/bash
+
+# Step 7: delete what this tutorial created: the training job's namespace, the device plugin's namespace, the GPU node
+# pool, the Service Bus namespace with the identity's role on it, and the identity's federation with the training job.
+# The managed identity, its federation with the KEDA operator and the operator's binding to it are left in place for
+# the KEDA tutorials, which share them.
+
+# Variables
+source ./00-variables.sh
+
+for KUBERNETES_NAMESPACE in "$NAMESPACE" "$DEVICE_PLUGIN_NAMESPACE"; do
+ echo "Deleting the [$KUBERNETES_NAMESPACE] namespace..."
+ if ! kubectl delete namespace "$KUBERNETES_NAMESPACE" --ignore-not-found --wait=true --timeout=600s; then
+ echo "Failed to delete the [$KUBERNETES_NAMESPACE] namespace"
+ exit 1
+ fi
+done
+
+if az identity federated-credential show \
+ --name "$FEDERATED_IDENTITY_NAME_JOB" \
+ --identity-name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors &>/dev/null; then
+ echo "Deleting the [$FEDERATED_IDENTITY_NAME_JOB] federated identity credential of the [$MANAGED_IDENTITY_NAME] managed identity..."
+ if ! az identity federated-credential delete \
+ --name "$FEDERATED_IDENTITY_NAME_JOB" \
+ --identity-name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --yes \
+ --only-show-errors; then
+ echo "Failed to delete the [$FEDERATED_IDENTITY_NAME_JOB] federated identity credential"
+ exit 1
+ fi
+else
+ echo "The [$FEDERATED_IDENTITY_NAME_JOB] federated identity credential does not exist"
+fi
+
+SERVICE_BUS_NAMESPACE_ID=$(az servicebus namespace show \
+ --name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query id \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+if [[ -n $SERVICE_BUS_NAMESPACE_ID ]]; then
+ MANAGED_IDENTITY_PRINCIPAL_ID=$(az identity show \
+ --name "$MANAGED_IDENTITY_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query principalId \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+ if [[ -n $MANAGED_IDENTITY_PRINCIPAL_ID ]]; then
+ echo "Removing the [$ROLE] role of the [$MANAGED_IDENTITY_NAME] managed identity on the [$SERVICE_BUS_NAMESPACE_NAME] namespace..."
+ if ! az role assignment delete \
+ --assignee "$MANAGED_IDENTITY_PRINCIPAL_ID" \
+ --role "$ROLE" \
+ --scope "$SERVICE_BUS_NAMESPACE_ID" \
+ --only-show-errors; then
+ echo "Failed to remove the [$ROLE] role of the [$MANAGED_IDENTITY_NAME] managed identity"
+ exit 1
+ fi
+ fi
+ echo "Deleting the [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace..."
+ if ! az servicebus namespace delete \
+ --name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors; then
+ echo "Failed to delete the [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace"
+ exit 1
+ fi
+else
+ echo "The [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace does not exist"
+fi
+
+if az aks nodepool show \
+ --name "$GPU_NODE_POOL_NAME" \
+ --cluster-name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors &>/dev/null; then
+ echo "Deleting the [$GPU_NODE_POOL_NAME] node pool of the [$AKS_NAME] AKS cluster..."
+ if ! az aks nodepool delete \
+ --name "$GPU_NODE_POOL_NAME" \
+ --cluster-name "$AKS_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --only-show-errors; then
+ echo "Failed to delete the [$GPU_NODE_POOL_NAME] node pool"
+ exit 1
+ fi
+else
+ echo "The [$GPU_NODE_POOL_NAME] node pool does not exist"
+fi
+echo "Cleanup complete"
diff --git a/tutorials/gpu-node-pools/scripts/functions.sh b/tutorials/gpu-node-pools/scripts/functions.sh
new file mode 100644
index 0000000..325eeff
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/functions.sh
@@ -0,0 +1,196 @@
+# Functions shared by the scripts of the GPU node pools tutorial. Source it after 00-variables.sh.
+
+# Read the Service Bus endpoints the clients use, from the endpoint the namespace reports, and set:
+# - SERVICE_BUS_HOST: the namespace's host name;
+# - SERVICE_BUS_CUSTOM_ENDPOINT: sb://: when the namespace's AMQP port isn't 443, else empty. Azure reports
+# https://.servicebus.windows.net:443/, the emulator its own host and AMQP port, which a client that signs
+# in with Microsoft Entra ID reaches through a custom endpoint address;
+# - SERVICE_BUS_HTTPS_AUTHORITY and SERVICE_BUS_ENDPOINT_SUFFIX: where the HTTPS API answers, which the KEDA scaler
+# and the message sender use. It is served on the Azure Resource Manager port: 443 on Azure, the gateway port on the
+# emulator. The default port stays out of the authority, because a shared access signature signs the URI as written.
+read_service_bus_endpoints() {
+ local endpoint authority port resource_manager arm_port
+ endpoint=$(az servicebus namespace show \
+ --name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --query serviceBusEndpoint \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+ if [[ -z $endpoint ]]; then
+ echo "Failed to read the endpoint of the [$SERVICE_BUS_NAMESPACE_NAME] Service Bus namespace"
+ return 1
+ fi
+ authority=${endpoint#*://}
+ authority=${authority%%/*}
+ SERVICE_BUS_HOST=${authority%%:*}
+ port=443
+ if [[ $authority == *:* ]]; then
+ port=${authority##*:}
+ fi
+ SERVICE_BUS_CUSTOM_ENDPOINT=""
+ if [[ $port != 443 ]]; then
+ SERVICE_BUS_CUSTOM_ENDPOINT="sb://$SERVICE_BUS_HOST:$port"
+ fi
+ resource_manager=$(az cloud show --query endpoints.resourceManager --output tsv --only-show-errors | tr -d '\r')
+ resource_manager=${resource_manager%/}
+ arm_port=${resource_manager##*:}
+ if [[ $arm_port == "$resource_manager" || $arm_port == */* ]]; then
+ arm_port=443
+ fi
+ SERVICE_BUS_HTTPS_AUTHORITY=$SERVICE_BUS_HOST
+ if [[ $arm_port != 443 ]]; then
+ SERVICE_BUS_HTTPS_AUTHORITY="$SERVICE_BUS_HOST:$arm_port"
+ fi
+ SERVICE_BUS_ENDPOINT_SUFFIX=${SERVICE_BUS_HTTPS_AUTHORITY#"$SERVICE_BUS_NAMESPACE_NAME".}
+}
+
+# send_training_request SECONDS: send the message {"train_seconds": SECONDS} to the queue through the Service Bus HTTPS
+# API, signed with a shared access signature of the namespace's authorization rule. Prints nothing about the key.
+send_training_request() {
+ local body key resource_uri expiry signature
+ body="{\"train_seconds\": $1}"
+ read_service_bus_endpoints || return 1
+ key=$(az servicebus namespace authorization-rule keys list \
+ --namespace-name "$SERVICE_BUS_NAMESPACE_NAME" \
+ --resource-group "$AKS_RESOURCE_GROUP_NAME" \
+ --name "$SERVICE_BUS_AUTHORIZATION_RULE" \
+ --query primaryKey \
+ --output tsv \
+ --only-show-errors 2>/dev/null | tr -d '\r')
+ if [[ -z $key ]]; then
+ echo "Failed to read the key of the [$SERVICE_BUS_AUTHORIZATION_RULE] rule of the [$SERVICE_BUS_NAMESPACE_NAME] namespace"
+ return 1
+ fi
+ resource_uri=$(jq -rn --arg value "https://$SERVICE_BUS_HTTPS_AUTHORITY/$SERVICE_BUS_QUEUE_NAME" '$value | @uri')
+ expiry=$(($(date +%s) + 3600))
+ signature=$(printf '%s\n%s' "$resource_uri" "$expiry" |
+ openssl dgst -sha256 -hmac "$key" -binary |
+ openssl base64 -A |
+ jq -rRn 'input | @uri')
+ echo "Sending the message $body to the [$SERVICE_BUS_QUEUE_NAME] queue..."
+ if ! curl --silent --show-error --fail-with-body --request POST \
+ --header "Authorization: SharedAccessSignature sr=$resource_uri&sig=$signature&se=$expiry&skn=$SERVICE_BUS_AUTHORIZATION_RULE" \
+ --header "Content-Type: application/json" \
+ --data "$body" \
+ "https://$SERVICE_BUS_HTTPS_AUTHORITY/$SERVICE_BUS_QUEUE_NAME/messages" >/dev/null; then
+ echo "Failed to send a message to the [$SERVICE_BUS_QUEUE_NAME] queue"
+ return 1
+ fi
+ echo "The message was sent to the [$SERVICE_BUS_QUEUE_NAME] queue"
+}
+
+# Print the names of the jobs the ScaledJob created, oldest first
+list_jobs() {
+ kubectl get jobs \
+ --namespace "$NAMESPACE" \
+ --selector "scaledjob.keda.sh/name=$SCALED_JOB_NAME" \
+ --sort-by .metadata.creationTimestamp \
+ --output jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null
+}
+
+# wait_for_new_job KNOWN_JOBS: wait until the ScaledJob creates a job that isn't in KNOWN_JOBS (one name per line),
+# and print its name
+wait_for_new_job() {
+ local known_jobs=$1 deadline job
+ deadline=$((SECONDS + JOB_START_TIMEOUT))
+ while ((SECONDS < deadline)); do
+ for job in $(list_jobs); do
+ if ! printf '%s\n' "$known_jobs" | grep -qxF "$job"; then
+ echo "$job"
+ return 0
+ fi
+ done
+ sleep "$POLL_INTERVAL"
+ done
+ return 1
+}
+
+# wait_for_running_pod JOB: wait until a pod of JOB runs, and print the node it runs on. While the pod is pending, the
+# cluster autoscaler is adding a node to the GPU pool, and its status says why when it doesn't.
+wait_for_running_pod() {
+ local job=$1 deadline node phase
+ deadline=$((SECONDS + JOB_START_TIMEOUT))
+ while ((SECONDS < deadline)); do
+ node=$(kubectl get pods \
+ --namespace "$NAMESPACE" \
+ --selector "job-name=$job" \
+ --field-selector status.phase=Running \
+ --output jsonpath='{.items[0].spec.nodeName}' 2>/dev/null)
+ if [[ -n $node ]]; then
+ echo "$node"
+ return 0
+ fi
+ phase=$(kubectl get pods \
+ --namespace "$NAMESPACE" \
+ --selector "job-name=$job" \
+ --output jsonpath='{.items[*].status.phase}' 2>/dev/null)
+ echo "Waiting for a pod of the [$job] job to run (pods: ${phase:-none yet})..." >&2
+ sleep "$POLL_INTERVAL"
+ done
+ return 1
+}
+
+# wait_for_training_progress JOB: wait until a running pod of JOB logs training progress, the "Step N, loss X" line
+# train.py prints every 30 seconds of training, and print the node that pod runs on. A pod is Running as soon as its
+# container starts, while it still installs its libraries and receives its message.
+wait_for_training_progress() {
+ local job=$1 deadline pod node
+ deadline=$((SECONDS + TRAINING_PROGRESS_TIMEOUT))
+ while ((SECONDS < deadline)); do
+ pod=$(kubectl get pods \
+ --namespace "$NAMESPACE" \
+ --selector "job-name=$job" \
+ --field-selector status.phase=Running \
+ --output jsonpath='{.items[0].metadata.name}' 2>/dev/null)
+ if [[ -n $pod ]] && kubectl logs "$pod" --namespace "$NAMESPACE" 2>/dev/null | grep -q '^Step '; then
+ node=$(kubectl get pod "$pod" \
+ --namespace "$NAMESPACE" \
+ --output jsonpath='{.spec.nodeName}' 2>/dev/null)
+ if [[ -n $node ]]; then
+ echo "$node"
+ return 0
+ fi
+ fi
+ echo "Waiting for the pod of the [$job] job to report training progress..." >&2
+ sleep "$POLL_INTERVAL"
+ done
+ return 1
+}
+
+# wait_for_job_completion JOB: wait until JOB completes; fail as soon as it fails
+wait_for_job_completion() {
+ local job=$1 deadline conditions
+ deadline=$((SECONDS + JOB_TIMEOUT))
+ while ((SECONDS < deadline)); do
+ conditions=$(kubectl get job "$job" \
+ --namespace "$NAMESPACE" \
+ --output jsonpath='{range .status.conditions[?(@.status=="True")]}{.type}{" "}{end}' 2>/dev/null)
+ case " $conditions " in
+ *" Complete "*) return 0 ;;
+ *" Failed "*) return 1 ;;
+ esac
+ sleep "$POLL_INTERVAL"
+ done
+ return 1
+}
+
+# print_gpu_node NODE: print the labels AKS gives a GPU node of a Spot pool, its taints and the GPUs it advertises
+print_gpu_node() {
+ local node=$1
+ echo "The [$node] node:"
+ kubectl get node "$node" --output json | jq '{
+ name: .metadata.name,
+ labels: (.metadata.labels | with_entries(select(.key | test("accelerator|agentpool$|scalesetpriority|priority$|instance-type$|topology.kubernetes.io|^workload$")))),
+ taints: [.spec.taints[]? | "\(.key)=\(.value // ""):\(.effect)"],
+ gpus: .status.allocatable["nvidia.com/gpu"]
+ }'
+}
+
+# print_job_logs JOB: print the logs of the job's pods
+print_job_logs() {
+ kubectl logs \
+ --namespace "$NAMESPACE" \
+ --selector "job-name=$1" \
+ --tail=-1 \
+ --prefix 2>/dev/null
+}
diff --git a/tutorials/gpu-node-pools/scripts/nvidia-device-plugin.yml b/tutorials/gpu-node-pools/scripts/nvidia-device-plugin.yml
new file mode 100644
index 0000000..e5e8444
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/nvidia-device-plugin.yml
@@ -0,0 +1,52 @@
+# The NVIDIA device plugin DaemonSet from Microsoft's guide to installing it by hand
+# (https://learn.microsoft.com/en-us/azure/aks/use-nvidia-gpu#manually-install-the-nvidia-device-plugin), with two
+# additions: a node selector for the accelerator=nvidia label AKS gives GPU nodes, so it runs only there, and a
+# toleration for the taint AKS gives Spot nodes. 02-deploy-device-plugin.sh fills in the name, the namespace, the image
+# and the tolerations.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+ name: ""
+ namespace: ""
+spec:
+ selector:
+ matchLabels:
+ name: nvidia-device-plugin-ds
+ updateStrategy:
+ type: RollingUpdate
+ template:
+ metadata:
+ labels:
+ name: nvidia-device-plugin-ds
+ spec:
+ nodeSelector:
+ accelerator: nvidia
+ tolerations:
+ - key: "sku"
+ operator: "Equal"
+ value: "gpu"
+ effect: "NoSchedule"
+ - key: "kubernetes.azure.com/scalesetpriority"
+ operator: "Equal"
+ value: "spot"
+ effect: "NoSchedule"
+ # Mark this pod as a critical add-on, so that it is rescheduled first after a failure
+ priorityClassName: "system-node-critical"
+ containers:
+ - image: ""
+ name: nvidia-device-plugin-ctr
+ env:
+ # Wait instead of exiting when the node has no GPU the plugin can use
+ - name: FAIL_ON_INIT_ERROR
+ value: "false"
+ securityContext:
+ allowPrivilegeEscalation: false
+ capabilities:
+ drop: ["ALL"]
+ volumeMounts:
+ - name: device-plugin
+ mountPath: /var/lib/kubelet/device-plugins
+ volumes:
+ - name: device-plugin
+ hostPath:
+ path: /var/lib/kubelet/device-plugins
diff --git a/tutorials/gpu-node-pools/scripts/scaledjob.yml b/tutorials/gpu-node-pools/scripts/scaledjob.yml
new file mode 100644
index 0000000..71f418e
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/scaledjob.yml
@@ -0,0 +1,91 @@
+# The KEDA ScaledJob: one Kubernetes Job for each message on the queue. Its pod asks for one GPU, selects the GPU
+# pool's nodes and tolerates their taints, and signs in with workload identity. 04-deploy-scaled-job.sh fills in the
+# empty values, and sets the backoff limit, the node selector, the tolerations, the resources, the polling interval and
+# the replica bound from 00-variables.sh.
+apiVersion: keda.sh/v1alpha1
+kind: ScaledJob
+metadata:
+ name: ""
+ namespace: ""
+spec:
+ jobTargetRef:
+ backoffLimit: 3
+ template:
+ metadata:
+ labels:
+ azure.workload.identity/use: "true"
+ spec:
+ serviceAccountName: ""
+ restartPolicy: Never
+ securityContext:
+ runAsNonRoot: true
+ runAsUser: 1000
+ runAsGroup: 1000
+ nodeSelector:
+ workload: gpu
+ tolerations:
+ - key: "sku"
+ operator: "Equal"
+ value: "gpu"
+ effect: "NoSchedule"
+ - key: "kubernetes.azure.com/scalesetpriority"
+ operator: "Equal"
+ value: "spot"
+ effect: "NoSchedule"
+ containers:
+ - name: train
+ image: ""
+ command: ["sh", "-c"]
+ args:
+ # Install the libraries into the writable /tmp, then train; filled in with JOB_PYTHON_PACKAGES
+ - ""
+ env:
+ - name: SERVICE_BUS_FULLY_QUALIFIED_NAMESPACE
+ value: ""
+ - name: SERVICE_BUS_CUSTOM_ENDPOINT
+ value: ""
+ - name: SERVICE_BUS_QUEUE_NAME
+ value: ""
+ - name: DEFAULT_TRAIN_SECONDS
+ value: ""
+ - name: HOME
+ value: /tmp
+ resources:
+ requests:
+ nvidia.com/gpu: 1
+ cpu: "1"
+ memory: 2Gi
+ limits:
+ nvidia.com/gpu: 1
+ memory: 4Gi
+ securityContext:
+ allowPrivilegeEscalation: false
+ readOnlyRootFilesystem: true
+ capabilities:
+ drop: ["ALL"]
+ volumeMounts:
+ - name: tmp
+ mountPath: /tmp
+ - name: script
+ mountPath: /app
+ readOnly: true
+ volumes:
+ - name: tmp
+ emptyDir: {}
+ - name: script
+ configMap:
+ name: ""
+ pollingInterval: 10
+ maxReplicaCount: 2
+ triggers:
+ - type: azure-servicebus
+ authenticationRef:
+ name: ""
+ metadata:
+ queueName: ""
+ namespace: ""
+ # One job for each message waiting on the queue
+ messageCount: "1"
+ # cloud: Private is what lets the scaler use a custom endpointSuffix, which the emulator needs
+ cloud: Private
+ endpointSuffix: ""
diff --git a/tutorials/gpu-node-pools/scripts/serviceaccount.yml b/tutorials/gpu-node-pools/scripts/serviceaccount.yml
new file mode 100644
index 0000000..67b500f
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/serviceaccount.yml
@@ -0,0 +1,9 @@
+# The training job's service account. The annotation names the managed identity the job signs in as with workload
+# identity; 04-deploy-scaled-job.sh fills in the name, the namespace and the identity's client id.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: ""
+ namespace: ""
+ annotations:
+ azure.workload.identity/client-id: ""
diff --git a/tutorials/gpu-node-pools/scripts/triggerauthentication.yml b/tutorials/gpu-node-pools/scripts/triggerauthentication.yml
new file mode 100644
index 0000000..48c4776
--- /dev/null
+++ b/tutorials/gpu-node-pools/scripts/triggerauthentication.yml
@@ -0,0 +1,11 @@
+# Tells the KEDA operator to read the queue's message count as the managed identity, with workload identity.
+# 04-deploy-scaled-job.sh fills in the name, the namespace and the identity's client id.
+apiVersion: keda.sh/v1alpha1
+kind: TriggerAuthentication
+metadata:
+ name: ""
+ namespace: ""
+spec:
+ podIdentity:
+ provider: azure-workload
+ identityId: ""
diff --git a/tutorials/gpu-node-pools/src/train.py b/tutorials/gpu-node-pools/src/train.py
new file mode 100644
index 0000000..cca52eb
--- /dev/null
+++ b/tutorials/gpu-node-pools/src/train.py
@@ -0,0 +1,64 @@
+"""The training job: receive one training request from the Service Bus queue with workload identity, read how many
+seconds to train from it, and train a small PyTorch model on the GPU when PyTorch finds one, on the CPU otherwise.
+
+KEDA's ScaledJob starts one Kubernetes Job for each message on the queue. The pod installs the Service Bus and identity
+libraries into /tmp/pydeps when it starts (see scaledjob.yml), next to the PyTorch the image provides.
+"""
+
+import json
+import os
+import sys
+import time
+
+sys.path.insert(0, "/tmp/pydeps")
+
+import torch # noqa: E402
+from azure.identity import DefaultAzureCredential # noqa: E402
+from azure.servicebus import ServiceBusClient, ServiceBusReceiveMode # noqa: E402
+
+train_seconds = int(os.environ["DEFAULT_TRAIN_SECONDS"])
+# Empty on Azure. On the LocalStack emulator, the namespace's AMQP port isn't 443, so the client connects through a
+# custom endpoint address, and the library then uses AMQP over WebSockets (websocket-client).
+custom_endpoint = os.environ.get("SERVICE_BUS_CUSTOM_ENDPOINT") or None
+
+# The message is deleted when it is received: a pod that loses its node, such as a Spot node Azure evicts, doesn't
+# find it again, and the job's replacement pod trains for the default time.
+with ServiceBusClient(
+ os.environ["SERVICE_BUS_FULLY_QUALIFIED_NAMESPACE"],
+ DefaultAzureCredential(),
+ custom_endpoint_address=custom_endpoint,
+) as client:
+ with client.get_queue_receiver(
+ os.environ["SERVICE_BUS_QUEUE_NAME"], receive_mode=ServiceBusReceiveMode.RECEIVE_AND_DELETE
+ ) as receiver:
+ messages = receiver.receive_messages(max_message_count=1, max_wait_time=30)
+ if messages:
+ body = json.loads(str(messages[0]))
+ train_seconds = int(body.get("train_seconds", train_seconds))
+ print(f"Received message {messages[0].message_id}: {body}", flush=True)
+ else:
+ print("No message in the queue: training for the default time", flush=True)
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+print(f"PyTorch {torch.__version__}, CUDA available: {torch.cuda.is_available()}", flush=True)
+if device == "cuda":
+ print(f"GPU: {torch.cuda.get_device_name(0)}", flush=True)
+
+model = torch.nn.Linear(1024, 1024).to(device)
+optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
+inputs = torch.randn(256, 1024, device=device)
+targets = torch.randn(256, 1024, device=device)
+started = time.monotonic()
+next_report = started + 30
+steps = 0
+loss = torch.tensor(0.0)
+while time.monotonic() - started < train_seconds:
+ optimizer.zero_grad()
+ loss = torch.nn.functional.mse_loss(model(inputs), targets)
+ loss.backward()
+ optimizer.step()
+ steps += 1
+ if time.monotonic() >= next_report:
+ print(f"Step {steps}, loss {loss.item():.4f}", flush=True)
+ next_report += 30
+print(f"Trained {steps} steps on {device} in {train_seconds} seconds, final loss {loss.item():.4f}", flush=True)