Skip to content
by Adithya Rajendran

Kubernetes on the NVIDIA DGX Spark

How I set up MicroK8s and the NVIDIA GPU Operator on a DGX Spark, then configured GPU time-slicing for concurrent workloads.

I recently added an NVIDIA DGX Spark—an AI supercomputer built on the Grace Blackwell GB10 platform—to my lab. As a Field Software Engineer at Canonical, I deploy OpenStack, Kubernetes, and Ceph at scale. Rather than use the Spark only for NVIDIA’s AI stack, I wanted to treat it as an infrastructure node.

This post covers how I set up MicroK8s alongside Docker, installed the NVIDIA GPU Operator, and configured GPU time-slicing so several workloads can share the GB10 GPU. Along the way, I’ll explain the MicroK8s-specific containerd paths that make this setup work.

Installing MicroK8s#

bash
sudo snap install microk8s --classic --channel=1.32/stable
sudo microk8s status --wait-ready

Add your user to the microk8s group:

bash
sudo usermod -aG microk8s $USER
newgrp microk8s
microk8s kubectl get nodes

Enable the core add-ons:

bash
microk8s enable dns
microk8s enable hostpath-storage
microk8s enable helm3

Create aliases for kubectl and helm so you do not have to prefix every command with microk8s:

bash
echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc
echo "alias helm='microk8s helm3'" >> ~/.bashrc
source ~/.bashrc

Docker and MicroK8s Use Separate Image Stores#

Docker and MicroK8s use separate container runtimes, so an image pulled with docker pull is not automatically available to a Kubernetes pod, and vice versa. To use a locally built Docker image in MicroK8s, import it:

code
docker save myimage:latest | microk8s ctr images import -

For registry-hosted images, each runtime can pull its own copy from the same path. The distinction mainly matters when you build images locally.

Installing the NVIDIA GPU Operator#

The GPU Operator automates the components needed to expose NVIDIA GPUs to Kubernetes workloads, including driver management, the container toolkit, the device plugin, DCGM metrics, and MIG configuration. The DGX Spark’s driver is already installed and managed by DGX OS, so I disable the operator’s driver component and let it manage the remaining pieces.

Add the NVIDIA Helm Repository#

bash
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update nvidia
helm repo list | grep nvidia

Install the GPU Operator#

The key flag is --set driver.enabled=false. DGX OS already manages the host driver, so I leave that component disabled. The operator still manages the container toolkit—which configures containerd to use the NVIDIA runtime—and its other components.

Because MicroK8s is packaged as a snap, its containerd socket and configuration files live under /var/snap/microk8s/ rather than the usual system paths. The GPU Operator needs those exact locations, so I pass three environment variables explicitly:

bash
helm install gpu-operator -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v25.10.1 \
  --set driver.enabled=false \
  --set toolkit.enabled=true \
  --set toolkit.env[0].name=CONTAINERD_CONFIG \
  --set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd-template.toml \
  --set toolkit.env[1].name=CONTAINERD_SOCKET \
  --set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \
  --set toolkit.env[2].name=RUNTIME_CONFIG_SOURCE \
  --set-string toolkit.env[2].value=file=/var/snap/microk8s/current/args/containerd.toml

These paths are a common failure point when deploying the GPU Operator on snap-based Kubernetes. If the operator cannot reach containerd, its pods may enter CrashLoopBackOff with a socket error. Canonical Kubernetes uses different containerd locations that depend on its configured base directory, so verify the actual socket and configuration paths before adapting this command.

Verify GPU Access#

Create a test pod that runs a CUDA vector addition:

yaml
# cuda-test.yaml
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
    resources:
      limits:
        nvidia.com/gpu: 1
bash
kubectl apply -f cuda-test.yaml
kubectl logs cuda-vectoradd

Successful output looks like this:

code
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
bash
# Clean up
kubectl delete -f cuda-test.yaml

GPU Time-Slicing: Running Multiple Workloads on One GPU#

By default, Kubernetes advertises each GPU as an indivisible resource, so a node with one GPU can satisfy only one request for nvidia.com/gpu: 1 at a time. That default can be restrictive on the DGX Spark, which has 128 GB of coherent unified system memory. Workloads such as Jupyter notebooks, inference services, and training jobs can often coexist.

GPU time-slicing lets the NVIDIA device plugin advertise one physical GPU as several allocatable replicas and interleave workloads that share it. Unlike MIG, time-slicing provides no memory or fault isolation between replicas, so workloads share the same unified memory budget. Plan capacity accordingly.

Step 1: Create the Time-Slicing ConfigMap#

bash
cat <<'EOF' | kubectl apply -n gpu-operator -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4
EOF

replicas: 4 makes a one-GPU node advertise four nvidia.com/gpu resources. I use four here; choose the replica count based on the workload mix rather than treating it as a capacity guarantee.

Step 2: Patch the ClusterPolicy#

bash
kubectl patch clusterpolicy/cluster-policy -n gpu-operator --type merge -p \
  '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"any"}}}}'

Step 3: Restart the Device Plugin#

bash
kubectl delete pod -n gpu-operator -l app=nvidia-device-plugin-daemonset

Step 4: Verify#

Wait 30–60 seconds, then:

bash
kubectl describe node | grep nvidia.com/gpu

Expected output:

code
nvidia.com/gpu.replicas=4
nvidia.com/gpu.sharing-strategy=time-slicing
nvidia.com/gpu.product=NVIDIA-GB10-SHARED
nvidia.com/gpu:     4
nvidia.com/gpu:     4

The -SHARED product suffix, nvidia.com/gpu.replicas=4 label, and sharing-strategy=time-slicing label confirm that time-slicing is active. Each pod still requests nvidia.com/gpu: 1; the NVIDIA device plugin manages shared access to the physical GPU.

Adjusting or Disabling Time-Slicing#

bash
# Change the replica count
kubectl edit configmap time-slicing-config -n gpu-operator
kubectl delete pod -n gpu-operator -l app=nvidia-device-plugin-daemonset

# Revert to exclusive access
kubectl patch clusterpolicy/cluster-policy -n gpu-operator --type merge -p \
  '{"spec":{"devicePlugin":{"config":{"name":"","default":""}}}}'
kubectl delete configmap time-slicing-config -n gpu-operator
kubectl delete pod -n gpu-operator -l app=nvidia-device-plugin-daemonset

With this setup, the DGX Spark behaves like a small GPU-enabled infrastructure node rather than a single-purpose workstation. MicroK8s keeps the host lightweight, while the GPU Operator handles the Kubernetes integration and time-slicing lets several experiments share the GB10 without implying that they are isolated.