ContactCV

Writing

adithya-rajendran.com/blog/kubernetes-on-the-nvidia-dgx-spark · Filed 6 Mar 2026 · Adithya Rajendran

Writing

4 min read

Sharing the DGX Spark GPU with MicroK8s

Setting up MicroK8s, the NVIDIA GPU Operator, and GPU time-slicing on my DGX Spark, including the containerd paths and memory-sharing tradeoffs.

Contents

I added an NVIDIA DGX Spark, built on the GB10 Grace Blackwell platform, to my lab to explore GPU workloads as infrastructure. My field engineering work at Canonical involved OpenStack, Kubernetes, and Ceph, so I wanted to bring the Spark into that world: a node where several experiments could share the GPU.

This post covers how I set up MicroK8s alongside Docker, installed the NVIDIA GPU Operator, and configured GPU time-slicing so several workloads can share the GB10 GPU. Along the way, I’ll explain the MicroK8s-specific containerd paths that make this setup work.

Installing MicroK8s

Bash
sudo snap install microk8s --classic --channel=1.32/stable
sudo microk8s status --wait-ready

Add your user to the microk8s group:

Bash
sudo usermod -aG microk8s $USER
newgrp microk8s
microk8s kubectl get nodes

Enable the core add-ons:

Bash
microk8s enable dns
microk8s enable hostpath-storage
microk8s enable helm3

Create aliases for kubectl and helm so you do not have to prefix every command with microk8s:

Bash
echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc
echo "alias helm='microk8s helm3'" >> ~/.bashrc
source ~/.bashrc

Docker and MicroK8s Use Separate Image Stores

Docker and MicroK8s use separate container runtimes, so an image pulled with docker pull is not automatically available to a Kubernetes pod, and vice versa. To use a locally built Docker image in MicroK8s, import it:

Text
docker tag myimage:latest myimage:local
docker save myimage:local | microk8s ctr images import -

In the pod specification, use image: myimage:local and imagePullPolicy: Never to run the imported image without contacting a registry. The image must support the Spark’s Arm architecture. On a multi-node cluster, import it on every node where the pod could run, or publish it to a registry instead. Docker and MicroK8s can each pull registry-hosted images independently.

Installing the NVIDIA GPU Operator

The GPU Operator automates the components needed to expose NVIDIA GPUs to Kubernetes workloads, including driver management, the container toolkit, the device plugin, DCGM metrics, and MIG configuration. The DGX Spark’s driver is already installed and managed by DGX OS, so I disable the operator’s driver component and let it manage the remaining pieces.

Add the NVIDIA Helm Repository

Bash
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update nvidia
helm repo list | grep nvidia

Install the GPU Operator

The key flag is --set driver.enabled=false. DGX OS already manages the host driver, so I leave that component disabled. The operator still manages the container toolkit—which configures containerd to use the NVIDIA runtime—and its other components.

Because MicroK8s is packaged as a snap, its containerd socket and configuration files live under /var/snap/microk8s/ rather than the usual system paths. The GPU Operator needs those exact locations, so I pass three environment variables explicitly:

Bash
helm install gpu-operator -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v25.10.1 \
  --set driver.enabled=false \
  --set toolkit.enabled=true \
  --set toolkit.env[0].name=CONTAINERD_CONFIG \
  --set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd-template.toml \
  --set toolkit.env[1].name=CONTAINERD_SOCKET \
  --set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \
  --set toolkit.env[2].name=RUNTIME_CONFIG_SOURCE \
  --set-string toolkit.env[2].value=file=/var/snap/microk8s/current/args/containerd.toml

These paths are a common failure point when deploying the GPU Operator on snap-based Kubernetes. If the operator cannot reach containerd, its pods may enter CrashLoopBackOff with a socket error. Canonical Kubernetes uses different containerd locations that depend on its configured base directory, so verify the actual socket and configuration paths before adapting this command.

Verify GPU Access

Create a test pod that runs a CUDA vector addition:

YAML
# cuda-test.yaml
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
    resources:
      limits:
        nvidia.com/gpu: 1
Bash
kubectl apply -f cuda-test.yaml
kubectl logs cuda-vectoradd

Successful output looks like this:

Text
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
Bash
# Clean up
kubectl delete -f cuda-test.yaml

GPU Time-Slicing: Running Multiple Workloads on One GPU

By default, Kubernetes advertises each GPU as an indivisible resource, so a node with one GPU can satisfy only one request for nvidia.com/gpu: 1 at a time. That default can be restrictive on the DGX Spark, which has 128 GB of coherent unified system memory. Workloads such as Jupyter notebooks, inference services, and training jobs can often coexist.

GPU time-slicing lets the NVIDIA device plugin advertise one physical GPU as several allocatable replicas and interleave workloads that share it. Unlike MIG, time-slicing provides no memory or fault isolation between replicas, so workloads share the same unified memory budget. Plan capacity accordingly.

Step 1: Create the Time-Slicing ConfigMap

Bash
cat <<'EOF' | kubectl apply -n gpu-operator -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4
EOF

replicas: 4 makes a one-GPU node advertise four nvidia.com/gpu resources. I use four here; choose the replica count based on the workload mix rather than treating it as a capacity guarantee.

Step 2: Patch the ClusterPolicy

Bash
kubectl patch clusterpolicy/cluster-policy -n gpu-operator --type merge -p \
  '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"any"}}}}'

Step 3: Restart the Device Plugin

Bash
kubectl delete pod -n gpu-operator -l app=nvidia-device-plugin-daemonset

Step 4: Verify

Wait 30–60 seconds, then:

Bash
kubectl describe node | grep nvidia.com/gpu

Expected output:

Text
nvidia.com/gpu.replicas=4
nvidia.com/gpu.sharing-strategy=time-slicing
nvidia.com/gpu.product=NVIDIA-GB10-SHARED
nvidia.com/gpu:     4
nvidia.com/gpu:     4

The -SHARED product suffix, nvidia.com/gpu.replicas=4 label, and sharing-strategy=time-slicing label confirm that time-slicing is active. Each pod still requests nvidia.com/gpu: 1; the NVIDIA device plugin manages shared access to the physical GPU.

Adjusting or Disabling Time-Slicing

Bash
# Change the replica count
kubectl edit configmap time-slicing-config -n gpu-operator
kubectl delete pod -n gpu-operator -l app=nvidia-device-plugin-daemonset

# Revert to exclusive access
kubectl patch clusterpolicy/cluster-policy -n gpu-operator --type merge -p \
  '{"spec":{"devicePlugin":{"config":{"name":"","default":""}}}}'
kubectl delete configmap time-slicing-config -n gpu-operator
kubectl delete pod -n gpu-operator -l app=nvidia-device-plugin-daemonset

With this setup, the DGX Spark behaves like a small GPU-enabled infrastructure node rather than a single-purpose workstation. MicroK8s keeps the host lightweight, while the GPU Operator handles the Kubernetes integration and time-slicing lets several experiments share the GB10 without implying that they are isolated.