Back to blog

Running Kubernetes in Jails

May 30, 2026

Most Kubernetes security discussions stop at the API layer: RBAC, admission, PodSecurity, and NetworkPolicy. Those controls matter, but they are not the whole story. If an attacker reaches code execution inside a node-critical process such as kube-apiserver, etcd, kubelet, or containerd, the question changes completely.

At that point, the real security boundary is no longer the Kubernetes policy stack. It is the host itself: what the compromised process can still read, mount, signal, or execute. This is where nsjail becomes useful. Used correctly, it reduces the blast radius with restricted mounts, namespaces, capability dropping, resource limits, and seccomp. The same design can extend beyond kubelet and containerd to CSI sidecars, image fetchers, logging agents, mesh helpers, and selected control-plane services, each with its own rootfs, SELinux type, seccomp profile, and projected mount set.

Architecture

1. Jailed kubelet

kubelet runs under systemd and nsjail, with a dedicated rootfs, explicit bind mounts, and SELinux confinement.

The goal is not to hide only the process. The goal is to control what filesystem tree kubelet can see when it resolves:

  • Pod volume paths
  • plugin directories
  • pod state directories
  • CRI sockets
  • certificate and kubeconfig paths

If kubelet sees only a restricted root, then hostPath: / no longer means "real host root". It means "root of the kubelet jail view".

2. Jailed containerd in the same path model

Jailing kubelet alone is not enough.

The process that performs the actual bind mount must share the same restricted path model. Otherwise:

  • kubelet validates one path view
  • containerd or runc mounts from the real host view

That breaks the guarantee.

So containerd, its CRI plugin, and the runtime path must be aligned to the same constrained mount model. The simplest defensible design is:

  • kubelet jailed
  • containerd jailed
  • CRI socket inside the jail
  • runtime state directories inside the jail
  • only explicitly projected host directories visible

3. Limited hostPath allowlist

There is no general-purpose hostPath.

Instead, the host exposes only a fixed set of projected directories, for example:

/srv/allowed/readonly/node-metadata
/srv/allowed/runtime-logs
/srv/allowed/workload-data/<workload-id>

Those directories are:

  • pre-created by the host
  • labeled with SELinux types
  • mounted into the jail under fixed paths
  • bound into containers only through approved workload templates

4. No generic CNI dependency

This design can avoid dependence on a general third-party CNI plugin, but not on networking setup itself.

There is always a networking layer.

The closed-appliance version replaces "generic CNI plugin selected by Kubernetes" with "host-owned network provisioning":

  • host pre-creates veth pairs or tap devices
  • host pre-assigns pod subnets or routes
  • runtime receives an already-prepared network namespace path
  • no arbitrary CNI binary execution from mutable plugin directories

This removes one moving part and makes networking part of the trusted host image. It does not mean "no networking policy is needed". It means "network setup is part of the host appliance, not delegated to an external plugin chain".

5. SELinux as the real MAC layer

nsjail gives process layout, mount shaping, namespace boundaries, and seccomp entry points.

SELinux gives:

  • mandatory file access control
  • process domain separation
  • socket and port controls
  • relabel-backed guarantees that survive application mistakes

The serious version of this design uses both:

  • nsjail for shape
  • SELinux for enforcement

Target-State systemd Units

/etc/systemd/system/kubelet-nsjail.service

[Unit]
Description=Kubelet under nsjail
After=network-online.target containerd-nsjail.service
Wants=network-online.target
Requires=containerd-nsjail.service

[Service]
Type=simple
ExecStart=/usr/local/libexec/k8s/run-kubelet-nsjail.sh
Restart=always
RestartSec=5
TimeoutStopSec=30
KillMode=mixed
LimitNOFILE=1048576
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target

/etc/systemd/system/containerd-nsjail.service

[Unit]
Description=containerd under nsjail
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/libexec/k8s/run-containerd-nsjail.sh
Restart=always
RestartSec=5
TimeoutStopSec=30
KillMode=mixed
LimitNOFILE=1048576
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target

Target-State nsjail Wrappers

/usr/local/libexec/k8s/run-kubelet-nsjail.sh

#!/usr/bin/env bash
set -euo pipefail
exec /usr/local/bin/nsjail -Mo \
  --hostname kubelet-node \
  --chroot /opt/k8s-jails/kubelet-root \
  --cwd / \
  --user 0 --group 0 \
  --disable_clone_newnet \
  --bindmount_ro /usr/bin/kubelet:/usr/bin/kubelet \
  --bindmount_ro /etc/kubernetes:/etc/kubernetes \
  --bindmount_ro /etc/ssl/certs:/etc/ssl/certs \
  --bindmount /var/lib/kubelet:/var/lib/kubelet \
  --bindmount /var/lib/containerd-jail:/var/lib/containerd \
  --bindmount /run/containerd-jail:/run/containerd \
  --bindmount /srv/allowed:/srv/allowed \
  --bindmount /sys:/sys \
  --bindmount /proc:/proc \
  --bindmount /dev:/dev \
  --tmpfsmount /tmp \
  --seccomp_policy /etc/nsjail/kubelet.kafel \
  -- \
  /usr/bin/kubelet \
    --config=/etc/kubernetes/kubelet/config.yaml \
    --kubeconfig=/etc/kubernetes/kubelet/kubeconfig \
    --container-runtime-endpoint=unix:///run/containerd/containerd.sock \
    --root-dir=/var/lib/kubelet \
    --cgroup-driver=systemd \
    --seccomp-default=true

/usr/local/libexec/k8s/run-containerd-nsjail.sh

#!/usr/bin/env bash
set -euo pipefail
exec /usr/local/bin/nsjail -Mo \
  --hostname containerd-node \
  --chroot /opt/k8s-jails/containerd-root \
  --cwd / \
  --user 0 --group 0 \
  --disable_clone_newnet \
  --bindmount_ro /usr/bin/containerd:/usr/bin/containerd \
  --bindmount_ro /etc/containerd:/etc/containerd \
  --bindmount_ro /etc/ssl/certs:/etc/ssl/certs \
  --bindmount /var/lib/containerd-jail:/var/lib/containerd \
  --bindmount /run/containerd-jail:/run/containerd \
  --bindmount /srv/allowed:/srv/allowed \
  --bindmount /sys:/sys \
  --bindmount /proc:/proc \
  --bindmount /dev:/dev \
  --tmpfsmount /tmp \
  --seccomp_policy /etc/nsjail/containerd.kafel \
  -- \
  /usr/bin/containerd \
    --config /etc/containerd/config.toml \
    --root /var/lib/containerd \
    --state /run/containerd

Limited hostPath Contract

The supported contract is:

  • no arbitrary hostPath
  • only platform-defined mounts

Example:

  • /srv/allowed/readonly/node-metadata → mounted read-only into specific workloads
  • /srv/allowed/workload-data/<id> → dedicated writable subtree for one workload

Everything else is simply absent from the jailed root.

That means a pod asking for /etc, /root, /var/lib/etcd, or / does not gain access to the real host paths if they are not projected into the jail.

SELinux Policy Shape

The host labels the runtime pieces separately:

kubelet_nsjail_exec_t
kubelet_nsjail_t
containerd_nsjail_exec_t
containerd_nsjail_t
k8s_allowed_hostpath_t
k8s_runtime_state_t
k8s_runtime_log_t

Example file-context policy:

/usr/local/libexec/k8s/run-kubelet-nsjail\.sh      -- gen_context(system_u:object_r:kubelet_nsjail_exec_t,s0)
/usr/local/libexec/k8s/run-containerd-nsjail\.sh   -- gen_context(system_u:object_r:containerd_nsjail_exec_t,s0)
/srv/allowed(/.*)?                                 -- gen_context(system_u:object_r:k8s_allowed_hostpath_t,s0)
/var/lib/containerd-jail(/.*)?                     -- gen_context(system_u:object_r:k8s_runtime_state_t,s0)
/run/containerd-jail(/.*)?                         -- gen_context(system_u:object_r:k8s_runtime_state_t,s0)
/var/log/nsjail-k8s(/.*)?                          -- gen_context(system_u:object_r:k8s_runtime_log_t,s0)

Example policy intent:

  • kubelet domain can read Kubernetes config and runtime metadata
  • kubelet domain can access only k8s_allowed_hostpath_t and k8s_runtime_state_t
  • containerd domain can mount only projected paths and runtime state
  • neither domain gets broad access to unlabeled host files

The short version is:

  • We do not rely on the Kubernetes API as the primary trust boundary.
  • We do not allow arbitrary users to create arbitrary Pods.
  • We move enforcement below Kubernetes, into the host runtime layout.
  • kubelet, containerd, and the CRI/CNI-equivalent path all run inside the same restricted host-controlled view.
  • hostPath is not "allowed in general". It is replaced by an explicit host-side allowlist and path projection model.