Back to blog

Create Your Own Talos or CoreOS Linux

May 30, 2026

Talos shows that a Kubernetes-focused OS can be tiny, immutable, and purpose-built, much like CoreOS once did for container hosts. But the more interesting idea is that this approach is not magic or exclusive to one vendor. With tools like LinuxKit, a minimal boot image, and a small Kubernetes runtime, you can build your own stripped-down node OS: custom, repeatable, and designed to include only the components your cluster actually needs.

This walkthrough combines two labs. Part I builds three immutable LinuxKit kernel+initrd images, boots them as Firecracker microVMs, and turns them into a small Kubernetes cluster with one control-plane node and two workers using k3s. Part II builds one reusable guest rootfs, clones it into multiple Firecracker disks, and brings up a full kubeadm HA cluster: three control-plane nodes with stacked etcd, two workers, and the Kubernetes API exposed through a host-side HAProxy endpoint.

Part I — LinuxKit and k3s

This part runs directly on a Linux lab host. Run everything as root on the host that will run Firecracker. The host needs Docker, Firecracker, ip, iptables, ssh, ssh-keygen, openssl, and outbound internet access for pulling container images. Pick a subnet that does not overlap with other bridges on the host.

1. Export The Lab Variables

export RUN_ROOT=/var/lib/create-your-own-talos
export NODE_COUNT=3
export SUBNET_PREFIX=198.19.0
export SERVER_IP="${SUBNET_PREFIX}.10"
export GATEWAY="${SUBNET_PREFIX}.1"
export CIDR="${SUBNET_PREFIX}.0/24"
export BRIDGE_NAME=lkt198
export TAP_PREFIX=lkt198
export FIRECRACKER_BIN=/usr/local/bin/firecracker
export LINUXKIT_VERSION=v1.8.2
export LINUXKIT_BIN=/root/linuxkit/bin/linuxkit
export LINUXKIT_KERNEL_IMAGE=linuxkit/kernel:6.12.59
export K3S_IMAGE=rancher/k3s:v1.36.1-k3s1
export K3S_TOKEN="$(openssl rand -hex 24)"
export GUEST_SSH_KEY="${RUN_ROOT}/lab_ssh_key"
export GUEST_SSH_PUB="${GUEST_SSH_KEY}.pub"

2. Install LinuxKit And The Guest SSH Key

command -v docker ip iptables ssh ssh-keygen openssl >/dev/null
test -x "${FIRECRACKER_BIN}"
mkdir -p "$(dirname "${LINUXKIT_BIN}")" "${RUN_ROOT}"
if [[ ! -x "${LINUXKIT_BIN}" ]]; then
  curl -fsSL \
    "https://github.com/linuxkit/linuxkit/releases/download/${LINUXKIT_VERSION}/linuxkit-linux-amd64" \
    -o "${LINUXKIT_BIN}"
  chmod +x "${LINUXKIT_BIN}"
fi
if [[ ! -f "${GUEST_SSH_KEY}" || ! -f "${GUEST_SSH_PUB}" ]]; then
  ssh-keygen -q -t ed25519 -N "" -f "${GUEST_SSH_KEY}"
fi

3. Define A Few Helper Functions

node_ip() {
  printf '%s.%d\n' "${SUBNET_PREFIX}" "$((10 + $1))"
}
node_name() {
  printf 'linuxkit-%02d\n' "$1"
}
node_tap() {
  printf '%s%d\n' "${TAP_PREFIX}" "$1"
}
node_mac() {
  printf '06:19:80:00:00:%02x\n' "$((16 + $1))"
}
node_mem() {
  if [[ "$1" == "0" ]]; then
    echo 2048
  else
    echo 1536
  fi
}
SSH_OPTS=(-n -i "${GUEST_SSH_KEY}" -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR)
server_ssh() {
  ssh "${SSH_OPTS[@]}" "root@${SERVER_IP}" "$@"
}
k3s_exec() {
  local cmd="$1"
  local exec_id="x$(date +%s%N)"
  server_ssh "ctr -n services.linuxkit tasks exec --exec-id ${exec_id} k3s /bin/sh -lc $(printf '%q' "${cmd}")"
}

4. Write The LinuxKit YAML

Control plane:

cat > "${RUN_ROOT}/server.yml" <<EOF
kernel:
  image: ${LINUXKIT_KERNEL_IMAGE}
  cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
  - linuxkit/init:b5506cc74a6812dc40982cacfd2f4328f8a4b12a
  - linuxkit/runc:9442aa234715e751a16144f1d4ae3fd1a00fd492
  - linuxkit/containerd:ba19f64efd3331a8fd0a33e00eabd14f6ee1780e
  - linuxkit/ca-certificates:256f1950df59f2f209e9f0b81374177409eb11de
onboot:
  - name: sysctl
    image: linuxkit/sysctl:43ac1d39da329c3567fcb9689e5ca99de6d169b6
  - name: ip
    image: linuxkit/ip:3c0676ee83a0dc739685be1253b8326f08581ef7
    binds:
      - /etc/ip:/etc/ip
    command: ["ip", "-b", "/etc/ip/eth0.conf"]
services:
  - name: getty
    image: linuxkit/getty:a86d74c8f89be8956330c3b115b0b1f2e09ef6e0
    env:
      - INSECURE=true
  - name: rngd
    image: linuxkit/rngd:984eb580ecb63986f07f626b61692a97aacd7198
  - name: sshd
    image: linuxkit/sshd:08e5d4a46603eff485d5d1b14001cc932a530858
    binds.add:
      - /root/.ssh:/root/.ssh
  - name: k3s
    image: ${K3S_IMAGE}
    capabilities:
      - all
    net: host
    rootfsPropagation: shared
    devices:
      - path: /dev/kmsg
        type: c
        major: 1
        minor: 11
        mode: "0666"
    mounts:
      - type: cgroup
        options: ["rw","nosuid","noexec","nodev","relatime"]
    binds:
      - /etc/resolv.conf:/etc/resolv.conf
      - /lib/modules:/lib/modules
      - /var:/var:rshared,rbind
    env:
      - PATH=/bin:/bin/aux:/usr/bin:/sbin:/usr/sbin
      - K3S_TOKEN=${K3S_TOKEN}
    command:
      - /bin/k3s
      - server
      - --cluster-init
      - --node-name=linuxkit-00
      - --node-ip=${SERVER_IP}
      - --advertise-address=${SERVER_IP}
      - --bind-address=0.0.0.0
      - --tls-san=${SERVER_IP}
      - --write-kubeconfig-mode=0644
      - --disable=traefik
      - --disable=servicelb
      - --disable=metrics-server
      - --disable=local-storage
      - --disable-cloud-controller
      - --disable-network-policy
      - --flannel-iface=eth0
      - --flannel-backend=host-gw
files:
  - path: etc/ip/eth0.conf
    contents: |
      address add ${SERVER_IP}/24 dev eth0
      link set eth0 up
      route add default via ${GATEWAY} dev eth0
  - path: etc/resolv.conf
    contents: |
      nameserver 1.1.1.1
      nameserver 8.8.8.8
  - path: root/.ssh/authorized_keys
    source: ${GUEST_SSH_PUB}
    mode: "0600"
EOF

Workers:

cat > "${RUN_ROOT}/agent.yml.tmpl" <<EOF
kernel:
  image: ${LINUXKIT_KERNEL_IMAGE}
  cmdline: "console=tty0 console=ttyS0 console=ttyAMA0"
init:
  - linuxkit/init:b5506cc74a6812dc40982cacfd2f4328f8a4b12a
  - linuxkit/runc:9442aa234715e751a16144f1d4ae3fd1a00fd492
  - linuxkit/containerd:ba19f64efd3331a8fd0a33e00eabd14f6ee1780e
  - linuxkit/ca-certificates:256f1950df59f2f209e9f0b81374177409eb11de
onboot:
  - name: sysctl
    image: linuxkit/sysctl:43ac1d39da329c3567fcb9689e5ca99de6d169b6
  - name: ip
    image: linuxkit/ip:3c0676ee83a0dc739685be1253b8326f08581ef7
    binds:
      - /etc/ip:/etc/ip
    command: ["ip", "-b", "/etc/ip/eth0.conf"]
services:
  - name: getty
    image: linuxkit/getty:a86d74c8f89be8956330c3b115b0b1f2e09ef6e0
    env:
      - INSECURE=true
  - name: rngd
    image: linuxkit/rngd:984eb580ecb63986f07f626b61692a97aacd7198
  - name: sshd
    image: linuxkit/sshd:08e5d4a46603eff485d5d1b14001cc932a530858
    binds.add:
      - /root/.ssh:/root/.ssh
  - name: k3s
    image: ${K3S_IMAGE}
    capabilities:
      - all
    net: host
    rootfsPropagation: shared
    devices:
      - path: /dev/kmsg
        type: c
        major: 1
        minor: 11
        mode: "0666"
    mounts:
      - type: cgroup
        options: ["rw","nosuid","noexec","nodev","relatime"]
    binds:
      - /etc/resolv.conf:/etc/resolv.conf
      - /lib/modules:/lib/modules
      - /var:/var:rshared,rbind
    env:
      - PATH=/bin:/bin/aux:/usr/bin:/sbin:/usr/sbin
      - K3S_TOKEN=${K3S_TOKEN}
    command:
      - /bin/k3s
      - agent
      - --server=https://${SERVER_IP}:6443
      - --node-name=__NODE_NAME__
      - --node-ip=__NODE_IP__
      - --flannel-iface=eth0
files:
  - path: etc/ip/eth0.conf
    contents: |
      address add __NODE_IP__/24 dev eth0
      link set eth0 up
      route add default via ${GATEWAY} dev eth0
  - path: etc/resolv.conf
    contents: |
      nameserver 1.1.1.1
      nameserver 8.8.8.8
  - path: root/.ssh/authorized_keys
    source: ${GUEST_SSH_PUB}
    mode: "0600"
EOF

5. Build The Images

mkdir -p "${RUN_ROOT}/nodes/node0"
cp "${RUN_ROOT}/server.yml" "${RUN_ROOT}/nodes/node0/node.yml"
"${LINUXKIT_BIN}" build \
  --decompress-kernel \
  --dir "${RUN_ROOT}/nodes/node0" \
  --name node \
  -f kernel+initrd \
  "${RUN_ROOT}/nodes/node0/node.yml"

for i in 1 2; do
  ip="$(node_ip "${i}")"
  name="$(node_name "${i}")"
  dir="${RUN_ROOT}/nodes/node${i}"
  mkdir -p "${dir}"
  sed \
    -e "s/__NODE_NAME__/${name}/g" \
    -e "s/__NODE_IP__/${ip}/g" \
    "${RUN_ROOT}/agent.yml.tmpl" > "${dir}/node.yml"
  "${LINUXKIT_BIN}" build \
    --decompress-kernel \
    --dir "${dir}" \
    --name node \
    -f kernel+initrd \
    "${dir}/node.yml"
done

6. Prepare The Bridge And NAT

ip link add name "${BRIDGE_NAME}" type bridge 2>/dev/null || true
ip addr add "${GATEWAY}/24" dev "${BRIDGE_NAME}" 2>/dev/null || true
ip link set "${BRIDGE_NAME}" up
sysctl -w net.ipv4.ip_forward=1
iptables -C FORWARD -i "${BRIDGE_NAME}" -j ACCEPT 2>/dev/null || iptables -A FORWARD -i "${BRIDGE_NAME}" -j ACCEPT
iptables -C FORWARD -o "${BRIDGE_NAME}" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || \
  iptables -A FORWARD -o "${BRIDGE_NAME}" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -C POSTROUTING -s "${CIDR}" ! -o "${BRIDGE_NAME}" -j MASQUERADE 2>/dev/null || \
  iptables -t nat -A POSTROUTING -s "${CIDR}" ! -o "${BRIDGE_NAME}" -j MASQUERADE

The explicit FORWARD rules matter on hosts where the default FORWARD policy is DROP. NAT alone is not enough.

7. Boot The MicroVMs

for i in 0 1 2; do
  dir="${RUN_ROOT}/nodes/node${i}"
  tap="$(node_tap "${i}")"
  mac="$(node_mac "${i}")"
  mem="$(node_mem "${i}")"
  ip tuntap add dev "${tap}" mode tap 2>/dev/null || true
  ip link set "${tap}" master "${BRIDGE_NAME}"
  ip link set "${tap}" up
  cat > "${dir}/vm.json" <<EOF
{"boot-source":{"kernel_image_path":"${dir}/node-kernel","initrd_path":"${dir}/node-initrd.img","boot_args":"$(tr -d '\n' <"${dir}/node-cmdline") reboot=k panic=1 pci=off random.trust_cpu=on"},"drives":[],"machine-config":{"vcpu_count":2,"mem_size_mib":${mem}},"network-interfaces":[{"iface_id":"eth0","host_dev_name":"${tap}","guest_mac":"${mac}"}],"logger":{"log_path":"${dir}/firecracker.log","level":"Info","show_level":true,"show_log_origin":true}}
EOF
  "${FIRECRACKER_BIN}" \
    --api-sock "${dir}/fc.sock" \
    --config-file "${dir}/vm.json" \
    > "${dir}/console.log" 2>&1 &
  echo $! > "${dir}/pid"
done

8. Wait For SSH And Cluster Readiness

for ip in "$(node_ip 0)" "$(node_ip 1)" "$(node_ip 2)"; do
  until ssh "${SSH_OPTS[@]}" "root@${ip}" true >/dev/null 2>&1; do
    sleep 2
  done
done

for _ in $(seq 1 180); do
  k3s_exec "kubectl get nodes -o wide > /var/log/create-your-own-talos.nodes 2>&1 || true" >/dev/null 2>&1 || true
  ready_count="$(server_ssh "awk '\$2==\"Ready\"{c++} END{print c+0}' /var/log/create-your-own-talos.nodes 2>/dev/null || echo 0")"
  if [[ "${ready_count}" -ge "${NODE_COUNT}" ]]; then
    break
  fi
  sleep 3
done
server_ssh "cat /var/log/create-your-own-talos.nodes"

Expected result:

linuxkit-00   Ready   control-plane,etcd
linuxkit-01   Ready   <none>
linuxkit-02   Ready   <none>

9. Run A Smoke Test

k3s_exec "$(cat <<'EOF'
kubectl create namespace smoke --dry-run=client -o yaml | kubectl apply -f -
cat <<'YAML' | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-smoke
  namespace: smoke
spec:
  selector:
    matchLabels:
      app: node-smoke
  template:
    metadata:
      labels:
        app: node-smoke
    spec:
      tolerations:
        - operator: Exists
      containers:
        - name: node-smoke
          image: busybox:1.36
          command: ["sh", "-lc", "sleep 3600"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: smoke
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: smoke
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 80
YAML
kubectl -n smoke rollout status daemonset/node-smoke --timeout=240s
kubectl -n smoke rollout status deployment/echo --timeout=240s
kubectl -n kube-system rollout status deployment/coredns --timeout=240s
cat <<'YAML' | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: dns-http
  namespace: smoke
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: dns-http
          image: busybox:1.36
          command:
            - sh
            - -lc
            - |
              for _ in $(seq 1 30); do
                nslookup echo.smoke.svc.cluster.local && \
                wget -qO- http://echo.smoke.svc.cluster.local >/tmp/echo.html && \
                test -s /tmp/echo.html && exit 0
                sleep 2
              done
              exit 1
YAML
kubectl -n smoke wait --for=condition=complete job/dns-http --timeout=240s
kubectl get pods -A -o wide > /var/log/create-your-own-talos.pods 2>&1
kubectl -n smoke get pods -o wide > /var/log/create-your-own-talos.smoke-pods 2>&1
kubectl -n smoke logs job/dns-http > /var/log/create-your-own-talos.smoke-job.log 2>&1
kubectl config view --raw > /var/log/create-your-own-talos.kubeconfig 2>&1
EOF
)"
server_ssh "cat /var/log/create-your-own-talos.smoke-pods"
server_ssh "cat /var/log/create-your-own-talos.smoke-job.log"

That proves:

  • all three nodes can schedule workloads;
  • in-cluster DNS resolves echo.smoke.svc.cluster.local;
  • a pod can reach the service over HTTP.

10. Save The Artifacts

mkdir -p "${RUN_ROOT}/artifacts"
server_ssh "cat /var/log/create-your-own-talos.nodes" > "${RUN_ROOT}/artifacts/nodes.txt"
server_ssh "cat /var/log/create-your-own-talos.pods" > "${RUN_ROOT}/artifacts/pods.txt"
server_ssh "cat /var/log/create-your-own-talos.smoke-pods" > "${RUN_ROOT}/artifacts/smoke-pods.txt"
server_ssh "cat /var/log/create-your-own-talos.smoke-job.log" > "${RUN_ROOT}/artifacts/smoke-job.log"
server_ssh "cat /var/log/create-your-own-talos.kubeconfig" \
  | sed -e "s#https://127.0.0.1:6443#https://${SERVER_IP}:6443#g" \
        -e "s#https://0.0.0.0:6443#https://${SERVER_IP}:6443#g" \
  > "${RUN_ROOT}/artifacts/kubeconfig.yaml"

From there:

export KUBECONFIG="${RUN_ROOT}/artifacts/kubeconfig.yaml"
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl -n smoke get pods -o wide

11. Tear It Down

for pid_file in "${RUN_ROOT}"/nodes/*/pid; do
  [[ -f "${pid_file}" ]] || continue
  kill "$(cat "${pid_file}")" 2>/dev/null || true
done
sleep 1
for pid_file in "${RUN_ROOT}"/nodes/*/pid; do
  [[ -f "${pid_file}" ]] || continue
  kill -9 "$(cat "${pid_file}")" 2>/dev/null || true
done
for i in 0 1 2; do
  ip link del "$(node_tap "${i}")" 2>/dev/null || true
done
iptables -D FORWARD -i "${BRIDGE_NAME}" -j ACCEPT 2>/dev/null || true
iptables -D FORWARD -o "${BRIDGE_NAME}" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || true
iptables -t nat -D POSTROUTING -s "${CIDR}" ! -o "${BRIDGE_NAME}" -j MASQUERADE 2>/dev/null || true
ip link set "${BRIDGE_NAME}" down 2>/dev/null || true
ip link del "${BRIDGE_NAME}" type bridge 2>/dev/null || true
rm -rf "${RUN_ROOT}"

Part II — kubeadm HA

The second lab runs on the same Linux host. It builds one reusable guest rootfs, clones it into multiple Firecracker microVM disks, boots three kubeadm control-plane nodes with stacked etcd (one member per control-plane node), two workers by default, and exposes the Kubernetes API through a host-side HAProxy endpoint.

The host needs Docker, Firecracker, curl, ip, iptables, ssh, scp, ssh-keygen, unsquashfs from squashfs-tools, mkfs.ext4, e2fsck, resize2fs from e2fsprogs, mount, umount, truncate, chroot, and outbound internet access for the LinuxKit kernel image, the Firecracker CI Ubuntu rootfs, Kubernetes packages from pkgs.k8s.io, the Docker containerd package repo, the Flannel manifest, and the HAProxy container image.

1. Export The Lab Variables

This guide creates 3 control-plane nodes and 2 workers by default. Change WORKER_COUNT if you want fewer or more workers.

export RUN_ROOT=/var/lib/firecracker-kubeadm-ha
export CACHE_ROOT=/var/cache/firecracker-kubeadm-ha
export CONTROL_PLANE_COUNT=3
export WORKER_COUNT=2
export NODE_COUNT="$((CONTROL_PLANE_COUNT + WORKER_COUNT))"
export SUBNET_PREFIX=198.19.0
export GATEWAY="${SUBNET_PREFIX}.1"
export CIDR="${SUBNET_PREFIX}.0/24"
export API_LB_IP="${SUBNET_PREFIX}.5"
export API_LB_PORT=6443
export PRIMARY_CONTROL_PLANE_IP="${SUBNET_PREFIX}.10"
export BRIDGE_NAME=k8sha198
export TAP_PREFIX=k8sha198
export FIRECRACKER_BIN=/usr/local/bin/firecracker
export FIRECRACKER_ARCH="$(uname -m)"
export LINUXKIT_KERNEL_IMAGE=linuxkit/kernel:6.12.59
export ROOTFS_SIZE_GIB=12
export CONTROL_PLANE_MEM_MIB=2048
export WORKER_MEM_MIB=1536
export VCPU_COUNT=2
export KUBERNETES_MINOR=v1.36
export KUBERNETES_VERSION=v1.36.1
export POD_CIDR=10.244.0.0/16
export SERVICE_CIDR=10.96.0.0/12
export CNI_PLUGINS_VERSION=v1.3.0
export NETWORK_PLUGIN=flannel
export FLANNEL_MANIFEST_URL="https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml"
export HAPROXY_IMAGE=haproxy:3.2.19-alpine
export API_LB_CONTAINER_NAME="kubeadm-ha-api-lb-${BRIDGE_NAME}"
export GUEST_SSH_KEY="${CACHE_ROOT}/lab_ssh_key"
export GUEST_SSH_PUB="${GUEST_SSH_KEY}.pub"
export ARTIFACT_PREFIX=/var/log/kubeadm-ha-lab
export KERNEL_BOOT_ARGS="console=ttyS0 reboot=k panic=1 pci=off root=/dev/vda rw random.trust_cpu=on [email protected] systemd.mask=systemd-random-seed.service"

2. Check The Host Dependencies And Create The Guest SSH Key

command -v awk chroot curl docker e2fsck grep ip iptables mkfs.ext4 mount \
  resize2fs sha256sum sort ssh scp ssh-keygen truncate umount unsquashfs >/dev/null
test -x "${FIRECRACKER_BIN}"
mkdir -p "${RUN_ROOT}" "${CACHE_ROOT}"
if [[ ! -f "${GUEST_SSH_KEY}" || ! -f "${GUEST_SSH_PUB}" ]]; then
  ssh-keygen -q -t ed25519 -N "" -f "${GUEST_SSH_KEY}"
fi

3. Define A Few Helper Functions

node_role() {
  if (( $1 < CONTROL_PLANE_COUNT )); then
    echo control-plane
  else
    echo worker
  fi
}
node_ip() {
  printf '%s.%d\n' "${SUBNET_PREFIX}" "$((10 + $1))"
}
node_name() {
  printf 'k8s-%02d\n' "$1"
}
node_tap() {
  printf '%s%d\n' "${TAP_PREFIX}" "$1"
}
node_mac() {
  printf '06:36:00:00:00:%02x\n' "$((16 + $1))"
}
node_mem() {
  if [[ "$(node_role "$1")" == "control-plane" ]]; then
    echo "${CONTROL_PLANE_MEM_MIB}"
  else
    echo "${WORKER_MEM_MIB}"
  fi
}
SSH_OPTS=(-i "${GUEST_SSH_KEY}" -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=5)
node_ssh() {
  local idx="$1"
  shift
  ssh "${SSH_OPTS[@]}" "root@$(node_ip "${idx}")" "$@"
}
server_ssh() {
  node_ssh 0 "$@"
}
copy_guest_file() {
  local remote_path="$1"
  local local_path="$2"
  if server_ssh "test -f $(printf '%q' "${remote_path}")" >/dev/null 2>&1; then
    server_ssh "cat $(printf '%q' "${remote_path}")" > "${local_path}"
  fi
}
wait_for_ssh() {
  local ip="$1"
  for _ in $(seq 1 120); do
    if ssh "${SSH_OPTS[@]}" "root@${ip}" true >/dev/null 2>&1; then
      return 0
    fi
    sleep 2
  done
  return 1
}
wait_for_node_ready() {
  local name="$1"
  for _ in $(seq 1 240); do
    if server_ssh "kubectl --kubeconfig /etc/kubernetes/admin.conf get node ${name} --no-headers 2>/dev/null | awk '\$2==\"Ready\"{ok=1} END{exit(ok?0:1)}'"; then
      return 0
    fi
    sleep 2
  done
  return 1
}

4. Download The Guest Kernel And Rootfs Inputs

This guide uses a LinuxKit kernel image and the latest Firecracker CI Ubuntu squashfs for your Firecracker major/minor.

4.1 Pull The LinuxKit Kernel And Extract vmlinux

export LINUXKIT_KEY="$(printf '%s\n' "${LINUXKIT_KERNEL_IMAGE}" | sha256sum | awk '{print substr($1,1,16)}')"
export LINUXKIT_CACHE_DIR="${CACHE_ROOT}/downloads/linuxkit-${LINUXKIT_KEY}"
export KERNEL_PATH="${LINUXKIT_CACHE_DIR}/vmlinux"
export KERNEL_BZIMAGE_PATH="${LINUXKIT_CACHE_DIR}/kernel"
export KERNEL_MODULES_TAR_PATH="${LINUXKIT_CACHE_DIR}/kernel.tar"
export KERNEL_DEV_TAR_PATH="${LINUXKIT_CACHE_DIR}/kernel-dev.tar"
mkdir -p "${LINUXKIT_CACHE_DIR}"
if [[ ! -f "${KERNEL_PATH}" || ! -f "${KERNEL_MODULES_TAR_PATH}" ]]; then
  rm -rf "${LINUXKIT_CACHE_DIR}.tmp"
  mkdir -p "${LINUXKIT_CACHE_DIR}.tmp"
  docker pull "${LINUXKIT_KERNEL_IMAGE}" >/dev/null
  cid="$(docker create "${LINUXKIT_KERNEL_IMAGE}" /bin/sh)"
  docker cp "${cid}:/kernel" "${LINUXKIT_CACHE_DIR}.tmp/kernel"
  docker cp "${cid}:/kernel.tar" "${LINUXKIT_CACHE_DIR}.tmp/kernel.tar"
  docker cp "${cid}:/kernel-dev.tar" "${LINUXKIT_CACHE_DIR}.tmp/kernel-dev.tar"
  docker rm -f "${cid}" >/dev/null
  export LINUXKIT_HEADERS_DIR="$(
    tar -tf "${LINUXKIT_CACHE_DIR}.tmp/kernel-dev.tar" \
      | sed -n 's#^\(usr/src/linux-headers-[^/]*\)/scripts/extract-vmlinux$#\1#p' \
      | head -n 1
  )"
  tar -xOf "${LINUXKIT_CACHE_DIR}.tmp/kernel-dev.tar" "${LINUXKIT_HEADERS_DIR}/scripts/extract-vmlinux" > "${LINUXKIT_CACHE_DIR}.tmp/extract-vmlinux"
  chmod +x "${LINUXKIT_CACHE_DIR}.tmp/extract-vmlinux"
  "${LINUXKIT_CACHE_DIR}.tmp/extract-vmlinux" "${LINUXKIT_CACHE_DIR}.tmp/kernel" > "${LINUXKIT_CACHE_DIR}.tmp/vmlinux"
  mv "${LINUXKIT_CACHE_DIR}.tmp/kernel" "${KERNEL_BZIMAGE_PATH}"
  mv "${LINUXKIT_CACHE_DIR}.tmp/vmlinux" "${KERNEL_PATH}"
  mv "${LINUXKIT_CACHE_DIR}.tmp/kernel.tar" "${KERNEL_MODULES_TAR_PATH}"
  mv "${LINUXKIT_CACHE_DIR}.tmp/kernel-dev.tar" "${KERNEL_DEV_TAR_PATH}"
  rm -rf "${LINUXKIT_CACHE_DIR}.tmp"
fi

4.2 Download The Firecracker Ubuntu Rootfs Squashfs

export FIRECRACKER_CI_VERSION="$("${FIRECRACKER_BIN}" --version | awk '{print $2}' | sed 's/\.[0-9]*$//')"
export UBUNTU_KEY="$(
  curl -fsSL "https://s3.amazonaws.com/spec.ccfc.min?prefix=firecracker-ci/${FIRECRACKER_CI_VERSION}/${FIRECRACKER_ARCH}/ubuntu-&list-type=2" \
    | grep -oP "(?<=<Key>)(firecracker-ci/${FIRECRACKER_CI_VERSION}/${FIRECRACKER_ARCH}/ubuntu-[0-9]+\.[0-9]+\.squashfs)(?=</Key>)" \
    | sort -V | tail -1
)"
export ROOTFS_SQUASHFS_PATH="${CACHE_ROOT}/downloads/$(basename "${UBUNTU_KEY}")"
mkdir -p "${CACHE_ROOT}/downloads"
if [[ ! -f "${ROOTFS_SQUASHFS_PATH}" ]]; then
  curl -fsSL "https://s3.amazonaws.com/spec.ccfc.min/${UBUNTU_KEY}" -o "${ROOTFS_SQUASHFS_PATH}"
fi

5. Create The Base ext4 Rootfs

This converts the squashfs into an ext4 image and injects the guest SSH key.

export BASE_KEY="$(
  {
    sha256sum "${ROOTFS_SQUASHFS_PATH}"
    sha256sum "${GUEST_SSH_PUB}"
  } | sha256sum | awk '{print substr($1,1,16)}'
)"
export BASE_ROOTFS_PATH="${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.ext4"
mkdir -p "${CACHE_ROOT}/base"
if [[ ! -s "${BASE_ROOTFS_PATH}" ]]; then
  rm -rf "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs"
  rm -f "${BASE_ROOTFS_PATH}.tmp"
  mkdir -p "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs"
  unsquashfs -d "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs" "${ROOTFS_SQUASHFS_PATH}" >/dev/null
  mkdir -p "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs/root/.ssh"
  chmod 700 "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs/root/.ssh"
  cp "${GUEST_SSH_PUB}" "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs/root/.ssh/authorized_keys"
  chmod 600 "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs/root/.ssh/authorized_keys"
  if [[ -f "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs/etc/ssh/sshd_config" ]]; then
    sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin yes/' "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs/etc/ssh/sshd_config" || true
    sed -i 's/^#\?PubkeyAuthentication.*/PubkeyAuthentication yes/' "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs/etc/ssh/sshd_config" || true
    sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs/etc/ssh/sshd_config" || true
  fi
  chown -R root:root "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs"
  truncate -s 4G "${BASE_ROOTFS_PATH}.tmp"
  mkfs.ext4 -d "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs/rootfs" -F "${BASE_ROOTFS_PATH}.tmp" >/dev/null
  mv "${BASE_ROOTFS_PATH}.tmp" "${BASE_ROOTFS_PATH}"
  rm -rf "${CACHE_ROOT}/base/ubuntu-${BASE_KEY}.rootfs"
fi

6. Prepare One Reusable kubeadm-Capable Guest Image

This installs kernel modules from the LinuxKit kernel tarball, containerd, kubelet, kubeadm, kubectl, CNI plugins, Kubernetes sysctls, and base service wiring. 

export PREPARED_KEY="$(
  {
    sha256sum "${BASE_ROOTFS_PATH}" "${KERNEL_PATH}" "${GUEST_SSH_PUB}" "${KERNEL_MODULES_TAR_PATH}"
    printf 'kubernetes_minor=%s\n' "${KUBERNETES_MINOR}"
    printf 'kubernetes_version=%s\n' "${KUBERNETES_VERSION}"
    printf 'cni_plugins=%s\n' "${CNI_PLUGINS_VERSION}"
    printf 'rootfs_size_gib=%s\n' "${ROOTFS_SIZE_GIB}"
    printf 'generation=manual-kubeadm-firecracker-ha-v1\n'
  } | sha256sum | awk '{print substr($1,1,16)}'
)"
export PREPARED_ROOTFS_PATH="${CACHE_ROOT}/prepared-${PREPARED_KEY}.ext4"
if [[ ! -s "${PREPARED_ROOTFS_PATH}" ]]; then
  cp --reflink=auto "${BASE_ROOTFS_PATH}" "${PREPARED_ROOTFS_PATH}.tmp" 2>/dev/null || cp "${BASE_ROOTFS_PATH}" "${PREPARED_ROOTFS_PATH}.tmp"
  e2fsck -fy "${PREPARED_ROOTFS_PATH}.tmp" >/dev/null 2>&1 || true
  truncate -s "${ROOTFS_SIZE_GIB}G" "${PREPARED_ROOTFS_PATH}.tmp"
  resize2fs "${PREPARED_ROOTFS_PATH}.tmp" >/dev/null
  export MNT="${CACHE_ROOT}/mnt-${PREPARED_KEY}"
  mkdir -p "${MNT}"
  mount -o loop "${PREPARED_ROOTFS_PATH}.tmp" "${MNT}"
  mount -t proc proc "${MNT}/proc"
  mount -t sysfs sysfs "${MNT}/sys"
  mount --bind /dev "${MNT}/dev"
  mount -t devpts devpts "${MNT}/dev/pts"
  mount --bind /run "${MNT}/run"
  rm -f "${MNT}/etc/resolv.conf"
  printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\noptions timeout:2 attempts:3\n' > "${MNT}/etc/resolv.conf"
  rm -rf "${MNT}/lib/modules"
  mkdir -p "${MNT}/lib"
  tar -xf "${KERNEL_MODULES_TAR_PATH}" -C "${MNT}" ./lib/modules
  chmod 1777 "${MNT}/tmp"
  mkdir -p "${MNT}/dev/pts" "${MNT}/var/cache/apt/archives/partial" "${MNT}/var/lib/apt/lists/partial" "${MNT}/var/log/apt"
  touch "${MNT}/var/log/dpkg.log"
  mkdir -p "${MNT}/root/.ssh"
  chmod 700 "${MNT}/root/.ssh"
  cp "${GUEST_SSH_PUB}" "${MNT}/root/.ssh/authorized_keys"
  chmod 600 "${MNT}/root/.ssh/authorized_keys"
  if [[ -f "${MNT}/etc/ssh/sshd_config" ]]; then
    sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin yes/' "${MNT}/etc/ssh/sshd_config" || true
    sed -i 's/^#\?PubkeyAuthentication.*/PubkeyAuthentication yes/' "${MNT}/etc/ssh/sshd_config" || true
    sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' "${MNT}/etc/ssh/sshd_config" || true
  fi
  chroot "${MNT}" apt-get -o Acquire::Retries=5 -o Acquire::http::Timeout=20 -o Acquire::https::Timeout=20 update
  DEBIAN_FRONTEND=noninteractive chroot "${MNT}" apt-get -o Acquire::Retries=5 -o Acquire::http::Timeout=20 -o Acquire::https::Timeout=20 \
    -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install -y --no-install-recommends \
    apt-transport-https ca-certificates conntrack curl ebtables ethtool gpg iptables ipset jq openssh-server socat tar xz-utils
  mkdir -p "${MNT}/etc/apt/keyrings" "${MNT}/etc/apt/sources.list.d"
  docker_arch="$(chroot "${MNT}" dpkg --print-architecture)"
  docker_codename="$(awk -F= '/^VERSION_CODENAME=/{print $2}' "${MNT}/etc/os-release")"
  chroot "${MNT}" curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | chroot "${MNT}" gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  printf 'deb [arch=%s signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu %s stable\n' "${docker_arch}" "${docker_codename}" > "${MNT}/etc/apt/sources.list.d/docker.list"
  chroot "${MNT}" curl -fsSL "https://pkgs.k8s.io/core:/stable:/${KUBERNETES_MINOR}/deb/Release.key" | chroot "${MNT}" gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
  printf 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/%s/deb/ /\n' "${KUBERNETES_MINOR}" > "${MNT}/etc/apt/sources.list.d/kubernetes.list"
  chroot "${MNT}" apt-get -o Acquire::Retries=5 -o Acquire::http::Timeout=20 -o Acquire::https::Timeout=20 update
  DEBIAN_FRONTEND=noninteractive chroot "${MNT}" apt-get -o Acquire::Retries=5 -o Acquire::http::Timeout=20 -o Acquire::https::Timeout=20 \
    -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install -y containerd.io \
      "kubelet=${KUBERNETES_VERSION#v}-*" "kubeadm=${KUBERNETES_VERSION#v}-*" "kubectl=${KUBERNETES_VERSION#v}-*"
  chroot "${MNT}" apt-mark hold kubelet kubeadm kubectl >/dev/null 2>&1 || true
  case "$(uname -m)" in
    x86_64) cni_arch=amd64 ;;
    aarch64) cni_arch=arm64 ;;
    *) echo "unsupported CNI architecture: $(uname -m)" >&2; exit 1 ;;
  esac
  export CNI_ARCHIVE="${CACHE_ROOT}/downloads/cni-plugins-linux-${cni_arch}-${CNI_PLUGINS_VERSION}.tgz"
  if [[ ! -f "${CNI_ARCHIVE}" ]]; then
    curl -fsSL "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/cni-plugins-linux-${cni_arch}-${CNI_PLUGINS_VERSION}.tgz" -o "${CNI_ARCHIVE}"
  fi
  mkdir -p "${MNT}/opt/cni/bin"
  tar -C "${MNT}/opt/cni/bin" -xzf "${CNI_ARCHIVE}"
  pause_image="$(chroot "${MNT}" kubeadm config images list 2>/dev/null | awk '/pause/{print $1; exit}')"
  mkdir -p "${MNT}/etc/containerd"
  chroot "${MNT}" containerd config default > "${MNT}/etc/containerd/config.toml"
  sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' "${MNT}/etc/containerd/config.toml"
  if [[ -n "${pause_image}" ]]; then
    sed -i "s#sandbox_image = \".*\"#sandbox_image = \"${pause_image}\"#" "${MNT}/etc/containerd/config.toml"
  fi
  mkdir -p "${MNT}/etc/modules-load.d" "${MNT}/etc/sysctl.d" "${MNT}/etc/systemd/system/multi-user.target.wants"
  cat > "${MNT}/etc/modules-load.d/kubernetes.conf" <<'EOF'
overlay
br_netfilter
EOF
  cat > "${MNT}/etc/sysctl.d/99-kubernetes.conf" <<'EOF'
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
EOF
  touch "${MNT}/etc/cloud/cloud-init.disabled" 2>/dev/null || true
  chroot "${MNT}" update-alternatives --set iptables /usr/sbin/iptables-legacy >/dev/null 2>&1 || true
  chroot "${MNT}" update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy >/dev/null 2>&1 || true
  ln -sf /lib/systemd/system/ssh.service "${MNT}/etc/systemd/system/multi-user.target.wants/ssh.service"
  ln -sf /lib/systemd/system/systemd-networkd.service "${MNT}/etc/systemd/system/multi-user.target.wants/systemd-networkd.service"
  umount "${MNT}/dev/pts" || true
  umount "${MNT}/proc" || true
  umount "${MNT}/sys" || true
  umount "${MNT}/dev" || true
  umount "${MNT}/run" || true
  umount "${MNT}" || true
  mv "${PREPARED_ROOTFS_PATH}.tmp" "${PREPARED_ROOTFS_PATH}"
fi

7. Prepare The Bridge, NAT, And Host-Side API Load Balancer

The explicit FORWARD rules matter on hosts where the default FORWARD policy is DROP.

docker rm -f "${API_LB_CONTAINER_NAME}" >/dev/null 2>&1 || true
ip link add name "${BRIDGE_NAME}" type bridge 2>/dev/null || true
ip addr add "${GATEWAY}/24" dev "${BRIDGE_NAME}" 2>/dev/null || true
ip addr add "${API_LB_IP}/24" dev "${BRIDGE_NAME}" 2>/dev/null || true
ip link set "${BRIDGE_NAME}" up
sysctl -w net.ipv4.ip_forward=1 >/dev/null
iptables -C FORWARD -i "${BRIDGE_NAME}" -j ACCEPT 2>/dev/null || iptables -A FORWARD -i "${BRIDGE_NAME}" -j ACCEPT
iptables -C FORWARD -o "${BRIDGE_NAME}" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || \
  iptables -A FORWARD -o "${BRIDGE_NAME}" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -t nat -C POSTROUTING -s "${CIDR}" ! -o "${BRIDGE_NAME}" -j MASQUERADE 2>/dev/null || \
  iptables -t nat -A POSTROUTING -s "${CIDR}" ! -o "${BRIDGE_NAME}" -j MASQUERADE

cat > "${RUN_ROOT}/haproxy.cfg" <<EOF
global
  maxconn 2048
defaults
  mode tcp
  timeout connect 5s
  timeout client 60s
  timeout server 60s
frontend kube_api
  bind ${API_LB_IP}:${API_LB_PORT}
  default_backend kube_apis
backend kube_apis
  balance roundrobin
  option tcp-check
  default-server inter 2s fall 3 rise 2
EOF
for i in $(seq 0 "$((CONTROL_PLANE_COUNT - 1))"); do
  printf '  server cp%s %s:6443 check\n' "${i}" "$(node_ip "${i}")" >> "${RUN_ROOT}/haproxy.cfg"
done
docker run -d --name "${API_LB_CONTAINER_NAME}" --network host \
  -v "${RUN_ROOT}/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro" \
  "${HAPROXY_IMAGE}" >/dev/null

8. Create And Boot The MicroVMs

Each VM gets its own cloned ext4 disk, hostname, /etc/hosts, and systemd-networkd config.

for i in $(seq 0 "$((NODE_COUNT - 1))"); do
  vm_dir="${RUN_ROOT}/nodes/node${i}"
  mkdir -p "${vm_dir}"
  cp --reflink=auto "${PREPARED_ROOTFS_PATH}" "${vm_dir}/rootfs.ext4" 2>/dev/null || cp "${PREPARED_ROOTFS_PATH}" "${vm_dir}/rootfs.ext4"
  e2fsck -fy "${vm_dir}/rootfs.ext4" >/dev/null 2>&1 || true
  mkdir -p "${vm_dir}/mnt"
  mount -o loop "${vm_dir}/rootfs.ext4" "${vm_dir}/mnt"
  printf '%s\n' "$(node_name "${i}")" > "${vm_dir}/mnt/etc/hostname"
  {
    printf '127.0.0.1 localhost\n'
    printf '127.0.1.1 %s\n' "$(node_name "${i}")"
    printf '%s api-lb\n' "${API_LB_IP}"
    for j in $(seq 0 "$((NODE_COUNT - 1))"); do
      printf '%s %s\n' "$(node_ip "${j}")" "$(node_name "${j}")"
    done
  } > "${vm_dir}/mnt/etc/hosts"
  mkdir -p "${vm_dir}/mnt/etc/systemd/network" "${vm_dir}/mnt/etc/systemd/system/multi-user.target.wants"
  cat > "${vm_dir}/mnt/etc/systemd/network/20-eth0.network" <<EOF
[Match]
Name=eth0
[Network]
Address=$(node_ip "${i}")/24
Gateway=${GATEWAY}
DNS=1.1.1.1
DNS=8.8.8.8
EOF
  rm -f "${vm_dir}/mnt/etc/resolv.conf"
  printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' > "${vm_dir}/mnt/etc/resolv.conf"
  rm -f "${vm_dir}/mnt/etc/machine-id" "${vm_dir}/mnt/var/lib/dbus/machine-id" 2>/dev/null || true
  touch "${vm_dir}/mnt/etc/machine-id"
  rm -rf "${vm_dir}/mnt/etc/kubernetes" "${vm_dir}/mnt/var/lib/etcd" "${vm_dir}/mnt/var/lib/cni" "${vm_dir}/mnt/var/lib/kubelet" "${vm_dir}/mnt/var/lib/containerd" "${vm_dir}/mnt/etc/cni/net.d"
  mkdir -p "${vm_dir}/mnt/var/lib/containerd" "${vm_dir}/mnt/etc/cni/net.d"
  ln -sf /lib/systemd/system/ssh.service "${vm_dir}/mnt/etc/systemd/system/multi-user.target.wants/ssh.service"
  ln -sf /lib/systemd/system/systemd-networkd.service "${vm_dir}/mnt/etc/systemd/system/multi-user.target.wants/systemd-networkd.service"
  umount "${vm_dir}/mnt"
  tap="$(node_tap "${i}")"
  mac="$(node_mac "${i}")"
  mem="$(node_mem "${i}")"
  ip tuntap add dev "${tap}" mode tap 2>/dev/null || true
  ip link set "${tap}" master "${BRIDGE_NAME}"
  ip link set "${tap}" up
  cat > "${vm_dir}/vm.json" <<EOF
{"boot-source":{"kernel_image_path":"${KERNEL_PATH}","boot_args":"${KERNEL_BOOT_ARGS}"},"drives":[{"drive_id":"rootfs","path_on_host":"${vm_dir}/rootfs.ext4","is_root_device":true,"is_read_only":false}],"machine-config":{"vcpu_count":${VCPU_COUNT},"mem_size_mib":${mem}},"network-interfaces":[{"iface_id":"eth0","host_dev_name":"${tap}","guest_mac":"${mac}"}],"logger":{"log_path":"${vm_dir}/firecracker.log","level":"Info","show_level":true,"show_log_origin":true}}
EOF
  "${FIRECRACKER_BIN}" --api-sock "${vm_dir}/fc.sock" --config-file "${vm_dir}/vm.json" > "${vm_dir}/console.log" 2>&1 &
  echo $! > "${vm_dir}/pid"
done

9. Wait For SSH And Prepare The Nodes

for i in $(seq 0 "$((NODE_COUNT - 1))"); do
  wait_for_ssh "$(node_ip "${i}")"
done

for i in $(seq 0 "$((NODE_COUNT - 1))"); do
  node_ssh "${i}" "cat >/root/prepare-node.sh" <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
swapoff -a || true
sed -i '/ swap / s/^/#/' /etc/fstab 2>/dev/null || true
modprobe overlay || true
modprobe br_netfilter || true
sysctl --system >/dev/null
systemctl daemon-reload || true
systemctl enable --now ssh containerd kubelet >/dev/null 2>&1 || true
systemctl restart containerd
kubeadm reset -f >/dev/null 2>&1 || true
rm -rf /etc/kubernetes /var/lib/etcd /var/lib/cni /var/lib/kubelet/* /etc/cni/net.d/*
mkdir -p /etc/cni/net.d
kubeadm config images pull --kubernetes-version="$(kubeadm version -o short)" >/dev/null 2>&1 || true
EOF
  node_ssh "${i}" "chmod +x /root/prepare-node.sh && /root/prepare-node.sh"
done

10. Initialize The First Control Plane

server_ssh "cat >/root/init-primary.sh" <<EOF
#!/usr/bin/env bash
set -euo pipefail
version="${KUBERNETES_VERSION}"
kubeadm init \
  --kubernetes-version "\${version}" \
  --control-plane-endpoint "${API_LB_IP}:${API_LB_PORT}" \
  --apiserver-advertise-address "${PRIMARY_CONTROL_PLANE_IP}" \
  --pod-network-cidr "${POD_CIDR}" \
  --service-cidr "${SERVICE_CIDR}" \
  --upload-certs \
  --ignore-preflight-errors=all
mkdir -p /root/.kube
cp /etc/kubernetes/admin.conf /root/.kube/config
certificate_key="\$(kubeadm init phase upload-certs --upload-certs 2>/dev/null | tail -n 1)"
join_cmd="\$(kubeadm token create --print-join-command)"
printf '%s\n' "\${certificate_key}" >/root/certificate-key.txt
printf '%s\n' "\${join_cmd}" >/root/join-command.txt
EOF
server_ssh "chmod +x /root/init-primary.sh && /root/init-primary.sh"

11. Install Flannel

server_ssh "kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f $(printf '%q' "${FLANNEL_MANIFEST_URL}")"
for _ in $(seq 1 180); do
  flannel_ns="$(server_ssh "kubectl --kubeconfig /etc/kubernetes/admin.conf get daemonset -A --no-headers 2>/dev/null | awk '\$2==\"kube-flannel-ds\"{print \$1; exit}'" || true)"
  if [[ -n "${flannel_ns}" ]]; then
    if server_ssh "kubectl --kubeconfig /etc/kubernetes/admin.conf -n ${flannel_ns} rollout status daemonset/kube-flannel-ds --timeout=5s" >/dev/null 2>&1; then
      break
    fi
  fi
  sleep 2
done

12. Join The Other Two Control-Plane Nodes

join_cmd="$(server_ssh "cat /root/join-command.txt")"
certificate_key="$(server_ssh "cat /root/certificate-key.txt")"
for i in 1 2; do
  ip="$(node_ip "${i}")"
  node_ssh "${i}" "cat >/root/join-control-plane.sh" <<EOF
#!/usr/bin/env bash
set -euo pipefail
${join_cmd} --control-plane --certificate-key ${certificate_key} --apiserver-advertise-address ${ip} --ignore-preflight-errors=all
EOF
  node_ssh "${i}" "chmod +x /root/join-control-plane.sh && /root/join-control-plane.sh"
  wait_for_node_ready "$(node_name "${i}")"
done

13. Join The Worker Nodes

for i in $(seq "${CONTROL_PLANE_COUNT}" "$((NODE_COUNT - 1))"); do
  node_ssh "${i}" "cat >/root/join-worker.sh" <<EOF
#!/usr/bin/env bash
set -euo pipefail
${join_cmd} --ignore-preflight-errors=all
EOF
  node_ssh "${i}" "chmod +x /root/join-worker.sh && /root/join-worker.sh"
  wait_for_node_ready "$(node_name "${i}")"
done

14. Wait For Full Cluster Readiness

for _ in $(seq 1 240); do
  server_ssh "kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes -o wide > ${ARTIFACT_PREFIX}.nodes 2>&1 || true
kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes --no-headers > ${ARTIFACT_PREFIX}.nodes-plain 2>&1 || true
kubectl --kubeconfig /etc/kubernetes/admin.conf get endpoints kubernetes -o wide > ${ARTIFACT_PREFIX}.apiserver-endpoints 2>&1 || true" >/dev/null 2>&1 || true
  ready_count="$(server_ssh "awk '\$2==\"Ready\"{c++} END{print c+0}' ${ARTIFACT_PREFIX}.nodes-plain 2>/dev/null || echo 0")"
  control_plane_ready="$(server_ssh "awk '\$2==\"Ready\" && \$3 ~ /control-plane/ {c++} END{print c+0}' ${ARTIFACT_PREFIX}.nodes-plain 2>/dev/null || echo 0")"
  endpoint_count="$(server_ssh "kubectl --kubeconfig /etc/kubernetes/admin.conf get endpoints kubernetes -o jsonpath='{range .subsets[*].addresses[*]}{.ip}{\"\\n\"}{end}' 2>/dev/null | awk 'NF{c++} END{print c+0}' || echo 0")"
  if [[ "${ready_count}" -ge "${NODE_COUNT}" && "${control_plane_ready}" -ge "${CONTROL_PLANE_COUNT}" && "${endpoint_count}" -ge "${CONTROL_PLANE_COUNT}" ]]; then
    break
  fi
  sleep 3
done
server_ssh "cat ${ARTIFACT_PREFIX}.nodes"

Expected result:

k8s-00   Ready   control-plane
k8s-01   Ready   control-plane
k8s-02   Ready   control-plane
k8s-03   Ready   <none>
k8s-04   Ready   <none>

15. Capture The etcd Membership Proof

etcd_pod="etcd-$(node_name 0)"
server_ssh "kubectl --kubeconfig /etc/kubernetes/admin.conf -n kube-system exec ${etcd_pod} -- etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key member list -w table > ${ARTIFACT_PREFIX}.etcd-members 2>&1
kubectl --kubeconfig /etc/kubernetes/admin.conf -n kube-system exec ${etcd_pod} -- etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key endpoint status -w table > ${ARTIFACT_PREFIX}.etcd-endpoints 2>&1"
server_ssh "cat ${ARTIFACT_PREFIX}.etcd-members"

That should show three etcd members.

16. Run A Smoke Test

server_ssh "cat >/root/run-smoke.sh" <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
kubectl --kubeconfig /etc/kubernetes/admin.conf create namespace smoke --dry-run=client -o yaml | kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f -
cat <<'YAML' | kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-smoke
  namespace: smoke
spec:
  selector:
    matchLabels:
      app: node-smoke
  template:
    metadata:
      labels:
        app: node-smoke
    spec:
      tolerations:
        - operator: Exists
      containers:
        - name: node-smoke
          image: busybox:1.36
          command: ["sh", "-lc", "sleep 3600"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: smoke
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      tolerations:
        - operator: Exists
      containers:
        - name: echo
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: smoke
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 80
YAML
kubectl --kubeconfig /etc/kubernetes/admin.conf -n smoke rollout status daemonset/node-smoke --timeout=240s
kubectl --kubeconfig /etc/kubernetes/admin.conf -n smoke rollout status deployment/echo --timeout=240s
kubectl --kubeconfig /etc/kubernetes/admin.conf -n kube-system rollout status deployment/coredns --timeout=240s
cat <<'YAML' | kubectl --kubeconfig /etc/kubernetes/admin.conf apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: dns-http
  namespace: smoke
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      tolerations:
        - operator: Exists
      containers:
        - name: dns-http
          image: busybox:1.36
          command:
            - sh
            - -lc
            - |
              for _ in $(seq 1 30); do
                nslookup echo.smoke.svc.cluster.local && \
                wget -qO- http://echo.smoke.svc.cluster.local >/tmp/echo.html && \
                test -s /tmp/echo.html && exit 0
                sleep 2
              done
              exit 1
YAML
kubectl --kubeconfig /etc/kubernetes/admin.conf -n smoke wait --for=condition=complete job/dns-http --timeout=240s
kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes -o wide > /var/log/kubeadm-ha-lab.nodes 2>&1
kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -A -o wide > /var/log/kubeadm-ha-lab.pods 2>&1
kubectl --kubeconfig /etc/kubernetes/admin.conf get svc -A -o wide > /var/log/kubeadm-ha-lab.services 2>&1
kubectl --kubeconfig /etc/kubernetes/admin.conf -n smoke get pods -o wide > /var/log/kubeadm-ha-lab.smoke-pods 2>&1
kubectl --kubeconfig /etc/kubernetes/admin.conf -n smoke get svc echo -o wide > /var/log/kubeadm-ha-lab.smoke-service 2>&1
kubectl --kubeconfig /etc/kubernetes/admin.conf -n smoke logs job/dns-http > /var/log/kubeadm-ha-lab.smoke-job.log 2>&1
kubectl --kubeconfig /etc/kubernetes/admin.conf config view --raw > /var/log/kubeadm-ha-lab.kubeconfig 2>&1
EOF
server_ssh "chmod +x /root/run-smoke.sh && /root/run-smoke.sh"
server_ssh "cat ${ARTIFACT_PREFIX}.smoke-pods"
server_ssh "cat ${ARTIFACT_PREFIX}.smoke-job.log"

That proves:

  • all nodes can schedule workloads
  • in-cluster DNS resolves echo.smoke.svc.cluster.local
  • a pod can reach the service over HTTP

17. Save The Artifacts

mkdir -p "${RUN_ROOT}/artifacts"
copy_guest_file "${ARTIFACT_PREFIX}.nodes" "${RUN_ROOT}/artifacts/nodes.txt"
copy_guest_file "${ARTIFACT_PREFIX}.pods" "${RUN_ROOT}/artifacts/pods.txt"
copy_guest_file "${ARTIFACT_PREFIX}.services" "${RUN_ROOT}/artifacts/services.txt"
copy_guest_file "${ARTIFACT_PREFIX}.apiserver-endpoints" "${RUN_ROOT}/artifacts/apiserver-endpoints.txt"
copy_guest_file "${ARTIFACT_PREFIX}.etcd-members" "${RUN_ROOT}/artifacts/etcd-members.txt"
copy_guest_file "${ARTIFACT_PREFIX}.etcd-endpoints" "${RUN_ROOT}/artifacts/etcd-endpoints.txt"
copy_guest_file "${ARTIFACT_PREFIX}.smoke-pods" "${RUN_ROOT}/artifacts/smoke-pods.txt"
copy_guest_file "${ARTIFACT_PREFIX}.smoke-service" "${RUN_ROOT}/artifacts/smoke-service.txt"
copy_guest_file "${ARTIFACT_PREFIX}.smoke-job.log" "${RUN_ROOT}/artifacts/smoke-job.log"
copy_guest_file "${ARTIFACT_PREFIX}.kubeconfig" "${RUN_ROOT}/artifacts/kubeconfig.yaml"
curl -ksSf --max-time 5 "https://${API_LB_IP}:${API_LB_PORT}/version" > "${RUN_ROOT}/artifacts/api-lb-version.json"

From there:

export KUBECONFIG="${RUN_ROOT}/artifacts/kubeconfig.yaml"
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl -n smoke get pods -o wide

18. Tear It Down

docker rm -f "${API_LB_CONTAINER_NAME}" >/dev/null 2>&1 || true
for pid_file in "${RUN_ROOT}"/nodes/*/pid; do
  [[ -f "${pid_file}" ]] || continue
  kill "$(cat "${pid_file}")" 2>/dev/null || true
done
sleep 1
for pid_file in "${RUN_ROOT}"/nodes/*/pid; do
  [[ -f "${pid_file}" ]] || continue
  kill -9 "$(cat "${pid_file}")" 2>/dev/null || true
done
for i in $(seq 0 "$((NODE_COUNT - 1))"); do
  ip link del "$(node_tap "${i}")" 2>/dev/null || true
done
iptables -D FORWARD -i "${BRIDGE_NAME}" -j ACCEPT 2>/dev/null || true
iptables -D FORWARD -o "${BRIDGE_NAME}" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || true
iptables -t nat -D POSTROUTING -s "${CIDR}" ! -o "${BRIDGE_NAME}" -j MASQUERADE 2>/dev/null || true
ip link set "${BRIDGE_NAME}" down 2>/dev/null || true
ip link del "${BRIDGE_NAME}" type bridge 2>/dev/null || true
rm -rf "${RUN_ROOT}"

CACHE_ROOT is intentionally left in place so you can reuse the downloaded kernel, Ubuntu squashfs, base ext4 image, and prepared guest image on the next run.