ssh, ssh-copy-id$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"sudo hostnamectl set-hostname cplane1# worker 1
sudo hostnamectl set-hostname worker1
# worker 2
sudo hostnamectl set-hostname worker2/ect/hosts192.168.178.200 cplane1
192.168.178.201 worker1
192.168.178.201 worker2cplane1 & worker1maciej@cplane1:~$ ping worker1 -c3
PING worker1 (192.168.178.201) 56(84) bytes of data.
64 bytes from worker1 (192.168.178.201): icmp_seq=1 ttl=64 time=0.853 ms
64 bytes from worker1 (192.168.178.201): icmp_seq=2 ttl=64 time=0.586 ms
64 bytes from worker1 (192.168.178.201): icmp_seq=3 ttl=64 time=0.581 msmaciej@worker1:~$ ping cplane1 -c3
PING cplane1 (192.168.178.200) 56(84) bytes of data.
64 bytes from cplane1 (192.168.178.200): icmp_seq=1 ttl=64 time=0.606 ms
64 bytes from cplane1 (192.168.178.200): icmp_seq=2 ttl=64 time=0.602 ms
64 bytes from cplane1 (192.168.178.200): icmp_seq=3 ttl=64 time=0.607 mscplane1.$ ssh-copy-id -i ~/.ssh/id_ed25519.pub maciej@192.168.178.200
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/maciej/.ssh/id_ed25519.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
maciej@192.168.178.200's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -i /Users/maciej/.ssh/id_ed25519 'maciej@192.168.178.200'"
and check to make sure that only the key(s) you wanted were added.~/.ssh/config like the one following:Host cplane1
HostName 192.168.178.200
User maciej
Port 22 # adapt if you use different port
IdentityFile ~/.ssh/id_ed25519
PreferredAuthentications publickeyssh cplane1.Protocol Direction Port Range Purpose Used By
-----------------------------------------------
TCP Inbound 6443 Kubernetes API server All
TCP Inbound 2379-2380 etcd server client API kube-apiserver, etcd
TCP Inbound 10250 Kubelet API Self, Control plane
TCP Inbound 10259 kube-scheduler Self
TCP Inbound 10257 kube-controller-manager Selfsudo ufw allow "OpenSSH"
sudo ufw enable
# Test
sudo ufw status
sudo ufw allow 6443/tcp
sudo ufw allow 2379:2380/tcp
sudo ufw allow 10250/tcp
sudo ufw allow 10259/tcp
sudo ufw allow 10257/tcp
# Test
$ sudo ufw status
[sudo] password for maciej:
Status: active
To Action From
-- ------ ----
OpenSSH ALLOW Anywhere
6443/tcp ALLOW Anywhere
2379:2380/tcp ALLOW Anywhere
10250/tcp ALLOW Anywhere
10259/tcp ALLOW Anywhere
10257/tcp ALLOW Anywhere
OpenSSH (v6) ALLOW Anywhere (v6)
6443/tcp (v6) ALLOW Anywhere (v6)
2379:2380/tcp (v6) ALLOW Anywhere (v6)
10250/tcp (v6) ALLOW Anywhere (v6)
10259/tcp (v6) ALLOW Anywhere (v6)
10257/tcp (v6) ALLOW Anywhere (v6)sudo ufw allow "OpenSSH"
sudo ufw enable
sudo ufw allow 10250/tcp
sudo ufw allow 30000:32767/tcp
# Test
$ sudo ufw status
[sudo] password for maciej:
Status: active
To Action From
--
$ sudo modprobe overlay
$ sudo modprobe br_netfilter
$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF# add kernel parameters to load with k8s configuration
$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# load modules above
$ sudo sysctl --system# disable with command
$ sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
# or comment the line manually
$ sudo vim /etc/fstab # disable /swap
$ sudo swapoff -a
$ free -m
total used free shared buff/cache available
Mem: 15783 261 15085 1 436 15255
Swap: 0 0 0$ sudo apt -y update && sudo apt -y upgrade
# if having info about outdated gpg do following
sudo apt -y install containerd
# remove old things if existing
sudo apt remove $(dpkg --get-selections docker.io docker-compose docker-doc docker-buildx podman-docker containerd runc | cut -f1)
# Add Docker's official GPG key:
sudo apt -y install apt-transport-https ca-certificates software-properties-common curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt -y update
# stop containers
sudo systemctl stop containerd
# copy orginal configuration
sudo mv /etc/containerd/config.toml /etc/containerd/config.toml.orig
sudo containerd config default > /etc/containerd/config.toml
-bash: /etc/containerd/config.toml: Permission denied
sudo su
root@worker1:/home/maciej# sudo containerd config default > /etc/containerd/config.toml
exit
# replace "SystemdCgroup = false" to "SystemdCgroup = true".
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
# or do it manually
sudo vim /etc/containerd/config.toml
# start
sudo systemctl start containerd
$ sudo systemctl status containerd
● containerd.service - containerd container runtime
Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-04-19 23:05:59 UTC; 1min 12s ago
Docs: https://containerd.io
Process: 2285 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 2286 (containerd)
Tasks: 10
Memory: 12.7M
CPU: 248ms
CGroup: /system.slice/containerd.service
└─2286 /usr/bin/containerd
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.574566257Z" level=info msg=serving... address=/run/container>
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.574626153Z" level=info msg=serving... address=/run/container>
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.574685614Z" level=info msg="containerd successfully booted i>
Apr 19 23:05:59 worker1 systemd[1]: Started containerd container runtime.
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.574660458Z" level=info msg="Start subscribing containerd eve>
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.576694989Z" level=info msg="Start recovering state"
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.576767152Z" level=info msg="Start event monitor"
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.576792721Z" level=info msg="Start snapshots syncer"
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.576807855Z" level=info msg="Start cni network conf syncer fo>
Apr 19 23:05:59 worker1 containerd[2286]: time="2023-04-19T23:05:59.576816067Z" level=info msg="Start streaming server"
$ sudo systemctl is-enabled containerdkubeadm requirements (v1.37):sudo curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
echo "$(cat kubectl.sha256) kubectl" | sha256sum --check
kubectl: OK
# install(move) binary to install folder
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
# prepare
$ sudo apt update
$ sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
# Download the public signing key for the Kubernetes package repositories.
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.37/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
sudo chmod 644 /etc/apt/keyrings/kubernetes-apt-keyring.gpg # allow unprivileged APT programs to read this keyring
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo chmod 644 /etc/apt/sources.list.d/kubernetes.list # helps tools such as command-not-found to work correctly
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
# check installed versions
$ kubelet --version [0:29:33]
Kubernetes v1.37.0
$ kubectl version [0:30:07]
Client Version: v1.37.0
Kustomize Version: v5.8.1
The connection to the server localhost:8080 was refused - did you specify the right host or port?
$ kubeadm version [0:30:32]
kubeadm version: &version.Info{Major:"1", Minor:"37", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.37.0", GitCommit:"f54c212e3a2f75d674b717a9b29052b20b60aefc", GitTreeState:"clean", BuildDate:"2026-08-26T10:47:17Z", GoVersion:"go1.26.6
# Prevent kubernetes packages from being upgraded between versions
$ sudo apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.# check is kubelet is running
sudo systemctl enable --now kubeletNetworkPolicy. Therefore, with it, I was not accomplish some of the CKAD exercises.If you're setting up the Kubernetes cluster also for learning for the CKAD, I encourage you to use Calico CNI instead. It's more complex than Flannel but it has a lot of features. Moreover I provide an example how to configure it. 😉NetworkPolicy is not working with it)
If you want to use flannel, then you you have to install flanneld program
sudo mkdir -p /opt/bin/
sudo curl -fsSLo /opt/bin/flanneld https://github.com/flannel-io/flannel/releases/download/v0.25.5/flanneld-amd64
sudo chmod +x /opt/bin/flanneldkubeadm init - later, in further steps we will complete CNI configuration.$ kubeadm init
[init] Using Kubernetes version: v1.37.0
[preflight] Running pre-flight checks
E0902 00:33:43.577254 19372 remote_runtime.go:1359] "RuntimeConfig from runtime service failed" err="rpc error: code = Unimplemented desc = unknown method RuntimeConfig for service runtime.v1.RuntimeService"
[WARNING ContainerRuntimeVersion]: You must update your container runtime to a version that supports the CRI method RuntimeConfig. Falling back to using cgroupDriver from kubelet config will be removed in 1.38. For more information, see https://git.k8s.io/enhancements/keps/sig-node/4033-group-driver-detection-over-cri
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
W0902 00:33:43.698412 19372 checks.go:920] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended to use "registry.k8s.io/pause:3.10.2" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cplane1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.178.200]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [cplane1 localhost] and IPs [192.168.178.200 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [cplane1 localhost] and IPs [192.168.178.200 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.996368ms
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.178.200:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-scheduler is healthy after 19.085472ms
[control-plane-check] kube-controller-manager is healthy after 21.829877ms
[control-plane-check] kube-apiserver is healthy after 2.502960902s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node cplane1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node cplane1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 482fay.wf71zoya8t3wpwxb
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.178.200:6443 --token 482fay.wf71zoya8t3wpwxb \
--discovery-token-ca-cert-hash sha256:4676b2500958363245797bc126250f4d893cc60c2d61d3357b56e76ff867f846root@cplane1:/home/root# cat <<EOF | tee /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.0/16
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF
root@cplane1:/home/root# kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created$ kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/v1_crd_projectcalico_org.yaml
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/gatewayapis.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/goldmanes.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/istios.operator.tigera.io
[20:50:23.203697 11745 checks.go:846] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.
[20:50:23.203697 11745 checks.go:846] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.# open containerd config and update the pause image to desired version 3.10
vim /etc/containerd/config.toml
# after restart service to apply
systemctl restart containerdkubens, kubectx, k9s/etc/apt/source.list.d/snap ubuntu package manager.sudo apt update
sudo apt install snapd
# installing programs
sudo snap install kubectx --classic # contains kubens
sudo snap install k9skubeadm reset and repeat seteps (recommend)# delete calico
kubectl delete -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/custom-resources.yaml
kubectk delete deploy tigera-operator -n tigera-operator
# reset cplane and workers
kubeadm reset
sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni
sudo apt-get autoremove
# Remove Related Files and Directories
sudo rm -rf ~/.kube
sudo rm -rf /etc/cni /etc/kubernetes rm -f /etc/apparmor.d/docker /etc/systemd/system/etcd*
sudo rm -rf /var/lib/dockershim /var/lib/etcd /var/lib/kubelet /var/lib/etcd2/ /var/run/kubernetes
# Clear out the Firewall Tables and Rules (for some must be root user)
#
# flushing and deleting the filter table
sudo iptables -F && iptables -X
# lush and delete the NAT (Network Address Translation) table
sudo iptables -t nat -F && iptables -t nat -X
# flush and remove the chains and rules in the raw table
sudo iptables -t raw -F && iptables -t raw -X
# remove the chains and rules in the mangle table
sudo iptables -t mangle -F && iptables -t mangle -X
# optional: remove containerd
sudo apt-get remove containerd.io
sudo apt-get purge --auto-remove containerd
# if cplane continue to init
kubeadm init ... # and rest of flags
# if worker continue to join
sudo apt-get purge --auto-remove containerd
kubeadm join ... # and rest from kubeadm initmaciej@cplane1:~$ kubectl get pv
E1225 20:37:45.232733 7857 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.178.200:6443/api?timeout=32s\": dial tcp 192.168.178.200:6443: connect: connection refused"
E1225 20:37:45.234160 7857 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.178.200:6443/api?timeout=32s\": dial tcp 192.168.178.200:6443: connect: connection refused"
E1225 20:37:45.235670 7857 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.178.200:6443/api?timeout=32s\": dial tcp 192.168.178.200:6443: connect: connection refused"
E1225 20:37:45.237148 7857 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.178.200:6443/api?timeout=32s\": dial tcp 192.168.178.200:6443: connect: connection refused"
E1225 20:37:45.238673 7857 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.178.200:6443/api?timeout=32s\": dial tcp 192.168.178.200:6443: connect: connection refused"
The connection to the server 192.168.178.200:6443 was refused - did you specify the right host or port?maciej@cplane1:~$ k get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
vol1 1Gi RWO Retain Bound default/registry-claim0 <unset> 117d
vol2 200Mi RWO Retain Bound default/nginx-claim0 <unset> 117d
There is no need to add this to workers machines as only cplane1 is used for communication with cluster, but will not stop you if you want to do it anyway 😉
curl -L https://github.com/projectcalico/calico/releases/download/v3.32.2/calicoctl-linux-amd64 -o calicoctl
chmod +x ./calicoctl
# move calicoctl to accessible on system path
sudo mv calicoctl /usr/local/bin/