Deploying a Kubernetes Cluster on Ubuntu 22.04 with Containerd – Part I

by Posted on

In this series of blog posts I’ll be writing about deploying a Kubernetes cluster running on Ubuntu servers. This blog series is divided into the following parts:

  • Part I: Setting up the cluster nodes (this post)
  • Part II(a): Setting up Calico as the cluster CNI
  • Part II(b): Setting up Cilium as the cluster CNI
  • Part III: Adding a second master node to the cluster
  • Part IV: Adding Rancher as Kubernetes Cluster Management Platform
  • Part V: Setting up Veeam Backup (Kasten) as the backup solution for the cluster
  • Part VI: Setting up a monitoring service for the cluster using Prometheus and Grafana
  • Part VII: Adding a network storage system to the cluster using Ceph and Rook
  • Part VIII(a): Adding Nginx Ingress Controller to the cluster nodes
  • Part VIII(b): Adding Traefik Ingress Controller to the cluster nodes

So, with the roadmap set, let’s begin…

For our cluster, we should meet the following requirements, and will use the latest stable versions:

  • Three Ubuntu 22.04 servers running as VMs or on bare-metal (one master node, two worker nodes)
  • CNI Plugins v1.2.0
  • Containerd v1.7.0 as the CRI (Container Runtime Interface)
  • Kubernetes components (Kubelet, Kubeadm, Kubectl) v1.26.3

Some Background

Before version 1.20, Kubernetes used to use a component called dockershim, which was an overlay to support the use of Docker as the container runtime. But they decided to make things simpler and remove the shim and directly support the use of Containerd and CRI-O as the container runtimes for Kubernetes. You may read all the details of the change in here.

Preparing the Operating Systems for Our Cluster

This section includes the steps needed to take for preparing all nodes in the cluster. Please apply these steps on all nodes.

1. Update Operating System Components and Install Prerequisites
Keep the Operating Systems components up-to-date. You may achieve this by running the following commands.

sudo apt update
sudo apt upgrade -y
sudo apt install -y apt-transport-https

After running the mentioned commands and applying the updates, make sure you reboot the nodes to boot them with the latest Kernel.

2. Form a Naming System for The Cluster Nodes
Make sure that all nodes have unique hostnames and MAC address on the NIC cards. For this deployment I'll be using the following naming and addressing (you may modify to fit your environment), and I will append the data below to the end of /etc/hosts file on the nodes.

192.168.10.131 kube-master-01
192.168.10.133 kube-worker-01
192.168.10.134 kube-worker-02

You should also change the hostnames by using the sudo hostnamectl set-hostname <HOSTNAME> command. 

3. Load required Kernel Modules, Configure System Parameters, and Install Containerd
Kubernetes uses overlay and br_netfilter for Containerd networking. You can load them on runtime with the following commands.

sudo modprobe overlay
sudo modprobe br_netfilter

However, in order to make sure these components load on each OS boot, make sure you add them to Kernel modules in /etc/modules-load.d/ using the following command.

sudo tee /etc/modules-load.d/containerd.conf << EOF
overlay
br_netfilter
EOF

Next, we need to configure system parameters for container networking. We're going to add the following to /etc/sysctl.d directory to be loaded on boot. 

sudo tee /etc/sysctl.d/containerd.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

After that, we need to tell the OS to read the configuration and apply it. This can be achieved by using the command sudo sysctl --system.

Installing Containerd could be done from three different ways: 1. from native Ubuntu repository, 2. from Docker repository that must be manually added to apt repositories, and 3. by downloading the latest release from Containerd GitHub repository. For the purposes of this deployment, we'll be using the GitHub repository.

wget https://github.com/containerd/containerd/releases/download/v1.7.0/containerd-1.7.0-linux-amd64.tar.gz
tar Cxzvf /usr/local containerd-1.7.0-linux-amd64.tar.gz
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service -P /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now containerd

What we did was to basically download the binaries from GitHub, extract them to /usr/local, where it would make the containerd directory, and also download the service file and added it to the system.

After installing Containerd, we must also install runc and CNI Plugins. This can be achieved with the following commands.

wget https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc

wget https://github.com/containernetworking/plugins/releases/download/v1.2.0/cni-plugins-linux-amd64-v1.2.0.tgz
mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.2.0.tgz

Finally, we need to configure Containerd to use Systemd as cgroup driver. To do this, we save the default configuration of Containerd, and then modify it.

sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo vi /etc/containerd/config.toml

Modify the section [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] and set systemdcgroup to true (default is false).

containerd-config

After modifying the parameter, restart the Containerd service.

sudo systemctl restart containerd

4. Disable Swap
In order for Kubelet to work properly, we must disable Swap. This can be achieved during runtime by using the command sudo swapoff -a. Also, we must comment the corresponding line for mounting the Swap partition on boot in the /etc/fstab file.

disable-swap-mount 

5. Install Kubernetes Packages
The final step in this section is to install the required packages for Kubernetes.

First we need to add the Kubernetes repository to apt.

curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list

Then we need to update the repository list by issuing the command sudo apt update and then we can install the required packages.

sudo apt install -y kubelet kubeadm kubectl

Lastly, it is recommended to mark these packages so that they won't be updated with the system packages.

sudo apt-mark hold kubelet kubeadm kubectl

6. Pull Kubernetes Images on The Nodes
Before we initialize the cluster, we should pull the container images for the cluster on all nodes. This can be done with the following command.

sudo kubeadm config images pull

image-pull

Initializing The Cluster and Joining The Workers

This section is where everything becomes connected.

First we must initialize the master node.

sudo kubeadm init --pod-network-cidr=10.0.0.0/16

Above command will initialize the containers and start them, making the master mode the controller of the cluster. The output of the command will give you the guide to join the worker nodes to the cluster.

kubeadm-done

After joining the nodes to the cluster, you can check the cluster status with the sudo kubectl get nodes command.

worker-joined



The output will show the nodes and NotReady, which is normal.

In the next post, we will be adding the CNI to our cluster.

Published by Mazi

Maziar is a Network Engineer focusing on Service Provider and Telecom networks, but he keeps poking around in other technology areas and figure out how to connect them together.

Leave a Reply

Your email address will not be published. Required fields are marked *