K3s is a lightweight, highly available open source Kubernetes cluster platform designed for easy and resource-efficient installation. K3s is provided in a package of less than 60 MB. The package is optimized for ARM platforms and can therefore also be run on hardware such as a Raspberry Pi, or as a guest VM on ESXi-on-ARM.

Prerequisites and collection of information

K3s is a cluster solution. That is why the order in which the nodes are updated is important. The update starts on the master node. So first we need to find out which node has this role. The easiest way to do this is with a kubectl command:

kubectl get node

NAME STATUS ROLES AGE VERSION
k3node1.lab.local Ready master 2y43d v1.19.3+k3s3
k3node2.lab.local Ready none 2y42d v1.19.3+k3s3
k3node3.lab.local Ready none 2y42d v1.19.3+k3s3

From the output above we see my three K3s nodes with FQDN, status, role, age and version. So here k3node1 has the master role.

As an alternative, you can also execute the command in verbose mode:

kubectl get node -o wide

Check Version

The K3s version of the current node can also be queried with the following k3s command:

k3s -v

k3s version v1.19.3+k3s3 (0e4fbfef)

Check for active pods

Before upgrading / updating, make sure that there are no active pods on the node.

kubectl get pods --all-namespaces

Alternative short form of the command above:

kubectl get pods -A

There were no active pods on this node. However, if there are pods on that node, the node must be drained first.

kubectl drain k3node1 --ignore-daemonsets

Upgrade

After the preliminary tests we are ready to upgrade. We will start on the node with the master role.

Start Upgrade on Master Node

curl -sfL https://get.k3s.io | sh -

Check Versions

kubectl get node

The master node has been updated to version v1.25.4.

Caution. In production systems, the versions on the nodes should extend over a maximum of three minor versions. I.e. we should have gone first from v1.19.3 to v1.21.x here. Since it is a lab environment, the jump was made directly to the latest available version (v1.25.4).

Update Agent Node

Before we can proceed with updating the agent nodes, we need to determine the master node’s token. This token was also used in the process of creating the cluster. To do this, we execute the following command directly on the master node:

cat /var/lib/rancher/k3s/server/node-token

We note the token or copy it to the clipboard. The next step is performed on an agent node. To do this, we open an SSH session to the agent node and run the setup script specifying the master node and the master token. The IP address of my master node is in this case 10.0.10.171 and the port is 6443. Behind the parameter K3S_TOKEN= we copy the master token from the clipboard.

curl -sfL https://get.k3s.io | K3S_URL=https://10.0.10.171:6443 K3S_TOKEN=K109fc629c6ff2feb3e94a6e6d7c972cdbab2cfa39e19697db67b6455d10cc13c98::server:47581903e817d3cc683a20bb1d1f1448 sh -

After running the setup script, we check the version on the master node. (This time in the verbose variant).

kubectl get node -o wide

Master and first agent node are updated. Repeat the above command on the second agent node. As before, we connect to the third node (k3node3) via SSH. Again, specifying the master node and master token. So it is the identical command as on k3node2.

curl -sfL https://get.k3s.io | K3S_URL=https://10.0.10.171:6443 K3S_TOKEN=K109fc629c6ff2feb3e94a6e6d7c972cdbab2cfa39e19697db67b6455d10cc13c98::server:47581903e817d3cc683a20bb1d1f1448 sh -

Check kubectl version

It is not enough to update only the K3s version, but also the CLI kubectl must be updated. First we check the currently installed version and have the result output as YAML (-o=yaml).

kubectl version -o=yaml

There are two ways to update:

Update to the latest version
Update to a specific version

Update to the latest kubectl version

The URL to stable.txt returns the latest version. The string (e.g. v1.25.4) is taken over into the actual path of the curl command. Depending on the platform, the URL in the trailing part must be adapted to arm64 or amd64.

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl"

Update to a specific version

There is no need to query for the latest version here and the URL points directly to the desired version.

curl -LO https://dl.k8s.io/release/v1.25.4/bin/linux/arm64/kubectl

Check and troubleshoot

After updating, it is useful to perform a small test of the connection. For example, by creating a simple pod.

kubectl run testpod --image=nginx

We can check for successful deployment with the describe command.

kubectl describe pod testpod

In my case, the pod could not be created.

Error: failed to create containerd container: get apparmor_parser version: apparmor_parser resolves to executable in current directory (./apparmor_parser)

The package apparmor was missing on my K3s hosts. Since my platform is PhotonOS, the package manager tdnf is used for that purpose.

tdnf install apparmor-parser

The corresponding command on Ubuntu systems:

sudo apt-get install apparmor
sudo apt-get install apparmor-utils

After the installation, the deployment of the pod testpod can be tried again.

kubectl get pods

That obviously worked. Our environment is now up to date.

Upgrade of a K3s Lightweight Kubernetes Cluster