Upgrade of a K3s Lightweight Kubernetes Cluster

K3s is a lightweight, highly available open source Kubernetes cluster platform designed for easy and resource-efficient installation. K3s is provided in a package of less than 60 MB. The package is optimized for ARM platforms and can therefore also be run on hardware such as a Raspberry Pi, or as a guest VM on ESXi-on-ARM.

Prerequisites and collection of information

K3s is a cluster solution. That is why the order in which the nodes are updated is important. The update starts on the master node. So first we need to find out which node has this role. The easiest way to do this is with a kubectl command:

kubectl get node

NAME STATUS ROLES AGE VERSION
k3node1.lab.local Ready master 2y43d v1.19.3+k3s3
k3node2.lab.local Ready none 2y42d v1.19.3+k3s3
k3node3.lab.local Ready none 2y42d v1.19.3+k3s3

From the output above we see my three K3s nodes with FQDN, status, role, age and version. So here k3node1 has the master role.

As an alternative, you can also execute the command in verbose mode:

kubectl get node -o wide

7. December 202214. December 2022

Heads-up: Linux Start- and Reboot-Issues with Maxtang NX6412 solved

Cohesity vExpert gift

I recently became the owner of a Maxtang NX6412-B11 Mini PC. Cohesity gave away these barebones to vExperts at the VMware Explore EMEA in Barcelona. Once again a big thank you to Cohesity for their support of the community!

The fanless MiniPC with Elkhart Lake chipset is well-equipped. It has 2x 1 Gbit LAN, 1x USB-C (front), 2x USB 3.2 (front), 2x USB 2.0, 2x HDMI 2.0, and an audio jack.

The MiniPC will be a great addition to my homelab. I had intended to install the Tanzu community edition for it. Unfortunately, the project has since been discontinued by VMware and the removal of the packages from GitHub has been announced. 🙁

Hardware finish

The barebone still had to be provided with RAM and a flash disk. I installed a Samsung SSD 860 EVO Series 1TB M.2 SATA and two SO-DIMM DDR4 3200 16 GB from Crucial.

Reboot Issues with Linux

With the SATA SSD and the RAM, the machine was ready to boot. Ubuntu 22.04 LTS was used as operating system. After installation, a usual reboot was requested. However, the PC did not shut down completely and remained in the “Reached target shutdown” state. The PC had to be powered off hard. The reboot also took several minutes, which is very unusual for Ubuntu. To rule out the possibility that the problem is specific to Ubuntu, I tried an installation with Fedora. The result was exactly the same here too.

The solution

After a lengthy search, I found a clue that was specific to the EHL hardware platform. The fix is to disable a kernel module for the Intel Elkhart Lake SoC chipset. This can be done by adding it to the blacklist.conf file.

sudo vi /etc/modprobe.d/blacklist.conf

The line below must be added to blacklist.conf:

blacklist pinctrl_elkhartlake

Quit the vi editor with [ESC] [:] wq! (save and exit)

update-initramfs –u

The next shutdown was still delayed, but after a cold boot the OS came up within a few seconds.

I hope this hint helps someone – especially my vExpert colleagues who received the Cohesity gift too. Sharing is caring. 🙂

17. November 202215. March 2024

Manage ESXi Coredump Files

Okay, admit it, this is not a new topic, but it cost me some time in a client project. Since this blog also acts as a swap partition of my brain, I wrote it down for future reference. It is important to follow the steps correctly so that the changes are preserved after a reboot.

Why a Coredump-File?

Modern ESXi installations starting with version 7 use a new partition layout of the boot device. Coredumps are also located there. But only when the boot medium is not a USB flash medium and not an SD card. In such cases the coredump is relocated to a VMFS datastore with at least 32GB capacity.

This is exactly the case I found in a customer environment. The system was migrated from vSphere 6.7 and therefore still had the old boot layout on a ( at that time still fully supported) SD-Card RAID1. We found a vmkdump folder with files for each host on one of the shared VMFS datastores. This (VMFS5) datastore was supposed to be decommissioned and replaced with a VMFS6 datastore. (Side note from the VCI: there is no online migration path from VMFS5 to VMFS6) 😉 So the vmkdump files had to be removed from there.

Procedure

First, we get an inventory of the coredump files.

esxcli system coredump file list

All coredump files of all ESXi hosts are listed here. Each line contains the path and the Active and Configured (true or false) states. Active means that this is the current coredump file of this host. It is important that the value for Configured also has the status ‘true’. Otherwise the setting will not survive a reboot. Only the coredump file of the current host has the status ‘active’. All other files belong to other hosts and are therefore active=false.

By default, the host chooses the first matching VMFS datastore. This is not necessarily the desired one.

Remove the current Coredump-File

First we delete the active coredump file of the host. We have to force the removal because it is set as active=true.

esxcli system coredump file remove --force

If we execute the list command from above again, there should be one line less.

Add a new Coredump File

The next command creates a new coredump file at the destination. If it does not already exist, a vmkdump folder is created and the dumpfile is created in it. We specify the desired file name without extension, because it will be created automatically (.dumpfile).

esxcli system coredump file add -d <Name | UUID> -f <filename>

Example: Name of the host is “ESX-01” and the VMFS datastore has the name “Service”. The datastore may be specified as either DisplayName or Datastore_UUID.

esxcli system coredump file add -d Service -f ESX-01

A folder vmkdump will be created on the designated datastore and a file named ESX-01.dumpfile will be created in it. We can check this using the list command.

esxcli system coredump file list

A new line will appear with the full path to the new dumpfile. However, the status is still active=false and configured=false. It might be useful to copy this full path to the clipboard, because it is required in the next step.

Activate Dumpfile

In the following step, we set the created dumpfile to active. This way, the setting is retained even after a host reboot. We specify the complete path to the dumpfile. The copy from the clipboard is helpful here and avoids typos.

esxcli system coredump file set -p <path_to_dumpfile>

Example:

esxcli system coredump file set -p /vmfs/volumes/<UUID>/vmkdump/ESX-01.dumpfile

A final List command validates the result.

VMware Explore EMEA – Review and Outlook

VMware Explore 2022 has come to a close. Four days packed with information, conversations and activities. Now it’s time to pause and take a recap of the last few days. I won’t go into technical innovations, products or announcements here. I will write separate technical articles for that and there is also the VMware Explore Video Library with numerous videos including a recording of the General Session and many of the Breakout Sessions. Instead, I would like to go into things that I liked very much and also those that have potential for improvement. Not all points concern VMware itself – in many cases they are things that were good or not so good at the sidelines. Perhaps it will inspire some readers to consider a visit to VMware Explore in the future.

A Plus – VMware Explore 2022 on site in Barcelona

The fact that after two cancelled VMworlds (yes, World) an on-site event took place again is a big plus. No matter who I asked in the last few days – everyone was tired of the virtual meetings and was looking forward to the personal contact. From this perspective, VMware-Explore 2022 was bound to be a success.

A Plus – Public Transport

The Fira is located just outside the city center. So for those who have accommodation in the center, there are twice daily trips by public transport. Barcelona’s metro system is well organized and the trains run at a tight schedule. I rarely had to wait more than 5 minutes for a connecting train. The tickets are cheap and VMware gave out cards of 10-rides at registration. But you have to ask for them. Even if you buy a 10-rides ticket for the Barcelona city area yourself, it currently costs less than €8. This has even become a little cheaper compared to 2019.

The transfer to and from the airport is also cheap and reliable. A special airport bus runs at short intervals between the terminal and the city center (Placa Espana and Placa Catalunya). Pro tip: Buy a return ticket straight away, it’s cheaper.

Attention. After midnight, many lines close down. So night owls should have some cash ready for a cab, or plan a long walk to the hotel.

Blogger Program with Potential for Improvement

There were only 15 blogger passes in total, which is very few if seen in terms of the entire EMEA region. This also hit some of my fellow bloggers who had been providing excellent VMworld contributions for many years in a row and now surprisingly missed out.

At this point I would like to thank Elisabeth Kefal, who included me in the official program in the follow-up.

To clarify, this is not a full travel package. All bloggers always finance flight and accommodation on their own. Only the admission pass to the VMware Explore is sponsored by VMware. So costs can only have played a minor role.

After all, it’s the bloggers who have a deep reach into the community in their respective countries and beyond. I don’t like the word “influencer,” but it does come pretty close to the role of tech bloggers. Let’s call them “community networkers.” They are mostly the same people who are active in the VMware User Group (VMUG) and maintain close contact with the user base there.

The current practice should be reconsidered here. Instead, it would be nice to prioritize inviting bloggers in the future who are not only seasonally active during the 4 days of VMware-Explore, but who keep the community together all year round and constantly deliver valuable content.

Author: Michael Schroeder