ElasticSky

23. December 20209. January 2021

Homelab Upgrade – from IKEA to 19″

I reported in detail last year about the planning and hardware of my homelab vSAN cluster (part 1 and part 2). For the sake of simplicity, the four small Supermicro E300-9Ds were placed on an ordinary IKEA rack.

This worked quite well for some time, but changes in cabling, or hardware maintenance turned out to be difficult. Disconnecting patch cables and power cords was always a challenge. Restoring the cabling was even more difficult. It was not uncommon to end up with swapped ports after such actions.

In other words, a solution was needed. A 19″ rack would be ideal, but I don’t have room for a 48 U cabinet in my office. After some research, I found a small rack with 12 U at one of my hardware suppliers, which can also be equipped with rolls. This seemed perfect as it was just about as high as my desk. Also the depth is variable from 57,5cm to 101cm. No problem for the compact E300-9D and a few network components.

Time for Tinkering

The frame comes in a very compact pack size and turns out to be solid metal design with proper bolting.

Refitting Components

Some of the existing components had to be refitted to 19 inches. For example, the Ubiquiti EdgeRouter 10X, for which a conversion kit is available from the manufacturer (SKU: ER-RMKIT). The same applies to the Supermicro E300-9D mini servers. Supermicro also offers a conversion kit (MCP-290-30002-0B), which extends the servers to 19″ width and includes a bracket for the power supply.

Preliminary expansion

In the early design phase, I didn’t think I would be able to fill all 12 height units. But step by step it became clear that further components were necessary and had to be added. For example, the power switch strip (top), which can be used to switch individual device groups. It is connected to the power distribution strips on the rear. Cable guides are also a must if you don’t want to end up in a mess. It quickly became clear that a separate switch would be needed for the 1 GBit connections if you didn’t want to waste 10 GBit ports for 1 GBit.

Although I am a big fan of the Ubiquiti UniFi system, I decided to use EdgeMax series devices for the Homelab. These can be configured individually on the CLI or the web GUI.

The E300-9D are equipped with a total of four 10 GBit adapters. Two of these are RJ45 and two are SFP+. This led to the upgrade with an Ubiquit EdgeSwitch 16 XG.

There are storage shelves on the back, which can be used for the Raspberry Pi and other accessories. A Raspberry Pi acts as DNS and NTP server in the lab.

Since the whole rack is mounted on rollers, it can be easily pulled out for maintenance. Only one patch cable and one power supply has to be disconnected for this.

Noise emission

As the number of devices increases, so does the noise. I already described how to silence the Netgear 10G switch in an older article. The Ubiquiti switches work surprisingly silent and are only a bit louder during startup. At a distance of one meter, all switches reach a sound pressure of about 20 dB. This corresponds to a flying mosquito. Not too annoying, but audible. The four E300 servers currently have the highest noise level, but the original fans could be replaced with Noctua models as well. Maybe a future project…

6. December 20206. December 2020

vSAN File Service for SMB

One new feature of vSAN 7.0 update 1 is SMB support with vSAN File Service. Version 7.0 GA was limited to the NFS protocol.

Where’s the SMB option?

I was a little bit confused that I wasn’t able to choose SMB protocol for vSAN File Service after I’ve updated my homelab cluster to version 7 U1.

As always it’s a good idea to open your eyes and read the fine-print. Just below the dropdown menu there’s the unmistakable quote:

Enable active directory configuration in the File Service configuration before using SMB protocol.

Sounds logic and explains why I could not choose the SMB protocol. My homelab works without ADS. Instead a small bind9 server is responsible for DNS resolution.

You can find a hint in the properties of an existing file share too.

Long story short: No Active Directory – no SMB in vSAN File Service 🙂

5. December 20209. January 2021

NSX-T Update Procedure

On October 20th 2020 VMware released NSX-T version 3.1 (release notes).

Upgrade from version 3.0

I’ll outline the process of upgrading from version 3.0.x to version 3.1. In the example shown, a base version 3.0.2 is upgraded, but the process is the same for all versions from 3.0.

Requirements

We’ll need an upgrade bundle (MUB) from VMware download site (login required).

Upgrade

First we need to login to NSX-T Manager. Go to section Lifecycle Management and select Upgrade. You’ll see your current version on the right. Start the process with Upgrade NSX.

29. November 20206. December 2020

Using ESXi on Arm as a tiny Kubernetes cluster

ESXi on Intel x86 architecture has been a commodity for many years now. In recent years and during VMworld for example we’ve seen early alpha versions of ESXi running on Arm architecture like smart NICs or even Raspberry Pi. Meanwhile VMware developers published a Fling named ESXi Arm Edition to deploy ESXi on Arm architecture. Of course this is a lab project and not supported by VMware for production workloads. But anyway, it’s a great opportunity to play around with ESXi on a cheap and tiny computer like Raspberry Pi. I will not explain how to deploy ESXi on Arm. Check the detailed documentation on the Fling project page (PDF). I will focus on day-2 operation.

I would like to thank William Lam for providing a lot of background information, hacks and tricks around PhotonOS and ESXionArm.

Now I’ve got an ESXi host on my Raspi. What can I do with it?

Just a few remarks before we start:

You can’t run any workload on the ESXi on Arm platform. As the project name says, it’s an Arm architecture, So you can’t run operating systems based on Intel architecture. All guest VMs need to be made for Arm architecture. That will rule out Windows guest systems and also most Linux distributions. But luckily there are a couple of Linux distributions made specific for Arm architecture like Ubuntu Server for Arm, or Photon OS. For my demonstration I chose the latest Photon OS (version 4 beta). As host hardware I’m using the “big” Raspberry Pi 4 with 8 GB RAM. You can imagine that 8 GB of RAM isn’t very much for host OS and guest VMs. We have to use resources sparingly.

Our aim is to deploy a 3 node Kubernets cluster on an ESXi on Arm host on Raspberry Pi with just 8 GB RAM and 4 cores. Sounds crazy, but it’s possible. Thanks to K3s lightweight Kubernetes on Arm.

Hardware used

Raspberry Pi 4, Broadcom BCM2711, Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
Heat sink for Raspberry Pi4 (your Raspi will become hot without)
SD-card (only for UEFI BIOS)
USB stick for ESXi installation
USB 3 hub with external power supply (Raspi doesn’t provide reliable power on USB port for an NVMe SSD)
USB 3 NVMe M.2 case
Samsung NVMe EvoPlus 250 GB M.2

Using ESXi on Arm in standalone mode

Although I have joined my ESXi on Raspi to my vCenter 7, I will not use any vCenter features. All works are done like on a standalone ESXi (with all the shortcomings and limitations).

First we need 3 VMs for the 3 K3s nodes. It’s a good idea to build a VM with everything we need except K3s and then clone it. Well, if you think cloning a VM on a standalone ESXi on Arm host is just a mouse click in the UI, then welcome to the real world. 😉 I will come to that point later. Let’s build our first Photon OS VM.

Pages: 1 2 3