On Monday night we’ve encountered massive problems on the blog ElasticSky.de. Page load was affected and visitors got an Internal Server Error 500.
We do apologize for the inconvenience.
Although it looked like an attack, the disturbance originated from a configuration issue that struck the website a couple of hours after the configuration change. Worse than that we’ve applied two changes which made troubleshooting even more difficult. The most likely suspect wasn’t the one that caused the problems.
Note to self: Careful with not-so-well known hosting functions! Do only change one parameter at a time. 😉
There have been many new releases in the first quarter of 2020. The long anticipated release of Veeam Backup & Replication version 10, we’ve been waiting for since 2017 and also the latest generation of VMware vSphere. While I had vSAN 7 beta running on my homelab cluster before GA, I’ve worked with Veeam Backup 10 only in customer projects. There’s unfortunately no room for playing with new features unless the customer requests it. One of the new features of Veeam v10 is the ability to use Linux proxies and repositories. With XFS filesystem on the repository you can use the fast clone feature which is similar to ReFS on Windows.
In this tutorial I will show how to:
Deploy and size the Veeam server
Show base configuration to integrate vCenter
Build, configure and deploy a Linux proxy and its integration into backup infrastructure
Build, configure and deploy a Linux XFS repository
Using Veeam Backup on a vSAN Cluster has special design requirements. There’s no direct SAN backup on VMware vSAN because there’s neither a SAN, nor a fabric and nor HBAs. There are only two backup methods available: Network Mode (nbd) and Virtual Appliance Mode (hotadd). The latter is recommended for vSAN, but you should deploy one proxy per host to avoid unnecessary traffic on the vSAN interfaces. Hotadd also utilizes Veeam Advanced Data Fetcher (ADF).
Left: Backup from vSAN cluster with single proxy appliance; Backup traffic may be redirected over vSAN network if proxy and datadisk are not on same host. Right: One proxy appliance per vSAN node; Direct data transfer without impact on vSAN network.
Talking about licenses: Having Linux proxies on each host will reduce the cost of Windows licensing. One more reason to play around with this new feature. A Veeam license will be required too, but as a vExpert I can get a NFR (not for resale) license which is valid for one year. Just one of the advantages of being a vExpert. 🙂
Let the games begin. We’ll need a Veeam server that holds the job database and the main application. The proxy and repository role will be kept on individual (Linux) servers.
Recently I’ve upgraded my homelab from 6.7U3 to vSphere7. The workflow is straightforward and very easy. The VMware Design team did a very good job with the UI.
First steps
I cannot point that out enough: check the VMware HCL. Just because your system is supported under your current vSphere version, doesn’t mean it’ll be supported under vSphere7 too. On the day I’ve upgraded, vSphere7 was brand new and there were just a few entries in the HCL. But it’s a homelab and if something breaks I don’t care to rebuild it from scratch. Don’t do this in production!
Although my Supermicro E300-9D is not yet certified for version 7.0, it works like a charm. I guess it’s just a matter of time, because the VMware Nano-Edge cluster is based on that hardware.
Before we can start, you need to download the vCenter Server Appliance 7.0 (VCSA) from VMware downloads (Login required). You also need to have new license keys for vCenter, ESXi and vSAN (if yor cluster is hyperconverged).
There have been issues with VMware network driver igbn which is responsible for Intel 82580, I210, I350, and I354 Gigabit Ethernet Controllers. Under certain conditions this can lead to a PSOD, which makes it a critical issue for all hosts with one of the ethernet controllers mentioned above.
Currently there’s no VMware patch to solve the problem. It is recommended to replace the VMware driver with a newer version (1.4.10) of Intels native driver.
If we start SSH service on the host, we can check the installed igbn version.
First we have to download the driver package from VMware (login required) and extract the archive. It contains a documentation with release notes and update guide, a VMware Installation Bundle (VIB) and an offline bundle (ZIP). While it is possible to install the VIB on a command shell from an ESXi host, it is more convenient to use VMware Update Manager (VUM). The latter is the procedure I will explain here.
Open vSphere-Client and go to Menu > Update Manager. If you’re not running vSphere 6.7 U1 or later, you’ll have to use the infamous Web-Client (Flash-Client). Select Updates and click on “Upload from File”.
Select the extracted ZIP File (Offline Bundle). Just to avoid some confusion: The file you’ve downloaded from VMware is a ZIP-archive. Extract it once. Within that archive there’s another ZIP-archive. Do not extract that one! From the dialogue we select that ‘inner’ ZIP-file for upload to VUM.
