ElasticSky

8. March 202127. March 2021

Don’t confuse a blog post with a deployment guide

Lab environments are a great thing. We can test new products on a small scale platform and demonstrate them as a proof of concept (PoC).

Like many of my fellow bloggers I write down my lab experience in little blog posts that I share with the community. I regularly read blogs and tutorials to keep myself informed about new products and techniques. There is hardly a topic in the field of virtualization that someone hasn’t written something about at some point. This is invaluable, as it gives you a quick introduction to what is usually a complex subject.

When reading my (and other) blog posts, you may get the impression that the described setup procedure follows the simple skip-skip-finish principle. In other words, accept the default values, click three times and the installation is complete. This might be true in the lab, but a real life deployment is miles away from a lab setup.

In the lab many things are simplified to the max according to the KISS principle (keep it simple and stupid). Some of the methods used are not necessarily in compliance with the manufacturer’s recommendations, or are outright forbidden in productive environments.

This means : Having read a tutorial by my favorite blogger [insert name here] does not enable me to transfer what I have learned 1:1 to a real project.

I have had several discussions about this in preliminary project meetings. People have asked why the planning phase takes so much time. They said that (they thought) the product was totally easy to install, as you can read on [insert name here]’s blog.

As a blogger and lab user, I know how to view these posts. They are to be understood as a quick introduction and an easy to understand overview of a new technology. This has very little to do with real world deployments. In this posting, I would like to point this out with the help of a few examples:

7. March 20217. March 2021

Heads up! Watch your NIC order when adding more hosts to VCF

VMware Cloud Foundation is a unified SDDC platform for the hybrid cloud. It is based on VMware’s compute, storage, and network virtualization.

VCF can be expanded with more workload domains by adding further hosts, or it can be stretched over two availability zones (AZ). The expansion is initiated by and under control of the SDDC-Manager. The procedure is fairly straightforward and SDDC-Manger does all the configuration tasks in the background, i.e. forming vSAN clusters, networks, kernel ports, vCenters and NSX control planes.

setup hosts with ESXi base image
confige a management IP address
set root credentials
configure DNS and NTP
import new hosts into SDDC-Manager
deploy new WLD

There is a pitfall that can be easily overseen: The order of the new host’s NICs. Before we can import new hosts, we’ll get to see a checklist about the host requirements. The hosts need to have two NICs with at least 10 GBit.

While reading the list there’s a little detail that is often overlooked. Traditional numbering means that both NICs must have numbers vmnic0 and vmnic1. Unfortunately this seems to be hard coded and cannot be changed (as of current version 4.2). To make matters worse, many server systems have onboard 1 GBit network adapters. There’s a KB article that explains how VMware ESXi determines the order in which names are assigned to network devices. It’ll start with onboard NICs and then continues with PCIe cards. As a result you’ll might end up with two 1 GBit onboard NICs as vmnic0 and vmnic1. In this case the bringup of the VCF expansion will fail.

While you can choose NICs during initial VCF bringup, this is not possible during expansion and this time there’s no such thing as a bringup sheet. You can’t select more than two NICs either when using SDDC-Manager. In that case you’ll need to use API-calls.

Workaround

Currently there’s no other way than to disable onboard NICs in the system BIOS. If your desired NICs still show a higher number you’ll need to put the PCIe card into a lane with lower number.

15. February 202123. February 2021

Activate Immutability flag on existing Veeam XFS Repositories

Veeam Backup & Replication v11 is about to be launched on February 24th. Partners and service providers already have access to the RTM build.

One of my favourite features of version 11 are immutable backup files. You can read about it in my recent post “Hardening your Veeam Backup strategy with immutable repositories on Linux XFS“.

Some of you might already use Linux repository servers based on XFS with fast clone. Once you upgrade to Veeam Backup & Replication version 11 you may wish to activate immutable backups. I will explain in this blog post how you can use this new feature on existing XFS repository servers.

Activate immutability

To activate the immutability feature edit your existing backup repository servers and go to step “repository” (picture below). Starting with version 11 there’s a checkbox “make recent backups immutable for [x] days” where 7 is the minimum value to enter. Once you activate the checkbox you will most likely see a message “Unable to set the immutability option: cannot find the transport service on server <your repo server>. Configure the server anyway?“. Click on [Yes].

To make use of the immutability flag you’ll have to reconfigure the transport service on the Linux server.

22. January 202128. January 2021

Hardening your Veeam Backup strategy with immutable repositories on Linux XFS

Attackers on IT infrastructure have become more sophisticated in recent times. Not only do they ecrypt live data and entire virtual machines, they also have learned to delete or encrypt entire backups too. No matter if your online backups are stored on a local repository or a cloud share. If that happens and you do not have an offline backup like tape or any other air gapped solution you’re up a creek without a paddle. Even worse: If you’re responsible for data backup, you might be facing unpleasant questions by the management.

Now the question is how to protect backup data in case an attacker gets hold on your backup server. Once they get credentials to the server, they can do whatever the backup-admin can do. Deleting backups for example. One first step is to separate your backup systems from your domain. In case the domain administrator account gets compromised there’s still (a little) barrier between attackers and your backup systems. But still, you’ll never know if attackers have knowledge of a zeroday exploit to takeover your backup server. It would be a better approach to protect backups against deletion for a defined timespan. There’s a similar option on AWS S3 buckets, which are usually used as offsite backup copies. Getting back offsite backups takes a considerable amount of time. In case of a crypto attack time is crucial and the clock is ticking. Wouldn’t it be great to have immutable backups on your primary repositories on site? Good news. There’s light at the end of the tunnel (and I promise it’s not the oncoming train).

Immutable Backups

Before we go into detail, we need to clarify what immutable backups mean. The idea is that you write once and then files (backups) are protected for a self defined period of time. Even the backup administrator cannot delete them before the defined timespan has passed.

Veeam Backup & Replication v11 will make use of a native Linux XFS filesystem feature. XFS can set an extended file attribute [i] which will protect the file from renaming, modification, deletion or hard-linking. The key point is that backups are transferred and written with non-privileged accounts and the immutable attribute is set and removed by a privileged service on the repository server which cannot be accessed from outside.

All you need is a Linux repository formatted with XFS and Veeam Backup & Replication v11, which will be released in the near future. (Update: Expected release date will be February 24th 2021)