Hardening your Veeam Backup strategy with immutable repositories on Linux XFS

Attackers on IT infrastructure have become more sophisticated in recent times. Not only do they ecrypt live data and entire virtual machines, they also have learned to delete or encrypt entire backups too. No matter if your online backups are stored on a local repository or a cloud share. If that happens and you do not have an offline backup like tape or any other air gapped solution you’re up a creek without a paddle. Even worse: If you’re responsible for data backup, you might be facing unpleasant questions by the management.

Now the question is how to protect backup data in case an attacker gets hold on your backup server. Once they get credentials to the server, they can do whatever the backup-admin can do. Deleting backups for example. One first step is to separate your backup systems from your domain. In case the domain administrator account gets compromised there’s still (a little) barrier between attackers and your backup systems. But still, you’ll never know if attackers have knowledge of a zeroday exploit to takeover your backup server. It would be a better approach to protect backups against deletion for a defined timespan. There’s a similar option on AWS S3 buckets, which are usually used as offsite backup copies. Getting back offsite backups takes a considerable amount of time. In case of a crypto attack time is crucial and the clock is ticking. Wouldn’t it be great to have immutable backups on your primary repositories on site? Good news. There’s light at the end of the tunnel (and I promise it’s not the oncoming train).

Immutable Backups

Before we go into detail, we need to clarify what immutable backups mean. The idea is that you write once and then files (backups) are protected for a self defined period of time. Even the backup administrator cannot delete them before the defined timespan has passed.

Veeam Backup & Replication v11 will make use of a native Linux XFS filesystem feature. XFS can set an extended file attribute [i] which will protect the file from renaming, modification, deletion or hard-linking. The key point is that backups are transferred and written with non-privileged accounts and the immutable attribute is set and removed by a privileged service on the repository server which cannot be accessed from outside.

All you need is a Linux repository formatted with XFS and Veeam Backup & Replication v11, which will be released in the near future. (Update: Expected release date will be February 24th 2021)

Continue reading “Hardening your Veeam Backup strategy with immutable repositories on Linux XFS”

Veeam Backup v10 on vSAN 7.0

There have been many new releases in the first quarter of 2020. The long anticipated release of Veeam Backup & Replication version 10, we’ve been waiting for since 2017 and also the latest generation of VMware vSphere. While I had vSAN 7 beta running on my homelab cluster before GA, I’ve worked with Veeam Backup 10 only in customer projects. There’s unfortunately no room for playing with new features unless the customer requests it. One of the new features of Veeam v10 is the ability to use Linux proxies and repositories. With XFS filesystem on the repository you can use the fast clone feature which is similar to ReFS on Windows.

In this tutorial I will show how to:

  • Deploy and size the Veeam server
  • Show base configuration to integrate vCenter
  • Build, configure and deploy a Linux proxy and its integration into backup infrastructure
  • Build, configure and deploy a Linux XFS repository

Using Veeam Backup on a vSAN Cluster has special design requirements. There’s no direct SAN backup on VMware vSAN because there’s neither a SAN, nor a fabric and nor HBAs. There are only two backup methods available: Network Mode (nbd) and Virtual Appliance Mode (hotadd). The latter is recommended for vSAN, but you should deploy one proxy per host to avoid unnecessary traffic on the vSAN interfaces. Hotadd also utilizes Veeam Advanced Data Fetcher (ADF).

Talking about licenses: Having Linux proxies on each host will reduce the cost of Windows licensing. One more reason to play around with this new feature. A Veeam license will be required too, but as a vExpert I can get a NFR (not for resale) license which is valid for one year. Just one of the advantages of being a vExpert. 🙂

Let the games begin. We’ll need a Veeam server that holds the job database and the main application. The proxy and repository role will be kept on individual (Linux) servers.

Continue reading “Veeam Backup v10 on vSAN 7.0”

Setting timezone in Photon-OS

Many of VMware’s appliances are based on Photon OS. Sometimes these appliances aren’t using suitable timezones for your current location, which makes it difficult to match logs and results.

If that appliance has no GUI, you have to adjust it on the shell. Use SSH or the appliance’s console to login as ‘root’. To get a list of all available timezones issue the command below.

ls -lsa /usr/share/zoneinfo | more

Some timezones are divided into sub-zones. For example “Europe”.

ls -lsa /usr/share/zoneinfo/Europe | more

We’re piping the results to the ‘more‘ command in order to achieve better readability. It’s optional. Once you’ve found your timezone, you can set it. In my example it’s “Europe/Berlin”.

set Europe/Berlin timezone

Next we’ll create a symbolic link from localtime to “Europe/Berlin”.

ln -sf /usr/share/zoneinfo/Europe/Berlin /etc/localtime

Finally we can check settings with the ‘date’ command. Date and Time is correct and also the timezone CEST.

VCSA 6.7 archive partition filling up

During a cluster routine check with RVtools I’ve seen a warning.

Warnings like this on a vCenter appliance let my alarm bells ring. Partitions filling up close to 100% are not a desirable condition. But let’s have a look into VAMI (vCenter Appliance Management Interface).

https://<vcsa_FQDN>:5480

Now that’s strange. Everything seems to be in a green state. It’s a VCSA 6.7 Update 2a (6.7.0.31000). Older versions did show a storage warning. “File system /storage/archive is low on storage space“. But this appliance seems to be happy. Let’s have a look at the shell, to see what’s really happening.

It looks like mount point /storage/archive is almost filled up.

Continue reading “VCSA 6.7 archive partition filling up”