Michael Schroeder - ElasticSky

22. January 202128. January 2021

Hardening your Veeam Backup strategy with immutable repositories on Linux XFS

Attackers on IT infrastructure have become more sophisticated in recent times. Not only do they ecrypt live data and entire virtual machines, they also have learned to delete or encrypt entire backups too. No matter if your online backups are stored on a local repository or a cloud share. If that happens and you do not have an offline backup like tape or any other air gapped solution you’re up a creek without a paddle. Even worse: If you’re responsible for data backup, you might be facing unpleasant questions by the management.

Now the question is how to protect backup data in case an attacker gets hold on your backup server. Once they get credentials to the server, they can do whatever the backup-admin can do. Deleting backups for example. One first step is to separate your backup systems from your domain. In case the domain administrator account gets compromised there’s still (a little) barrier between attackers and your backup systems. But still, you’ll never know if attackers have knowledge of a zeroday exploit to takeover your backup server. It would be a better approach to protect backups against deletion for a defined timespan. There’s a similar option on AWS S3 buckets, which are usually used as offsite backup copies. Getting back offsite backups takes a considerable amount of time. In case of a crypto attack time is crucial and the clock is ticking. Wouldn’t it be great to have immutable backups on your primary repositories on site? Good news. There’s light at the end of the tunnel (and I promise it’s not the oncoming train).

Immutable Backups

Before we go into detail, we need to clarify what immutable backups mean. The idea is that you write once and then files (backups) are protected for a self defined period of time. Even the backup administrator cannot delete them before the defined timespan has passed.

Veeam Backup & Replication v11 will make use of a native Linux XFS filesystem feature. XFS can set an extended file attribute [i] which will protect the file from renaming, modification, deletion or hard-linking. The key point is that backups are transferred and written with non-privileged accounts and the immutable attribute is set and removed by a privileged service on the repository server which cannot be accessed from outside.

All you need is a Linux repository formatted with XFS and Veeam Backup & Replication v11, which will be released in the near future. (Update: Expected release date will be February 24th 2021)

31. December 20201. January 2021

2020 – A retrospective view

I don’t think it’s an exaggeration when I say “2020 was a year like no other.” Pretty much everyone in the IT industry and elsewhere will confirm that. I’m taking this last day of 2020 as an opportunity to write my last blog article of the year and review important events for me in it.

January

The year began quite normally, almost like any other. After the turn of the year, tasks and projects for the coming months slowly started. According to experience, it is a quiet month, because many customers usually don’t resume work before the second week.

February

February had some of my personal highlights of the year.

We held a VMUG meeting in Nuremberg on February 13th. Today I have to add that it was an on-site meeting. At that time a matter of course. Today it sounds like a tale from a distant past. I myself had given a presentation there on the topic of “Strategies for proactive error prevention”. A closer look at the products VMware Skyline and Runecast Analyzer. What do they have in common and what are their unique selling points?

A long planned private trip took us to the high latitudes of the Arctic. Spitsbergen in winter. No other event left such deep impressions this year like this one. On February 14th, after 84 days of darkness, the long polar night ends on Spitsbergen and the sun comes above the horizon again for the first time. In mid-April, the sun will not set at all for another 99 days. In the time in between the ice desert shines in wonderful light. We love the Arctic and wanted to experience this natural spectacle and – of course – capture it in pictures.

Polar bear warning applies to the whole archipelago of Svalbard.

Exploring Spitsbergen means being away from civilization for many hours at temperatures down to -38°C. Always accompanied by an armed guide, because the island is polar bear country and you’re required by law to have at least one armed member in your party once leaving the settlement.

We’ve learned, for example, that such simple things like eating can be a real challenge at very low temperatures We’ve also witnessed what it means to be caught in a white-out when the sky merges into the coulour of the ice, visibility is close to zero and everything around you is just white.

While being abroad I got a pleasant notification in my inbox. I was nominated to be a vExpert for another year.

March

Our stay lasted until the first week of March. But the world we returned to was a different to the one we left. Already on the way we read news about Corona outbreaks and sold-out toilet paper. When we left Germany, Corona was hardly a topic. Now it was THE topic and would remain so for months.

On March 11th, the last German on site VMUG took place at VMware’s premises in Munich. I myself was co-organizer and speaker and had no concerns about the train ride and the meeting at that time. Looking back and with the knowledge of today, it was certainly not the brightest idea. We were unnecessarily putting ourselves at great risk. Washing hands alone is not enough against SARS-CoV-2. Luckily (as far as I know), the event had no further health consequences for anyone involved.

23. December 20209. January 2021

Homelab Upgrade – from IKEA to 19″

I reported in detail last year about the planning and hardware of my homelab vSAN cluster (part 1 and part 2). For the sake of simplicity, the four small Supermicro E300-9Ds were placed on an ordinary IKEA rack.

This worked quite well for some time, but changes in cabling, or hardware maintenance turned out to be difficult. Disconnecting patch cables and power cords was always a challenge. Restoring the cabling was even more difficult. It was not uncommon to end up with swapped ports after such actions.

In other words, a solution was needed. A 19″ rack would be ideal, but I don’t have room for a 48 U cabinet in my office. After some research, I found a small rack with 12 U at one of my hardware suppliers, which can also be equipped with rolls. This seemed perfect as it was just about as high as my desk. Also the depth is variable from 57,5cm to 101cm. No problem for the compact E300-9D and a few network components.

Time for Tinkering

The frame comes in a very compact pack size and turns out to be solid metal design with proper bolting.

Refitting Components

Some of the existing components had to be refitted to 19 inches. For example, the Ubiquiti EdgeRouter 10X, for which a conversion kit is available from the manufacturer (SKU: ER-RMKIT). The same applies to the Supermicro E300-9D mini servers. Supermicro also offers a conversion kit (MCP-290-30002-0B), which extends the servers to 19″ width and includes a bracket for the power supply.

Preliminary expansion

In the early design phase, I didn’t think I would be able to fill all 12 height units. But step by step it became clear that further components were necessary and had to be added. For example, the power switch strip (top), which can be used to switch individual device groups. It is connected to the power distribution strips on the rear. Cable guides are also a must if you don’t want to end up in a mess. It quickly became clear that a separate switch would be needed for the 1 GBit connections if you didn’t want to waste 10 GBit ports for 1 GBit.

Although I am a big fan of the Ubiquiti UniFi system, I decided to use EdgeMax series devices for the Homelab. These can be configured individually on the CLI or the web GUI.

The E300-9D are equipped with a total of four 10 GBit adapters. Two of these are RJ45 and two are SFP+. This led to the upgrade with an Ubiquit EdgeSwitch 16 XG.

There are storage shelves on the back, which can be used for the Raspberry Pi and other accessories. A Raspberry Pi acts as DNS and NTP server in the lab.

Since the whole rack is mounted on rollers, it can be easily pulled out for maintenance. Only one patch cable and one power supply has to be disconnected for this.

Noise emission

As the number of devices increases, so does the noise. I already described how to silence the Netgear 10G switch in an older article. The Ubiquiti switches work surprisingly silent and are only a bit louder during startup. At a distance of one meter, all switches reach a sound pressure of about 20 dB. This corresponds to a flying mosquito. Not too annoying, but audible. The four E300 servers currently have the highest noise level, but the original fans could be replaced with Noctua models as well. Maybe a future project…