Some of you might already use Linux repository servers based on XFS with fast clone. Once you upgrade to Veeam Backup & Replication version 11 you may wish to activate immutable backups. I will explain in this blog post how you can use this new feature on existing XFS repository servers.
Activate immutability
To activate the immutability feature edit your existing backup repository servers and go to step “repository” (picture below). Starting with version 11 there’s a checkbox “make recent backups immutable for [x] days” where 7 is the minimum value to enter. Once you activate the checkbox you will most likely see a message “Unable to set the immutability option: cannot find the transport service on server <your repo server>. Configure the server anyway?“. Click on [Yes].
To make use of the immutability flag you’ll have to reconfigure the transport service on the Linux server.
Attackers on IT infrastructure have become more sophisticated in recent times. Not only do they ecrypt live data and entire virtual machines, they also have learned to delete or encrypt entire backups too. No matter if your online backups are stored on a local repository or a cloud share. If that happens and you do not have an offline backup like tape or any other air gapped solution you’re up a creek without a paddle. Even worse: If you’re responsible for data backup, you might be facing unpleasant questions by the management.
Now the question is how to protect backup data in case an attacker gets hold on your backup server. Once they get credentials to the server, they can do whatever the backup-admin can do. Deleting backups for example. One first step is to separate your backup systems from your domain. In case the domain administrator account gets compromised there’s still (a little) barrier between attackers and your backup systems. But still, you’ll never know if attackers have knowledge of a zeroday exploit to takeover your backup server. It would be a better approach to protect backups against deletion for a defined timespan. There’s a similar option on AWS S3 buckets, which are usually used as offsite backup copies. Getting back offsite backups takes a considerable amount of time. In case of a crypto attack time is crucial and the clock is ticking. Wouldn’t it be great to have immutable backups on your primary repositories on site? Good news. There’s light at the end of the tunnel (and I promise it’s not the oncoming train).
Immutable Backups
Before we go into detail, we need to clarify what immutable backups mean. The idea is that you write once and then files (backups) are protected for a self defined period of time. Even the backup administrator cannot delete them before the defined timespan has passed.
Veeam Backup & Replication v11 will make use of a native Linux XFS filesystem feature. XFS can set an extended file attribute [i] which will protect the file from renaming, modification, deletion or hard-linking. The key point is that backups are transferred and written with non-privileged accounts and the immutable attribute is set and removed by a privileged service on the repository server which cannot be accessed from outside.
All you need is a Linux repository formatted with XFS and Veeam Backup & Replication v11, which will be released in the near future. (Update: Expected release date will be February 24th 2021)
As part of a VMware Cloud Foundation (VCF) greenfield deployment, the Cloud Builder appliance is used for one-time use. It automatically deploys the management infrastructure of a VCF cluster and can be discarded afterwards.
The ideal situation is that the previously created workbook or JSON is processed and the cluster is successfully created.
In the UI of the Cloud Builder, however, there is no option to reset the wizard and restart it from zero. For example, when requirements have changed and a new or adapted workbook is to be used. Or you want to use the appliance for another rollout. In this case, the appliance would have to be completely redeployed. Any errors in the JSON file cannot be corrected this way either.
However, there is a trick to reset the Cloud Builder to zero and feed it with a modified JSON file. This is thanks to an API call that may have been ‘forgotten’ during development. In order to do so, we have to log in to the console of the Cloud Builder as user root.
[Optional] It may be easier to grant the root user temporary SSH access. Log in to the VM console as root and edit the sshd configuration.
sudo vi /etc/ssh/sshd_config
Browse the sshd_config and look for the line PermitRootLoginno. Disable the line by putting a # in front of it.
# PermitRootLogin no
Save the configuration and open a SSH session as user root. We now can execute an API call as user root.
curl -X GET http://localhost:9080/bringup-app/bringup/sddcs/test/deleteAll
Login to the web UI of the Cloud Builder appliance. You now can start from the beginning.
I don’t think it’s an exaggeration when I say “2020 was a year like no other.” Pretty much everyone in the IT industry and elsewhere will confirm that. I’m taking this last day of 2020 as an opportunity to write my last blog article of the year and review important events for me in it.
January
The year began quite normally, almost like any other. After the turn of the year, tasks and projects for the coming months slowly started. According to experience, it is a quiet month, because many customers usually don’t resume work before the second week.
February
February had some of my personal highlights of the year.
We held a VMUG meeting in Nuremberg on February 13th. Today I have to add that it was an on-site meeting. At that time a matter of course. Today it sounds like a tale from a distant past. I myself had given a presentation there on the topic of “Strategies for proactive error prevention”. A closer look at the products VMware Skyline and Runecast Analyzer. What do they have in common and what are their unique selling points?
A long planned private trip took us to the high latitudes of the Arctic. Spitsbergen in winter. No other event left such deep impressions this year like this one. On February 14th, after 84 days of darkness, the long polar night ends on Spitsbergen and the sun comes above the horizon again for the first time. In mid-April, the sun will not set at all for another 99 days. In the time in between the ice desert shines in wonderful light. We love the Arctic and wanted to experience this natural spectacle and – of course – capture it in pictures.
Polar bear warning applies to the whole archipelago of Svalbard.
Exploring Spitsbergen means being away from civilization for many hours at temperatures down to -38°C. Always accompanied by an armed guide, because the island is polar bear country and you’re required by law to have at least one armed member in your party once leaving the settlement.
We’ve learned, for example, that such simple things like eating can be a real challenge at very low temperatures We’ve also witnessed what it means to be caught in a white-out when the sky merges into the coulour of the ice, visibility is close to zero and everything around you is just white.
Svalbard reindeer
While being abroad I got a pleasant notification in my inbox. I was nominated to be a vExpert for another year.
March
Our stay lasted until the first week of March. But the world we returned to was a different to the one we left. Already on the way we read news about Corona outbreaks and sold-out toilet paper. When we left Germany, Corona was hardly a topic. Now it was THE topic and would remain so for months.
On March 11th, the last German on site VMUG took place at VMware’s premises in Munich. I myself was co-organizer and speaker and had no concerns about the train ride and the meeting at that time. Looking back and with the knowledge of today, it was certainly not the brightest idea. We were unnecessarily putting ourselves at great risk. Washing hands alone is not enough against SARS-CoV-2. Luckily (as far as I know), the event had no further health consequences for anyone involved.
