System choked by data – Why you should remove the default repository after installation
A typical Veeam Backup & Replication installation consists of several sub-components. There is the Backupserver with the database, there are backup proxies, Mount Server, Gateway server and Backup-repositories. Repositories are datastores which hold your backup data. Right after initial setup the installer will create a repositoty on your system partition which is the default repository. Normally your system partition isn’t very big. Maybe 100 GB or less. One of the first tasks after installation is to define a new backup repository with Terabytes of free space. Sometimes you might forget about the default repository, which is pointing at your system partition. Under certain conditions this can turn into a timebomb which I witnessed in the wild recently.
What happened?
Im referring about a rather simple Veeam Backup infrastructure. A virtual Veeam Server with a physical proxy which has two iSCSI Targets as repositories. First repo for backups and second repo for backup copies – nothing special.
One day the Windows OS on the proxy stopped responding and had to be restarted. After reboot the second repo with backup copies was down. We managed to bring it back online and triggered some retrys of failed backup jobs from the last night. Everything seemed fine.
After all job retries finished successfully, the backup copy job kicked in and failed soon after transferring 2 or 3 VMs. I’ve realized that the system partition of the (virtual) Veeam Server was full with 0 bytes free. WTF?!
Luckyily it was a VM and I could expand the vDisk. But in no time the partition filled up again. After inspecting the system partition of the backup server I’ve found backup files growing rapidly. Time to disable all active jobs now! Just in time before the system partition was about to be flooded again.
While checking the jobs I realized that somehow the repository for the backup copy job made a fallback to default after repo two went down.
You don’t need much fantasy to imagine how a backup copy job will choke your system partition in no time.
Conclusion
I made this to one of my standard operation procedures (SOP). Right after adding the first Veeam repository, I will remove the default repository. Forgetting to do so might plant a timebomb in your backup infrastructure.