ESXi host restore with obstacles

Unable to re-join EVC cluster after restore of ESXi system

Changing boot media of ESXi hosts (unfortunately) has become a routine job. It is based on the fact, that many flash media have a limited lifespan. To be fair, I need to point out that many customers use (cheap and dirty) USB flash sticks as boot media. But what is good in a homelab, turns out to be a bad idea in enterprise environments.

The usual procedure for media replacement is fairly simple:

  • export host configuration
  • evacuate and shut down host
  • prepare fresh boot medium with installation ISO that has the same or lower patchlevel as the old installation
  • boot freshly installed host
  • apply (intermediate) IP address if no DHCP available
  • restore host configuration
  • re-connect to cluster
  • apply patches if neccessary

So far so good. But last week I had a nasty experience with a recovered ESXi host. Continue reading “ESXi host restore with obstacles”

vCenter and AD Domain Functional Level

If you’re running a vCenter appliance with Active Directory integration you should take care about your Domain Functional Level. It is crucial to closely work together with the domain administrators team, for some vCenter versions may not support the latest level supported by Windows Server 2016.

What is the Domain Functional Level?

Functional levels determine the available Active Directory Domain Services domain capabilities. They also determine which Windows Server operating systems you can run on domain controllers in the domain or forest. Choosing a Functional Level of Windows Server 2012 implies that there can’t be any Domain Controllers prior that level (like Server 2008 R2).

Functional levels do not affect which operating systems you can run on workstations or servers that are joined to the domain.

Set the domain and forest functional levels to the highest value that your environment can support. This way, you can use as many ADS features as possible. Continue reading “vCenter and AD Domain Functional Level”

vSphere 6.7 U1 and Veeam Backup 9.5 U3a

Failed Backup Jobs after Update to vSphere 6.7 U1

On October 16th 2018 vSphere 6.7 Update 1 became available. An update we’ve been desperately waiting for. Finally vSphere-Client (HTML5) has become fully functional. Until that some tasks had to be done with the infamous flash client.

Unfortunately problems emerged soon after users updated their clusters to vSphere 6.7 U1 in combination with Veeam Backup jobs.

Workaround

VMware and Veeam worked hard to identify the root cause of the problem. It turned out that there was a change in the vSphere API which caused communication issues with Veeam Backup.

Latest API version is 6.7.1, but this one seems to be incompatible with Veeam Backup 9.5 U3a. According to Veeam sources, the issue will be settled with the soon to come Veeam Backup Update 4.

For all of those who have already updated their clusters to vSphere 6.7 U1 there’s a workaround. You need to enter a registry key to force Veeam Backup using the older API 6.7.

Warning! This solution is not recommended by Veeam Support. If you’re not yet on vSphere 6.7 U1 and you’re using Veeam Backup, you better wait until release of Veeam Backup 9.5 Update 4. Do not upgrade. Read this passage again!

The workaround outlined below has to be reverted as soon as Veeam 9.5 Update 4 is available.

HKLM\SOFTWARE\Veeam\Veeam Backup and Replication

You need to add a multi-string-value (REG_MULTI_SZ). Enter the Value below:

6.7.1 = 6.7

This registry key forces Veeam Backup to use API 6.7, but might lead to other yet unknown problems. But it enables to run your Veeam Backup jobs again.

NSX-Manager Permissions and Groups

Using vSphere-Client 6.7 to synchronize NSX-Manager with Active Directory

Once you’ve deployed NSX-Manager to a vSphere 6.7 cluster, you may have noticed an error  on the dashboard.

“No NSX Managers available. Verify current user has role assigned on NSX Manager.”

Assuming you have configured vCenter connections correctly, there’s a simple explanation for the error (KB2080740).

Usually initial setup of NSX-Manager is done by the default SSO User administrator@vsphere.local. If you log into vCenter using that user, there will be no error on the dashboard. The point is that NSX-Manager has its own permissions and roles which are not coupled to vCenter permissions. That means a user with administrator rights in vCenter does not automatically get administrator rights in NSX-Manager. Without any permissions that user can’t even see NSX-Manager. Continue reading “NSX-Manager Permissions and Groups”