What happened to vSphere 7.0 U3 ?
vSphere 7.0 Update 3 was initially released on October 5, 2021. Shortly after release, there were a number of issues reported by customers, so on November 18, 2021, all ESXi versions 7.0 U3a, U3b, U3c, as well as vCenter 7.0 U3b were withdrawn from VMware’s download area. VMware explains details of the issue in KB 86191.
The main reason was a duplicate driver i40en and i40enu for Intel 10 GBit NICs X710 and X722 in the system. A check on the CLI returns a result quickly. Only one result may be returned here.
esxcli software vib list | grep -i i40
Hosts with both drivers will potentially have HA issues when updating to U3c, as well as issues with NSX.
What’s new with Update 3c ?
On 27 January 2022 ( 28 January 2022 CET) the new Update 3c was released and is available for download. Besides fixing the issues from previous Update 3 versions (KB 86191), the main feature is the fix for the Apache Log4j vulnerability (VMSA-2021-0028.10).
All users and customers who had installed one of the withdrawn updates 3 at an early stage are highly recommended to update to version U3c.
Update Precheck
Before updating to U3c, ensure that the hosts do not have both drivers on board. In an exceptional case, the update of the hosts to U3c must be done before the update of the vCenter Appliance. There are two situations where the precheck will raise an issue:
- ESXi 7.0 U3/ U3a exists anywhere in the VC inventory
- ESXi 7.0 U2c/U2d exists in a vSphere Lifecycle Manager (vLCM) image-enabled cluster. Clusters using the legacy baseline method are not affected.
VMware KB 86447 explains the precheck process.
Source: VMware KB86447
As soon as the precheck finds a problematic host version, a more granular check is started. The hosts found are checked for the presence of both drivers and, if applicable, listed in a text file on the VCSA (dual_driver_check_faulty_hosts _[timestamp].txt). The admin team then has to take action.
- Upgrade to ESXi U3c prior to upgrading vCenter (Inverted Upgrade)
- Remove the i40enu driver from the hosts (Precheck will fail again but set the flag)
If the refined check finds a host with the problematic version but without duplicate drivers, a flag is set in vCenter to skip the precheck on the next run.
config.SDDC.VCUpgradeVLCMPrecheck.Skip = True
Once this is done, the upgrade must be restarted.
The script can also be started separately from the update process. It needs to be executed in an SSH session to the vcsa. SSH and the Bash shell must be enabled in the VAMI of the vcsa. The Python script is uploaded to the appliance via SCP.
python vSphere_upgrade_assessment.py
More details at KB 87258.
After a succesful update of vcenter the flag should be reset.
config.SDDC.VCUpgradeVLCMPrecheck.Skip = False
Final Word
If you’re using baselines, please make sure you upgrade your hosts with an imported iso image and NOT the predefines non-critical baseline. Your vCenter will remind you with the banner below.
