VIB Conflict

Host upgrades with custom images offer extended driver support for vendor specific hardware or agents. You’ll get drivers that are not included in a standard VMware (Vanilla) image. Upgrading with customized images may lead into trouble while updating existing driver packages. There used to be a nasty bug with the lsiprovider package on Fujitsu ESXi 5.1 images. Another example was the “death by upgrade” bug (blog post in German) when upgrading a customized Fujitsu installation to ESXi 6.0. There are other examples from different vendors in the hall of shame.

Recently I had lots of fun while upgrading customized ESXi 6.0 hosts. You can find out if it is a customized installation when you have a look at the image profile in vCenter. This one is from Fujitsu.

I tried to upgrade the customized host with a customized image 6.5 U2. After booting the ISO I got this error/warning message.

Our VIB in question has the name Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560. Now we need to find out its internal name. For further analysis we need a SSH shell to the ESXi host.

First we have to look for Emulex packages.

esxcli software vib list | grep Emulex

The package we’re looking for has the internal name scsi-lpfc820 and it is certified by VMware. Before we remove the driver, we need to know if any crucial component is using the package. If not, we can just delete it.

Check LAN adapters

Does the host have any network adapters, that rely on the emulex package?

esxcli network nic list

As you can see in the result, there are only Intel made network adapters in use. So there will be no problem on the network side.

Check SCSI devices

Next we have to check for SCSI devices that might use the driver.

 esxcfg-scsidevs -a

We can see that there are two Emulex FibreChannel HBA, but they’re using the lpfc driver package. So no problem to expect from this side either.

Remove VIB

After checking the host, we can now remove the conflicting driver package and have another try.

esxcli software vib remove -n scsi-lpfc820

After successful removal of the VIB, the host need a reboot. We can directly boot into our Upgrade ISO image.

Without the conflicting package there was no issue during upgrade and we finally have an ESXi host 6.5 U2 with Fujitsu customized image.

More conflicting VIB packages

The VIB scsi-lpfc820 wasn’t the only one causing trouble. On a different host I got the message shown below.

Here’s a path on the boot-media locked by multiple VIBs. It is related to LSI, so let’s search for LSI VIBs.

esxcli software vib list | grep LSI

lsiprovider                  500.04.V0.63-0005                       LSI    VMwareAccepted    2018-04-06
scsi-mpt3sas                 06.00.00.00.1vmw-1OEM.550.0.0.1331820   LSI    VMwareCertified   2018-04-06
vmware-esx-storcli-1.17.08   1.17.08-01                              LSI    PartnerSupported  2018-04-06

Seems that we have to get rid of three LSI packages. Package lsiprovider offers drivers for SCSI controllers LSI MegaRAID, LSI SAS IR, LSI SAS IR2 and MegaSR. VIB scsi-mpt3sas offers support for SAS2 and SAS3 HBA. To interact with LSI SAS-Controllers there’s vmware-esx-storcli. Because there were no LSI SAS controllers on our host, it was safe to remove all three packages.

esxcli software vib remove -n lsiprovider

esxcli software vib remove -n scsi-mpt3sas

esxcli software vib remove -n vmware-esx-storcli-1.17.08

A new custom image comes around with latest versions of the packages we’ve just removed before. Ready for potential fun in future upgrades. 😉