Details
If a Virtual Machine has more than 127 vCPUs, it must use IOMMU interrupt remapping. ESX currently doesn't fully support enabling a virtual IOMMU in a Virtual Machine that is also using vGPU. ESXi 7.x and older versions will fail to Power On Virtual Machines, with the following message:"The virtual machine cannot be powered on because IOMMU virtualization is not allowed with VGPU. To power on the Virtual Machine either disable the virtual IOMMU or remove the VGPU device(s)."
Solution
ESXi 8.0 has partial support for virtual IOMMU in conjunction with SR-IOV based vGPU configurations. This support is not available with older non-SR-IOV based vGPU implementations. Also, when used in this way, DMA remapping is not supported by the virtual IOMMU. A VI Admin can enable the functionality by adding the following Virtual Machine configuration parameters: AMD CPU:amd.iommu.supportsPcip=”TRUE”vvtd.enable=“TRUE”pciPassthru.vmiop.allowViommu=”TRUE”pciPassthru.vmiop.enableViommu=”TRUE” Intel CPU:vvtd.enable=“TRUE”pciPassthru.vmiop.allowViommu=”TRUE”pciPassthru.vmiop.enableViommu=”TRUE”
Furthermore, in the Linux Guest OS the IOMMU must be in passthrough mode, by setting kernel option “iommu=pt” to disable DMA remapping.
