Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Errors manifest in vmware.log when attempting to power on a VM, the VM is being vMotioned to another host, or a Fast-Suspend and Resume (FSR)operation is performed. FSR is used when live-migrating VM storage with Storage vMotion, but also for VM Hot Add. Hot Add is the capability to add vCPUs, memory, and other selected VM hardware devices to a powered-on VM.Errors visible in the vmware.log file: Module Disk power on failed. Cannot open the disk '/vmfs/volumes/datastore1/vm-1/vm-1.vmdk' or one of the snapshot disks it depends on. Operation failed.The migration has exceeded the maximum switchover time of 100 second(s)LWD: Failed to connect IPC client to serve request 23409792 for disk 0; error: Connection timed out in vmware.log
This issue may arise when a protected VM is migrated to a host where the DPD service is not running or where lightweight data protection is not functioning.The ioFilter (named vmwarelwd), that is leveraged by VMware Cloud Disaster Recovery and other data protection solutions, might also still be attached to a VM when the solution is no longer used to protect it.
Removal of the vmwarelwd filter from a VM that is being protected will result in a full-sync should protection be re-enabled on the VM at a later point. The execution of the attached script should be performed only against VMs that are no longer to be protected by VCDR or other solutions that leverage the vmwarelwd filter.The attached utility will not reconfigure a VM to remove any filter other than that of the vmwarelwd ioFilter.
Option 1In clusters where DRS is enabled, its important to ensure that the DPD service on each host the LWD-protected VMs can migrate to is running and functional. If there are hosts within the cluster that are not enabled for lightweight data protection or can not support vmwarelwd for some reason, use anti-affinity rules to block LWD-protected VMs from migrating to them. This applies to manual migrations as well.You may also need to ensure the DPD service is configured to start with the host from the vSphere Client. Select the ESXi host in the inventory.Click the Configure tab.Under System, click Services.Select the dpd service and use the Edit Startup Policy button to change it to "Start and Stop with host" Option 2For situations where it is no longer desired for the virtual machine to be protected by vmwarelwd-related solutions, you can use the following instructions to remove the IO filter from them. A full-sync will be necessary should protection be re-enabled on the VM at a later point.The attached script (lwd-cleanup-util.py) is a pyVmomi-based utility that allows users to: List LWD-protected virtual machines in vCenter inventory listDisable LWD protection for selected virtual machines on a vSphere vCenter server where the problematic VM resides. The script requires the VMware vSphere API Python Bindings. In particular, it requires: pyVim (vm, folder, connect)pyVmomi (with vim version of at least vim.version.v7_0_2_1)six (a single-file opensource dependency used by pyVim and pyVmomi) All of these dependencies are satisfied by the VMware vCenter Appliance. Move the script into the vCenter Appliance, SSH to it, and execute it using the directions below. Execution Steps HelpCommand Line Helppython lwd-cleanup-util.py --helpUsageList all LWD-protected VM(s) which are protected by any vendors (including VMs protected by a vendor without a vendor ID). python ./lwd-cleanup-util.py -u 'administrator@vsphere.local' --listpython ./lwd-cleanup-util.py -u 'administrator@vsphere.local' --list --all-vendors List all LWD-protected VM(s) which are protected by 'myVendor'. python ./lwd-cleanup-util.py -u 'administrator@vsphere.local' --list -i myVendor List all LWD-protected VM(s) which are protected but do not have an associated vendor ID. python ./lwd-cleanup-util.py -u 'administrator@vsphere.local' --list -i "" Disable LWD protection provided by 'myVendor' if present on the specified VM. python ./lwd-cleanup-util.py -u 'administrator@vsphere.local' -d -i myVendor -m test-vm-001 Disable LWD protection provided by a vendor which didn't specify a vendor ID, if present on the specified VM. python ./lwd-cleanup-util.py -u 'administrator@vsphere.local' -d -i "" -m test-vm-001 Disable LWD protection provided by any vendor, if present on the specified VM. python ./lwd-cleanup-util.py -u 'administrator@vsphere.local' -d --all-vendors -m test-vm-001 Disable LWD protection provided by 'myVendor' if present on the specified VMs. python ./lwd-cleanup-util.py -u 'administrator@vsphere.local' -d -i myVendor -m vm1 -m vm2 -m vm3 Disable LWD protection provided by 'myVendor' for all VMs. python ./lwd-cleanup-util.py -u 'administrator@vsphere.local' -d -i 'myVendor' --all Connect to a remote VC '10.1.2.34' to list LWD protected VMs. python ./lwd-cleanup-util.py -r 10.1.2.34 -u 'administrator@vsphere.local' --skipCertCheck -l Connect to a remote VC '10.1.2.34' to disable LWD protection provided by 'myVendor' for all LWD-protected VMs. python ./lwd-cleanup-util.py -r 10.1.2.34 -u 'administrator@vsphere.local' -d -i myVendor -all Use custom pyVpx library from 'pyVpx' subfolder to connect to a remote VC and disable LWD protection provided by 'myVendor' for all LWD-protected VMs. python ./lwd-cleanup-util.py -r 10.1.2.3 -u 'administrator@vsphere.local' -d -i myVendor --all --pyVpx pyVpx