Symptoms
This issue affects the automated generated File Service VMs (= FSVM).Reference: "When you configure vSAN file service, vSAN creates a single VDFS distributed file system for the cluster which will be used internally for management purposes. A file service VM (FSVM) is placed on each host. The FSVMs manage file shares in the vSAN datastore. Each FSVM contains a file server that provides both NFS and SMB service."
A File Service VM (= FSVM) shows the Warning: "Virtual Machine Consolidation Needed Status"The vCenter /var/log/vmware/vsan-health/vmware-vsan-health-service.log indicate that the destroy of the File Service VM (= FSVM) failed by showing similar entries like the following:
2020-04-21T06:05:20.787Z ERROR vsan-mgmt[35201] [VsanEamUtil::DestroyEamAgentVMs opID=W79787] Failed to destroy agent VM 'vim.VirtualMachine:vm-333' on host host-108Traceback (most recent call last): File "/build/mts/release/bora-16051639/bora/build/vcenter/release/pkg/vsanmgmt/stage/usr/lib/vmware-vpx/vsan-health/pyMoVsan/VsanEamUtil.py", line 734, in DestroyEamAgentVMs File "/usr/lib/vmware/site-packages/pyVim/vimutil.py", line 30, in InvokeAndTrack return WaitForTask(task) File "/usr/lib/vmware/site-packages/pyVim/task.py", line 140, in WaitForTask raise info.errorVsanVmomiPatcher.vim.fault.InvalidPowerState: (vim.fault.InvalidPowerState) { msg = 'The attempted operation cannot be performed in the current state (Powered on).', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [], requestedState = 'poweredOff', existingState = 'poweredOn'}Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Cause
This issue occurs when the File Service Remediation Service detected one or more File Service VM (= FSVM) is not working and tries to remove and deploy a new File Service VM (= FSVM).In the process of removing the bad File Service VM (= FSVM), the host side File Service Recovery Service might power it back. This will make the File Service VM (= FSVM) to be partially removed.
Resolution
This issue has been resolved by upgrading vCenter/ESXi to version 7.0 Update 1.
Workaround
Workaround to resolve this issue:
1.) Power off the File Service VM (= FSVM) that reported the Warning: "Virtual Machine Consolidation Needed Status"2.) Right-click the VM and select "Delete from Disk" to remove the VM completely. 3.) The File Service Remediation process will deploy a new File Service VM (= FSVM) onto the same Host after Step 2 has been completed by the System.
