Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
ESXi host is not responding to vCenter server. vpxa process on the ESXi host was shut down by itself and soon restarted automatically. A dump file of vmx process was created under a VM folder. In the vmware.log under the VM directory, you see entries similar to: 2017-05-11T08:38:18.497Z| svga| I125: MKS-RenderMain: Stopping MKSBasicOps 2017-05-11T08:38:18.498Z| svga| W115: GLWindow: Unable to reserve host GPU resources 2017-05-11T08:38:18.498Z| vmx| I125: Msg_Post: Information 2017-05-11T08:38:18.498Z| vmx| I125: [msg.mks.noGPUResourceFallback] Hardware GPU resources are not available. The virtual machine will use software rendering. 2017-05-11T08:38:18.498Z| vmx| I125: ---------------------------------------- 2017-05-11T08:38:18.499Z| svga| I125: MKS-SWP: plugin started - llvmpipe (LLVM 3.3, 256 bits) 2017-05-11T08:38:18.500Z| svga| I125: Started Shim3D 2017-05-11T08:38:18.500Z| svga| I125: MKS-RenderMain: Starting SWRenderer 2017-05-11T08:38:18.506Z| svga| I125: Stopped Shim3D 2017-05-11T08:38:18.506Z| svga| I125: MKS-SWP: plugin stopped 2017-05-11T08:38:18.506Z| svga| I125: MKS-RenderMain: Stopping SWRenderer 2017-05-11T08:38:18.506Z| svga| I125: MKS-RenderMain: Starting MKSBasicOps 2017-05-11T08:38:18.512Z| svga| I125: MKS-RenderMain: Stopping MKSBasicOps 2017-05-11T08:38:18.513Z| svga| W115: GLWindow: Unable to reserve host GPU resources 2017-05-11T08:38:18.514Z| svga| I125: MKS-SWP: plugin started - llvmpipe (LLVM 3.3, 256 bits) 2017-05-11T08:38:18.514Z| svga| I125: Started Shim3D 2017-05-11T08:38:18.514Z| svga| I125: MKS-RenderMain: Starting SWRenderer 2017-05-11T08:38:19.024Z| vmx| I125: Vigor_MessageRevoke: message 'msg.mks.noGPUResourceFallback' (seq 17226643) is revoked 2017-05-11T08:38:19.025Z| vmx| I125: Msg_Post: Information 2017-05-11T08:38:19.025Z| vmx| I125: [***] ************ 2017-05-11T08:38:19.025Z| vmx| I125: ---------------------------------------- 2017-05-11T08:38:19.035Z| vmx| I125: Vigor_MessageRevoke: message '***' (seq 17226644) is revoked 2017-05-11T08:38:19Z[+0.000]| vmx| W115: Caught signal 11 -- tid 2946475 (addr 51CB83E0) where area marked as *** in the log messages above are some string with garbled letters depending on cases. In the hostd log under /var/run/log directory, you see entries similar to: 2017-05-11T08:38:19.024Z info hostd[60980B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 124098 : Message on <VM_name> on <ESXi_name> in ha-datacenter: Hardware GPU resources are not available. The virtual machine will use software rendering. .. 2017-05-11T08:38:19.035Z info hostd[60980B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 124100 : Message on <VM_name> on <ESXi_name> in ha-datacenter: *** .. 2017-05-11T08:38:21.173Z info hostd[611C4B70] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 124104 : An application (/bin/vmx) running on ESXi host has crashed (1 time(s) so far). A core file might have been created at /vmfs/volumes/bae09318-1962d078/<VM_name>/vmx-zdump.000. Similar to the vmware.log, *** in hostd log above will be garbled letters. In the vpxa log under /var/run/log directory, you see entries similar to: 2017-05-11T08:38:19.757Z error vpxa[FFEB3B70] [Originator@6876 sub=vmomi.soapStub[1] opID=WFU-76958b15] Error deserializating SOAP response body: --> Error returned by expat parser: not well-formed (invalid token) --> --> while parsing serialized value of type string --> at line 7, column 626 --> --> while parsing property "fullFormattedMessage" of static type string --> --> while parsing serialized DataObject of type vim.event.VmMessageEvent --> at line 7, column --> --> while parsing return value of type vim.event.Event[], version vim.version.version --> at line 7, column 0 --> --> while parsing SOAP body --> at line 6, column 0 --> --> while parsing SOAP envelope --> at line 2, column 0 --> --> while parsing HTTP response for method readNext --> on object of type vim.event.EventHistoryCollector --> at line 1, column 0 2017-05-11T08:38:19.758Z error vpxa[FFEB3B70] [Originator@6876 sub=vpxaVmomi opID=WFU-76958b15] [VpxaClientAdapter::InvokeCommon] Got exception while invoking readNext on vim.event.EventHistoryCollector:session[52524d42-09c9-7049-b373-1909e8ca4582]5284e251-e630-1e82-5dff-df13ab09949a: 'N7Vmacore4Soap24InvalidResponseExceptionE( --> Error returned by expat parser: not well-formed (invalid token) --> --> while parsing serialized value of type string --> at line 7, column 626 --> --> while parsing property "fullFormattedMessage" of static type string --> --> while parsing serialized DataObject of type vim.event.VmMessageEvent --> at line 7, column 42 --> --> while parsing return value of type vim.event.Event[], version vim.version.version10 --> at line 7, column 0 --> --> while parsing SOAP body --> at line 6, column 0 --> --> while parsing SOAP envelope --> at line 2, column 0 --> --> while parsing HTTP response for method readNext --> on object of type vim.event.EventHistoryCollector --> at line 1, column 0)' 2017-05-11T08:38:19.758Z error vpxa[FFEB3B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-76958b15] [WaitForUpdatesDone] Got error while processing updates: N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication) 2017-05-11T08:38:19.758Z error vpxa[FFEB3B70] [Originator@6876 sub=VpxaHalCnxHostagent opID=WFU-76958b15] [WaitForUpdatesDone] Fatal error while listening-for/processing updates from hostd. --> N5Vmomi5Fault17HostCommunication9ExceptionE(vmodl.fault.HostCommunication) 2017-05-11T08:38:19.763Z info vpxa[FFEB3B70] [Originator@6876 sub=vpxaInvtHost opID=WFU-76958b15] [VpxaInvtHost] ServerId has been changed from 54132 to 0 2017-05-11T08:38:19.763Z error vpxa[FFEB3B70] [Originator@6876 sub=vpxaInvtHostCnx opID=WFU-76958b15] [VpxaInvtHost] Can't connect to hostd/serverd. Shutting down... 2017-05-11T08:38:19.763Z info vpxa[FFEB3B70] [Originator@6876 sub=Default opID=WFU-76958b15] [Vpxa] Shutting down now
The vmx process crashed around processing the message event "[msg.mks.noGPUResourceFallback] Hardware GPU resources are not available. The virtual machine will use software rendering". As the vmx process passed a message event with invalid string to hostd just before the crash, hostd can't create a proper response to GetChange request from vpxa, hence vpxa gets an error on deserializing SOAP response body and decided to exit by itself. It results in ESXi host not to respond to a heartbeat from vCenter.
This is a known issue affecting ESXi 6.0 VMware vSphere ESXi 6.5 P02 (ESXi650-201712001) To download, go to Customer Connect Downloads.
To workaround the issue, restart the ESXi host daemon and vCenter Agent services using these commands: /etc/init.d/hostd restart/etc/init.d/vpxa restart
vmx プロセス クラッシュが原因で、ESXi ホストが vCenter Server に応答しない