Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
An ESXi host is disabled (grayed out) and displays as Not Responding.An ESXi host is disabled (grayed out) and displays as Disconnected.Clients connected to services running in one or more virtual machines are no longer accessible.Applications dependent on services running in one or more virtual machines are reporting errors.One or more virtual machines are no longer responding to network connections.
This article provides steps to determine if an ESXi host was powered off or restarted.
Hardware, User Interaction, Physical Infrastructure related, Third party calls using API etc.,
None
Note: This article assumes that you have completed the steps described in Troubleshooting an unresponsive host and multiple Disconnected virtual machines (1019082). ESX 4.x To determine the reason for abrupt shut down or reboot an ESX host: If the host is currently turned off, turn the host back on. Ensure that there are no hardware lights that may indicate a hardware issue. For more information, engage the hardware vendor. Log in to the host at the console as the root user. Run the command: # cat /var/run/log/vmksummary Determine if the ESX host was deliberately rebooted. When a user or script reboots a VMware ESX host, it generates a series of events under /var/run/log/vmksummary similar to: localhost logger: (1265803308) hb: vmk loaded, 1746.98, 1745.148, 0, 208167, 208167, 0, vmware-h-59580, sfcbd-7660, sfcbd-3524 localhost vmkhalt: (1268148282) Rebooting system... localhost vmkhalt: (1268148374) Starting system... localhost logger: (1268148407) loaded VMkernel Hostd: [<YYYY-MM-DD> <time>.284 27D13B90 info 'TaskManager'] Task Created : haTask-ha-host-vim.HostSystem.reboot-50</time> If your ESX host is deliberately restarted, review the vCenter Server logs to identify any recent tasks that may have made the ESX host to reboot. These are a list of other resources that help determine the reason for reboot of an ESX host: For information on tracking user login and activities, see Tracking ESX host user logins and activities (1010026).Third-party products that reside in the service console or using the VMware vSphere API may manipulate the functionality of the VMware ESX host. For more information about third-party software in the service console, see Third-Party Software in the Service console.When applying patches to VMware ESX, unless you specify the --noreboot option when running the esxupdate command, the patches that require a reboot automatically reboot the ESX host. For more information, see Confirming if a patch or software installation caused an ESX host reboot (1004410). Determine if the VMware ESX host was deliberately shut down. When a user or script shuts down a VMware ESX host, it generates a series of events similar to: localhost logger: (1265803308) hb: vmk loaded, 1746.98, 1745.148, 0, 208167, 208167, 0, vmware-h-59580, sfcbd-7660, sfcbd-3524 localhost vmkhalt: (1268149354) Halting system... localhost vmkhalt: (1268149486) Starting system... localhost logger: (1268149540) loaded VMkernel If your VMware ESX host is deliberately shut down, review the vCenter Server logs to identify any recent tasks that may have told the VMware ESX host to reboot. Use this list of other resources to help determine the reason for shutting down of VMware ESX host: For information on tracking user login and activities, see Tracking ESX host user logins and activities (1010026).Third-party products that reside in the service console or using the VMware vSphere API may are able to manipulate the functionality of the VMware ESX host. For more information about third-party software in the service console, see Third-Party Software in the Service console.If a server hardware watchdog timer is enabled, it may automatically reboot the ESX host if it detects that the operating system is unresponsive. For more details about your server hardware watchdog timer, consult the applicable software documentation and support. For more information about Hewlett Packard's server hardware watchdog, see HP Automatic Server Recovery in a VMware ESX Environment (1010842) and if necessary engage Hewlett Packard documentation and support.Sometime virtual power shutdown/restart using an iLO on HP server can be the reason if the ESX server rebooted or shutdown. Determine if the ESX host experienced a kernel error. When an ESX host experiences a kernel error, it generates a series of events similar to: vsphere5 logger: (1251788469) hb: vmk loaded, 3597562.98, 3597450.113, 13, 164009, 164009, 356, vmware-h-79976, vpxa-54148, sfcbd-12600 vsphere5 vmkhalt: (1251797195) Starting system... vsphere5 logger: (1251797206) VMkernel error vsphere5 logger: (1251797261) loaded VMkernel If your ESX host has experienced a kernel error, see Interpreting an ESXi/ESX host purple diagnostic screen (1004250). Run this command to check if the ESXi host is configured to automatically reboot after a purple diagnostic screen: esxcfg-advcfg -g /Misc/BlueScreenTimeout If the value is different than 0, then the ESXi host reboots automatically after a purple diagnostic screen. For more information, see Configuring an ESX/ESXi host to restart after becoming unresponsive with a purple diagnostic screen (2042500). When the host is rebooted after a failure and if the core dump is successful, the /var/run/log/vmksummary.log shows that a core dump is found. For example: <YYYY-MM-DD>T<time>Z bootstop: Host has booted <YYYY-MM-DD>T<time>Z bootstop: file core dump found</time></time> Note: The preceding information indicates the ESXi host failure rather than indicating the ESXi host is restarted automatically. Determine if the VMware ESX host hardware abruptly rebooted. When the VMware ESX host hardware abruptly reboots, it generates a series of events similar to: localhost logger: (1265803308) hb: vmk loaded, 1746.98, 1745.148, 0, 208167, 208167, 0, vmware-h-59580, sfcbd-7660, sfcbd-3524 localhost vmkhalt: (1268149486) Starting system... localhost logger: (1268149540) loaded VMkernel If your VMware ESX host has experienced an outage and it was not the result of a kernel error, deliberate reboot, or shut down, then the physical hardware may have abruptly restarted on its own. Hardware is known to reboot abruptly due to power outages, faulty components, and heating issues. To investigate further, engage the hardware vendor. Alternatively, the outage may have been deliberately triggered by an administrator by physically pressing the power button to turn off the hardware or using the hardware tools such as iLO, DRAC, RAS, etc. This occurrence may generate this event in the /var/run/log/vmkernel log of the ESX host: VMKAcpi: 1865: In PowerButton Helper If your VMware ESX host experiences an outage that is not the result of a kernel error, deliberate reboot, or shut down, then the physical hardware may have abruptly restarted on its own. Hardware may reboot abruptly due to power outages, faulty components, and heating issues. To investigate further, engage the hardware vendor. Alternatively, if an administrator has physically turned off or restarted the physical hardware because the console is not responding to user interaction, see Determining why an ESXi/ESX host does not respond to user interaction at the console (1017135). Notes: This message is also logged when the server is powered down through the System Management Interface (such as HP iLO).If the server is powered off by pressing the power button and the button is held for more than 10 seconds, this event is not logged. If an administrator has physically turned off or restarted the physical hardware because the console was not responding to user interaction, see Determining why an ESXi/ESX host does not respond to user interaction at the console (1017135) ESXi 4.x/5.x/6.x/7.x To determine the reason for abrupt shut down or reboot of a VMware ESXi host: Note: By default, VMware ESXi logs do not persist upon a reboot. If a VMware ESXi host experiences an abrupt reboot due to reasons other than a VMkernel error, the logs do not persist and you do not have access to the logs prior to the reboot to determine the cause. The steps in this section assume that the VMware ESXi host is configured to redirect the logs to a location where the logs persist. For more information on how to configure a VMware ESXi host to redirect the logs to an alternate location, see Configure Syslog on ESXi Hosts in the Basic Administration Guide for your version of ESXi. If the ESXi host is currently turned off, turn the host back on. Ensure that there are no hardware lights that may indicate a hardware issue. For more information, engage the hardware vendor. Determine where the logs are being redirected to: Open vSphere Client.Connect to the ESXi host or vCenter Server managing the ESXi host.Provide the credentials of an administrative user.Select the ESXi host in the Inventory.Click the Configuration tab.Click Advanced Settings.In the Advanced Settings dialog, verify the location where the log files are being redirected: Note: If either of these settings are not properly configured, then logs do not persist upon a reboot and may limit the amount of information that can be gathered for troubleshooting. Syslog > Local > Syslog.Local.DatastorePath contains the location of the logs if they are redirected to a VMFS volume.Syslog > Remote > Syslog.Remote.Hostname contains the IP address or hostname of the syslog server that houses the logs for this host. Navigate to the location of the log files, and based on the modified date of the files, open the log file using your preferred editor. Determine if the ESXi host was deliberately restarted. If an ESXi host was restarted deliberately, the /var/run/log/hostd.log file will contain events similar to these: Hostd: [12:51:54.284 27D13B90 info 'TaskManager'] Task Created : haTask-ha-host-vim.HostSystem.reboot-50 or DCUI: reboot Note: In ESXi 5.5, these entries will be in /var/run/log/shell.log. If your host is deliberately shut down, review the vCenter Server logs to identify any recent tasks that may have made the host to power off. Determine if the ESXi host was deliberately shut down. If an ESXi server was shut down deliberately, it contains an event similar to: Hostd: [<YYYY-MM-DD> <time>.550 2FEDEB90 info 'TaskManager'] Task Created : haTask-ha-host-vim.HostSystem.shutdown-78</time> or DCUI: poweroff If your host is deliberately shut down, review the vCenter Server logs to identify any recent tasks that may have made the host to power off. ESXi 5.x may also include PowerButton Helper events in the vmkernel.log file, similar to: T02:04:13.069Z cpu6:8222)VMKAcpi: 217: In PowerButton Helper Verify whether the virtual machine or ESXi host has generated a core dump: Log in to Tech Support mode. For more information, see Tech Support Mode for Emergency Support (1003677). ESXi hosts do not automatically collect the core dumps. To collect the core dump, manually run the esxcfg-dumppart command. For more information, see Extracting a core dump file from the VMKCore diagnostic partition following a purple diagnostic screen error (1002769). Note: Not configuring a core dump partition could interfere with the analysis of the abrupt reboots. For information on setting up a core dump partition, see Configuring an ESXi/ESX host to capture a VMkernel coredump from a purple diagnostic screen (1000328). If your VMware ESXi host has experienced a kernel error, see Interpreting an ESX host purple diagnostic screen (1004250). Check if ESXi is configured to automatically reboot after a purple screen by executing this command: esxcfg-advcfg -g /Misc/BlueScreenTimeout If the value is different than 0, then ESXi reboots automatically after the purple screen. For more details: Configuring an ESX/ESXi host to restart after becoming unresponsive with a purple diagnostic screen (2042500) When the host is rebooted after a crash and if the core dump was successful, the /var/log/vmksummary.log shows that a core dump is found. For example: <YYYY-MM-DD>T<time>Z bootstop: Host has booted <YYYY-MM-DD>T<time>Z bootstop: file core dump found</time></time> Note: This information does not necessarily means that ESXi restarted automatically but gives an indication when ESXi crashed. If your VMware ESXi host experiences an outage that is not the result of a kernel error, deliberate reboot, or shut down, then the physical hardware may have abruptly restarted on its own. Hardware may reboot abruptly due to power outages, faulty components, and heating issues. To investigate further, engage the hardware vendor. Alternatively, if an administrator has physically turned off or restarted the physical hardware because the console is not responding to user interaction, see Determining why an ESXi/ESX host does not respond to user interaction at the console (1017135). The ESXi 5.x log file /var/run/log/vmksummary.log contains information regarding ESXi host startup and shutdown and an hourly heartbeat with uptime and other metrics. For more information, see Format of the ESXi 5.0 vmksummary log file (2004566. Direct Web-Client to VC and shutdown from GUI, you will see the below messages in hostd.log . 2T04:15:50.984Z info hostd[264057] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=kq62iv14-100271-auto-25dd-h5:70006796-21-29-f331 user=vpxuser:VSPHERE.LOCAL\Administrator] Event 1662 : User logged event: UserAction shutdown: Initiated by VMware VirtualCenter 2021-10-22T04:15:50.985Z info hostd[264052] [Originator@6876 sub=Vimsvc.TaskManager opID=kq62iv14-100271-auto-25dd-h5:70006796-21-29-f332 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Created : haTask-ha-host-vim.HostSystem.shutdown-3101607641 --> eventTypeId = "esx.audit.host.stop.shutdown", 2021-10-22T04:15:51.039Z info hostd[264052] [Originator@6876 sub=Vimsvc.TaskManager opID=kq62iv14-100271-auto-25dd-h5:70006796-21-29-f332 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask-ha-host-vim.HostSystem.shutdown-3101607641 Status success 2021-10-22T04:18:47.230Z info hostd[264036] [Originator@6876 sub=Solo.VmwareCLI] (vim.EsxCLI.system.shutdown) ha-cli-handler-system-shutdown created 2021-10-22T04:18:47.279Z info hostd[264036] [Originator@6876 sub=Solo.VmwareCLI] CreateDynMoType (Type vim.EsxCLI.system.shutdown) (Wsdl VimEsxCLIsystemshutdown) (Version vim.version.version5). Direct Web-Client to host and shutdown from GUI you will see the below messages in hostd.log .. 2021-10-22T04:26:19.608Z info hostd[264466] [Originator@6876 sub=Vimsvc.TaskManager opID=esxui-c891-f8d4 user=root] Task Created : haTask-ha-host-vim.HostSystem.shutdown-1063352434 --> obj = 'vim.Task:haTask-ha-host-vim.HostSystem.shutdown-1063352434', --> object = 'vim.Task:haTask-ha-host-vim.HostSystem.shutdown-1063352434', --> eventTypeId = "esx.audit.host.stop.shutdown", 2021-10-22T04:26:19.662Z info hostd[264471] [Originator@6876 sub=Vimsvc.TaskManager opID=esxui-c891-f8d4 user=root] Task Completed : haTask-ha-host-vim.HostSystem.shutdown-1063352434 Status success 2021-10-22T04:28:24.305Z info hostd[264022] [Originator@6876 sub=Solo.VmwareCLI] (vim.EsxCLI.system.shutdown) ha-cli-handler-system-shutdown created 2021-10-22T04:28:24.348Z info hostd[264022] [Originator@6876 sub=Solo.VmwareCLI] CreateDynMoType (Type vim.EsxCLI.system.shutdown) (Wsdl VimEsxCLIsystemshutdown) (Version vim.version.version5).
VMware Skyline Health Diagnostics for vSphere - FAQFor all other ESXi 5.x log file locations, see Location of ESXi 5.0 log files (2004201). To be alerted when this document is updated, click the Subscribe to Article link in the Actions boxConfiguring an ESXi/ESX host to capture a VMkernel coredump from a purple diagnostic screenExtracting a core dump file from the diagnostic partition following a purple diagnostic screen errorTech Support Mode for Emergency SupportInterpreting an ESX/ESXi host purple diagnostic screenConfirming if a patch or software installation caused an ESX host rebootTracking ESX host user logins and activitiesHP Automatic Server Recovery (ASR) in an ESX environmentDetermining why an ESX/ESXi host does not respond to user interaction at the consoleLocation of ESXi 5.0 log filesFormat of the ESXi 5.x vmksummary log fileConfiguring an ESX/ESXi host to restart after becoming unresponsive with a purple diagnostic screenESXi/ESX ホストがパワーオフまたは再起動した原因の特定确定 ESXi/ESX 主机关闭电源或重新启动的原因