Symptoms

Updating the firmware on Kioxia CM6 NVMe drives may cause an operating system (OS) crash. A fatal error is reported in the LifeCycle controller log for the PCIe switch associated with the NVMe drives.After the OS crash and subsequent reboot, the drives will report the updated firmware as being successfully installed. 10/24/24 7:00 RED110 Informational Downloading the Express-Flash-PCIe-SSD_Firmware_GMM6V_WN64_2.2.2_A08.EXE update package. 10/24/24 7:00 RED054 Informational An update job JID_298144489144 was created. 10/24/24 7:00 RED002 Informational Package successfully downloaded. 10/24/24 7:01 SUP0516 Informational Updating firmware for PCIe SSD In Slot 0 In Bay 1 to version 2.2.2. 10/24/24 7:01 SUP0516 Informational Updating firmware for PCIe SSD In Slot 1 In Bay 1 to version 2.2.2. 10/24/24 7:01 SUP0516 Informational Updating firmware for PCIe SSD In Slot 2 In Bay 1 to version 2.2.2. 10/24/24 7:01 SUP0516 Informational Updating firmware for PCIe SSD In Slot 3 In Bay 1 to version 2.2.2. 10/24/24 7:02 SUP0518 Informational Successfully updated the PCIe SSD In Slot 0 In Bay 1 firmware to version 2.2.2. 10/24/24 7:02 SUP0518 Informational Successfully updated the PCIe SSD In Slot 1 In Bay 1 firmware to version 2.2.2. 10/24/24 7:02 SUP0518 Informational Successfully updated the PCIe SSD In Slot 2 In Bay 1 firmware to version 2.2.2. 10/24/24 7:02 SUP0518 Informational Successfully updated the PCIe SSD In Slot 3 In Bay 1 firmware to version 2.2.2. 10/24/24 7:02 PR36 Informational Version change detected for PCIe SSD In Slot 0 In Bay 1 firmware. Previous version:2.2.0, Current version:2.2.2 10/24/24 7:02 PR36 Informational Version change detected for PCIe SSD In Slot 1 In Bay 1 firmware. Previous version:2.2.0, Current version:2.2.2 10/24/24 7:02 PR36 Informational Version change detected for PCIe SSD In Slot 2 In Bay 1 firmware. Previous version:2.2.0, Current version:2.2.2 10/24/24 7:02 PR36 Informational Version change detected for PCIe SSD In Slot 3 In Bay 1 firmware. Previous version:2.2.0, Current version:2.2.2 10/24/24 7:04 PCI1318 Critical A fatal error was detected on a component at bus 227 device 0 function 0. 10/24/24 7:04 PCI1318 Critical A fatal error was detected on a component at bus 226 device 2 function 0.

Cause

The server detects an NVMe drive timeout during the firmware update, causing the host Operating System to crash, while the iDRAC reports a PCIe fatal error.

Resolution

Dell Technologies has released CM6 firmware version 2.2.2 to resolve this behavior. Dell Technologies recommends updating the drive firmware during a maintenance window to reduce the risk to production. NOTE: While the issue is fixed in firmware 2.2.2, enhancements become effective only after the update is completed. This issue may still occur during an update to 2.2.2 or later versions from any previous firmware. The Dell Kioxia CM6 and many other NVMe devices support rebootless firmware updates that apply without the need for a server reboot.The iDRAC installs the firmware immediately instead of staging it to the LifeCycle Controller for installation on the next reboot. Support for rebootless firmware updates was added in iDRAC9 firmware 6.10.00.00; further details can be found in the iDRAC User Guides available from Dell Support.Solutions that use automated updates may be impacted if firmware updates are run before systems are moved into maintenance mode. Kioxia CM6 2.2.2 firmware (GMM6V)https://www.dell.com/support/home/drivers/DriversDetails?driverID=GMM6V Kioxia CM6 firmware 2.2.2 contains improvements to extend timeout conditions while drive firmware is being updated.

Support Cases