Symptoms

How to uninstall PowerPath 5.7 SPX on the cluster node where it says migrations are pending ? The user earlier started setup of PPME Migrations on PowerPath 5.7 SPX node in a 2 node cluster but later faced some issues and tried to uninstall PowerPath 5.7 SPX. It failed on the node where PPME commands were run, gives the message that migrations are pending.As per PowerPath Uninstall Logs we see the following error:Action start 11:35:42: CA_CheckPendingClusterMigrations.MSI (s) (34!90) [11:35:47:258]: PROPERTY CHANGE: Adding PpMigrationPending property. Its value is '1'.PpCheckPendingClusterMigrations: Entering PpCheckPendingClusterMigrations in C:\Windows\Installer\MSIDD8F.tmp, version 5.7.1.343PpCheckPendingClusterMigrations: Pending cluster migrations were foundMSI (s) (34:48) [11:35:47:258]: Skipping action: CA_PPMigrationsError (condition is false)MSI (s) (34:48) [11:35:47:258]: Doing action: CA_PPMigrationsPendingAction ended 11:35:47: CA_CheckPendingClusterMigrations. Return value 1.MSI (s) (34:48) [11:35:47:258]: Transforming table CustomAction.MSI (s) (34:48) [11:35:47:258]: Transforming table CustomAction.MSI (s) (34:48) [11:35:47:258]: Note: 1: 2262 2: CustomAction 3: -2147287038Action start 11:35:47: CA_PPMigrationsPending.MSI (s) (34:48) [11:35:48:584]: Product: EMC PowerPath 5.7 Service Pack 1 (64bit) -- Migration(s) is(are) pending. Cannot uninstall.MSI (c) (68:8C) [11:35:47:258]: Font created. Charset: Req=0, Ret=0, Font: Req=MS Shell Dlg, Ret=MS Shell DlgMigration(s) is(are) pending. Cannot uninstall.MSI (s) (34:48) [11:35:48:584]: Transforming table InstallExecuteSequence.MSI (s) (34:48) [11:35:48:584]: Transforming table InstallExecuteSequence.MSI (s) (34:48) [11:35:48:584]: Doing action: CA_WexExecuteExitActions_OnErrorAction ended 11:35:48: CA_PPMigrationsPending. Return value 3. Audit Logs show the following Entries regarding the setup:2/6/2014:6:35:25 PM:PPME: Info: handle=1, event=stateChanged, state=setup, cmd=setup, tech=HostCopy, src=harddisk12 (XXXXXXXXXXXX:1B32), tgt=harddisk1 (XXXXXXXXXXXX:2026), clus=PPME Cluster Disk 142/6/2014:7:16:31 PM:PPME: Info: handle=2, event=stateChanged, state=setup, cmd=setup, tech=HostCopy, src=harddisk16 (XXXXXXXXXXXX:0537), tgt=harddisk2 (XXXXXXXXXXXX:204A), clus=PPME Cluster Disk 9"powermig info -all" Shows the following output:-C:\Windows\system32>powermig info -allHnd Source Target Tech State 1 harddisk?? harddisk?? HostCopy(cl) needsRecovery(cleanup) 2 harddisk?? harddisk?? HostCopy(cl) needsRecovery(cleanup) Trying to cleanup the handle 1 gives the following Error:-C:\Windows\system32>powermig cleanup -handle 1Cleanup migration for handle 1 ? [yes]/no: yPPME error(7): Needs recovery Trying to cleanup the handle 1 using force option gives the following error:-C:\Windows\system32>powermig cleanup -handle 1 -forceCleanup migration for handle 1 ? [yes]/no: yPPME error(74): Cluster open failed, cluster migration must be initiated from a cluster node Using the recover command on handle 1 returns the following error:C:\Windows\system32>powermig recover -handle 1Recover migration for handle 1 ? [yes]/no: yPPME error(74): Cluster open failed, cluster migration must be initiated from a cluster node Powermigcl command displays the following result and shows all the groups are unconfigured:- C:\Windows\system32>powermigcl display -all Group name | Current owner | Status |================================|=================|===================| Cluster Group | CH1BAASGSQLA01 | Unconfigured | Available Storage | CH1BAASGSQLA02 | Unconfigured | DDServicesGroupCH1 | CH1BAASGSQLA01 | Unconfigured | SQL Server (BY2BAASGSQL) | BY2BAASGSQLA02 | Unconfigured | Query commands show the following result:- C:\Windows\system32>powermig query -handle 2 Handle: 2Source: harddisk?? (1.96 TB)Target: harddisk?? (1.96 TB, thin)Technology: HostCopyMigration state: needsRecovery(cleanup)Throttle Value: 0 Trying to run the cleanup gives the following error:-C:\Windows\system32>powermig cleanup -handle 2 -force Cleanup migration for handle 2 ? [yes]/no: y PPME error(74): Cluster open failed, cluster migration must be initiated from a cluster node The user wants to cleanup these handles so that they can uninstall PowerPath 5.7 SP1 and reinstall.

Cause

The problem here is that migration node contains information about the migration stored in User mode database (umd). This stores the ppme resource name which was associated with the migration at setup (PPME Cluster Disk 14 , and PPME Cluster Disk 9 accordingly). When you try to cleanup the migration, it tries to open the resource and check the state of all the nodes. As the resource was removed, it cannot do it.Please note, if running PowerPath 5.7 SP1, this may have been caused by a known bug whereby if you run the powermigcl config command prior to the target devices being presented and initialized on the host the ppme.dll saves an incomplete device mapping in umd_dev_conf in kmd database.

Resolution

Before cleaning up any migrations the customer needs to know which disks are source and which are targets and what state the copy got to.PowerPath Migration Enabler copies every block of the LUN. This means when the source and target are in Sync, they will look identical.After the Commit state the source and target will be no longer be in-Sync but will share the same Disk Signature / Label.The powemig cleanup command scrubs the original source disk so that it cannot be used any further (wipes sector 0 where the signature resides).If you take the example below you will notice one pair of devices has been cleaned up (although requires recovery) and the other pair is in the committed state.You don't want to uninstall PowerPath in this state because we still have identical signatures on the handle 5 (Harddisk7 & harddisk9) and a reboot will cause issues with cluster.Removing the original source devices from all nodes for this configuration would be preferred so that there is no way of confusing the server about which disk to use in the cluster. This may require removing zones or devices from storage groups. If there are devices that are still not committed then do not remove those devices.If all handles have been cleaned up there is no need to remove the source devices unless you want to. POWERMIG INFO -ALL Hnd Source Target Tech State 1 harddisk3 harddisk15 HostCopy(cl) needsRecovery(cleanup) 5 harddisk7 harddisk9 HostCopy(cl) committed Once the source devices have been removed: 1. Manually remove all dependencies between the PPME resource and the Cluster Disk resource in Failover Cluster. Please also check the Available Storage group, in case that was also converted. Removing dependencies is done in the cluster gui > click the cluster group > right click the cluster disk > go to properties and under dependencies you will see PPME resource, remove that dependency. At the bottom there are other resources and will you see the PPME resources > right click and delete. WARNING: if you get a message that a resource depends on this, STOP and go back to check that the device is not depending on the resource. 2. Offline and delete all PPME resources from the Failover Cluster.3. Failover all cluster groups to another node.4. Stop cluster service on the node where powermig shows the migrations.5. Uninstall PowerPath 5.7 SPX and reboot the server. If the uninstall fails showing migrations still in progress, please contact EMC for some manual steps to cleanup the PowerPath database.6. Install PowerPath 5.7 SPX. and reboot the server

Support Cases