PowerPath: Running the command "powermig sync -handle X" or "powermig cleanup -handle X" returns an error "PPME error(79):

Running the command "powermig sync -handle X" returns an error "PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later".
Running the command "powermig cleanup -handle X" returns an error "PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later".

We are trying to use PowerPath Migration Enabler(PPME) to migrate the physical disk resources that are part of the Cluster group "XYZ-ClusterResources" which is online on node NODE-3.

C:\Windows\system32>powermigcl display
Group name | Current owner | Status |
================================|=================|===================|
Available Storage              | NODE-2 | Unconfigured |
XYZ-ClusterResources      | NODE-3 | Configured |
LMN-CLuster Resources   | NODE-2 | Unconfigured |
ABC_08CLDTC                 | NODE-2 | Unconfigured |
Cluster Group                    | NODE-2 | Unconfigured |

We are able to run the setup command:
C:\Windows\system32>powermig setup -techType hostcopy -src harddisk6 -tgt harddisk16 -cluster

Setup migration? [yes]/no: y
Migration Handle = 9

C:\Windows\system32>powermig setup -techType hostcopy -src harddisk7 -tgt harddisk21 -cluster

Setup migration? [yes]/no: y
Migration Handle = 10

C:\Windows\system32>powermig info -all
================================================
Hnd Source Target Tech State
=== ========== ========== ============ =====
9 harddisk6 harddisk16 HostCopy(cl) setup
10 harddisk7 harddisk21 HostCopy(cl) setup
11 harddisk8 harddisk20 HostCopy(cl) setup
12 harddisk9 harddisk18 HostCopy(cl) setup
13 harddisk10 harddisk15 HostCopy(cl) setup
14 harddisk11 harddisk14 HostCopy(cl) setup
15 harddisk12 harddisk17 HostCopy(cl) setup
16 harddisk13 harddisk19 HostCopy(cl) setup

As Soon as we run the sync command, it gives this error:
C:\Windows\system32>powermig sync -handle 9

Start sync for handle 9 ? [yes]/no: y

PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later

Trying to cleanup the migration also gives the same error:

C:\Windows\system32>powermig cleanup -all

Cleanup migration for handle 9 ? [yes]/no: y

PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later

At that stage cleanup -force also gives the same error:
C:\Windows\system32>powermig cleanup -handle 9 -force

Cleanup migration for handle 9 ? [yes]/no: y

PPME error(79): Not all nodes have reached the correct state to complete this operation, please try this or the recover command again later

C:\Windows\system32>powermig info -all
=================================================================
Hnd Source Target Tech State
=== ========== ========== ============ ======================
9 harddisk6 harddisk16 HostCopy(cl) needsRecovery(cleanup)
10 harddisk7 harddisk21 HostCopy(cl) setup
11 harddisk8 harddisk20 HostCopy(cl) setup
12 harddisk9 harddisk18 HostCopy(cl) setup
13 harddisk10 harddisk15 HostCopy(cl) setup
14 harddisk11 harddisk14 HostCopy(cl) setup
15 harddisk12 harddisk17 HostCopy(cl) setup
16 harddisk13 harddisk19 HostCopy(cl) setup

C:\Windows\system32>powermig cleanup -handle 9 -force

Cleanup migration for handle 9 ? [yes]/no: y

PPME error(74): Cluster open failed, cluster migration must be intiate from a cluster node

In Cluster.log, there is below PPME error instead of "connection error":

ERR [RES] ppme <PPME 2404-RESOURCE-036-TEMPDB>: Cannot set funnel state in non-mig node: error:fffffffd:81c0
0000111c.00001b20::2014/03/14-19:50:24.850 ERR [RES] ppme <PPME 2404-RESOURCE-036-TEMPDB>: Cannot set funnel state in non-mig node: error:fffffffd:81c0
0000111c.00001b20::2014/03/14-19:50:27.869 ERR [RES] ppme <PPME 2404-RESOURCE-036-TEMPDB>: Cannot set funnel state in non-mig node: error:fffffffd:81c0

Or you see the following logs in the Cluster logs:-In cluster logs on node on PH1CRMSGSQL0A we see the following Logs:

00001ba8.000048b0::2014/03/24-13:30:50.879 WARN [RCM] Possible owners: <vector len='3'>
00001ba8.000048b0::2014/03/24-13:30:50.879 WARN <item>1</item>
00001ba8.000048b0::2014/03/24-13:30:50.879 WARN <item>3</item>
00001ba8.000048b0::2014/03/24-13:30:50.879 WARN <item>2</item>
00001ba8.000048b0::2014/03/24-13:30:50.879 WARN </vector>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] DllIsPresent( 4 ) returned false for restype ppme.
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] PPME Disk L cannot be hosted on node 4
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] Possible owners: <vector len='0'>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN </vector>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] Disk O cannot be hosted on node 4
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] Possible owners: <vector len='3'>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>2</item>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>3</item>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>1</item>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN </vector>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] SQL Server (INT) cannot be hosted on node 4
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN [RCM] Possible owners: <vector len='3'>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>2</item>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>1</item>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN <item>3</item>
00001ba8.000048b0::2014/03/24-13:30:50.957 WARN </vector>
00001ba8.000048b0::2014/03/24-13:30:51.035 WARN [RCM] DllIsPresent( 4 ) returned false for restype ppme.
00001ba8.000048b0::2014/03/24-13:30:51.035 WARN [RCM] PPME Disk M cannot be hosted on node 4
00001ba8.000048b0::2014/03/24-13:30:51.035 WARN [RCM] Possible owners: <vector len='0'>
00001ba8.000048b0::2014/03/24-13:30:51.035 WARN </vector>
00001ba8.000034a8::2014/03/24-13:30:51.129 WARN [RCM] PPME Disk O cannot be hosted on node 4
00001ba8.000034a8::2014/03/24-13:30:51.129 WARN [RCM] Possible owners: <vector len='0'>
00001ba8.000034a8::2014/03/24-13:30:51.129 WARN </vector>

Target devices are not presented to all the nodes of the cluster.
All the cluster nodes are not up and running.
Cluster service is not running on all the cluster nodes.
Failover nodes have funneling and access control set on for the devices.

Please make sure:

1. PowerPath version is same on all the nodes of the Cluster.

2. Find the total number of nodes in the cluster from SERVERNAME_CLUSTER_MPS_INFORMATION.TXT in the Cluster Folder in EMC reports.

3, The cluster service is running on all cluster nodes.

4. The target device is connected and is available to all the nodes of the cluster.

5. Once the above is taken care of, Run the powermig recover command on node where PPME commands were run. Then try sync.

6. If it still fails with the same error message, check the registry key:
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath" on all the nodes of the cluster and perform the following action after taking the backup of the registry:

All dm-related KMD entries should be removed on all servers in the cluster.
1. Delete the UMD by deleting the files from C:\Program Files\EMC\PPME\db\*.*

2. Delete all the subkeys with Prefix "dm_" EXCEPT for dev_conf under:
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_0" (or
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_1"). The Keys would be dm_ac, dm_control_io_to_clones, dm_funnel_io, dm_wc.

3. Delete the subkey umd_dev_conf (NOT dev_conf) under:
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_0" (or
"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\EmcPowerPath\KMD_1").

4. Reboot Each node of the cluster after performing the above steps on that node.

5. For better results, reinstall powerpath on each node and reboot.