VxRail or VxRack: Dial home event MYSTIC010011, VXR010011 on host power status, Host lost Power Supply

VxRail or VxRack generates dial home events "MYSTIC010011 or VXR010011" on "host power status" or "Host lost Power Supply" in later VxRail release and "MYSTIC020002 or VXR020002" on Disk Status after a time, from a couple of hours to weeks. The events also appear in the event log for VxRail Manager. These events occur even when hosts' power supplies and disks are healthy.

VxRail Manager's Physical Health may show PSUs or disks or both as missing as in this image:

This part of control path hardware monitoring relies on PTAgent or iDRAC responses in Dell Platforms. Sometimes VxRail Manager (VXRM) can raise a false alarm on Power Supplies or Disks when hardware information is missing from a PTAgent response or the response format that is retrieved is not expected. Usually this is transient, and the Physical Health view in VXRM could show everything fine by the time it is checked.

False positive events of these kinds (and dial homes they can generate) may also be caused by inability of VxRail Manager to poll for host information due to other reasons. This KB focuses on Dell platform, and events generated due to failure between the hosts' PTAgent and the VXRM and would not apply to Phoenix or Quanta platforms.

To confirm the hardware status from vCenter:

1. Log in to vCenter and go to Hosts and Clusters > VxRail or VxRack Datacenter > VxRail or VxRack vSAN Cluster.
2. Select each host, go to the Monitor tab > Hardware Status.
3. Check for 'power state' and 'disk state' to ensure they are not really failing as reported from the hosts to vCenter.

To confirm the hardware status from iDRAC:

1. Use a Web Browser to connect and log in to the iDRAC interface of a host (https://<host_iDRAC_IP>).
2. Select Hardware > Power Supplies, and check for power supply status.
3. Select Storage > Physical Disks, and check disk state.

How to proceed:

• If vCenter or iDRAC shows issues with host hardware, VxRail or VxRack Support does further analysis and troubleshooting to resolve it.
• If no issues are found in vCenter or on the host through iDRAC, and VXRM's Physical Health page looks fine, no further intervention should be needed.
• If VXRM's Physical Health page still shows missing PSUs or disks after confirming the above, this indicates a false alarm due to either the PTAgent > VXRM issue or VXRM being unable to properly monitor host health for another reason. In this case, follow the steps below:

Follow below steps for Dell platforms. For Phoenix or Quanta platforms, or if VXRM still shows PSUs or disks that are lost after following the steps below, log collection and further troubleshooting by VxRail Support is needed. 

1. In VxRail Manager, go to the physical health page. Use the refresh icon at the top of this page if the PSU or disks are still missing or 'lost', 
2. Check the IDRAC under IDRAC Settings > Network > Operating System to IDRAC Pass-through, ensure that the IP address for Operating System to IDRAC is set to USB NIC with 169.254.0.1 set. 
3. Ensure that the host can ping 169.254.01.
4. Restart the IDRAC and PTAgent to try to clear VxRail Manager showing PSU or temporarily work around disk issues and PSU lost:
   1. Open a PuTTY session to the host showing disk issues in VxRail Manager. Then run these commands:
   • To reset iDRAC: /scratch/dell/DellPTAgent/bin/ipmitool_static mc reset cold
   • To restart PTAgent process: /etc/init.d/DellPTAgent restart
   2. Reboot the VxRail Manager after running these steps then log back in to the Web UI and check to see if it is showing up properly.
5. If VXRM now shows PSU and disks properly, then this confirms an issue with the PTAgent > VXRM. Resetting the IDRAC and PTAgent is a temporary fix for this, and upgrading VxRail code version is likely required for a permanent fix.