VPLEX: Performance regression for VS2 Metro-IP systems on GeoSynchrony 6.1.x

Impact:
Higher (worse) write latency occurs for VS2 Metro-IP systems running GeoSynchrony 6.1.x vs. VS2 Metro-IP systems running GeoSynchrony 5.x/6.0.x. 
 

Issue:

After upgrading a VS2 Metro-IP system running GeoSynchrony 5.x/6.0.x to GeoSynchrony 6.1.x end users may notice higher I/O response times.

The performance regression is greatest  when there is zero latency between the clusters in the VS2 Metro IP system (0ms or near 0ms WAN Round Trip Time or RTT),  offered load overhead latency is at times 50% higher.

A VS2 Metro-IP system with non-zero WAN latency (1+ms WAN RTT) may only experience as little as a sub-ms (microsecond) increase in write latency, but this still has the potential to impact latency-sensitive environments.

When the issue occurs the VPLEX firmware.log will have been logging  floor/33  events for  port show .

Example log excerpts:

128.221.253.68/cpu0/log:5988:W/"00601636d7c624637-2":5911:<5>2018/07/27 14:26:28.02: floor/33 TowerDomain domain has been stuck on pid 4720 (port show) for 150000 usecs, backtrace: none

128.221.253.68/cpu0/log:5988:W/"00601636d7c624637-2":5912:<5>2018/07/27 14:26:32.86: floor/33 TowerDomain domain has been stuck on pid 5164 (port show) for 150000 usecs, backtrace: none

128.221.253.67/cpu0/log:5988:W/"00601638320023139-2":8151:<5>2018/07/27 14:26:30.91: floor/33 TowerDomain domain has been stuck on pid 9936 (port show) for 180000 usecs, backtrace: none

In GeoSynchrony 6.1.x the management-server CLI may excessively call the directors with a port listing command (port show) that on VS2 Metro-IP systems may result in long-running firmware tower domain threads, which causes an increase in I/O response time (particularly write response times).

If you encounter this issue please contact Dell EMC Customer Support by opening a Live Chat and mention this article.

Note: To open a live chat with support go to https://www.dell.com/support/ and look for the green tag on the right side of the screen labeled  Contact Us , click on it and it will slide open, then click on  Chat with Support . This will take you to the  Live Chat  page where you will follow the steps for opening a Live Chat with support.

Permanent Fix:
This issue is fixed in GeoSynchrony 6.2 and later.

If there are other VPLEX currently running on pre-GeoSynchrony 6.1.x the recommendation is to upgrade to 6.2 where there is a fix for this issue.