Skip to main content
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
  • Manage your Dell EMC sites, products, and product-level contacts using Company Administration.

Article Number: 000167139


VPLEX systems may be at risk of experiencing unresponsive or reset conditions after December 31, 2016 23:59:59 UTC from the Leap Second addition which may result in potential data unavailability

Summary: This KB is to make users aware of the planned Leap Second event scheduled for 31 December 2016 at 23:59:59 where a second will be added to all atomic clocks and NTP servers.

Article Content


Symptoms

Every now and then a leap second is added to Coordinated Universal Time (UTC) in order to synchronize clocks worldwide with the Earth's ever slowing rotation.

Systems running VPLEX GeoSynchrony 5.3.x and later may experience unresponsive or reset conditions after the addition of the Leap Second on 31-December-2016 23:59:59 Coordinated Universal Time (UTC) which may result in potential data unavailability.

Impacted Hardware:
EMC Hardware: VPLEX Series
EMC Hardware: VPLEX VS1
EMC Hardware: VPLEX VS2
EMC Hardware: VPLEX VS6
EMC Hardware: VPLEX Local
EMC Hardware: VPLEX  Metro
EMC Hardware: VPLEX  Geo

Impacted VPLEX GeoSynchrony:
EMC Software: GeoSynchrony 5.3
EMC Software: GeoSynchrony 5.3 Patch 1
EMC Software: GeoSynchrony 5.3 Patch 2
EMC Software: GeoSynchrony 5.3 Patch 3
EMC Software: GeoSynchrony 5.3 Patch 4
EMC Software: GeoSynchrony 5.4
EMC Software: GeoSynchrony 5.4 Service Pack 1
EMC Software: GeoSynchrony 5.4 Service Pack 1 Patch 1
EMC Software: GeoSynchrony 5.4 Service Pack 1 Patch 3
EMC Software: GeoSynchrony 5.4 Service Pack 1 Patch 4
EMC Software: GeoSynchrony 5.4 Service Pack 1 Patch 5
EMC Software: GeoSynchrony 5.5
EMC Software: GeoSynchrony 5.5 Patch 1
EMC Software: GeoSynchrony 5.5 Service Pack 1
EMC Software: GeoSynchrony 5.5 Service Pack 1 Patch 1
EMC Software: GeoSynchrony 5.5 Service Pack 1 Patch 2
EMC Software: GeoSynchrony 5.5 Service Pack 2
EMC Software: GeoSynchrony 5.5 Service Pack 2 Patch 1
EMC Software: GeoSynchrony 5.5 Service Pack 2 Patch 2
EMC Software: GeoSynchrony 6.0
EMC Software: GeoSynchrony 6.0 Patch 1
EMC Software: GeoSynchrony 6.0 Patch 2
EMC Software: GeoSynchrony 6.0 Service Pack 1

VPLEX systems running SUSE Linux Kernel 10.2, 11 SP1, 11 SP2, or 11 SP3 releases may become unresponsive, go offline, reset, or experience deadlock conditions on the Management Server (VS1 and VS2), MMCS-A (VS6), or directors, after the addition of the Leap Second on 31-December-2016 23:59:59 Coordinated Universal Time (UTC). EMC is proactively issuing this EMC Technical Advisory (ETA) to make users aware of the issue. EMC strongly recommends that affected users follow the instructions in the Resolution section PRIOR TO 31-December-2016 to preemptively address this issue.

Note: GeoSynchrony 5.2.x and earlier are not at risk of impact by  the 2016 Leap Second event as they have the Network Time Protocol (NTP) "slew" mode enabled by default. If you did configure NTP for a 5.2.x and earlier VPLEX cluster you will need to manually enable NTP SLEW mode prior to December 30, 2016 and then disable SLEW mode after January 1, 2017. Use the instructions provided in leapsecond2016_manual_v4a.zip available from support.emc.com under the Product/tools page.

Cause

The International Earth Rotation and Reference Systems Service (IERS) has scheduled the addition of a Leap Second for 31-December-2016 at 23:59:59 Coordinated Universal Time (UTC) which may result in time-related issues with the SUSE Kernel running on the VPLEX Management Servers (VS1 and VS2) and MMCS-A (VS6) and directors.

Information about these issues, SUSE Linux Enterprise Server (SLES) releases, and which kernels may be impacted are found in the SUSE KB article 7017873, "General information on Leap Second 2016".

All VPLEX releases listed in the Issue section of this EMC Technical Advisory running the affected SUSE SLES releases may be exposed to this issue.

Resolution

Workaround:
SUSE recommends switching Network Time Protocol (NTP) to "slew" mode at least 24 hours before the Leap Second addition and switch the setting back on or after 1-January-2017.
 
                                                                        !!!!!!! IMPORTANT NOTICE !!!!!!!!
                   FOR USERS WHO MAY HAVE ALREADY DEPLOYED THE EARLIER LEAP SECOND SCRIPT

In the prior Leap Second script a bug was found in the code that would have prevented the script from implementing the NTP setting changes on December 31st and might have left the VPLEX clusters at risk of the leap second event. The issue has been fixed and an updated script, Leapscecond2016_v9.zip, is now available under the Tools page on support.emc.com VPLEX product page.
 
For those users who may have already installed the prior Leap Second script you will need to download and run the updated script, prior to December 30th, to install the corrected portion of the script code that monitors the system date to know when to implement the NTP setting changes on the clusters.
 
Once the Leap Second script is loaded this protection will remain in place on the VPLEX even if you upgrade to a higher code level.

For all VPLEX users:
  • To avoid the impact of the leap second, the design is to enable the common industry workaround of enabling NTP Slew mode, where the time is adjusted incrementally throughout the day so that there is no  Leap Second inserted , but rather accumulated throughout the day.
  • Requirement is all clusters with 5.3 and above. It is recommended to apply to all 6.0+ version as well, in the event that there are undiscovered Kernel Leap Second bugs, and this solution will always avoid them.
  • The new script, once installed, has the following high level design:
                  o  EVERY June 30th, and Dec. 31st, VPLEX will enable SLEW mode for the 24 hours leading up to the
                       potential leap second event, and disable the slew mode starting 12 hours after the event.  
  • Leap seconds will be inserted during these times, and is announced 6 months ahead. This design will require no changes going forward.
  • Additional robustness design in the script is to schedule checks every hour, in the case that the initial check is missed due to a director reboot.  Typically this means SLEW mode will be enabled 24 hours before, and disabled 12 hours after.
                   o  VPLEX Engineering has determined that the least risk to users is to not run with the SLES 11
                        implementation of NTP SLEW enabled all the time, hence the design only enables it for 2 days of the year
                        that leap seconds are possible.
                   o  The new script has three commands:
  • configure: Activate the solution as described, resulting in SLEW being enabled twice a year.  No further changes required. This command can be issued multiple times with no negative outcome, and will always leave the configuration in a valid state.
  • status: Check if Slew mode is active.
  • check-system: Validate the configuration is active.  This command is always run during RPM installation.

For customers running GeoSynchrony 5.3 and above, if you do not wish to load the Leap Second script at this time you can check your NTP configuration and if set you can unconfigure it until you plan to load the script.

The best way to check for the use of external NTP servers is the run the vplexcli command:
 
configuration sync-time-show
 
*Note that Cluster-2 should ALWAYS be configured to sync its time with cluster-1*
 
Output examples:
 
With External NTP:
 
VPlexcli:/> configuration sync-time-show
This cluster is configured to get its time from the following servers: xx.xx.xx.xx
 
Without External NTP:
 
VPlexcli:/> configuration sync-time-show
No NTP server is configured.
 
To Clear External NTP (also a temporary strategy to avoid leap second):
 
VPlexcli:/> configuration sync-time-clear
Shutting down network time protocol daemon (NTPD)..done
Starting network time protocol daemon (NTPD)..done
executing sudo /etc/init.d/ntp restart
VPlexcli:/> configuration sync-time-show
No NTP server is configured.
 
To establish External NTP:
 
VPlexcli:/> configuration sync-time
Running the Sync Time task
 
Please enter the IP address of the NTP server in order to synchronize the system clock. IP Address of NTP server:xx.xx.xx.xx
 
Syncing time on the management server of a Metro/Geo system could cause the VPN to require a restart. Please Confirm (Yes: continue, No: exit) (Yes/No) Yes
 
Shutting down network time protocol daemon (NTPD)..done
PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data.
64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=246 time=23.2 ms
 
--- 10.xx.xx.xx ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 23.215/23.215/23.215/0.000 ms
ntpd: time slew -0.000250s
Starting network time protocol daemon (NTPD)..done
executing sudo /etc/init.d/ntp stop
executing /bin/ping xx.xx.xx.xx -c 1
executing sudo /usr/sbin/ntpd -q -g
executing sudo /sbin/hwclock --systohc -u
executing sudo /etc/init.d/ntp start
Synchronize Time task completed.
 
VPlexcli:/> configuration sync-time-show
This cluster is configured to get its time from the following servers: xx.xx.xx.xx
 
 
Further details can be found in the Command Reference (CLI) guide for the appropriate GeoSynchrony version.

EMC has the following script, Leapsecond2016_v9.zip, to address the leap second issue. The script is to be run on all VPLEX Management Server (VS1 and VS2) and MMCS-A (VS6). The script is available for download from EMC Online Support https://support.emc.com/products/29264_VPLEX-VS2/Tools/.

The README file, included with the script .zip file, refers the users to the "quick_install_general_v9.txt" instructions for deploying the script on the VPLEX system. Within the instructions, there is a check that may be run to confirm changes are completed successfully by the script. This script must be run before 31-December-2016.

If, after running the script and checking the status of the changes, it does not appear that changes were made, contact EMC Customer Support and mention this article.

NOTE: Before running the script ensure that ALL directors are online.

Run the health-check cli command then in the output look for "Director OS Uptime", this will show
if all directors in the VPLEX setup are running or not.

Example output:
VPLEX-Local:
Director OS Uptime:
-------------------
Director        OS Uptime
--------------  --------------------------
director-1-1-A   17:51pm  up 52 days  0:46
director-1-1-B   18:14pm  up 101 days 11:24
director-1-2-A   18:11pm  up 137 days 22:11
director-1-2-B   18:11pm  up 137 days 22:11


VPLEX-Metro/Geo:
Director OS Uptime:
-------------------
Director        OS Uptime
--------------  -------------------------
director-1-1-A  17:47pm  up 33 days 22:46
director-1-1-B  17:47pm  up 55 days  3:19
director-2-1-A  Director unreachable 
<<< not reachable, need to investigate
director-2-1-B  18:14pm  up 55 days  3:24

Note:
Once the script available in the Leapsecond2016_v9.zip file is loaded, the VPLEX is protected against the current 2016 Leap Second event and any future Leap Second events. Should a user later upgrade the VPLEX to a higher level of code from the one they ran the Leap Second script on, the changes that were made with the prior running of the script are not impacted by a new non-disruptive upgrade (NDU). The fix for the Leap Second issue remains in place.

After the Leap Second script is run, if the a Management Server or the MMCS-A or a director requires a Field Replacement Unit (FRU) before 31-December-2016, the script must be run again.

Article Properties


Affected Product

VPLEX Series

Product

VPLEX Series, VPLEX VS1, VPLEX VS2, VPLEX VS6

Last Published Date

20 Nov 2020

Version

2

Article Type

Solution