Symptoms
Upgrading to, or already running OE version 05.33.009.5.217, and using
D@RE encryption.
There is an issue with
D@RE encrypted VNX2 systems, either during an upgrade to OE version 05.33.009.5.217 (aka R33.217), or when running at R33.217, that may result in one or more disk drives being marked offline and not able to fail-over to the alternate SP during an SP reboot. The result of an incomplete drive fail-over may cause a Storage Pool to go offline, with a loss of access to LUNs/File Systems. Data loss and corruption may also occur.
Example Log Errors:
Pool 2 ( File_Pool) is went offline but not marked for recovery. The recovery operation for Pool 2 could not proceed because its associated FLUs object is in error state with status 0xe12d8707
06:10:11.760 11896 FFFFFA814F9B7040 2 std:INFO SERV ELOG 100C0 : 0xe1688008 Uncorrectable Sector RAID Group: 0x1a8 Position: 0x1 LBA: 0x1b1f2998 Blocks: 0x56 Error in
06:10:11.760 15 FFFFFA814F9B7040 2 std:INFO SERV ELOG 100C0 : 0xe1688001 Data Sector Invalidated RAID Group: 0x1a8 Position: 0x1 LBA: 0x1b1f2998 Blocks: 0x56 Error
06:10:11.760 8 FFFFFA814F9B7040 2 std:ERR LIB RAID 1A8 : unexpected multi-bit crc w/lba st error pos: 2 lba: 0x1b1f2998 bl: 0x31
Cause
At version R33.217, and under certain conditions, such as when an SP is rebooting, if a drive path experiences a Single Loop Failure (SLP), its encryption key may become de-registered, and it is then unable to be brought online from the alternate SP. I/O that is being driven to this drive will result in multibit CRC errors, and invalidated data, resulting in the Storage Pool and LUN/File System resources to go offline.
Resolution
This issue has been fully resolved with VNX OE Version R33.218. However, if running R33.217 with
D@RE encryption, the array is still vulnerable until the system has been successfully upgraded to R33.218.
- If you are running VNX2 with D@RE encryption enabled, and are NOT running R33.217, you may safely upgrade to R33.218.
- If you are running VNX2 OE version 05.33.009.5.217, with D@RE encryption enabled, you should upgrade to R33.218 as soon as possible. However, we recommend that you adhere to the following guidelines before upgrading your system to R33.218:
Perform the following steps prior to upgrading from R33.217 to R33.218 if running D@RE encryption:
1. Ensure that there are current data Backups of your system
2. Gather a Data Collect log bundle and store in a safe location
3. Backup the Encryption Keys from the array and store in a safe location
4. Perform a Health Check on the system
5. Minimize I/O to the array before performing the NDU upgrade
6. Upgrade to R33.218
7. Gather a Data Collect log bundle and store in a safe location
8. After the upgrade to R33.218, the system is no longer exposed to the issue outlined by this KB article
Please contact your Service Provider if you require guidance and/or assistance for the above steps.