Dell Unity: How To Gather The Necessary Information For A Unity Performance Analysis

Issue

• Poor Read and/or Write performance or high response times for connected hosts.
• Response times for host applications have increased or are unacceptably high.
• Performance problems on a Dell Unity array, which requires performance analysis.

Performance issues can have a number of causes including network configuration, congestion, host configuration, hardware faults, disk contention (several busy LUNs sharing the same drives), or the storage array being at its maximum utilization.

Unity Technical Support will not be able to fully investigate and troubleshoot Performance issues until the following have been provided:

• Service Data File (Data Collect)
• UPA files (Unity Performance Archives)
• A detailed description of the symptoms experienced in the customer environment

Failure to promptly provide the information in this LKB can result in longer resolution times.

The details of the performance issues (symptoms, time stamps, etc.) are crucial so that the engineer knows where to focus (times/LUNs/hosts/etc.), rather than spending several hours just to find the issue (not the cause). Without this detail, hours may be spent searching for an issue and we may only be able to possibly only analyze the general performance statistics.
 
Scope
Customer Support Services investigate performance cases to identify issues that are affecting customer sites. Performance tuning (to achieve peak performance for a host or application) and reconfiguration is a task for a Storage Administrator or a Solutions Architect, so this will not be handled in a Break/Fix Service Request.

Service Requests that do not involve an actual problem, such as requesting a report of the current level of workload on an array, are outside the charter of Support. For self-help or those needing help to determine the level of workload on an array can use the LiveOptics service. Please see the following article for detailed information about this option:

• 000010770:  Dell EMC Unity/VNX KB: How to use LiveOptics for a general health-check, storage assessment, and/or host/environment sizing (User Correctable)  https://www.dell.com/support/kbdoc/000010770

 
Data And Information Requirements For A Detailed Performance Analysis
For each event, Technical Support needs the following logs and information which cover the incident time frame:

• Service Data Files (also known as Data Collects), generated shortly after the issue occurred.
  • If Replication is in use, then Data Collects and UPA files from both arrays will be needed.
• Unity UPA files (Unity Performance Archives), which cover the whole period when the problem was occurring.
• Host Grabs and Switch logs may also be required if the problem appears to be outside of the array.
• Unity Network Traces may also be required.

• Provide a clear and detailed problem description:
• When did/does the problem happen? (Date / Time & Time Zone)
• Which LUNs are affected?
  • This is very important. We need to narrow down the search.
• Is Data Reduction or Advanced Deduplication enabled on the LUNs experiencing issues? Is Data Reduction enabled for other LUNs? (Please see the Unity Best Practices White Paper - CPU Utilization section - page 9)
  • https://support.emc.com/docu69891_Best-Practices-Guide---Best-Practices-for-Performance-and-Availability.pdf?language=en_US
• Is Replication in use? Please provide information about the RPO for Replication and if the performance issue improves if Replication is paused.
• Is there or was there at any time active Data Unavailable?
• Is anything else running at the specified time? (Backup, Jobs, Batches, Virus Scans, Host Disk Optimization (TRIM/UNMAP), etc.)
• If something is not working as it should, did it ever work? Or is this part of a new implementation/setup? Were there any new applications or hosts added recently?
• How is performance being measured or what is the impact? Delays or errors on the host side? Are end users reporting slowness?
• Any recent changes in the SAN environment (hosts, switches, array)?
• Impact (Affected applications, # of affected servers, # of affected users, etc.)
• How often does the problem happen? (Constantly / Regularly / One-Time / Randomly)
• What are the names of the affected hosts?
• What topology is used? FC, iSCSI or Ethernet? (If using iSCSI all hosts MUST have TCP Delayed ACK disabled - see KB 000079979 and search for TCP Delayed ACK.)
  • 000079979: Dell EMC Unity/VNX/CLARiiON: Facts, limitations, and recommended settings when using iSCSI https://www.dell.com/support/kbdoc/000079979
• If the issue may be around connectivity, we will likely need a topology diagram.
• Does a known problem exist in the network at the same time or has the network been upgraded recently? Also check that iSCSI is not used on the internal LAN and any switches used must be Enterprise class switches.

 

• Provide a clear and detailed problem description:
• What are the specific protocols involved?
  • Example: NFS (version), CIFS (version), NDMP, iSCSI, FTP, etc.
• Is the issue related to a specific export/share, filesystem/LUN?
• What time of day does the issue occur? (Date / Time & Time Zone)
• What is the active workload at the time?
  • Example: login, logout, backup, batch jobs, etc.
• What hosts are involved?
  • Example: Backup, VMs, workstations, servers, etc.? 
  • Get as much detail on the application(s) as possible.
• What does the Network configuration look like?
  • Provide specific interface/NIC detail.
• Are the network interfaces isolated on separate NICs?
  • Example: replication traffic on it's own/single NIC

Collecting Service Data File (Data Collect)

To gather Service Data File (Data Collect), see LKB 000023676. The output files from running Service Data file on the array will be necessary to show the detailed array configuration and any events which occurred on the array which may have had a performance impact. We recommend that you gather the Service Data File (Data Collect) as soon as possible after the event occurs.

• 000023676: Dell Unity: How to Gather Service Data (Logs) from a Unity Array (User Correctable) https://www.dell.com/support/kbdoc/000079979

Keep in mind that the Service Data file will only contain the last complete UPA file (one hour) and any available -tmp.archive UPA file. Often, this is not enough information for a proper performance review and additional UPAs will be required.

To find the UPA file within the Service Data File (Data Collect), go to spx\cmd_outputs\metrics\. 

• spx = Primary SP

• Example: If SPA is the active SP, the UPA will be under spa\cmd_outputs\metrics

 
Collecting UPA Files

• Support may ask for additional UPA files (versus the UPA available in the Service Data File (Data Collect)).
• Unlike the Service Data File (Data Collect), the array will store at up to 48 hours of UPA files. 
• The UPA files will look like the below example. Each file name contains the date and time of the first data point in the file. The interval will be 10 seconds and cover one hour minus 10 seconds. The date and times in the filenames are UTC/GMT.
  • Note that the last UPA file in the example below is the most current archive (-tmp.archive). This contains the current performance data and can be copied from the array (via SSH/CLI only) to look at the most current data. We can use this file to see performance events without needing to wait until the next hour.

Example of UPA files stored on the Dell Unity array:
01/05/2017  10:59 AM        11,017,216 _default_20170105_150000.archive
01/05/2017  11:59 AM        11,017,216 _default_20170105_160000.archive
01/05/2017  01:00 PM        11,017,216 _default_20170105_170000.archive
01/05/2017  01:59 PM        10,983,424 _default_20170105_180000.archive
01/05/2017  02:45 PM         8,308,736 _default_20170105_190000-tmp.archive
   

 
Host Data Collection
Host Grab (or EMC Report) output files may be needed, especially if the issue is unique to specific hosts.

• Go to https://www.dell.com/support/home/ (Dell Support Site) and search for the Grab Utility for the correct host OS (Example: ESXi host grab)
• Download the utility and the README file.
• Follow the README file to run the utility for the host operating system chosen.

 
Switch Logs
If the performance issue only exists on certain paths, switch logs may be required. Please see the following LKB for instructions (based on the model being used):
 

Brocade/Connectrix

• 000028508: Connectrix Switches: How to collect Switch Logs (Primary) https://www.dell.com/support/kbdoc/000028508
• 000018926: Connectrix: How to collect a switch log file with a PuTTY session, step by step https://www.dell.com/support/kbdoc/000018926

• 000191330: How Do I collect Tech-Support Logs From A MDS Switch? https://www.dell.com/support/kbdoc/000191300
• 000010120: Connectrix: How to collect logs from a Cisco MDS 9000 Family Switch using the CLI https://www.dell.com/support/kbdoc/00010120

Network Trace Captures (TCP Dump)
Some File Performances issues may be related to network issues. In order to fully determine the cause of a problem, support may request network traces. Please see the following LKB for instructions:

• 000022522: Dell EMC Unity: How to gather the necessary information for a Unity performance analysis https://www.dell.com/support/kbdoc/000022522 

Please refer to this video: