Skip to main content
  • Place orders quickly and easily
  • View orders and track your shipping status
  • Enjoy members-only rewards and discounts
  • Create and access a list of your products
  • Manage your Dell EMC sites, products, and product-level contacts using Company Administration.

Avamar: How to gather information for troubleshooting replication issues

Summary: This article discusses what information should be gathered in order to understand and facilitate troubleshooting of Avamar replication issues.

This article may have been automatically translated. If you have any feedback regarding its quality, please let us know using the form at the bottom of this page.

Article Content


Symptoms

The purpose of this article is to provide a template of information to gather when investigating an Avamar replication failure. This makes facilitates troubleshooting and reduces the time for Support to understand the problem. 

Examples of errors which might be seen during a replication problem are listed below.
  • Dial home error: repl_cron - failed replication
  • Could not login to destination DPN
  • Exit code 20: Insufficient privileges
  • MSG_ERR_PSWD_INVALID
  • Error <5445>: INTERNAL CYCLE ERROR: aborting repldata
  • ERROR! Exit code 18: Could not locate requested information (Missing data? Invalid parameters?)
  • ERROR: Source and Destination servers are identical.
  • Error <5698>: Source (hostname-x) and destination
  • FATAL ERROR: <0001> Fatal signal 11
  • FATAL error: "Failed to record backup 
  • Connection killed
  • Error <0000>: Login error 74: Account not found in the Avamar database
  • (hostname-y) DPNs are the same!  Replication ABORTING.
  • CYCLE ERROR: bad response in repldata DST_ADD_DATA: SETSTATE
  • Network error occurred: Rebuilding connection 0

Cause

Not Applicable

Resolution

Note the following terminology:
Replication Source - An Avamar which *sends* data to a replication target.
Replication Target - An Avamar which *receives* data from a replication source.
Replication Pair - The collective name for two Avamars which engage in replication.
Cross replication - Where two Avamars replicate to one another. Both send, and both receive, replication data.

An Avamar can be both a replication source and a target device if it cross-replicates data with other Avamar servers.

If there are multiple issues with an Avamar replication environment, break the problem into smaller parts. Work with one replication pair at a time. 
 

Information to gather and items to check:

  1. The full hostname of the replication source.
  2. The full hostname of the replication target.
  3. Does the replication pair cross-replicate?
    a) If yes, are the replication failures occurring in one direction or both? 
  4. Has the failed replication job ever worked before? If yes, gather the log for the last successful job.
  5. On both Avamars, run the proactive check script (This article is only accessible when logged in to Dell Support Avamar: How to Run the proactive_check.pl health check script on an Avamar Server).
    a) Gather the "hc_results" file.
    b) Rerun the script with the --sched flag. Gather the output.
    b) Rerun the script with the --capacity flag. Gather the output.
  6. What is the expected bandwidth between source and target Avamar?
  7. From the source, ping the target and gather the output.
  8. Run status.dpn on each Avamar and gather the output.
  9. Gather the replication log for the jobs which failed. See the 'Notes' section of this article for details.

Additional Information

Gathering the Replication Activity logs from the Avamar HTML user interface (AUI).

The Activity Monitor in the Avamar Web User Interface enables provides access to a detailed session log. This enables analysis and troubleshooting.

Steps:
1. In the AUI navigation pane on the left, click >>, and then click Activity.
The Activity Monitor appears and displays a list of all activities.

2. Select an activity from the list, and then click VIEW LOGS.
The Log details window appears. By default, the Activity Monitor displays a detailed log of all client backup activity for the past 72 hours.
example screenshot
3. To filter the content based on a search string, in the search field, type the string.

4. To download the log file, click Download.

Avamar replication log



How to gather replication logs:

From the Administrator UI Activity monitor, select Actions -> Filter -> Filter by Type "All Replication Source & Destination".

Screenshot showing replication activity filtering

By default, the activity monitor displays only replication jobs which occurred within 72 hours or since the MCS service was restarted (if less than 72 hrs).

Screenshot showing list of Avamar replication jobs

Select the job to troubleshoot and double-click it to retrieve the log.

Activity drill down

The activity session drill-down screen appears with either the requested log, or a list of log files in a hyperlinked file.

Individual logs can be selected. Also there is an option to export logging to an html file on the Avamar Administrator desktop.

Activity drill down summary - save session log

Alternatively, download logs from the Avamar source utility node over SCP.  
Logs are in: /usr/local/avamar/var/client/
-rw-r----- 1 root  root  9.4K Feb 27 07:03 Default_Replication_Schedule-new-1456556402264#0-1008-Replicate-avtar.log
-rw-r----- 1 root  root   11K Feb 27 07:03 progress-Default_Replication_Schedule-new-1456556402264-1001-Unix_Progress.log
-rw-r----- 1 root  root  5.8K Feb 27 07:03 Default_Replication_Schedule-new-1456556402264-1008-Replicate.log
-rw-r----- 1 root  root  8.2K Feb 27 07:03 Default_Replication_Schedule-new-1456556402264-1008-Replicate.alg 

Article Properties


Affected Product

Avamar

Product

Avamar

Last Published Date

24 Jan 2024

Version

5

Article Type

Solution