The purpose of this article is to provide a template of information to gather when investigating an Avamar replication failure. This makes facilitates troubleshooting and reduces the time for Support to understand the problem.
Examples of errors which might be seen during a replication problem are listed below.
-
Dial home error: repl_cron - failed replication
-
Could not login to destination DPN
-
Exit code 20: Insufficient privileges
-
MSG_ERR_PSWD_INVALID
-
Error <5445>: INTERNAL CYCLE ERROR: aborting repldata
-
ERROR! Exit code 18: Could not locate requested information (Missing data? Invalid parameters?)
-
ERROR: Source and Destination servers are identical.
-
Error <5698>: Source (hostname-x) and destination
-
FATAL ERROR: <0001> Fatal signal 11
-
FATAL error: "Failed to record backup
-
Connection killed
-
Error <0000>: Login error 74: Account not found in the Avamar database
-
(hostname-y) DPNs are the same! Replication ABORTING.
-
CYCLE ERROR: bad response in repldata DST_ADD_DATA: SETSTATE
-
Network error occurred: Rebuilding connection 0
Note the following terminology:
Replication Source - An Avamar which *sends* data to a replication target.
Replication Target - An Avamar which *receives* data from a replication source.
Replication Pair - The collective name for two Avamars which engage in replication.
Cross replication - Where two Avamars replicate to one another. Both send, and both receive, replication data.
An Avamar can be both a replication source and a target device if it cross-replicates data with other Avamar servers.
If there are multiple issues with an Avamar replication environment, break the problem into smaller parts. Work with one replication pair at a time.
Information to gather and items to check:
-
The full hostname of the replication source.
-
The full hostname of the replication target.
-
Does the replication pair cross-replicate?
a) If yes, are the replication failures occurring in one direction or both?
-
Has the failed replication job ever worked before? If yes, gather the log for the last successful job.
-
-
What is the expected bandwidth between source and target Avamar?
- From the source, ping the target and gather the output.
- Run status.dpn on each Avamar and gather the output.
- Gather the replication log for the jobs which failed. See the 'Notes' section of this article for details.
Gathering the Replication Activity logs from the Avamar HTML user interface (AUI).
The
Activity Monitor in the Avamar Web User Interface enables provides access to a detailed session log. This enables analysis and troubleshooting.
Steps:
1. In the AUI navigation pane on the left, click
>>, and then click
Activity.
The Activity Monitor appears and displays a list of all activities.
2. Select an activity from the list, and then click
VIEW LOGS.
The
Log details window appears. By default, the Activity Monitor displays a detailed log of all client backup activity for the past 72 hours.
3. To filter the content based on a search string, in the search field, type the string.
4. To download the log file, click
Download.
How to gather replication logs:
From the Administrator UI Activity monitor, select Actions -> Filter -> Filter by Type "All Replication Source & Destination".
By default, the activity monitor displays only replication jobs which occurred within 72 hours or since the MCS service was restarted (if less than 72 hrs).
Select the job to troubleshoot and double-click it to retrieve the log.
The activity session drill-down screen appears with either the requested log, or a list of log files in a hyperlinked file.
Individual logs can be selected. Also there is an option to export logging to an html file on the Avamar Administrator desktop.
Alternatively, download logs from the Avamar source utility node over SCP.
Logs are in: /usr/local/avamar/var/client/
-rw-r----- 1 root root 9.4K Feb 27 07:03 Default_Replication_Schedule-new-1456556402264#0-1008-Replicate-avtar.log
-rw-r----- 1 root root 11K Feb 27 07:03 progress-Default_Replication_Schedule-new-1456556402264-1001-Unix_Progress.log
-rw-r----- 1 root root 5.8K Feb 27 07:03 Default_Replication_Schedule-new-1456556402264-1008-Replicate.log
-rw-r----- 1 root root 8.2K Feb 27 07:03 Default_Replication_Schedule-new-1456556402264-1008-Replicate.alg