Avamar - How to set the size of Avamar client caches to prevent cache overflows from causing backup performance problems

Check if the client cache needs tuning.  

• Avamar - How to check the Avamar client log for file cache or hash cache sizing issues

Do not change the cache file if it is appropriately sized - there will be no beneficial effect on performance.

Rules for tuning the maximum cache sizes

Ensure that the caches do become so large that the client ends up swapping because it has insufficient physical RAM to handle all the processes. Swapping is the movement of memory pages between RAM and disk.

Best practices for tuning the maximum cache sizes:

• Never allow the total combined cache sizes to exceed one-fourth of the total available physical RAM.
• Set the maximum file and hash cache sizes to a fraction of the total available physical RAM. Specify the file and hash cache sizes by using negative integers.
• Limit the total cache sizes to approximately one-fourth of the physical RAM.
• Set one of the caches to be -5 (this is equivalent to 20% of the client's RAM), and set the other cache to be -32 (3% of the client's RAM).
• For example, for a large database client use the following settings:
  --filecachemax=-32
  --hashcachemax=-5
• If using something other than the default cache sizes, include the customized maximum cache settings in the avtar.cmd file on the client.
• Sometimes the only choice may be to increase the amount of physical RAM on the client. You might also be able to back up the client by using multiple smaller datasets.
• If necessary, limit the sizes of the caches below the optimum values:
  • For a typical file server, first allocate the required RAM to the file cache.
  • For a typical database client, first allocate the required RAM to the hash cache.

Tuning the file cache

If the file cache is deleted, unused, or undersized, every file that is not a hit in the file cache must be processed. Processing involves reading, chunking, compressing and hashing the file contents.
This occurs before the Avamar process can check that the hashes were previously sent to the Avamar server.
If a file exists in the file cache, it does not need to be read by avtar.
This saves significant time and processing and minimises backup duration.

By default, the file cache could consume up to one-eighth of the physical RAM on the Avamar client.
For example, if the client has 4 GB of RAM, the file cache is limited to 4 GB divided by 8, or 512 MB maximum.

The file cache doubles in size each time it needs to increase.
The file cache sizes can be (in Megabytes): 5.5 MB, 11 MB, 22 MB, 44 MB, 88 MB, 176 MB, 352 MB, 704 MB, and 1,408 MB.

Because the avtar program is a 32-bit application, the maximum file cache size that avtar can use is limited to less than 2 GB.
In an example where a client has 4 GB of RAM, the maximum size of the file cache is 352 MB.

Each entry in a file cache consists of a 4-byte header plus two 20-byte SHA-1 hashes (44 bytes total):

• SHA-1 hash entry of the file attributes.
  The file attributes include: File name, file path, modification time, file size, owner, group, and permissions.
• SHA-1 hash entry for the hash of the actual file content, independent of the file attributes.

File cache rule

If the client consists of N million files, the file cache must be at least N million files x 44 million bytes/million files. This means that the file cache must be at least N x 44 MB, where N is the number of millions of files in the backup.

Example - File cache

When a client has 4 million files, the file cache must be at least 176 MB (4 x 44 MB). The file cache must be allowed to increase to 176 MB to accommodate all the files.

Best practice:

The file cache must be a minimum of N x 44 MB, where N is the number of millions of files in the backup. The file cache doubles in size each time it grows. To adequately size the file cache:

1. Set the --filecachemax value as follows:
    

   --filecachemax = 2 x N x 44

   
   Where N is the number of millions of files in the backup.
2. Set the --hashcachemax to a small value, such as:

   --hashcachemax=30

Tuning the hash cache

If the avtar process finds that a hash of a chunk is not contained in the hash cache, it queries the Avamar server for the presence of the hash.
By default, the hash cache could consume up to one-sixteenth of the physical RAM on the Avamar client.
Using the same client with 4 GB of RAM described in "Tuning the file cache", the hash cache is limited to 4 GB/16, or 256 MB maximum.

The hash cache also doubles in size each time it needs to increase.
The current hash cache sizes are in megabytes: 24 MB, 48 MB, 96 MB, 192 MB, 384 MB, 768 MB, and so forth. In this example where a client has 4 GB of RAM, the maximum size of the hash cache is 192 MB.

Each entry in a hash cache consists of a 4-byte header plus one SHA-1 hash per chunk or composite, which is the hash of the contents of the chunk or composite.

Hash cache rule

If the client consists of Y GB of database data, the hash cache must be at least YGB/average chunk size x 24 million bytes/million chunks.
Use 24 KB as the average chunk size for all backups.
The hash cache must be at least Y MB, where Y is the number of GB of database data in the backup.

Example - Hash cache

When a database client has 500 GB of database data, the hash cache must be allowed to increase to at least 500 MB.
The hash cache must be allowed to increase to the next incremental size (768 MB) to accommodate the hashes for all the chunks in a database backup.

Best practice

The hash cache must be a minimum of Y MB, where Y is the size of the database being backed up in GB.

The hash cache doubles in size each time it grows.
To adequately size the hash cache, set the --hashcachemax value as follows:

--hashcachemax = 2 x Y

Where Y is the size of the database to be backed up in GB.