Symptoms

An Avamar client requires memory, CPU, network, and I/O resources to perform a backup. Other operating system services share these resources, including processes which may be more 'mission critical' than Avamar. This article discusses throttling and prioritization of CPU, network, and I/O resources consumed by the avtar process during a backup.Avamar avtar process uses as much network bandwidth, CPU cycles, and I/O as are made available to it by the operating system. Avamar can be set with parameters which determine how aggressive it is when laying claim to some of these system resources.CPU UtilizationAn Avamar client machine may be a filer server, a mail server, a database or have some other role that requires system resources.If a backup runs while those system resources are under heavy load, both the backup and the client's regular role may be impacted. If it is not possible to schedule the backup to run during a quiet period, you may want to throttle CPU usage by avtar to ensure that sufficient CPU cycles remain available for client's primary function.Avtar CPU utilization can be reviewed in the Avamar client backup log as shown below: avtar Info <5100>: Status 2009-04-10 20:37:02, 0 files, 25.00 GB (0 files, 473.7 MB, 1.85% new) 117 MB 99% CPU The CPU percentage value may be seen to reach higher than 100% in environments where multiple CPU cores are available.CPU Process PrioritizationLinux Behavior: The avagent process automatically sets all processes that it spawns to be at a "nice" level.In Linux, priority levels range from -20 to 20, where -20 is the highest priority.By default, processes are spawned with priority 0.Avtar is created with priority 10, which is lower than the default priority. Windows Behavior: Windows manages priority levels ranging from 1 (lowest) to 31 (highest).Avtar is created with BELOW_NORMAL_PRIORITY_CLASS, which means that threads owned by avtar by default are created with priority 6.For reference, threads created by processes on Windows have a default priority of 8. Network UtilizationRelative to traditional backup software, Avamar consumes little bandwidth. However, if an Avamar client is separated from an Avamar server by a low-bandwidth link it may be helpful to throttle network usage. This can prevent backup data from swamping a line and slowing down transmission of more time critical data.I/O PrioritizationThe I/O priority of Avamar client processes is not explicitly set.Linux Behavior: I/O niceness ranges from 0 (highest priority) to 7 (lowest priority).The algorithm that is used is (cpu niceness + 20)/5 (source: http://linux.die.net/man/1/ionice)For avtar, I/O priority would be (10 + 20)/5 = 6, which is the second lowest I/O Priority.In comparison, a process s I/O priority with a default thread priority of 0 would be 4. Windows Behavior: For Windows, I/O prioritization is less clear. Microsoft claims that a process' I/O priority is tied to the thread priority. Avamar Engineering investigation for issue #240762 demonstrated that, in practice, this does not appear to be the case. Setting avtar to run with a lower priority does not change the I/O priority from the default.Memory utilizationThe amount of memory that is required during a backup depends the cache file sizing and the type of cache files in use. Traditional 'monolithic' cache files load the entire cache files into memory whereas the paging cache files 'page' data into memory as needed.Memory utilization and cache file sizing are beyond the scope of this KB article. See the Operational Best Practices Guide for more information.

Cause

Avtar's usage of CPU cycles, network bandwidth, and I/O is not restricted by default. This is normal and expected behavior as it is most desirable that Avamar backups run and complete as quickly as possible.

Resolution

The following flags exist to throttle network and CPU usage of the avtar process: --net-throttle (throttles network usage)--cpu-throttle (throttles CPU usage) To implement either of these flags, edit the 'avtar.cmd' file found in the Avamar client's avs/var directory. Note: Create a new 'avtar.cmd' file if a file does not exist.CPU ThrottlingThe flag should be in the following format, where '80' is the maximum average percentage of CPU that avtar should use. --cpu-throttle=80 This flag was introduced before multiple core CPUs were widely used. The percentage to use may require experimentation. A backup that is carried out on a dual core desktop machine with avtar throttled to 10% appeared to use around 5% of the CPU resource.Network ThrottlingThe entry that is made in the file should be in the following format: --throttle=Mbit/sec When implemented, avtar pauses if necessary after sending each packet. This ensures that network usage does not exceed the specified maximum bandwidth (specified in mega bits per second). For example, --throttle=5 uses half of a 10Mbit/sec connection, --throttle=0.772 restricts usage to one-half of a T1 link. After you have made the change, save the avtar.cmd file.The changes should take effect the next time avagent spawns a new avtar process.Alternatively, using the Options tab in Avamar Administrator, select the appropriate file system plug-in from the drop-down list (Windows File System, Linux File System, so forth). Check the "Show Advanced Options" checkbox and scroll down until to find "Network rate throttle (Mbps)".Fill this in with the required bandwidth in Mbps (decimals are fine, for example 1.2).Avamar plug-in backupsFor plug-in backups (Exchange, Oracle, so forth), in the Avamar Administrator user interface, select the plug-in from the menu drop-down, click the "More" button and enter the attribute and value by hand.For the attribute, enter [avtar]net-throttle. For the attribute value, enter the required bandwidth. When happy with the setting, click the + sign under the fields to add the flag to the box below.Click OK to save the changes.The [avtar] in the flag tells the plug-in to pass the flag through to avtar which is the process that communicates with the server.

Support Cases