Symptoms

Metadata Symptoms: If the Metadata Capacity limit has been reached or exceeded, the Avamar backup scheduler can be suspended and prevent running further scheduled backup jobs. Any of the following items indicate a high or full Metadata Capacity: The Avamar Administrator UI dashboard may show a Metadata Capacity value equal to or greater than 100%. The Management Console Server (MCS) command-line output shows the Metadata Capacity value equal to or greater than 100%: Example command and output: mcserver.sh --status | grep -i metadataUtilization metadataUtilization (larger of (nstripes/totalStripesPermitted) or (stripeReserved/(nodeSizeAfterDiskReadOnly * stripeUtilizationCapacityFactor)) * 100): 100.3 In the Avamar Administrator UI, the scheduler and dispatcher are in a "suspend" state. In the mcserver log files, a message is seen similar to "FINE: Suspending dispatcher." From a command-line MCS status query, the scheduler is suspended: mcserver.sh --status | grep suspend Backup dispatching: suspended Introduction: In an Avamar environment with Data Domain integration, backups can be stored on the Data Doman. Although the backup data is sent to, and stored on Data Domain, Avamar keeps records of the backups' metadata.(Metadata, in this environment, is the information about the backup files, file attributes, directories, and so forth from the client operating system). See Metadata Capacity Reporting and Monitoring Technical Note for more information. History: Avamar v6.x with Data Domain integration allowed database backups (and a few others) to backup to Data Domain. As databases only had a few files, there were only a small amount of metadata for Avamar to track.Filesystem backups to Data Domain were not supported in Avamar v6. Avamar v7.x with Data Domain integration introduced the new type of capacity called "Metadata Capacity". This was necessary due to the newly supported Filesystem backups to Data Domain. With filesystem backups, many more different files are backed up, each of whose attributes and Metadata must be tracked by Avamar. As metadata is much smaller in size than backup file contents, it can build up and cause other types of resource limits to be reached. Metadata Capacity was added to protect the server from this new type of usage exceeding restrictions and limits.

Cause

There are generally one of three situations when Metadata Capacity becomes high or is a concern: 1. An Avamar grid (v7+) with Data Domain integration is impacted by full Metadata Capacity anytime that the capacity fills up. This situation can be caused from intentional or accidental backups to Avamar instead of Data DomainThis situation may also have been caused by a previously high Avamar GSAN capacity that occurred prior to integration of Data Domain. 2. An Avamar grid (v7+) that does NOT have a Data Domain Integration. High Metadata Capacity is falsely reported likely due to a hardware error, or misconfiguration. 3. The pre-checks for an Avamar grid (v6.x to v7.x) calculate and verify what the Metadata Capacity would be post upgrade. This process ensures that if an upgrade proceeds, that it will not put the newly upgraded grid into a full Metadata Capacity state.

Resolution

Note: If upgrading an Avamar v6.x to Avamar v7 grid (where Metadata Capacity is introduced and enforced) contact Support and submit a new upgrade service request. The Remote Proactive Support ("RemPro") team reviews the metadata capacity as part of the pre-upgrade procedure. 1. Log in to the Avamar Utility Node as admin. 2. Run the following command to verify the high or full metadata capacity: mcserver.sh --status | grep -i metadataUtilization Examples: metadataUtilization (larger of (nstripes/totalStripesPermitted) or (stripeReserved/(nodeSizeAfterDiskReadOnly * stripeUtilizationCapacityFactor)) * 100): 95.95 metadataUtilization (larger of (nstripes/totalStripesPermitted) or (stripeReserved/(nodeSizeAfterDiskReadOnly * stripeUtilizationCapacityFactor)) * 100): 100.3 If the utilization is not high, stop using this article. 3. Review the following article: Avamar: How to gather the information required to troubleshoot capacity issues. 4. Review the following article to verify that there is no other type of Avamar Capacity issues present: Avamar: Capacity Troubleshooting, Issues, and Questions - All Capacity (Resolution Path) 5. Verify that there is a Data Domain attached by running the following command: ddrmaint read-ddr-info Examples: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <avamar dpnid="1234567890" type="default" version="5"> <datadomain count="1"> <ddrconfig client-map-default="true" cloud_enabled="true" cloud_unit_name="ECS_Unit1" dd-cert-chain=" -----BEGIN CERTIFICATE----- ..... -----END CERTIFICATE-----" ddos-version="7.7.5.25-1078970" ddrcreatetime="1685672811" ddrid="ABCDEF0C18C7E2D75F20E0A1596E66AA2EE764E9" gsan-backup-target-default="true" hostname="dd.compamy.com" index="1" instant-access-limit="32" ipv4-hostname="dd.compamy.com" ipv6-hostname="" max-streams="50" max-streams-for-cp-backup="2" modelno="DD VE" mtree_name="/data/col1/avamar-1643691765" password="Onlh2XP9xEpvI2exqGQwqA==" policy_id="%2Fdata%2Fcol1%2Favamar-1234567890%3AECS_Unit1" serialno="AUDABCDEFGHIJK" token="AQBpud2RFZVF81Vv6NHMK/Y3jecFGhxv0KUif8BYoNk7iQ==" username="<ddboostuser"> <snmp community="noSejOLJ1lRsg1kEHjWdiA==" version="2"> <ports getter-setter="161" trap="163"> </ports> </snmp> <client-map> </client-map> <auth-type>credential</auth-type> </ddrconfig> </datadomain> </avamar> <4774>read-ddr-info: MSG_ERR_NOT_PRESENT: ddr_info does not exist. If there is no Data Domain attached, this is likely a false positive caused by hardware errors or a misconfiguration. Contact Support for assistance. If there is a Data Domain attached, and the Metadata Capacity limits have truly been reached or is very high, Contact Support for assistance.Where possible, include all the information collected and details to help expedite the time to resolution. In some instances, the Metadata Capacity value can be lowered by tuning parameters. Other situations may require new limitations or hardware changes. The Avamar Support team member will discuss any required actions.

Support Cases