Symptoms

Data Domain Virtual Edition(DDVE) is a software-only storage Appliance, which can be either on-premises or cloud deployed. Active Tier on Object Storage (ATOS) is another name for cloud deployed DDVE.On premises, DDVE supports VMware, Hyper-V, KVM, and VxRail.In the cloud, Data Domain Virtual Edition runs in: The Amazon Web Services (AWS) (cloud and gov cloud)Azure (cloud and gov cloud)VMware Cloud on AWS cloud platformsGoogle Cloud Platform (GCP) DDVE on Cloud is not the same as a Cloud Tier. Cloud Tier is a different product that can only be deployed on on-premises Hardware DDs and DDVEs.If a DDVE deployed on the Cloud fills up its Local-Metadata Storage, the following alert is issued: EVT-SPACE-00004: Space usage in metadata storage has exceeded 100% threshold.

Cause

DDVE deployed on Cloud (ATOS) supports two types of data storage: Block Storage (Used for Data and Metadata, or Metadata on Object Storage enabled DDVEs aka ATOS).Hot Blob Storage is also known as Object Storage (Used only for Data) Metadata Disks on an ATOS deployed DDVE are populated with: Index, LP segment references, CMETA (Metadata) containers, and Directory manager (DM) data Most deployments are using (Hot Blob or Object Storage) as that is the current recommendation on Cloud Deployments. All the data is written using Object Storage, but 10% of space is defined as Block Storage for the DD Filesystem metadata. The size of the local-metadata is a speculation based on the size of the Licensed Capacity. At deployment time, 10% Metadata usage is estimated for a 10x deduplication factor. A DDVE with a 100% Metadata Storage usage cannot ingest backups. Due to the following scenarios, the DDVE may run out of local-metadata storage space: The deduplication is higher than 10x, as such the metadata requirements are higher than what was deployed in metadata size(default 10%).A large number or old snapshots can increase the metadata size A larger amount of metadata may be noticed in ATOS DDVEs acting as Destinations for File Replication (Controlled Clone Replication CCR /Managed File Replication MFR): If not all base files are replicated, the destination ATOS DDVE has a higher usage in metadata.In this scenario Replication is NOT taking full advantage of the Virtual Synthetics optimizations, as such needing more space to store metadata.This type of problem was noticed in NW CCR, where it does not seem all the base files were being replicated. This causes the destination to have a higher metadata usage.

Resolution

The solution for this issue would be to add further disk/s as Local Metadata Storage. Only AWS and Azure allow metadata disk size expansion when the the maximum allowed number of disks have already been allocated. The metadata capacity for existing metadata disks can increased 1 TiB increments. This does not apply to the first metadata disk. See the "Expand metadata storage" section in the guide document for the details. PowerProtect DD Virtual Edition 7.x in the Azure Cloud Installation and Administration GuidePowerProtect DD Virtual Edition 7.x on Amazon Web Services Installation and Administration Guide IMPORTANT INFORMATINO WHEN ADDING DISKS FOR METADATA: Using different-sized metadata disks creates an index imbalance.Each metadata disk added to the active tier should have its own Spindle Group, the software automatically does this. The maximum known number of spindle groups is 16.If going over the max number of spindle groups (16), metadata disks will the same spindle group. I/O against disks on the same spindle groups is sequential.See the manual for the recommended sizes, most recommend a 1 TiB increment, higher sizes are possible depending on Cloud provider.Only expand metadata disks for AWS and Azure. Expanding metadata disks in any provider makes the DDVE unusable with corruption, or having the expanded space unused. Review the Metadata Usage, if it reached 100%, it needs more disks assigned. The example below is provided, some outputs may differ depending on Cloud Provider, see the Install and Admin Manual:Display the metadata Usage. # filesys show space tier active local-metadata -------------------------- Active Tier: local-metadata usage Size GiB Used GiB Avail GiB Use% -------- -------- --------- ------ 1293.0 1291.5 1.5 100.0% -> We can see Metadata space is full. -------- -------- --------- ------ The process of adding more Metadata disks differs depending on the Cloud Provider, see each of the Cloud Provider manuals at the bottom. Display which disks are used. # disk show hardware ------------------ Disk Slot Manufacturer/Model Firmware Serial No. Capacity Type (pci/idx) ---- --------- -------------------- -------- ---------- --------- ----- dev1 -/a Virtual BLOCK Device n/a (unknown) 250.0 GiB BLOCK -> DDOS Disk dev2 -/b Virtual BLOCK Device n/a (unknown) 10.0 GiB BLOCK -> NVRAM disk dev3 -/c Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Currently used for Metadata dev4 -/d Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Currently used for Metadata ---- --------- -------------------- -------- ---------- --------- ----- 4 drives present. Go to the AWS/GCP/AZURE console and add storage to the DDVE. In this case, a 1TiB disk is added. It is seen as dev5, and remember not to expand or touch any of the other existing disks. MANUALS for DDVE on Cloud (Only posting below for DDVE V4) DDVEV4 on Microsoft Azure Install and Admin Manual. https://support.emc.com/docu91981_Data-Domain-Virtual-Edition-4.0-with-DD-OS-6.2.0.10-in-Azure-Installation-and-Administration-Guide.pdf?language=en_US DDVEV4 on Google Cloud Platform(GC) https://support.emc.com/docu91982_Data-Domain-Virtual-Edition-4.0-with-DD-OS-6.2.0.10-in-Google-Cloud-Platform-(GCP)-Installation-and-Administration-Guide.pdf?language=en_US DDVE4 on Amazon web Services(AWS), Install, and Admin Manual https://support.emc.com/docu91980_Data_Domain_Virtual_Edition_4.0_with_DD_OS_6.2.0.10_in_Amazon_Web_Services_(AWS)_Installation_and_Administration_Guide.pdf?language=en_US ATTENTION: New and Updated Install or Admin Manuals may exist at the time of reading this article. At this stage assuming a new 1 TiB disk was added, this is seen on the output as disk dev5. # disk show hardware ------------------ Disk Slot Manufacturer/Model Firmware Serial No. Capacity Type (pci/idx) ---- --------- -------------------- -------- ---------- --------- ----- dev1 -/a Virtual BLOCK Device n/a (unknown) 250.0 GiB BLOCK -> DDOS Disk dev2 -/b Virtual BLOCK Device n/a (unknown) 10.0 GiB BLOCK -> NVRAM disk dev3 -/c Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Currently used for Metadata dev4 -/d Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Currently used for Metadata dev5 -/e Virtual BLOCK Device n/a (unknown) 1.0 TiB BLOCK -> Newly added disk for metadata, unused. ---- --------- -------------------- -------- ---------- --------- ----- Add the new disk to the active Tier. dev5 for this example: # storage add tier active dev5 ATTENTION: Adding more disks to a DDVE ATOS local-metadata from the CLI may produce, the message "Local storage size exceeds the maximum required metadata capacity for this configuration" is received: ** Local storage size exceeds the maximum required metadata capacity for this configuration. Do you want to continue? (yes|no) [no]: Then consult the article below for resolution: Data Domain: Local storage size exceeds the maximum required metadata capacity for this configuration. Expand the file system. # filesys expand Confirm the FS which was read-only due to lack of metadata space can now be written to. Check that backups are now working fineCheck that incoming replication resumed and traffic may be seen.

Support Cases