Symptoms

Before proceeding, read the document which covers CyberSense Delta-Block Analysis (DBA) Mode of Performance Operations with Cyber Recovery.The Cybersense diff operation may be reporting a high change rate (such as 99%) in cases where the virtual machine is not changing at all.For example: 2023/09/11 04:12:21 notice: [7f3382ffd700] ddboost_diff: /opt/ie/var/mnt/dispatch/crawl_M5eksh/CRS_MTREE/05/74/3fad5a57-00000006-78fb7066-64fb7066-ab8aae6c-5095318b/vm-277919-disk-key-2001-flat.vmdk vs /opt/ie/var/mnt/dispatch/crawl_M5eksh/CRS_MTREE/86/28/a6360b9f-00000006-13fcc21c-64fcc21c-b9efae6c-5095318b/vm-277919-disk-key-2001-flat.vmdk is terminated because reported change rate 99% reached 50% threshold This results in the Virtual Machine (VM) going for a full scan, whereas CyberSense is only meant to analyze the changed blocks. This leads to the performance of the job not being optimal since the full scan typically takes longer.

Cause

There are several reasons for this behavior, the most common reasons are listed in the resolution section below in a checklist format. Verify all these items are in place and the diff operation should work as expected.

Resolution

Data Domain operating system for both the production and vault Data Domain systems is (DDOS) 7.7 or other. Use the latest code levels where possible. The NFSv4 protocol is enabled on the vault Data Domain system along with the NFSv3 protocol. CyberSense version is 7.10 or other. Use the latest code levels where possible. The ddboost users are set up for CyberSense to leverage DBA. For example, PowerProtect Data Manager copies require ddboost users with the role of 'none'. Administrators must typically create three DD Boost users with different roles: admin user none Those users must be added to the CyberSense server. For information about how to add these DD Boost users to CyberSense, see the "Configure Delta-Block Analysis" topic in the CyberSense installation guide. The workload is a supported workload. The following table lists the supported backup applications and workloads supported as of CyberSense 7.10. See the CyberSense release notes for supported workloads. For unsupported workloads, CyberSense uses the traditional scan method. Cyber Recovery Manager supports the production backup application and version. Use the latest code levels where possible. A scratch pool is configured within CyberSense, and it is sized correctly for the environment. The below switch is enabled on the policies that are being analyzed (it is enabled by default). synthetic-optim The fast copy by default uses this switch in order to copy over full synthetic recipes of the backups within the MTree. Fast copy is leveraged when creating a sandbox or making a copy in Cyber Recovery. This feature must be enabled as a requirement for CyberSense if the ddboost diff is to work properly. Log in using the CLI: [root@crhost]# /opt/dellemc/cr/bin/crcli login --username admin1 Show the current settings for the below policy: synthetic-optim [root@crhost]# /opt/dellemc/cr/bin/crcli policy synthetic-optim --show -n Policy1 Policy : Policy1 synthetic-optim = Disabled [root@crhost]# Enable the setting for the policy (it is enabled by default): synthetic-optim [root@crhost]# /opt/dellemc/cr/bin/crcli policy synthetic-optim -n Policy1 --enable Policy : Policy1 modified successfully. [root@crhost]# Verify the current configuration for the policy after enabling it: synthetic-optim [root@crhost]# /opt/dellemc/cr/bin/crcli policy synthetic-optim --show -n Policy1 Policy : Policy1 synthetic-optim = Enabled [root@crhost]# The backup applications "Synthetic full metadata" are available at the vault Data Domain system. Ideally there is no 'subcopy' and CyberSense is working with the original data. The term 'subcopy' is used to represent a subset of data that is copied to another MTree. That copied data is then replicated into the Cyber Recovery vault. There are different methods to create a subcopy, it is based on the backup application. For example, for an Avamar use case, a second Avamar or AVE server is used to retrieve important data to be protected with a reduced retention period by the Cyber Recovery solution. The subcopy is then replicated to the Data Domain system in the Cyber Recovery vault. This extra step of copying the data before it is replicated into the Cyber Recovery vault, has the possibility to lose the metadata on the files (also called a recipe) which is required for the diff operation to work correctly. Performance Optimized VM backups are less likely to have this issue. Capacity optimized backups are not supported with subcopy at production and use traditional method. Scenario A: Backup data is sent to the vault Data Domain system without making a subcopy at the production side (the original MTree is replicated into the vault). Scenario B: Backup data is copied at the production side (ensuring recipes are retained), and the copied data is replicated into the vault (copied MTree is replicated into the vault). NOTE: Sometimes, Data Domain support may be called upon to verify if the metadata (recipe) for the files are available within the sandbox that is presented to CyberSense. We are working on 'pairs' of backups. Sometimes the CyberSense cutoff time may require extending to ensure that CyberSense is working on pairs of backups. Check the below CyberSense log for the listed error messages. If they are present, consult with the Cyber Recovery support team before any change to the cutoff time is made. Log: ie_run.log Error: Objects are unrelated If these errors are not listed, then ignore this point.

Support Cases