Symptoms

Backups are running very unstable.. Often we have an error code "83" in Netbackup which means "Media open error". 11/25/2016 02:37:26 - Critical bptm (pid=3074) sts_get_server_prop_byname failed: error 2060001 one or more invalid arguments 11/25/2016 02:37:26 - Critical bptm (pid=3074) failure to open sts for storage server DFC-XXXXXXXX: plug-in reports error 2060001 one or more invalid arguments

Cause

Check the Netbackup bptm log.If you find "busy" as shown and the message is "No paths discovered to DFC server":02:29:56.719 [3074] 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c8t5002188136612175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c8t5002188136612175d0 failed, error 16: Device busy02:29:56.719 [3074] 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c8t5002188136612175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c8t5002188136612175d0 failed, error 16: Device busy02:29:56.720 [3074] 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c11t5002188136602175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c11t5002188136602175d0 failed, error 16: Device busy02:29:56.720 [3074] 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO scan: open /p/c11t5002188136602175d0 failed, err DFC operation failed, DFC_SIO: open dev /dev/scsi/processor/c11t5002188136602175d0 failed, error 16: Device busy02:29:56.720 [3074] 26071847:bptm:3074:dnmbckcph001: [C02:100CF6470] DFC_SIO: No paths discovered to DFC server XXXXXXAll devices are busy, the plugin didn't found a device available for new backup.

Resolution

You need more devices or limit the number of backups in parallel.This is documented in the DDBOOST Admin Guide: https://support.emc.com/docu61771_Data-Domain-Boost-for-OpenStorage-3.2.1-Administration-Guide.pdf?language=en_US"Solaris Media Servers For Solaris, device entries are exclusively locked on a per-process basis one and only one process can use a device entry. Calculations are based on application instance usage. If an application spawns multiple processes, each process exclusively locks at least one device entry. Multi-threaded applications lock one device per thread. For these reasons, you should configure the Data Domain system to advertise as many DFC devices as possible to avoid 'in use' errors from the sgen device driver. A Device Busy error may result if there are not enough devices accessible to the Solaris clients. The number of sgen devices is the number of Fibre Channel ports accessible to the Solaris instance times the number of different paths to the Data Domain system endpoint(s) times the number of LUNs in the access group.""Windows Media Servers : The Data Domain server path management logic spreads out connections across available logical paths (Initiator, Target Endpoint, DFC Device). Configure enough DFC devices such that each connection uses its own generic SCSI device (logical path) on the media server, with a max DFC device count of 64.certain client-side operating systems impose a restriction on the number of outstanding IO requests which can be conducted simultaneously over a given generic SCSI device. For example, the Windows SCSI Pass-Through Interface mechanism will only conduct 1 SCSI request at a time through each of its generic SCSI devices. This impacts the performance of the DD Boostover FC solution, if multiple connections (e.g. backup jobs) are trying to use the same generic SCSI device. Additionally, the Data Domain system also imposes a limit on the number of outstanding IO requests per advertised SCSI device. For performance reasons with larger workloads,multiple SCSI devices may need to be advertised on the Data Domain system. We use the term queue-depth to describe the system-imposed limit on the number of simultaneous SCSI requests on a single device. Client systems (like Windows) whose queue depth is so low as to impact performance are considered queue-depth constrained.

Support Cases