Symptoms

NetWorker server is deployed on a Red Hat Pacemaker (pcs) High Availability cluster.NetWorker was upgraded to 19.7.0.4.NetWorker service startup succeeds, all cluster resources show as "started" on one of the cluster nodes: root@NWrhelNodeG:~# pcs resource * Resource Group: NW_group: * fs (ocf::heartbeat:Filesystem): Started NWrhelNodeG.emclab.local * ip (ocf::heartbeat:IPaddr): Started NWrhelNodeG.emclab.local * nws (ocf::EMC_NetWorker:Server): Started NWrhelNodeG.emclab.local nsrlogin command fails with HTTP error 404 (not found): [admin@NWrhelNodeG linux_x86_64]$ nsrlogin -u Administrator 130136:nsrlogin: Please enter password: 117849:nsrlogin: Authentication library error: GET failed with HTTP-ERROR: 404 Server Message : Could not parse server-response from json string Server Message : Make sure that server is running The 'Server Protection' bootstrap backup fails to backup the authcdb: Figure 1: Bootstrap Backup Fails to Back up the authcdb

Cause

Changes to authc during upgrade were not committed to cluster shared authcdb. During the upgrade process, the pcs NWS resource is disabled or PCS cluster resource is stopped completely. When NWS is not running, the /nsr directory is symbolically linked (pointing) to /nsr.NetWorker.local instead of /nsr_share./nsr.NetWorker.local is the physical nodes /nsr directory and only contains client-related folders. A /nsr.NetWorker.local/authc folder exists but it does not contain any of the files specific to the NetWorker server's authcdb, this is located under /nsr_share/nsr/authc. After the NetWorker upgrade, authc is expecting "version B" of the authc files but is seeing "version A" of the files from before upgrade.

Resolution

Upgrade NetWorker to one of the following releases (or later): 19.8.0.419.9.0.2 If you are using an earlier release, upgrade to one of the versions recommended above. NetWorker packages can be downloaded from Dell Support Product Page for NetWorker.If you are using 19.8.0.4 or 19.9.0.2 and later, perform the following: On each node in the cluster, rename the /opt/nsr/authc-server/conf/h2_db.properties: mv /opt/nsr/authc-server/conf/h2_db.properties /opt/nsr/authc-server/conf/h2_db.properties.bak On each node, rerun the /opt/nsr/authc-server/scripts/authc_configure.sh to reconfigure authc. This does not delete any settings or configurations previously done in authc. On the active node, this looks something like: root@NWrhelNodeH:~# /opt/nsr/authc-server/scripts/authc_configure.sh Specify the directory where the Java Standard Edition Runtime Environment (JRE) software is installed [/opt/nre/java/latest]: The installation process will install an Apache Tomcat instance. For optimum security, EMC NetWorker Authentication Service will use a non-root user (nsrtomcat) to start the Apache Tomcat instance. If your system has special user security requirements, ensure that proper operational permissions are granted to this non-root user (nsrtomcat). Please refer to NetWorker Installation Guide. WARNING: Port 9090 is already in use. Do you wish to specify a different port number [y]? n The Apache Tomcat will use "NWrhelNodeH.emclab.local" as the host name. The Apache Tomcat will use "9090" as the port number. The NetWorker Authentication Service requires a keystore file to configure encryption and to provide SSL support. EMC recommends that you specify a keystore password that has a minimum of six characters. Do you want to use the existing keystore /nsr/authc/conf/authc.keystore [y]? Specify password for the existing keystore: The install will use the existing certificate "emcauthctomcat" for Apache Tomcat. The install will use the existing certificate "emcauthcsaml" for Authentication Service. Creating the installation log in /opt/nsr/authc-server/logs/install.log. Performing initialization. Please wait... The installation completed successfully. On the passive nodes, this looks something like: root@NWrhelNodeG:~# /opt/nsr/authc-server/scripts/authc_configure.sh Specify the directory where the Java Standard Edition Runtime Environment (JRE) software is installed [/opt/nre/java/latest]: The installation process will install an Apache Tomcat instance. For optimum security, EMC NetWorker Authentication Service will use a non-root user (nsrtomcat) to start the Apache Tomcat instance. If your system has special user security requirements, ensure that proper operational permissions are granted to this non-root user (nsrtomcat). Please refer to NetWorker Installation Guide. The Apache Tomcat will use "NWrhelNodeG.emclab.local" as the host name. The Apache Tomcat will use "9090" as the port number. The NetWorker Authentication Service requires a keystore file to configure encryption and to provide SSL support. EMC recommends that you specify a keystore password that has a minimum of six characters. Do you want to use the existing keystore /nsr/authc/conf/authc.keystore [y]? Specify password for the existing keystore: The install will use the existing certificate "emcauthctomcat" for Apache Tomcat. The install will use the existing certificate "emcauthcsaml" for Authentication Service. The NetWorker Authentication Service defines automatically an administrator user account named administrator in the NetWorker Authentication Service local database. This account is specific to the administration of the NetWorker Authentication Service, and is not related to other administrator accounts on this system. ******************************************************************************************* Password criteria: Minimum required characters - 9 and Maximum allowed characters - 126 Minimum [alphabetic - 2, Uppercase - 1, Lowercase - 1, Numeric - 1, Special character - 1] ******************************************************************************************** Specify an initial password for administrator: Confirm the password: Creating the installation log in /opt/nsr/authc-server/logs/install.log. Performing initialization. Please wait... The installation completed successfully. NOTE: On the passive node, you are prompted to create a new password for the NetWorker Administrator account. This does not mean that the existing password is lost. This happens because the authcdb used by the cluster is under /nsr_share/nsr/authc which only exists on the active node. When a passive node becomes the new active node, it uses the shared authcdb. The authc_configure.sh script is run on each node to re-create the /opt/nsr/authc-server/conf/h2_db.properties which is local to each node. Restart the NWS resource: pcs resource restart nws Confirm that the NWS resource has started: pcs resource root@NWrhelNodeH:~# pcs resource * Resource Group: NW_group: * fs (ocf::heartbeat:Filesystem): Started NWrhelNodeH.emclab.local * ip (ocf::heartbeat:IPaddr): Started NWrhelNodeH.emclab.local * nws (ocf::EMC_NetWorker:Server): Started NWrhelNodeH.emclab.local Nsrlogin attempts and bootstrap backups should succeed.

Support Cases