OPERATIONAL DEFECT DATABASE
...
...
VMware Identity Manager cluster health status is shown as CRITICAL in vRealize Suite Lifecycle Manager Health Notification.
Remediation of health issues associated with VMware Identity Manager clusters.
From VMware vRealize Suite Lifecycle Manager 8.1 to 8.4, cluster health can be remediated by clicking on the Power ON button. From VMware vRealize Suite Lifecycle Manager 8.5+, cluster health can be remediated by clicking on the REMIDIATE CLUSTER button.
Prerequisites Ensure vRSLCM's inventory is up-to-date. Sometimes there will be a drift/changes in the environment outside vRSLCM and for vRSLCM to know the current state of the system, its inventory needs to be updated. To do so, its advisable to perform an inventory sync of vIDM (global environment) in vRSLCM by clicking on the Trigger Inventory Sync button.For the notification to be precise, vRSLCM should be able to reach all the vIDM nodes and also be able to log-in to all nodes to perform health checks. Make sure all the vIDM nodes in the cluster are powered ON and running. If any of the nodes are powered OFF, then the health notification in vRSLCM would show a description message similar to Host(s) <IPs> not reachable from vRSLCM. Ensure vRSLCM is able to ssh into each vIDM node, if not a notification with a description message similar to Unable to get postgres status from any of the nodes in the cluster. and Unable to get pgpool status if postgres is unavailable from any of the nodes in the cluster.Ensure vRSLCM has the right root passwords else the following note will be displayed root passwords are expired In any of these cases, ensure that vRSLCM can reach vIDM nodes and all the nodes credentials are updated in vRSLCM's inventory. Note: vRSLCM does a ping to all the vIDM nodes to check if they are available or not. Once inventory is updated, cURL for the current cluster health status curl -H "Authorization: Basic token" -k https://vRSLCMFQDN/lcm/authzn/api/vidmcluserhealth Syntax Help:vRSLCMFQDN: The hostname / IP of vRSLCM managing the vIDM cluster.token: Run the following command to get the Base64 encoded value of username:password. Here username is admin@local, and password is admin@local user's password. echo -n 'admin@local:password' | base64 Note: On VCF mode replace admin@local with vcfadmin@local and its respective password. Note: The API will trigger a request to re-calculate the cluster health which will generate a notification on the current overall status in vRSLCM.Note: The automatic health check triggered by vRSLCM against a vIDM cluster is on a 1 hour default. Remediate cluster health Note: If the file /usr/local/etc/pgpool.pwd is present on any vIDM node, then while executing all the below steps replace the password with the one present in the file.Note: If an install, upgrade or scale out request is IN PROGRESS or FAILED state in vRSLCM, Do not follow the below steps. Ensure the pgpool service is running on all the nodes in the cluster /etc/init.d/pgService status Expected Response Getting pgpool service status pgpool service is running Note: This is critical since the pgpool service is responsible for orchestrating the postgres fail-over. Without pgpool service running on a given node, quorom will not be maintained and the node will not be understood to be part of the cluster. If the response is not as expected and pgpool service is not running, you will receive an error similar to Pgpool service not running on the node(s) <IPs> in the cluster. Start the service if it is not running /etc/init.d/pgService start To find the pgpool MASTER, run the below command on any of the vIDM nodes su root -c "echo -e 'password'|/usr/local/bin/pcp_watchdog_info -p 9898 -h localhost -U pgpool" Note: vIDM versions 3.3.2 and below will use the explicit string text of password in echo -e 'password'. For vIDM versions 3.3.3 and above, if /usr/local/etc/pgpool.pwd exists, use this value in "echo -e 'password'. Command parameters help -h : The host against which the command would be run, here its 'localhost'. -p : Port on which pgpool accepts connections, which is 9898 -U : The Pgpool health check and replication delay check user, which is pgpool Expected Response 3 YES <Host1>:9999 Linux <Host1> <Host1> <Host1>:9999 Linux <Host1> <Host1> 9999 9000 4 MASTER <Host2>:9999 Linux <Host2> <Host2> 9999 9000 7 STANDBY <Host3>:9999 Linux <Host3> <Host3> 9999 9000 7 STANDBY In the response, there needs to be a MASTER node present else vRSLCM will pop-up a notification with description message similar to Could not find a pgpool master node, as all nodes are in standby state. To check postgres status on all the vIDM nodes Run the command on the pgpool MASTER node found above in Step #2, this will generate a list of all configured nodes with corresponding postgres status. su root -c "echo -e 'password'|/opt/vmware/vpostgres/current/bin/psql -h localhost -p 9999 -U pgpool postgres -c \"show pool_nodes\"" Command parameters help -h : The host against which the command would be run, here it would be 'localhost' -p : The port on which Pgpool accepts connections, here its 9999 -U : The Pgpool user, which is pgpool -c : The command to run, which is 'show pool_nodes' Expected response node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change ---------+---------------+------+--------+-----------+---------+------------+-------------------+-------------------+--------------------- 0 | Host1 | 5432 | up | 0.333333 | primary | 0 | false | 0 | 2019-10-14 06:05:42 1 | Host2 | 5432 | up | 0.333333 | standby | 0 | false | 0 | 2019-10-14 06:05:42 2 | Host3 | 5432 | up | 0.333333 | standby | 0 | true | 0 | 2019-10-14 06:05:42 (3 rows) Role column - Informs postgres master node as known to pgpool serviceStatus column - Informs postgres status as known to pgpool service Note: In an ideal scenario, there must be at least one postgres master node, if so, proceed to the next step.Note: If none of the nodes are marked primary in the role column, vRSLCM will provide a notification with a message similar to Could not find a postgres master node, as all nodes are in standby state. Warning: This is a special case. Proceed go to the special cases section. Once complete, proceed to Step #4. Ensure the delegate IP is properly assigned. This will be the free IP that user provided while clustering vIDM via vRSLCM for the delegate IP field. vRLCM's postgres clustering solution assigns delegate IP to the postgres master node, so at any point in time the node which is marked master should hold the delegate IP. Check the role's column from the above show pool_nodes command response run on Step #3, for the postgres master node and ensure delegateIP is assigned on the same node. Run the following command on the postgres master node, to make sure delegate IP is still assigned ifconfig eth0:0 | grep 'inet addr:' | cut -d: -f2 Expected response delegateIP Bcast delegateIP: Make sure this is the same IP provided while clustering vIDM via vRSLCM for the delegate IP field. If so, skip this and continue to the step #5. If the above command does not return any output, it would mean delegateIP is not assigned on the master postgres node. In such event, the health notification in vRSLCM would show a description message similar to DelegateIP IP is not assigned to any nodes in the cluster. Make sure the delegateIP is not held by any other non-master nodes by running above ifconfig command on other nodes. If any of the non-master nodes still have the delegateIP run the following command first to detach it ifconfig eth0:0 down Run the below command on the master node to re-assign the delegateIP ifconfig eth0:0 inet delegateIP netmask Netmask delegateIP - This is a keyword and need not be substituted with the actual IPNetmask - Netmask currently configured for this node Once the above command is successful, ensure the delegateIP is assigned to eth0:0 by running ifconfig -a and make sure eth0:0 is holding the expected delegateIP. Note: Assigning the delegateIP explicitly needs a horizon-service restart on all the nodes. Run the following command to restart horizon service on all vIDM nodes service horizon-workspace restart From Step # 3's show pool_nodes response If any of the vIDM nodes in the cluster are marked as down in the 'status' column, vRSLCM would pop-up a notification with description message similar to 'Postgres status of node(s) <IPs> marked as down in the cluster.' This generally happens when the postgres on the node was down (or) the node went for a reboot (or) the node went out of network (or) even during a network glitch. If the replication_delay column has a value greater then 0, a node is not able to sync its data from the current postgres master. In such case, vRSLCM would pop-up a notification with a description message similar to Node(s) <IPs> have a replication delay with respect to the postgres master node. In both the above cases, the nodes would not be syncing its data from the master node and will not be able to participate in further master election in case of failover. Run the below command to recover/catch-up with master on all such nodes. This command will allow the standby nodes to be in-sync with the postgres master node and will mark the node as up in the cluster. Note: If replication delay has been corrected, the concerned node's postgres has to be brought down before performing the following command. To stop postgres for VMware Identity Manager 3.3.2 service vpostgres stop To stop postgres for VMware Identity Manager 3.3.3 or later /etc/init.d/vpostgres stop /usr/local/bin/pcp_recovery_node -h delegateIP -p 9898 -U pgpool -n node_id Command parameter help -h : The host against which the command would be run, its the delegateIP referring to the master node. -p : Port on which PCP process accepts connections, which is 9898 -U : The Pgpool user, which is pgpool -n : Node id which needs to be recovered. <node_id> will be the node that is being corrected. This can obtained from 'node_id' column from the show pool_nodes command. delegateIP : This is keyword. Need not be changed with IP. Use as is. pgpool : This is pgpool user. Need not be changed. Use as is. The above command would prompt for a password. Enter Password as password.Expected response pcp_recovery_node -- Command Successful Ensure to run the show pool_nodes command from Step # 3 to ensure all nodes are having the status column as up and the replication_delay column should have a value of 0. Note: If the above command execution is not successful follow step 6. Check if the file /etc/hosts in all the vIDM cluster nodes has the same master host entry, If master host entry is found missing on any of the nodes or if it is inconsistent across the nodes, follow the below steps Stop pgpool service on all the nodes (replica and then the master) /etc/init.d/pgService stop Check whether pgpool service is stopped, run the command /etc/init.d/pgService status Correct the master host entry in the file /etc/hosts. Note: Master host can be found by executing the command in Step #2. Ensure master host entry should be the same in all the vIDM nodes in the cluster. Start pgpool service on all the nodes (master and then the replica), run the command /etc/init.d/pgService start Check the status of the pgpool service, run the command /etc/init.d/pgService status Repeat Step # 5 to bring up the nodes in the cluster. Once all the above checks and corrections are done, re-trigger the health check API mentioned in Step #3 of the Prerequisites section of this article to ensure the cluster health notification turns GREEN displaying a message similar to vIDM postgres cluster health status is ok Special cases When all the nodes are marked standby in the role column from show pool_nodes command response in Step #3, it means there is no postgres master elected and it is a CRITICAL state. vIDM will not function. Choose one of the nodes as postgres master and the other nodes as replicas.Remediation depends on the value present in status column. Choose any one node which is marked as up in status column. This will be the new postgres master being elected.To promote the chosen node as postgres master, run the below command on the that node /usr/local/etc/failover.sh <Old master node_id> <Old master node_hostname> 5432 /db/data <choosen node_id> <choosen node_hostname> <Old master node_id> <Old master node_hostname> 5432 /db/data Command help node_id and hostname can be obtained from the 'show pool_nodes' response in step #3 <choosen node_id> - The node_id of the choosen node, which is marked as 'up' in the 'status' column. <choosen node_hostname> - The hostname of the current choosen node <Old master node_id> - This will be the node_id of the previous postgres master node. If not known, this can be any one of the unavailable ones. <Old master node_hostname> - This will be the hostname of the previous postgres master node. If not known, this can be any on of the unavailable ones. Once the command completes, stop pgpool service on all the nodes. Now sequentially start pgpool service on the new postgres master chosen above, followed by other nodes /etc/init.d/pgService stop /etc/init.d/pgService start Bring up the other nodes and follow the instructions present in Steps #4,5,6 to sync them against the new postgres master node. Running remediation commands using script Note: Remediation scripts work only if VMware Identity Manager is installed using vRealize Suite Lifecycle Manager 8.2 and above. Download the utility script KB_75080_vidm_postgres_cluster_remediation.sh from the attachments to /root partition in vIDM- KB_75080_vidm_postgres_cluster_remediation.shChange file permission of the script by executing the below command Command: chmod 777 KB_75080_vidm_postgres_cluster_remediation.shExecute the below command to find the command options which are executable using the utility script Command: ./KB_75080_vidm_postgres_cluster_remediation.sh --helpTo see the pgpool status of vIDM cluster nodes or to find the pgpool master execute the below command Command: ./KB_75080_vidm_postgres_cluster_remediation.sh --show_pgpool_statusTo see the postgres status of vIDM cluster nodes execute the below command Command: ./KB_75080_vidm_postgres_cluster_remediation.sh --show_postgres_statusTo recover/ bring up the vIDM nodes in the cluster execute the below command Command: ./KB_75080_vidm_postgres_cluster_remediation.sh --pcp_recovery <node_id> Command parameters help : <node_id> will be the node that is being corrected. This can obtained from 'node_id' column from the show pool_nodes command.
Note: These are involved steps, if you encounter issues with the procedure, please open a support ticket referencing this kb number and the area in which you are stuck and this can be evaluated.
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
