Symptoms

In the event logs you will see AppSync trying to add the target devices to mount host storage group and then mount failed with error: 09:28:06.757,Mount copy,63bfb2d1-c05b-4fb8-bdc3-8b489ee64ee1,MILE_000002,,Error occurred during the execution of service plan ServicePlan_Name 2022 06 23 12 26 55 PM 2.1 09:28:06.6,Mount copy,63bfb2d1-c05b-4fb8-bdc3-8b489ee64ee1,MILE_000005,,Mount copy phase for ServicePlan_Name completed with errors 08:27:40.776,Mount copy,63bfb2d1-c05b-4fb8-bdc3-8b489ee64ee1,VMAX_000013,,Removing device 00239,00240,00241,00244,0023C,0023D,0023A from storage group on VMAX array 000xxxxxxxxx. 08:27:38.776,Mount copy,63bfb2d1-c05b-4fb8-bdc3-8b489ee64ee1,GENL_000019,,Detaching the SCSI LUN on ESX: host-xxx. 07:27:09.19,Mount copy,63bfb2d1-c05b-4fb8-bdc3-8b489ee64ee1,VMAX_000014,,Adding copy device 00239,00240,00241,0023C,00244,0023D,0023A in storage group on VMAX array 000xxxxxxxxx. 07:26:09.827,Mount copy,63bfb2d1-c05b-4fb8-bdc3-8b489ee64ee1,HST_000011,,Mounting file systems on mount host . From Server logs you will see AppSync sends request to array to add the target devices to mount host storage group and the array returned with success: 07:27:09.326 INFO [EJB default - 14] [com.emc.archway.service.vmax.VMAXRestHelperBean] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> Adding the devices [00239, 00240, 00241, 0023C, 00244, 0023D, 0023A] to storage group in array 000xxxxxxxxx 07:27:09.328 INFO [EJB default - 14] [com.emc.archway.service.vmax.VMAXRestHelperBean] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> Content of VMAX REST PUT /univmax/restapi/92/sloprovisioning/symmetrix/000xxxxxxxxx/storagegroup/ in REST Interface : 07:27:09.332 INFO [EJB default - 14] [com.emc.archway.service.vmax.VMAXRestHelperBean] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> The payload data is :{ "volumeId" : [ "00239", "00240", "00241", "0023C", "00244", "0023D", "0023A" ] 07:27:09.391 INFO [EJB default - 14] [com.emc.archway.restclient.RESTClient] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> executeRequest: executing PUT: PUT https://:8443/univmax/restapi/92/sloprovisioning/symmetrix/000xxxxxxxxx/storagegroup/ HTTP/1.1 07:27:09.448 DEBUG [EJB default - 14] [com.emc.archway.restclient.RESTClientRequest] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> Request URL : https://:8443/univmax/restapi/92/sloprovisioning/symmetrix/000xxxxxxxxx/storagegroup/ Request Method : PUT 07:27:19.640 DEBUG [EJB default - 14] [com.emc.archway.restclient.RESTClientRequest] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> Response Content :{"jobId":"1656307733278","name":"Add volumes to SG ","symmetrixId":"000xxxxxxxxx","status":"SUCCEEDED", .......... Finished job execution on Mon 27 Jun 2022 05:28:57 UTC"} Next AppSync tried to connect to the vCenter to refresh the storage system on the parent ESX host to see if the target devices are visible on the host or not. But the connection gets timed out: 07:27:32.094 INFO [EJB default - 15] [com.emc.archway.service.vc.VCServiceBean] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> rescanHBA on ESX : 07:27:33.207 INFO [EJB default - 3] [com.emc.archway.service.vc.VCConnectionCache] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> --getConnection::Wakedup from wait on lock for entity:https:///sdk 07:57:32.681 INFO [EJB default - 15] [com.emc.archway.service.vc.VCServiceBean] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> refreshESX: Got HostSystem object for the ESX at org.apache.cxf@3.3.2//org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:65)Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking https:///sdk: Read timed out 08:27:32.823 DEBUG [EJB default - 15] [com.emc.archway.service.vc.VCServiceBean] [AppSync Server] [] 35c175b7-8035-40c2-8b58-fe2491d2d262->>> Exception while getting connection to vc or while executing vcservice ::Could not receive Message.: javax.xml.ws.WebServiceException: Could not receive Message.Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking https:///sdk: Read timed out 08:27:33.247 ERROR [EJB default - 13] [org.jboss.as.ejb3.invocation] [AppSync Server] [] WFLYEJB0034: EJB Invocation failed on component RescanESXClusterCommandBean for method public void com.emc.archway.commands.application.vcenter.RescanESXClusterCommandBean.rescanAndRefreshCluster(com.emc.archway.context.ExecutionContext,java.util.List) throws com.emc.archway.service.eventservice.exceptions.HostSystemRescanFailed: javax.ejb.EJBException: java.lang.RuntimeException: javax.ejb.EJBException: javax.xml.ws.WebServiceException: Could not receive Message. From vpxd logs: T08:32:47.233+02:00 error vpxd[53892] [Originator@6876 sub=MoDVSwitch opID=53fbb10a] RPC for host [vim.HostSystem:host-xx,] got status code 3T07:47:47.554+02:00 error vpxd[21221] [Originator@6876 sub=MoHost opID=sps-Main-460045-844-308867-1f] Null response from host [vim.HostSystem:host-xx,]; no errorT07:47:47.554+02:00 error vpxd[21221] [Originator@6876 sub=MoDVSwitch opID=sps-Main-460045-844-308867-1f] RPC for host [vim.HostSystem:host-xx,] got status code 3T07:41:45.520+02:00 warning vpxd[21073] [Originator@6876 sub=vmomi.soapStub[52] opID=TaskLoop-host-170] SOAP request returned HTTP failure; : 54396'>, >), /vpxa>, method: waitForUpdates; code: 500(Internal Server Error)

Cause

There are some connection/refresh issues on the ESX host to which the mount host VM belongs to.Due to rescan/refresh of the storage system failing on the ESX host, hence appsync is failing the mount operation.

Resolution

Rebooting the ESX host fixed the issue on ESX side. And post this the mount operation got succeeded.

Support Cases