Symptoms

Registering Secure Connect Gateway fails with the following error even though the local system can connect to the Dell Enterprise servers through ports 443 and 8443. We cannot connect to the Dell backend. Please check and verify your network configuration, ports 8443 and 443 are required and must be open to esrs3-core.emc.com,esrs3-coredr.emc.com. connectivityreport.log: 2023-02-12 07:12:41,863 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 8443 , IOException ConnectTimeoutException 2023-02-12 07:12:41,863 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 8443 , Exception - Connect to esrs3-core.emc.com:8443 [esrs3-core.emc.com/128.221.236.246] failed: Read timed out 2023-02-12 07:12:41,863 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] connStatus after skipProxy false 2023-02-12 07:12:41,863 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy not enabled, try bypass proxy 2023-02-12 07:12:41,863 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy is null so skipProxy() 2023-02-12 07:12:41,872 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 8443 , IOException ConnectTimeoutException 2023-02-12 07:12:41,872 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 8443 , Exception - Connect to esrs3-coredr.emc.com:8443 [esrs3-coredr.emc.com/168.159.224.236] failed: Read timed out 2023-02-12 07:12:41,872 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] connStatus after skipProxy false 2023-02-12 07:12:41,872 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy not enabled, try bypass proxy 2023-02-12 07:12:41,872 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy is null so skipProxy() 2023-02-12 07:12:52,068 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 8443 , IOException ConnectTimeoutException 2023-02-12 07:12:52,069 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 8443 , Exception - Connect to esrs3-core.emc.com:8443 [esrs3-core.emc.com/128.221.236.246] failed: Read timed out 2023-02-12 07:12:52,069 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] connStatus after skipProxy false 2023-02-12 07:12:52,069 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Call populateConnectivityBean: host esrs3-core.emc.com port: 8443 2023-02-12 07:12:52,069 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy not enabled, try bypass proxy 2023-02-12 07:12:52,069 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy is null so skipProxy() 2023-02-12 07:12:52,087 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 8443 , IOException ConnectTimeoutException 2023-02-12 07:12:52,088 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 8443 , Exception - Connect to esrs3-coredr.emc.com:8443 [esrs3-coredr.emc.com/168.159.224.236] failed: Read timed out 2023-02-12 07:12:52,088 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] connStatus after skipProxy false 2023-02-12 07:12:52,088 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Call populateConnectivityBean: host esrs3-coredr.emc.com port: 8443 2023-02-12 07:12:52,088 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy not enabled, try bypass proxy 2023-02-12 07:12:52,088 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy is null so skipProxy() 2023-02-12 07:12:54,485 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , IOException SSLHandshakeException 2023-02-12 07:12:54,485 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , Exception - Received fatal alert: handshake_failure 2023-02-12 07:12:54,485 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , SSLException or SSLHandshakeException. Consider this as success as just we are checking connectivity to backend core enterprise server. 2023-02-12 07:12:54,485 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] connStatus after skipProxy true 2023-02-12 07:12:54,485 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] avgCount: 1 timeTaken: 2397 host: esrs3-coredr.emc.com 2023-02-12 07:12:54,485 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy not enabled, try bypass proxy 2023-02-12 07:12:54,486 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Proxy is null so skipProxy() 2023-02-12 07:12:54,503 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , IOException SSLHandshakeException 2023-02-12 07:12:54,503 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , Exception - Received fatal alert: handshake_failure 2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , SSLException or SSLHandshakeException. Consider this as success as just we are checking connectivity to backend core enterprise server. 2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] connStatus after skipProxy true 2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] avgCount: 1 timeTaken: 2434 host: esrs3-core.emc.com 2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy not enabled, try bypass proxy 2023-02-12 07:12:54,503 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-1, , ] Proxy is null so skipProxy() 2023-02-12 07:12:56,838 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , IOException SSLHandshakeException 2023-02-12 07:12:56,838 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , Exception - Received fatal alert: handshake_failure 2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Host: esrs3-coredr.emc.com Port: 443 , SSLException or SSLHandshakeException. Consider this as success as just we are checking connectivity to backend core enterprise server. 2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] connStatus after skipProxy true 2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] avgCount: 2 timeTaken: 4750 host: esrs3-coredr.emc.com 2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Call populateConnectivityBean: host esrs3-coredr.emc.com port: 443 2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] Time taken: 4750 2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-2, , ] avgPingTime: 2375.0 2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] Latch count is 1 2023-02-12 07:12:56,838 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl$GetThread [pool-5-thread-2, , ] ---------------------------------------- 2023-02-12 07:12:56,945 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , IOException SSLHandshakeException 2023-02-12 07:12:56,946 ERROR com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , Exception - Received fatal alert: handshake_failure 2023-02-12 07:12:56,946 INFO com.emc.esrs.connectivityreport.service.ConnectivityReportServiceImpl [pool-5-thread-1, , ] Host: esrs3-core.emc.com Port: 443 , SSLException or SSLHandshakeException. Consider this as success as just we are checking connectivity to backend core enterprise server. The network traces captured by the Tcpdump command indicates that SCG-VE could connect the Dell Enterprise servers through ports 443 and 8443.

Cause

Secure Connect Gateway uses 1500 MTU size by default to connect to the Dell Enterprise servers. However, the customer's Internet provider does not allow the 1500 MTU size.

Resolution

Perform the below commands directly on the SCG-VE to change the MTU size to 1454.Changing the MTU size of eth0 on the SCG-VE does not resolve the issue since one of the dockers named "esrsde-app" on the SCG-VE connects to the Dell backend. Check the current MTU size of sae-srs-bridge: ifconfig sae-srs-bridge # ifconfig sae-srs-bridge sae-srs-b Link encap:Ethernet HWaddr 02:42:25:62:85:66 inet addr:172.18.0.1 Bcast:172.18.0.7 Mask:255.255.255.248 inet6 addr: fe80::42:25ff:fe62:8566/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2472 errors:0 dropped:0 overruns:0 frame:0 TX packets:2528 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:12309976 (11.7 Mb) TX bytes:289066 (282.2 Kb) Take a Snapshot of the VM before any changes are performed as detailed below: docker network disconnect sae-srs-bridge saede-app docker network disconnect sae-srs-bridge esrsde-app docker network disconnect sae-srs-bridge esrsalert-app docker network rm sae-srs-bridge docker network create --driver bridge --subnet 172.18.0.1/29 --opt com.docker.network.bridge.name=sae-srs-bridge --opt com.docker.network.driver.mtu=1454 sae-srs-bridge docker network connect sae-srs-bridge esrsde-app docker network connect sae-srs-bridge saede-app docker network connect sae-srs-bridge esrsalert-app Verify: ifconfig sae-srs-bridge # ifconfig sae-srs-bridge sae-srs-b Link encap:Ethernet HWaddr 02:42:DE:D8:AB:D2 inet addr:172.18.0.1 Bcast:172.18.0.7 Mask:255.255.255.248 inet6 addr: fe80::42:deff:fed8:abd2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:7 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:746 (746.0 b) FYI:If unable to access the UI after the above change, then either REBOOT the VM and wait for 4 minutes for containers to initialize. Or restart the service systemctl restart esrsve.service.

Support Cases