Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Host & vCenter/VRA appliances are syncing with the same NTP but you continuously hit the same error Updating DNS from VAMI will not work because hbrsvc & hms service must be in running state for it to take effect You cannot update DNS from CLI thru vami_config_net Switching from NTP to HOST within the time synchronization settings of vCenter & VR will not work (Even if the hosts are syncing with the same NTP)Rebooting vCenter & VRA does not help vCenter Server with an external Platform Services Controller to a vCenter Server with an embedded Platform Services Controller convergence causes this error. VR VAMI displays this message when you click on summary page - http://127.0.0.1:9286/sdk invocation failedThis error occurs when the ovfEnv.xml, HMS configuration file & VRMS DB does not get updated with the correct vCenter URL & thumbprints. You are bound to hit this error when you RECONFIGURE VR after performing an upgrade from 8.x to 8.4 etc. ERRORhttp://127.0.0.1:9286/sdk invocation failed with "java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-165 [ACTIVE]"30,000 milliseconds timeout on connection http-outgoing-165 [ACTIVE]Operation ID: 7fde2086-4972-49ca-903e-9a403ac75b7d4/15/21, 8:49:08 AM -0400Also look for these messages below, if its not DNS causing the problem./var/log/vmware/drconfigui/dr-config.log :2023-01-27 20:03:22,210 [srm-reactive-thread-5] WARN com.vmware.dr.configservice.summary.IsReconfigureRequiredHandler bd06a10d-bf5c-4193-81ab-8da29cd4509d getReconfigureRequired - DrRequestHandlerError:com.vmware.vim.vmomi.client.exception.ConnectionException: http://127.0.0.1:9286/sdk invocation failed with "java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-14 [ACTIVE]" at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:261) at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.setResponseError(HttpExchangeBase.java:321) at com.vmware.vim.vmomi.client.http.impl.HttpAsyncExchange$1$1.invokeWithinScope(HttpAsyncExchange.java:131) at com.vmware.vim.vmomi.client.http.impl.TracingScopedRunnable.run(TracingScopedRunnable.java:24) at com.vmware.dr.ui.tools.utilities.ThreadContext.lambda$wrap$10(ThreadContext.java:55) at com.vmware.dr.ui.tools.utilities.ThreadContext.execute(ThreadContext.java:209) at com.vmware.dr.ui.tools.utilities.ThreadContext.execute(ThreadContext.java:185) at com.vmware.dr.ui.tools.utilities.ThreadContext.setupContext(ThreadContext.java:76) at com.vmware.dr.ui.tools.utilities.ThreadContext.setupContext(ThreadContext.java:105) at com.vmware.dr.ui.tools.utilities.ExecutorUtils.lambda$wrap$6(ExecutorUtils.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-14 [ACTIVE] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:381) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ... 1 more/var/log/vmware/dr/drconfig.log : 2023-01-27T20:02:52.179Z info drconfig[00774] [SRM@6876 sub=DrConfigConfigurationManager opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] ReadVrmsCurrentConfig: Reading VRMS Configuration.2023-01-27T20:02:52.179Z verbose drconfig[00774] [SRM@6876 sub=Default opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Loading configuration /opt/vmware/hms/conf/hms-configuration.xml2023-01-27T20:02:52.180Z info drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Compatible VMODL version found: 'vim.version.version11/internalvim25/6.5'2023-01-27T20:02:52.180Z verbose drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Event broadcasted2023-01-27T20:02:52.180Z info drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Compatible VMODL version found: 'dr.version.version14/internaldr0/8.2'2023-01-27T20:02:52.180Z verbose drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Event broadcasted2023-01-27T20:02:52.180Z info drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Compatible VMODL version found: 'hms.version.version12/internalhmssrv/8.0'2023-01-27T20:02:52.180Z verbose drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Event broadcasted2023-01-27T20:02:52.180Z verbose drconfig[00774] [SRM@6876 sub=Default.SiteConnections opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] SiteConnections created for 'https://172.31.100.22:443/lookupservice/sdk' with thumbprint '1C:D4:B0:86:DD:90:C4:C4:7D:FD:04:D0:93:5F:72:DE:B4:77:AA:72:94:B9:1D:C5:19:54:78:55:96:FF:A1:FB'2023-01-27T20:02:52.184Z verbose drconfig[00774] [SRM@6876 sub=vmomi.soapStub[12] opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Connected to host <cs p:00007f964800f390, TCP:172.31.100.22:443> using internallookup/2.0 (lookup.version.version2)2023-01-27T20:03:13.780Z warning drconfig[01363] [SRM@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f9600004098, h:10, <TCP '172.31.11.7 : 50936'>, <TCP '172.31.100.22 : 443'>>, e: 110(Connection timed out)2023-01-27T20:03:13.780Z verbose drconfig[01362] [SRM@6876 sub=vmomi.soapStub[11] opID=8f3c8409-f5b3-4096-9a87-2f6eab6999d2-isReconfigureRequired] Resetting stub adapter for server <cs p:00007f961c019400, TCP:172.31.100.22:443> : Closed2023-01-27T20:03:13.780Z error drconfig[01362] [SRM@6876 sub=DrConfigConfigurationManager opID=8f3c8409-f5b3-4096-9a87-2f6eab6999d2-isReconfigureRequired] Failed to obtain VC instance id by URL.2023-01-27T20:03:13.781Z error drconfig[01362] [SRM@6876 sub=Default opID=8f3c8409-f5b3-4096-9a87-2f6eab6999d2-isReconfigureRequired] Failed to send response to the client--> N7Vmacore11IOExceptionE System exception while trasmitting HTTP Response:--> error id = 32-->--> [context]zKq7AVECAAQAAEnqDQEMZHJjb25maWcAAL3EG2xpYnZtYWNvcmUuc28AAEMOFwC7WyMAlVwjAY6OGmxpYnZtb21pLnNvAAKYCgZsaWJkci12bW9taS5zbwAC7xQGAO3/LgAyHC8AfjJAA4d/AGxpYnB0aHJlYWQuc28uMAAEvzUPbGliYy5zby42AA==[/context]--> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.4.0, build: build-17689161, tag: drconfig, cpu: x86_64, os: linux, buildType: release--> backtrace[03] libvmacore.so[0x001BC4BD]--> backtrace[04] libvmacore.so[0x00170E43]--> backtrace[05] libvmacore.so[0x00235BBB]--> backtrace[06] libvmacore.so[0x00235C95]--> backtrace[07] libvmomi.so[0x001A8E8E]--> backtrace[08] libdr-vmomi.so[0x00060A98]--> backtrace[09] libdr-vmomi.so[0x000614EF]--> backtrace[10] libvmacore.so[0x002EFFED]--> backtrace[11] libvmacore.so[0x002F1C32]--> backtrace[12] libvmacore.so[0x0040327E]--> backtrace[13] libpthread.so.0[0x00007F87]--> backtrace[14] libc.so.6[0x000F35BF]--> [backtrace end]2023-01-27T20:03:23.770Z warning drconfig[01362] [SRM@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f9604002d28, h:12, <TCP '172.31.11.7 : 50950'>, <TCP '172.31.100.22 : 443'>>, e: 110(Connection timed out)2023-01-27T20:03:23.770Z verbose drconfig[00774] [SRM@6876 sub=vmomi.soapStub[12] opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Resetting stub adapter for server <cs p:00007f964800f390, TCP:172.31.100.22:443> : Closed2023-01-27T20:03:23.770Z error drconfig[00774] [SRM@6876 sub=DrConfigConfigurationManager opID=bd06a10d-bf5c-4193-81ab-8da29cd4509d-isReconfigureRequired] Failed to obtain VC instance id by URL.
The connection to vCenter is taking a long time and the socket is timing out before a connection to vCenter is established because the first 2 DNS servers are decommissioned. Henceforth, when the appliance tries to communicate it times out before it can hit the 3rd DNS within 30 seconds.From the screenshot, 192.168.0.12 and 192.168.0.13 DNS servers have been decommissioned and 192.168.0.2 is the one that’s active & this is affecting the RECONFIGURE process because the appliance always hits the first DNS server found before moving on to others.
Unable to reconfigure VR to register with the newly converged vCenter, therefore you are unable to re-pair SRM sites.
Enable SSH from the replication appliance VAMIExecute the following command to update the proper DNS /opt/vmware/share/vami/vami_set_dns -d domain -s searchpath DNS_Server_1 DNS_Server_2 Replace domain, searchpath, DNS_Server_1, DNS_Server_2 with the correct values. The domain, searchpath, and second DNS Server are optional and can be skipped. Example: The following command will only set a domain and one DNS server: /opt/vmware/share/vami/vami_set_dns -d vr.local 192.168.0.2The following command will set all fields: /opt/vmware/share/vami/vami_set_dns -d vr.local -s vr.local 192.168.0.2 192.168.0.3 Verify the DNS changes by running the following command: resolvectl statusYou can also try Option 2 from this KB if this resolution fails to work RECONFIGURE should work now Continue with the CHECKS below, if fixing DNS does not resolve the problem. Before performing the checks below, its mandatory to ensure vCenter is working in a healthy condition. NOTE: Please take snapshots on vCenter & VR 1. External PSC, if any must be properly decommissioned. (This has to be validated when a convergence of vCenter Server with an external Platform Services Controller to a vCenter Server with an embedded Platform Services Controller occurs)2. SSL trust mismatch issues in the lookup service must be fixed by running the LSDoctor tool (Use the switches -l, -s & -t)3. All vCenter services must be restarted in the SSO site or ELM (Enhanced Linked Mode)Using the 'lsdoctor' Tool (80469) Refer to this KB to get thumbprints - Finding SHA-1 and SHA-256 thumbprints for ESXi, vCenter, vSphere Replication and Site Recovery Manager appliances (82548)CHECK # 1 : Check and update the ovfEnv.xml & hms-configuration.xml files. The following commands must be run in 'root' mode after logging in to VR Appliance. 1. Stop HMS & HBRSRV services root@VRMSPR [ ~ ]# systemctl stop hmsroot@VRMSPR [ ~ ]# systemctl stop hbrsrv2. cd /opt/vmware/etc/vami/root@VRMSPR [ /opt/vmware/etc/vami]# ls -lahtotal 28Kdrwxr-xr-x 3 root root 4.0K Jan 28 21:16 .drwxr-xr-x 15 root root 4.0K Apr 28 2022 ..drwxr-xr-x 2 root root 4.0K Apr 28 2022 flags-rw-r--r-- 1 root root 2.1K Jan 28 21:16 ovfEnv.xml-rw-r--r-- 1 root root 2.2K Jul 6 2022 ovfEnv.xml.bkp-rw-r--r-- 1 root root 166 Feb 26 2021 vami.xml-rw-r--r-- 1 root root 129 Apr 28 2022 vami_ovf_info.xml3. Edit the ovfEnv.xml file & modify the highlighted lines. I recommend to power OFF & power ON the VR appliance once or twice to see whether VR automatically fetches this information from vCenter (most times, it will). If not, we will need to update this file manually. evs:URL > vCenter URL evs:IP > vCenter IP evs:Address > vCenter URL </PropertySection> <ve:EthernetAdapterSection> <ve:Adapter ve:mac="00:50:56:83:53:59" ve:network="REPLICACION" ve:unitNumber="7"/> </ve:EthernetAdapterSection><ve:vServiceEnvironmentSection xmlns:ve="http://www.vmware.com/schema/ovfenv" xmlns:evs="http://www.vmware.com/schema/vservice/ExtensionVService" ve:bound="true" ve:id="installation" ve:type="com.vmware.vservice.extension"> <evs:GuestApi> <evs:URL>https://vc100.ulsa-df.lasalle.mx/vsm/extensionService</evs:URL> <evs:Token>93bdb6a3505b9aea6ac019b75b0b5859f5f4c227</evs:Token> <evs:X509Thumbprint>4b:c9:0b:75:20:23:d6:bf:bb:96:41:3f:e2:b9:4e:dc:eb:9a:f5:66</evs:X509Thumbprint> </evs:GuestApi> <evs:VCenterApi> <evs:IP>172.31.100.20</evs:IP> <evs:X509Thumbprint>4b:c9:0b:75:20:23:d6:bf:bb:96:41:3f:e2:b9:4e:dc:eb:9a:f5:66</evs:X509Thumbprint> <evs:HttpPort>80</evs:HttpPort> <evs:HttpsPort>443</evs:HttpsPort> <evs:SelfMoRef>VirtualMachine:vm-106</evs:SelfMoRef> <evs:Address>vc100.ulsa-df.lasalle.mx</evs:Address> </evs:VCenterApi></ve:vServiceEnvironmentSection></Environment>If you are making changes to ovfEnv.xml file, run the following command for it to take effect. This command also helps in setting the evs:Token if its changed.vmware-rpctool "info-set guestinfo.ovfEnv `cat /opt/vmware/etc/vami/ovfEnv.xml`"This command can also be used to check the ovfEnv.xml file : vmware-rpctool "info-get guestinfo.ovfEnv"4. Edit the hms-configuration.xml & update the items highlighted below. <!-- Local vCenter Server settings --> <hms-localvc-address>VCSA67PR.vr.local</hms-localvc-address> <hms-localvc-port>80</hms-localvc-port> <hms-localvc-thumbprint>12:6A:F7:33:15:AA:6B:24:4D:34:B3:3D:28:4B:87:20:E1:8F:82:66:47:BB:4B:70:29:2A:D2:EA:FE:A0:56:41</hms-localvc-thumbprint> <hms-localvc-adminmail>root@abc.com</hms-localvc-adminmail> <!-- LookupService connection settings --> <hms-ls-url>https://VCSA67PR.vr.local:443/lookupservice/sdk</hms-ls-url> <hms-ls-thumbprint>12:6A:F7:33:15:AA:6B:24:4D:34:B3:3D:28:4B:87:20:E1:8F:82:66:47:BB:4B:70:29:2A:D2:EA:FE:A0:56:41</hms-ls-thumbprint>5. Start HMS & HBRSRV services root@VRMSPR [ ~ ]# systemctl start hmsroot@VRMSPR [ ~ ]# systemctl start hbrsrvCHECK # 2 : Check and update the VRMS DB tables. 1. Stop HMS & HBRSRV services root@VRMSPR [ ~ ]# systemctl stop hmsroot@VRMSPR [ ~ ]# systemctl stop hbrsrv2. Note the VR EMB_DB_PASSWORD root@VRMSPR [ ~ ]# cat /opt/vmware/hms/conf/embedded_db.cfgEMB_DB_INSTALL_DIR='/opt/vmware/vpostgres/current'#EMB_DB_TYPE='PostgreSQL'#EMB_DB_SERVER='127.0.0.1'#EMB_DB_PORT='5432'EMB_DB_INSTANCE='vrmsdb'EMB_DB_USER='vrmsdb'EMB_DB_PASSWORD='iIUnKTHwccEZ5mhPwXoaBEZ9QSbWn6sx'EMB_DB_STORAGE='/var/lib/vrmsdb'EMB_DB_MAX_CONNECTIONS='150'3. Login to VRMSDB.root@VRMSPR [ /opt/vmware/vpostgres/current/bin ]# ./psql -U vrmsdbPassword for user vrmsdb:4. Check & update the following tables - vrmsdb=# Select movalue,vsrv_address,vsrv_thumbprint from hmsserverentity;vrmsdb=# Select dbid,address,sitename,thumbprint from localserverentity;vrmsdb=# Select dbid,address,sitename,thumbprint from localvcentity;vsrv_thumbprint > vCenter server thumbprint thumbprint in localserverentity > Local VR thumbprint thumbprint in localvcentity > Local vCenter thumbprint Example :vrmsdb=# Select movalue,vsrv_address,vsrv_thumbprint from hmsserverentity; movalue | vsrv_address | vsrv_thumbprint---------------------------------------------+-------------------+------------------------------------------------------------------------------------------------- HMSSRV-040f647f-a30a-42cc-9212-e463612779a8 | VCSA67PR.vr.local | 12:6A:F7:33:15:AA:6B:24:4D:34:B3:3D:28:4B:87:20:E1:8F:82:66:47:BB:4B:70:29:2A:D2:EA:FE:A0:56:41 HMSSRV-8dbaf307-f7ba-491e-9c7b-9ab72ccbf5a9 | VCSA67DR.vr.local | 74:94:A7:33:BB:DF:FD:75:8E:53:08:DF:3F:63:9B:CF:3C:35:EE:6A:BF:5F:1B:3E:8A:16:12:56:25:02:88:FF(2 rows)vrmsdb=# Select dbid,address,sitename,thumbprint from localserverentity; dbid | address | sitename | thumbprint------+-----------------+------------+------------------------------------------------------------------------------------------------- 1 | VRMSPR.vr.local | Production | 7F:4D:8E:53:E5:DA:82:49:93:2B:B1:58:22:83:A1:3C:E2:C4:F1:95:B9:09:85:ED:80:89:39:98:84:2D:25:E4(1 row)vrmsdb=# Select dbid,address,sitename,thumbprint from localvcentity; dbid | address | sitename | thumbprint------+-------------------+-------------------+------------------------------------------------------------------------------------------------- 2 | VCSA67PR.vr.local | VCSA67PR.vr.local | 12:6A:F7:33:15:AA:6B:24:4D:34:B3:3D:28:4B:87:20:E1:8F:82:66:47:BB:4B:70:29:2A:D2:EA:FE:A0:56:41(1 row)Update the URLs & thumbprint as needed (examples) -vrmsdb=# update hmsserverentity set vsrv_thumbprint='F4:C9:5C:54:DA:FD:23:7C:5B:B9:29:6A:BE:F4:84:41:80:16:A3:58:D4:9E:F4:BE:50:6C:7B:35:5B:C7:A9:41' where movalue='HMSSRV-ecbc15fb-bb9f-4fc4-bfe4-92a8de985647';vrmsdb=# update localserverentity set thumbprint='46:7C:ED:FB:15:1E:82:E9:D7:78:44:F4:F1:B0:63:EC:08:C1:E5:19:8A:C8:DE:5A:86:8F:58:27:CD:54:07:5B' where dbid=1; vrmsdb=# update hmsserverentity set vsrv_address='IP OR FQDN(whatever you find in the table)' where movalue='HMSSRV-ecbc15fb-bb9f-4fc4-bfe4-92a8de985647';vrmsdb=# update localserverentity set sitename='IP OR FQDN(whatever you find in the table)' where dbid=1;5. Start HMS & HBRSRV services root@VRMSPR [ ~ ]# systemctl start hmsroot@VRMSPR [ ~ ]# systemctl start hbrsrv
vSphere Replication 8.3.x gets disconnected post PSC Convergence and does not reconnect even after un-register/re -register (79948)vSphere Replication gets disconnected post PSC Convergence and does not reconnect even after un-register/re -register (78696)Unable to register replication with vCenter when vCenter/PSC certs gets changed or after Convergence. (80155)