No connection to VR Server for virtual machine in the cluster: SQLite error
Article ID: 423732
Updated On:
Products
VMware vSphere ESXi
VMware Live Recovery
Issue/Introduction
1. Replication status is Not Active
2. Replication starts working when moved to a different host that is not impacted by hardware changes or ESXi reinstallation.
3. VMs under the replication tab display the error -
A replication error occurred at the vSphere Replication Server for replication 'VMReplication'. Details: 'No connection to VR Server for virtual machine VMReplication on host Production.host.local in cluster Production in Datacenter: Unknown'.
Environment
vSphere Replication 9.x
VMware ESXi 7.x
Cause
These errors can arise when a servers motherboard is replaced or ESXi is reinstalled.
VMkwarning.log :
2025-12-10T17:14:29.860Z Wa(180) vmkwarning: cpu0:8847777)WARNING: Hbr: 5362: Failed to establish connection to [127.0.0.1]:32032 (groupID=GID-e36dbf9d-####-####-####-############): Broken pipe
2025-12-10T17:15:59.859Z Wa(180) vmkwarning: cpu1:8847777)WARNING: Hbr: 788: Failed to receive from 127.0.0.1 (groupID=GGID-e36dbf9d-####-####-####-############): Broken pipe
2025-12-10T17:15:59.859Z Wa(180) vmkwarning: cpu1:8847777)WARNING: Hbr: 2542: Failed to receive extended handshake response
2025-12-10T17:15:59.859Z Wa(180) vmkwarning: cpu1:8847777)WARNING: Hbr: 5362: Failed to establish connection to [127.0.0.1]:32032 (groupID=GID-e36dbf9d-####-####-####-############): Broken pipe
2025-12-10T17:17:29.859Z Wa(180) vmkwarning: cpu54:8847777)WARNING: Hbr: 788: Failed to receive from 127.0.0.1 (groupID=GID-e36dbf9d-####-####-####-############): Broken pipe
2025-12-10T17:17:29.859Z Wa(180) vmkwarning: cpu54:8847777)WARNING: Hbr: 2542: Failed to receive extended handshake response
hbrsrv.log:
2025-12-10T17:21:02.224Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] HbrError stack:
2025-12-10T17:21:02.225Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [0] Resource temporarily unavailable
2025-12-10T17:21:02.225Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [1] NfsVfs: While trying to lock '/vmfs/volumes/9cf961b6-########/.hbrroot_506cf186f462794634aa1ae469eb965129f11e1720c2bae565c00819fc316b88/persistent-cleanup-index-4c4c4544-005a-5610-8044-############.db'
2025-12-10T17:21:02.225Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [2] Ignored error.
2025-12-10T17:21:02.225Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] HbrError stack:
2025-12-10T17:21:02.225Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [0] SQLite error 14: unable to open database file
2025-12-10T17:21:02.225Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [1] Opening database /vmfs/volumes/9cf961b6-138ddad4/.hbrroot_506cf186f462794634aa1ae469eb965129f11e1720c2bae565c00819fc316b88/persistent-cleanup-index-4c4c4544-005a-5610-8044-############.db 'Retry #2 for operation 'Open database '/vmfs/volumes/9cf961b6-########/.hbrroot_506cf186f462794634aa1ae469eb965129f11e1720c2bae565c00819fc316b88/persistent-cleanup-index-4c4c4544-005a-5610-8044-############.db'', Total time retrying: 20.6158 secs.'
2025-12-10T17:16:25.390Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [0] SQLite error 14: unable to open database file
2025-12-10T17:16:35.645Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [0] SQLite error 14: unable to open database file
2025-12-10T17:16:45.974Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [0] SQLite error 14: unable to open database file
2025-12-10T17:16:56.337Z In(166) hbrsrv[8848852]: [Originator@6876 sub=Main opID=hs-init-2f01900d] [0] SQLite error 14: unable to open database file
hbr-agent.log :
When an ESXi host has been re-installed, it results in a change to the default certificate assigned to the host. The logs show that the hbr-agent is utilizing an outdated certificate.
2025-12-10T17:17:29.859Z In(166)[+] hbr-agent-bin[8848104]: thumbprint: 30:##:85:##:1D:##:8B:##:0C:##:B2:##:20:##:91:##:68:##:C7:##:4C:##:EA:##:89:##:02:##:CE:##:C2:##
2025-12-10T17:17:29.859Z In(166)[+] hbr-agent-bin[8848104]: certificate:-----BEGIN CERTIFICATE-----
2025-12-10T17:17:29.859Z In(166)[+] hbr-agent-bin[8848104]: ####/zCCA2egAwIBAgIJAO6pBrvU/########
2025-12-10T17:18:59.840Z In(166) hbr-agent-bin[8848104]: [0x000000fd97e3a700] info: [ConfigManager] No user configuration for key=hbrsvc_target_info in ConfigStore.
2025-12-10T17:18:59.840Z In(166) hbr-agent-bin[8848104]: [0x000000fd97e3a700] error: [ConfigManager] Failed to get config store object. Comp: esx, Grp: services, Key: hbrsvc_target_info, Id: 10.#.#.#, Prop: certificate
2025-12-10T17:18:59.840Z In(166) hbr-agent-bin[8848104]: [0x000000fd97e3a700] info: [ProxyConnection] Setting up secure tunnel to broker on 10.#.#.#:32032
2025-12-10T17:18:59.840Z In(166) hbr-agent-bin[8848104]: [0x000000fd97e3a700] info: [Proxy [Group: ] -> [10.#.#.#:32032]] Bound to vmk: vmk11 for connection to 10.#.#.#:32032
2025-12-10T17:18:59.843Z In(166) hbr-agent-bin[8848104]: [0x000000fd97cb7700] info: [Proxy [Group: ] -> [10.#.#.#::32032]] TCP Connect latency was 2634µs
2025-12-10T17:18:59.859Z In(166) hbr-agent-bin[8848104]: [0x000000fd97db9700] error: [Proxy [Group: GID-e36dbf9d-####-####-####-############] -> [10.#.#.#:32032]] The find server request failed: (1) Failed
2025-12-10T17:18:59.859Z In(166) hbr-agent-bin[8848104]: [0x000000fd97db9700] error: [Proxy [Group: GID-e36dbf9d-####-####-####-############] -> [10.#.#.#:32032]] Failed find server request additional error info: Thumbprint and certificate is not allowed to send replication data
hms.log:
2025-12-10 16:37:43.757 INFO com.vmware.hms.hbrsrvuw.HbrsrvuwRegistrarService [hms-main-thread-5133] (..hms.hbrsrvuw.HbrsrvuwRegistrarService) [operationID=db86ccf6-####-####-####-2d557468b521-HMSINT-12418064] | handleHmsTaskFinishedEvent: [host-161306][7da6ebf7-7c53-4e09-a6e9-############]: HmsTaskFinishedEvent[id: HTID-c655a092-ce43-4c56-928a-############; taskName: Register hbrsrvuw from host-161306; taskManagedEntityId: null; taskTypeId: RegisterHbrTask; requestedExecutorId: hbr-management; taskTag: hbrsrvuw-reg-host-161306--id--7da6ebf7-7c53-4e09-a6e9-############; autoAbort: true; queueTime: 1765395463672; startTime: 1765395463723; completeTime: null; error
: Cannot start connection to VR Server https://10.#.#.#:443/hbr: null; success: false; result: null]
2025-12-10 16:37:43.758 INFO com.vmware.jvsl.util.SingleThumbprintVerifier [hms-ping-scheduled-thread-8] (..jvsl.util.SingleThumbprintVerifier) [operationID=db86ccf6-####-####-####-2d557468b521-HMSINT-12418064, operationID=a05f7fa6-####-####-####-############-HMS-PING] | Failed to validate certificate chain for 10.#.#.# against HMS truststore. Error message: The certificate was not issued for use with the given hostname: 10.#.#.#
2025-12-10 16:37:43.758 ERROR com.vmware.hms.hbrsrvuw.HbrsrvuwRegistrarService [hms-main-thread-5133] (..hms.hbrsrvuw.HbrsrvuwRegistrarService$HostTask) [operationID=db86ccf6-####-####-####-############-HMSINT-12418064] | Register hbrsrvuw from host-161306...FAILED (HTID-c655a092-ce43-4c56-928a-############); will retry
com.vmware.vim.binding.hms.remote.fault.ConnectionFault: Cannot start connection to VR Server https://10.#.#.#:443/hbr: null
at com.vmware.hms.remote.SiteManagerImpl.registerHbrServerWithTask(SiteManagerImpl.java:704) ~[hms.jar:?]
at com.vmware.hms.remote.SiteManagerImpl$3.go(SiteManagerImpl.java:551) ~[hms.jar:?]
at com.vmware.hms.task.TaskRunnable.run(TaskRunnable.java:86) ~[hms.jar:?]
at com.vmware.hms.HmsTaskManager.lambda$0(HmsTaskManager.java:253) ~[hms.jar:?]
at com.vmware.hms.util.executor.LoggerOpIdConfigurator$RunnableWithDiagnosticContext.run(LoggerOpIdConfigurator.java:132) ~[hms.jar:?]
at com.vmware.hms.util.executor.LoggerOpIdConfigurator$2.run(LoggerOpIdConfigurator.java:99) ~[hms.jar:?]
at com.vmware.jvsl.sessions.net.impl.TlsPreservingWrapper$2.run(TlsPreservingWrapper.java:47) ~[jvsl-sessions-9.0.2.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
2025-12-10 16:37:43.758 INFO com.vmware.jvsl.util.SingleThumbprintVerifier [hms-ping-scheduled-thread-8] (..jvsl.util.SingleThumbprintVerifier) [operationID=db86ccf6-####-####-####-2d557468b521-HMSINT-12418064, operationID=a05f7fa6-####-####-####-############-HMS-PING] | Thumbprint 72:##:2C:##:E4:##:80:##:2A:##:E7:##:0A:##:79:##:12:##:DE:##:64:##:61:##:6A:##:CD:##:04:##:EE:## for host 10.#.#.# is trusted.
2025-12-10 16:37:43.758 INFO com.vmware.hms.hbrsrvuw.HbrsrvuwRegistrarService [hms-main-thread-5133] (..hms.hbrsrvuw.HbrsrvuwRegistrarService$HostTask) [operationID=db86ccf6-####-####-####-############-HMSINT-12418064] | Re-schedule registration of hbrsrvuw from host-161306
2025-12-10 16:37:43.758 ERROR com.vmware.hms.net.hbr.ping.svr.c8fa915b-b77c-4fd4-a972-############[hms-ping-scheduled-thread-8] (..net.impl.VmomiPingConnectionHandler) [operationID=db86ccf6-####-####-####-############-HMSINT-12418064, operationID=a05f7fa6-####-####-####-############-HMS-PING] | Ping for server 10.#.#.#:443/hbr for session: N/A failed:
com.vmware.vim.binding.vmodl.fault.SystemError: null
Resolution
Validate the current certificate used by ESXi -
COMMAND: openssl s_client -connect <Host FQDN/IP Address>:443 | openssl x509 -noout -fingerprint -sha256
STEP 1:
To ensure the same certificate is used by hbr-agent, restart the hbrsrv and hbr-agent services by going to Configuration > Services on the ESXi host.
Using commands to restart these services from ESXi SSH -
systemctl start hbrsrvsystemctl start hbr-agent
STEP 2:
To be followed, if STEP 1 does not work.
1. Take a snapshot of the VRMS appliance where you are seeing the SQLite errors
2. Stop the hbrsrv service from the VRMS VAMI or through SSH by running this command: systemctl stop hbrsrv
3. Backup the latest hbrsrv.***.db (Where * represents the highest DB number)
root@VRMS [ ~ ]# cd /etc/vmware
root@VRMS [ /etc/vmware ]# mkdir backup root@VRMS [ /etc/vmware ]# cp hbrsrv.***.db /backup root@VRMS [ /etc/vmware ]# sqlite3 hbrsrv.***.db (Login to the latest hbrsrv DB)
root@VRMS [ /etc/vmware ]# sqlite3 hbrsrv.100.dbSQLite version 3.22.0 2024-02-02 18:45:57Enter ".help" for usage hints.sqlite> select * from hostinfo; (Displays the contents of hostinfo table)sqlite> delete from hostinfo; (Deletes the contents of hostinfo table)
Feedback
Yes
No
Powered by
