OpsMan Upgrade fails with "Socket Error - BlobstoreVerifier" error

Products

• Tanzu Kubernetes Grid

Issue/Introduction

OpsMan Upgrade fails during apply-director-changes task with the following error:

"Socket Error 'http://<URL>', type: BlobstoreVerifier"

 

The issue is caused because on vSphere the Ops Manager VM configures the system DNS servers on startup with the information provided when configuring the VM in vCenter. If the Ops Manager process is able to startup before that DNS configuration happens, it gets only the system DNS servers and then is unable to resolve DNS queries correctly.

Environment

OpsMan v3.0.25 and older can be impacted.

v3.0.25 is more likely to be impacted due to a system update causing OpsMan process to startup faster.

Resolution

Workaround

1. In OpsMan VM, edit "/usr/lib/systemd/system/reset_network_from_ovf.service" and add "Before=network-online.target ntp.service tempest-web.service tempest-worker.service tempest-scheduler.service" right after "After=network.target"

E.g.
# vim /usr/lib/systemd/system/reset_network_from_ovf.service
[Unit]
Description=Resets network from ovf
After=network.target
Before=network-online.target ntp.service tempest-web.service tempest-worker.service tempest-scheduler.service

2. # sudo systemctl daemon-reload
3. # sudo service tempest-web restart
4. From OpsMan GUI, "Apply changes" to update healthwatch and exporter failing on upgrade-all-service errands

 

Permanent fix

v3.0.26