Protection and Recovery Proxy VMs do not deploy. Health status "In progress"

Products

VCF Automation

Issue/Introduction

When attempting to deploy Protection and Recovery Proxies within VCF Automation, the following symptoms are observed:

The proxy VMs, VREdge VMs do not get created in vCenter Server.
The health status remains stuck in "In progress".

Logs for the dr-operator pod show that it is unable to find to communicate to the vCenter Server and the Protection and Recovery Appliance through the proxy.

$ kubectl logs dr-operator-controller-manager-#### -n svc-dr-operator-####

E0609 HH:MM:SS.MS    1 hms_message_worker.go:134] "Failed to retrieve remote HMS server moref" err="failed to get DR provider config: vcProxy was not enabled. It is configuring now" logger="dr-operator-controller-manager-################.HmsMessageWorker"

E0609 HH:MM:SS.MS    1 controller.go:347] "Reconciler error" err="failed to get DR provider config: vcProxy was not enabled. It is configuring now" controller="VREdge" controllerGroup="<Domain>" controllerKind="VREdge" VREdge="svc-dr-operator-####/vr-edge-####" namespace="svc-dr-operator-####" name="vr-edge-####" reconcileID="########-####-####-####-############"

E0609 HH:MM:SS.MS     1 hms_message_worker.go:134] "Failed to retrieve remote HMS server moref" err="failed to get DR provider config: failed to get DP info: unable to create new vimclient: failed to create a vim client: Post \"<URL>": read tcp ##.###.#.##:43824->###.##.##.##:10097: read: connection reset by peer" logger="dr-operator-controller-manager-################.HmsMessageWorker"

The Supervisor Management proxy is missing proxy configuration for the vCenter server and Protection and Recovery Appliance.

In vCenter Server, navigate to Supervisor Management > Services > Supervisor Management Proxy service card > Actions > Manage.

The YAML config, does not have the below entries:
```
vcHTTPRemoteEndpoint:
  host: <VC FQDN>
  port: 443

drHTTPRemoteEndpoint:
  host: <PR FQDN>
  port: 443
```

Environment

VCF Automation 9.1

Protection and Recovery 9.1

Supervisor Management Proxy 0.4.1

Cause

This issue is caused by the presence of multiple (duplicate) configuration secrets within the vmware-system-supervisor-services namespace on the Supervisor Control Plane.

During startup, the dr-operator pod creates the proxy entries for vCenter Server and the Recovery Appliance in the Supervisor Management Proxy.

If multiple versions of the secret exist (e.g., one for version 0.4.0 and one for 0.4.1), the operator may identify and update the older or incorrect secret. Since the active Supervisor Management Proxy pod is mounted to the newer secret, the configuration changes applied by the operator are never realized by the running service and the proxy entries do not get created.

Resolution

Workaround 1: Manually add the proxy config to the Supervisor Management Proxy.

In vCenter Server, navigate to Supervisor Management > Services > Supervisor Management Proxy service card > Actions > Manage.

Update the YAML Service Config with the below entries:

vcHTTPRemoteEndpoint:
  host: <VC FQDN>
  port: 443

drHTTPRemoteEndpoint:
  host: <PR FQDN>
  port: 443

For eg.:

cpuRequests: 50m
memoryRequests: 64Mi
namespace: svc-supervisor-management-proxy-####
nsxManagers:
  - 192.168.1.100
nsxSvcLoadBalancerIP: ""

drHTTPRemoteEndpoint:
  host: dr.example.com
  port: 443

vcHTTPRemoteEndpoint:
  host: vc.example.com
  port: 443

Workaround 2: Delete the duplicate secret

Connect to the Supervisor Control Plane via SSH as root. See KB, How to SSH into Supervisor Control Plane VMs

Identify the duplicate secrets by running the following command:

kubectl get secret -n vmware-system-supervisor-services | grep supervisor-management-proxy | grep config-secret

Example output:
supervisor-management-proxy.vmware.com-0.4.0-config-secret-mpk
supervisor-management-proxy.vmware.com-0.4.1-config-secret-0x3

Verify the active secret being used by the Supervisor Management Proxy pod:
```
kubectl describe pod -n <smp-namespace> <smp-pod-name> | grep -A 5 Volumes
```
Remove the stale (unused) secret.
Caution: Ensure you are deleting the secret that is NOT in use by the active pod.
```
kubectl delete secret -n vmware-system-supervisor-services <stale-secret-name>
```

Restart the dr-operator pod to trigger a fresh reconciliation with the correct secret.

kubectl rollout restart deployment dr-operator-controller-manager -n $(kubectl get ns --no-headers | grep dr-operator | awk '{print $1}')