Data Services Manager (DSM) Certificate rotation blocks image pulls

Products

• VMware Data Services Manager

Issue/Introduction

When using the impacted versions of DSM and VKS, if the DSM Provider Appliance's certificates are rotated then the downstream provisioned workload clusters may face issues pulling new images onto the workload nodes blocking pod creation.

Dataservice clusters (like PostgresCluster, MySQLCluster, etc) are stuck in InProgress state.

Pods in them are not able to pull images from the DSM Appliance VM's image registry

Error:

tls: failed to verify certificate: x509: certificate signed by unknown authority

Environment

DSM 9.1.0.0 and DSM 9.1.1.0

3.6.0 <= VKS version < 3.7.1

Cause

This issue occurs because containerd in VKS fails to detect the new certificates on the node. As a result, it cannot pull images from the upstream DSM Appliance registry.

Resolution

To resolve this, either SSH into the node and restart the containerd service, or restart the workload cluster VMs directly from the vCenter UI.

When restarting VMs use the following order for a HA instance to limit the fail over impact of this action:

1. Monitor
2. Secondary
3. Primary

Additional Information

KB From upstream issue: https://knowledge.broadcom.com/external/article/448971