Steps to renew expired embedded harbor registry certificate and resolve Docker login failure.
Article ID: 323430
Updated On:
Products
VMware vCenter Server
Issue/Introduction
Currently, we don't have any automated way to renew the Embedded Harbor registry certificate. This Kb will provide the manual steps to rotate the certificate.
Symptoms:
From vCenter UI, you will observe Embedded harbor registry shows unhealthy.
From the cli when you are trying to do docker login it will fail with error: x509 certificate expired.
We can check if Harbor certificate is expired by looking at the “expire date” in the below output.
curl -vvv -k -I https://harbor-IP
Symptoms:
From vCenter UI, you will observe Embedded harbor registry shows unhealthy.
From the cli when you are trying to do docker login it will fail with error: x509 certificate expired.
We can check if Harbor certificate is expired by looking at the “expire date” in the below output.
curl -vvv -k -I https://harbor-IP
Environment
VMware vCenter Server 7.0.x
Resolution
Currently, there is no automated way to renew certificates.
Workaround:
Note: Below steps can be performed on any one of Supervisor Controlplane VM.
→ ssh into vCenter and follow KB 90194 to ssh into the Supervisor Control Plane VM as root.
PLEASE NOTE: When on the supervisor control plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the supervisor cluster from the SV VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere with Tanzu solution. Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permissions from VMware Support or specific instructions about what exactly you need to deploy/delete/edit from a kb.
1. Make a note of the IP of the Embedded Image Registry, which is available at the Cluster → Configure → Image Registry. As seen in the example below.
2. Create a file named openssl.conf with below content :
3. Generate a CSR for the Embedded Image Registry SSL certificate based on the IP noted in Step 1.
4. Generate the new Embedded Image Registry SSL certificate using the registry.csr CSR file created in Step 3. This will require submitting the CSR to VMCA certificate signing authority.
Copy the CSR file to vcenter using scp and then login to vCenter and run the below command.
6. List out the resource of the registry object where we will have to replace the certificate with below command.
kubectl get registry -A
Example:
NAMESPACE NAME AGE
vmware-system-registry-1735285855 harbor-1735285855 22h
Note: Please take a backup before proceeding to next step.
Example:
kubectl get registry harbor-1735285855 -n vmware-system-registry-1735285855 -o yaml > harbor-1735285855-backup.yaml
7. Edit the section which starts with sslCert:|- and replace it with the new cert (registry.crt) that was generated earlier.
kubectl edit registry harbor-xxxxxxx -n vmware-system-registry-xxxxxxx (replace xxxxxxx with the value you see in your cluster)
Note: Edit and replace the sslCert on two locations within the same file(resource), both start with sslCert:|-, you can compare the starting and ending lines to make sure you are editing the right cert. We are not editing any other sections apart from sslCert:|-
8. The key is stored in a secret. Fetch the secret using the command below
Example:
kubectl get secret harbor-1735285855-ssl -n vmware-system-registry-1735285855 -o yaml > harbor-1735285855-ssl.yaml
9. Before we edit the secret and replace the key, we need to convert it to base64.
Note: We are not changing anything else here apart from tls.key.
11. The harbor-nginx(harbor-xxx-harbor-nginx-xxxx-xx) pod will restart. If there are human errors in editing and adding the cert and key, the pod will go into crashloopbackoff. You can check the logs of the harbor-nginx pod to see what is the error if it is in crashloopbackoff state. You can try deleting and allowing it to be recreated if you think the changes are set correctly the second time.
Once the pod is in running state, you can check the certificate from Harbor UI and see if it's reflecting the new certificate with a new expiry date.
Workaround:
Note: Below steps can be performed on any one of Supervisor Controlplane VM.
→ ssh into vCenter and follow KB 90194 to ssh into the Supervisor Control Plane VM as root.
PLEASE NOTE: When on the supervisor control plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the supervisor cluster from the SV VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere with Tanzu solution. Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permissions from VMware Support or specific instructions about what exactly you need to deploy/delete/edit from a kb.
1. Make a note of the IP of the Embedded Image Registry, which is available at the Cluster → Configure → Image Registry. As seen in the example below.
2. Create a file named openssl.conf with below content :
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
x509_extensions = SAN
req_extensions = SAN
[ req_distinguished_name ]
[ SAN ]
subjectAltName = @alt_names
[alt_names]
IP.1=<registry IP>
Note: Replace <registry IP> with the IP address of your Embedded Image Registry noted in Step 1. 3. Generate a CSR for the Embedded Image Registry SSL certificate based on the IP noted in Step 1.
openssl req -nodes -new -keyout registry.key -out registry.csr -subj "/C=US/ST=CA/L=Palo Alto/O=VMware/OU=WCP/CN=<registry IP>" -config openssl.conf
Note: Replace <regisgtry IP> with the IP address of Embedded Image Registry noted in Step 1. Files named registry.key and registry.csr will be created in the current directory. 4. Generate the new Embedded Image Registry SSL certificate using the registry.csr CSR file created in Step 3. This will require submitting the CSR to VMCA certificate signing authority.
Copy the CSR file to vcenter using scp and then login to vCenter and run the below command.
$ /usr/lib/vmware-vmca/bin/certool --gencertfromcsr --csrfile registry.csr --cert registry.crt5. Copy the registry.crt file back to SCPVM using scp command.
6. List out the resource of the registry object where we will have to replace the certificate with below command.
kubectl get registry -A
Example:
NAMESPACE NAME AGE
vmware-system-registry-1735285855 harbor-1735285855 22h
Note: Please take a backup before proceeding to next step.
Example:
kubectl get registry harbor-1735285855 -n vmware-system-registry-1735285855 -o yaml > harbor-1735285855-backup.yaml
7. Edit the section which starts with sslCert:|- and replace it with the new cert (registry.crt) that was generated earlier.
kubectl edit registry harbor-xxxxxxx -n vmware-system-registry-xxxxxxx (replace xxxxxxx with the value you see in your cluster)
Note: Edit and replace the sslCert on two locations within the same file(resource), both start with sslCert:|-, you can compare the starting and ending lines to make sure you are editing the right cert. We are not editing any other sections apart from sslCert:|-
8. The key is stored in a secret. Fetch the secret using the command below
kubectl get secrets -A | grep -i registry
You will find secret harbor-xxxxxxxxxx-ssl under namespace vmware-system-registry-xxxxxxxxxx
Example :
vmware-system-registry-1735285855 harbor-1735285855-ssl Opaque 3 22h
Note: Take a backup of the secret.Example:
kubectl get secret harbor-1735285855-ssl -n vmware-system-registry-1735285855 -o yaml > harbor-1735285855-ssl.yaml
9. Before we edit the secret and replace the key, we need to convert it to base64.
$ cat registry.key | base64 | tr -d '\n'
10. Edit the secret and replace the section: tls.key with the output from step 9.Note: We are not changing anything else here apart from tls.key.
11. The harbor-nginx(harbor-xxx-harbor-nginx-xxxx-xx) pod will restart. If there are human errors in editing and adding the cert and key, the pod will go into crashloopbackoff. You can check the logs of the harbor-nginx pod to see what is the error if it is in crashloopbackoff state. You can try deleting and allowing it to be recreated if you think the changes are set correctly the second time.
Once the pod is in running state, you can check the certificate from Harbor UI and see if it's reflecting the new certificate with a new expiry date.
Additional Information
Impact/Risks:
If certificates are expired, we cannot log in to the registry using docker cli and therefore cannot push or pull the images to deploy applications in the cluster.
If certificates are expired, we cannot log in to the registry using docker cli and therefore cannot push or pull the images to deploy applications in the cluster.
Feedback
Yes
No
Powered by
