During upgrading our guest cluster from vKR 1.32.X to 1.33.X facing ImagePullBackOff on pods while using private repository with failed to verify certificate: x509: certificate specifies an incompatible key usage"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

During upgrading our guest cluster from VKR 1.32.X to 1.33.X facing ImagePullBackOff on pods while using private repository with failed to verify certificate: x509: certificate specifies an incompatible key usage

Image pull was working in VKR 1.32.X versions for same certificates without issues.

When checking the ImagePullBackOff pod logs:

E1209 09:38:40.80307   26467 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"<Repo URL/image>\": failed to resolve reference \"<Repo URL/image>\": failed to do request: Head \"<Repo URL>\": tls: failed to verify certificate: x509: certificate specifies an incompatible key usage" image="<Repo URL/image>"

FATA[0000] pulling image: failed to pull and unpack image "<Repo URL/image>": failed to resolve reference "<Repo URL/image>": failed to do request: Head "<Repo URL>": tls: failed to verify certificate: x509: certificate specifies an incompatible key usage

Environment

VMware vSphere Kubernetes Service (VKS)

VKR 1.33.x

Cause

From k8s 1.33 it uses go validation version 1.24.9 which is FIPS 140-3 compliance.

The error message indicates that the signed server certificate(includes root/intermediate/leaf certificates) has following:

1. Certificate has compatibility issue in Key usage/EKU parameters

2. Certificate missing EKU(Extended key usage)

3. Certificate has invalid bit RSA (length of the cryptographic key) other than 2048-bit

Resolution

From Kubernetes v1.33 release, it uses Go version 1.24.9 which is FIPS 140-3 compliance, so make sure the certificate used is FIPS140-3 compliance.

Some of the major FIPS140-3 compliances:

• RSA keys must be 2048 bits or higher

• ECDSA/ECC keys must use NIST-approved curves only (P-256, P-384, P-521)

• DH parameters (if used) must be 2048 bits or higher

• Certificates must be signed using SHA-256 / SHA-384 / SHA-512

• Signature algorithms must be RSA-PSS, RSA-PKCS#1 v1.5 (with SHA-256+), or ECDSA with SHA-256+

• No SHA-1, MD5, DSA, or non-NIST ECC curves in the certificate chain

• Entire certificate chain (root, intermediate, leaf) must be compliant

• Keys and certificates should be generated using a FIPS-approved DRBG (random number generator)

• Private keys should be stored/used in a FIPS-validated cryptographic module (e.g., OpenSSL FIPS provider, HSM, or cloud KMS in FIPS mode)

 

Additional Information

References: kubectl vsphere login fails with "tls: failed to verify certificate: x509: certificate specifies an incompatible key usage Cause Identification"

Kubernetes 1.33 notes:

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.33.md

Go 1.24:

https://go.dev/doc/go1.24
https://go.dev/doc/go1.24#fips140