The challenge is in pending status with an error "failed to change Route 53 record set: WebIdentityErr: failed to retrieve credentials"

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

- The certificate/tap-default-tls was created but it keeps in READY=False status.

$ kubectl get certificates -n tanzu-system-ingress
NAME              READY   SECRET                AGE
tap-default-tls   False   tap-default-tls       2d15h

 - By further checking on the related challenge, it's showing as a pending status.

$ kubectl get challenges -n tanzu-system-ingress
NAMESPACE              NAME                  STATE     DOMAIN    AGE
tanzu-system-ingress   tap-default-tls-xxx   pending   abc.com   2d15h

 - The reason of the failure is "failed to change Route 53 record set: WebIdentityErr: failed to retrieve credentials"

apiVersion: v1
items:
- apiVersion: acme.cert-manager.io/v1
  kind: Challenge
...
  status:
    presented: false
    processing: true
    reason: "failed to change Route 53 record set: WebIdentityErr: failed to retrieve
      credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus
      code: 403, request id: xyz123"
    state: pending

Environment

Product Version: Other

Resolution

When referencing Creating-an-issuer-or-clusterissuer - Route53 , it's necessary to configure an IAM Role and the trust policy. If any part of the step is misconfigured, such as the role with incorrect namespace configured in the trust relationship, annotation pointing the the wrong role, etc., you will see errors like WebIdentityErr: failed to retrieve credentials. Check the configuration and make corrections.