Avi HTTP 503s after certificate update on a virtual services with GSLB site persistence

Products

• VMware Avi Load Balancer

Issue/Introduction

HTTP requests to virtual services from a GSLB service with site persistence fail with HTTP 503 errors from Avi with client log significance:

Server certificate verification failed: certificate chaining error, Request ended abnormally: response code 5xx

This issue occurs in two scenarios:

1. When the virtual service's certificate is updated with a new certificate chain and the PKI profile configured in the GSLB service is not updated with the new CA certificate chain.
2. When the GSLB service's PKI profile is changed/updated wtih a new PKI profile with the new CA certificate chain.
   
   You can identify the SP pool for the virtual service from the GUI or CLI.  The Pool name will contain the following syntax: SP - GSLB_SERVICE_NAME - VS_NAME
   
   
   
   
   
   

   [admin:controller]: > show virtualservice vs_nginx | grep sp_pool_ref
   | sp_pool_refs[1]                    | SP-gslbservice_nginx-vs_nginx                       |

   
   The following command will show the SP pool and current PKI profile:
   
   

   [admin:controller]: > show pool SP-gslbservice_nginx-vs_nginx | grep pki_profile_ref
   | pki_profile_ref                           | gslb_pki                                       |

    

Environment

Affects Version(s):

22.1.1 - 22.1.1-2p6

22.1.2 - 22.1.2-2p7

22.1.3 - 22.1.3-2p14

22.1.4 - 22.1.4-2p7

22.1.5 - 22.1.5-2p8

22.1.6 - 22.1.6-2p9

22.1.7 - 22.1.7-2p9

30.1.1

30.1.2 - 30.1.2-2p3

30.2.1 - 30.2.1-2p6

30.2.2 - 30.2.2-2p6

30.2.3 - 30.2.3-2p3

30.2.4

31.1.1 - 31.1.1-2p3

Cause

Scenario#1: It is expected to observe HTTP 503 chaining errors if the PKI profile is not updated with the new CA chain before the virtual service certificate is changed.

Scenario#2: This has been identified to be a product issue with the site persistence pools are not updating with the new PKI profile if changed on the GSLB service.

Resolution

The correct sequence of updating the GSLB site persistence with a new certificate chain is as follows:

If you are keeping and reusing the same PKI profile configured on the GSLB service, please follow the steps in order:

1. Import the new CAs of the certificate chain to the existing PKI profile.
   
   -- This ensures the new certificate chain is pushed to the DNS VS service engines and to the SP pool of the GSLB service is updated.
   
   
2. Import your new certificate and all signing CA certificates to the Avi controller - Templates > Security > SSL/TLS Certificates
   
   
3. Update the virtual service(s) with the new certificate that has the new certificate chain.

If you are going to update a GSLB service with an entire new PKI profile, please follow the steps in order:

1. import the new CA chain to the new PKI profile
   
   
2. Disable the GSLB service
   
   
3. Update the virtual service(s) with the new certificate
   
   
4. Update the GSLB service with the new PKI profile
   
   
5. Enable the GSLB service
   
   
6. You can validate the new PKI profile is reference in the SP pool with CLI command: show pool SP_POOL NAME | grep pki_profile_ref

***NOTE***

• All signing CA certificates need to be included in the PKI profile, if there's a missing intermediate cert you will also observe certificate verification issues.

• The SP pool is only created and exists when the GSLB service is in enabled state.

The product Issue with the SP pool not updating with a new PKI profile will be addressed in the next GA version releases of the VMware Avi Load Balancer.  Please look for the issue ID below in the product release notes.

ID: AV-243818

Release Notes