Upgrade to 30.2.5 failed due to configuration migration issues.

Products

• VMware Avi Load Balancer

Issue/Introduction

• Upgrade to 30.2.5 can be unsuccessful due configuration migration failures.
• The upgrade transcripts will show errors similar to:
  • Certificate with Invalid SAN values:
    

    "UC::Migration failed for object of type SSLKeyAndCertificate, with uuid sslkeyandcertificate-xxxxxx-xxxxxx at function upgrade_SSLKeyAndCertificate for version 30.2.5 with exception: (\"Invalid SAN values found for certificate: ['test123', 'test789'].\",) ::UC"

  • Certificate with Duplicate SAN values:
    

    "UC::Migration failed for object of type SSLKeyAndCertificate, with uuid sslkeyandcertificate-xxxxxx-xxxxxx at function upgrade_SSLKeyAndCertificate for version 30.2.5 with exception: (\"Duplicate SAN values found for certificate: ['DNS:foo.org'].\",) ::UC"

  • Certificate with Invalid and Duplicate SAN values:
    

    "UC::Migration failed for object of type SSLKeyAndCertificate, with uuid sslkeyandcertificate-xxxxxx-xxxxxx at function upgrade_SSLKeyAndCertificate for version 30.2.5 with exception: (\"Invalid SAN values found for certificate: ['test1234'].Duplicate SAN values found for certificate: ['DNS:foo.bar'].\",) ::UC"

Environment

• Upgrade to version 30.2.5 from any version.

Cause

• Earlier versions prior to 30.2.5 had no strict validation for SAN values.
• During upgrade to 30.2.x versions, outages can occur for certificates with invalid wildcard FQDNs.
• Renewal may also fail for certificates containing SAN entries that do not comply with RFC 5280, such as those with underscores or unicode characters in FQDNs.
• To address this, strict RFC 5280 compliance validation was introduced.
• However, this change caused upgrade failures for certificates containing short names or duplicate SAN entries which were commonly used by customers for certificates signed by private CAs for internal consumption.

 

 

Resolution

• A fix is provided in patch, 30.2.5-2p1, for the releases with this known issue.
• The fix will relax SAN validation to allow short names and duplicate entries in SAN fields.
• The recommendation is to upgrade to 30.2.5 and apply the 30.2.5-2p1 patch in the SAME step.
• To perform the upgrade and patch in the same step, please follow the steps below:
  • Upload the 30.2.5 Image AND the 30.2.5-2p1 patch to the controller.
  • Navigate to Administration>System Update.
  • Select both the image and patch and then click on Upgrade as shown in the screenshot below.
    
  • This will apply the base image and the patch in the same step.
• Release Notes: 30.2.5 Release Notes.
• Bug ID: AV-254455
• Fix Versions: 30.2.5-2p1

Note:

• The fix added will only allow short names and repeated values in the SAN fields. 
• Starting version 30.2.5, we are still doing SAN validation checks as per RFC 5280 standards.
• This means that values like "*wildcard.com", "https://example.com", "**.wildcard.com", "*.*.wildcard.com" in the DNS SAN field will still be flagged causing the upgrade to fail.
• To verify if there are any offending certificates in your system, please upload the script attached to the kb and run it as instructed below:
  
  • Copy the script "find_offending_certs.py" to the controller.
  • You will need to provide the latest configuration backup to the script. Usually, the controller collects backup once everyday. You should be able to locate it under "/var/lib/avi/backups". 
  • Pick the latest config file and then run the script as follows:
    

    python3 find_offending_certs.py --config <path/to/config-file> [--verbose]

  • The script can be run with an optional "verbose" option to print out more details like the offending values and the VS names using the certificates, if any. 

Additional Information

• This sanity check has been added to avoid datapath issues as reported in: AV-233616.
• Known Issues on 30.2.2, documenting the impact of invalid SAN values on VSs.