GSLB service failed to resolve post upgrade from 21.1.x to 22.1.x (22.1.3-22.1.6)

Products

• VMware Avi Load Balancer

Issue/Introduction

Post Avi Load Balancer upgrade from version 21.1.x to 22.1.3 - 22.1.6, GSLB services FQDN containing a mix of uppercase and lowercase letters may experience an issue where their corresponding DNS records are intermittently removed upon subsequent updates or modifications to the GSLB service.

Environment

Avi Load Balancer configured with GSLB services

Cause

1. Starting with Avi version 22.1.3, a new feature was introduced to enforce DNS naming conventions by automatically converting all FQDNs to lowercase during creation or update. This ensures consistency and proper DNS record management.
2. However, a specific scenario was overlooked during the upgrade process for existing GSLB services. If a GSLB service's FQDN was originally configured in an older version (e.g., 21.1.x) with a mixed-case FQDN (e.g., MyService.example.com), this mixed-case FQDN was preserved in the database even after the upgrade to 22.1.6.
3. The conflict arises when such a GSLB service is subsequently modified or updated in Avi version 22.1.3 or later. The system attempts to apply the new lowercase conversion logic, but the underlying mixed-case FQDN, which was not properly normalized during the upgrade, leads to an inconsistency that can result in the removal of the associated DNS record.

Resolution

Modify GSLB Application Name:

1. Navigate to the GSLB service in the Avi UI.
2. Edit the GSLB application name (FQDN) by adding a temporary suffix (e.g., "-new") to the end.
3. Example: If the FQDN is MyService.example.com, change it to MyService-new.example.com.
4. Save the changes. 

This action forces the system to re-evaluate and normalize the FQDN, correcting the underlying case in the database.

Remove Temporary Suffix:

1. Immediately re-edit the same GSLB service.
2. Remove the temporary suffix (e.g., "-new") that you added in the previous step.
3. Example: Change MyService.example-new.com back to MyService.example.com.
4. Save the changes.

After performing this workaround, the GSLB service's FQDN will be correctly stored in lowercase in the database, and its DNS records will be properly maintained upon future updates.

Permanent Fix (Script Attached):

For environments with numerous affected GSLB services, manual application of the workaround can be time-consuming. A script is available to automate the correction of all GSLB services in the system that have mixed-case FQDNs.

1. Download the script file and save a copy to the controller leader node at /opt/avi/scripts
2. Name the script file as update_gs_domain_names.py

Script Usage:

• View Gslb Services, which contains the Capital letters

python3 update_gs_domain_names.py --verify

• To update the GSLB service with the correct FQDN format (all domain names to lower case letters in the GSLB service) - Without downtown time/traffic loss. 

python3 update_gs_domain_names.py --update 

• Verify the changes
  
  

  python3 update_gs_domain_names.py --verify

  The above command should not return any GSLB service.

Additional Information

The issue has been fixed in 22.1.7