Upgrading VCF 9.0 environment to 9.1 fails when trying to "Import VCF Operations in Fleet Lifecycle"

Products

VCF Operations for Networks

Issue/Introduction

Upgrading VCF 9.0.0 environment to 9.1.0 fails when trying to "Import VCF Operations in Fleet Lifecycle"

The error message seen is as below:

The Fleet lifecycle 'Component import' task with ID ######-####-####-####-######### failed. 
Internal errors: 
An unexpected error occurred in step discover_vrni_component_task_ref. Reference Code: DD4F27F4. Please contact support with this reference code.
 Detail: I/O error on POST request for "https://###.###.#####:443/api/auth/login": Certificate for <###.###.#####> doesn't match any of the subject alternative names: [], 
An unexpected error occurred in step discover_vrni_component_task_ref. Reference Code: 41E4134B. Please contact support with this reference code. Detail: I/O error on POST request for "https://###.###.#####/api/auth/login":

This issue is not seen if VCF Operations for Networks 9.1.0 is freshly deployed.

Environment

VCF Operations 9.1
VCF Operations for Networks 9.0.0

Cause

In VCF 9.0 and earlier releases, Ops LCM generated vRNI certificates without populating the SAN field correctly. Starting in VCF 9.1, Fleet LCM enforces strict certificate validity (including SAN matching) to comply with enhanced security requirements.
This connection fails in hostname verification if the IP of the VCF Operations for Networks appliance is not present in the certificate SAN presented by VCF Operations for Networks appliance to fleet while connection.

Resolution

To fix this, the VCF Operations for Networks Platform appliance(s) need to be updated with valid certificates containing IPs in SAN fields of certificates, via a script.

Script filename: rotate_vrni_certs.sh
Size: 52.8 kb
MD5 Checksum: 0A9CAC12A2AED222EFE0A490A38F9B17

See steps below:

Download the attached script named "rotate_vrni_certs.sh" attached to this Knowledge Base Article.
Transfer the script "rotate_vrni_certs.sh" to the platform node under /home/support/ directory. Use a tool such as WinSCP to the platform node and login with username support to transfer the file.
Take a SSH session to VCF Operations for Networks platform node and login with username support.
Validate the file exists under /home/support/ directory. Use below commands to validate.
```
ls -lrth
```
Validate the checksum of the script using below command:
```
 sudo md5sum rotate_vrni_certs.sh
```
Update the permission for script using below command:
```
sudo chmod 777 rotate_vrni_certs.sh
```
Validated if the VCF Operations for Networks appliances require certificate rotation or not.
Execute the script with below command to verify
```
sudo bash ./rotate_vrni_certs.sh verify
```
The above command will verify it vRNI deployment needs certificate rotation or not.

If above script output has message as shown below in screenshot, no rotation needs to be performed.

If above script output the message as shown below in screenshot, then you would need to perform certificate rotation for VCF Operations for Networks Appliance.
There are 2 options to rotate the certificates

a) By using certificate signed by CA authority, Recommended if customer has earlier applied external CA signed certificate to VCF Operations for Networks.
Copy the external CA signed certificate, private key and CA chain on the platform where we have copied rotate_vrni_certs.sh
Now execute the script as sudo bash ./rotate_vrni_certs.sh apply –-certs <certificate-pem-file-path-here> --key <private-key-pem-file-path-here> --ca-chain <ca-chain-pem-file-path-here>
After running the command you should get below output on successful execution.

b) By using self signed certificate generated by rotate_vrni_certs.sh script
You can provide option --self-signed to rotate_vrni_certs.sh and it will generate required self signed certificate for you.
The command to be ran is sudo bash ./rotate_vrni_certs.sh apply --self-signed
After running the command you should get below output on successful execution.
After the script is executed successfully, navigate VCF Operations 9.1.0 GUI.
Import VCF Operations for Networks using VCF Operations Fleet UI , see screenshots below:

Note:
Ensure you leverage Import Aria Operations for Networks 6.14.x option in order to import 9.x as well.
Import for VCF Operations for Networks should succeed.
Navigate to Build>Lifecycle>Tasks, where you should see most recent failed task such as "Install components: VCF Operations [***** #####-####-###-####-######### "

See screenshot below:
Now Click on the above Task to get the failed task id.
See screenshot below:
Copy the Task ID which is UUID for this failed task and make a note of this UUID.
SSH to the SDDC Manager appliance and login with vcf user and switch to root using command below:
```
su -
Password:
```
Enter the password to login
Use below command to edit the prod properties for the SDDC manager appliance. See command below:
```
sudo vim /etc/vmware/vcf/domainmanager/application-prod.properties
```
Add or update fleetlcm.task.id.to.recreate=<Copied TaskID from step 15> entry in the above file and save the file.
Revalidate the file using below command to ensure the UUID appended is correct.
```
cat /etc/vmware/vcf/domainmanager/application-prod.properties
```
Now restart the domain manager service on the SDDC manager appliance using below command:
```
systemctl restart domainmanager
```
Wait for 5 minutes for domain manager service to complete restart.
Navigate to section Build>Task on VCF Operations GUI In order to retry the "Deploy VCF Management Components" task

See screenshot below:
The retry task should complete successfully.