TCA 2.0.1 reconfigure on a network function NF reconfigure operation fails at 80 %

Products

• VMware
• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:

- Scale/reconfigure operations fail with an authentication error message after a CNF upgrade operation.
-  Helm Upgrade Failure: Helm API failed: authentication error



- Scale Operation failed with error. Auto rollback successful.

In the status detail you notice the following Error:



At TCA-CP app.log error snippet:
 

2023-03-27 09:23:06.001 UTC [NfvCnfLcmService_SvcThread-3819, Ent: HybridityAdmin, Usr: HybridityAdmin, , TxId: 0c5a54fc-0cea-47db-b3a4-65b82d05f0c9] INFO c.v.h.c.workflows.CnfReconfigureJob- Fetching the status of dispatched orchestration jobs


2023-03-27 09:23:06.005 UTC [NfvCnfLcmService_SvcThread-3819, Ent: HybridityAdmin, Usr: HybridityAdmin, , TxId: 0c5a54fc-0cea-47db-b3a4-65b82d05f0c9] ERROR com.vmware.hybridity.nfvm.JobUtils- CN_CHART_RECONFIGURE failed, errorCode:null. stacktrace:null, response: {"errors":[{"internalMessage":"Helm Upgrade Failure: Helm API failed: authentication error: invalid repository username and\/or password: authentication error: invalid repository username and\/or password","errorCode":"E_CNF_HELM_RECONFIGURATION"}]}

 

 

Environment

VMware Telco Cloud Automation 2.0.1

Cause

The root cause of the problem is, TCA loses the Harbor credentials stored in the TCA database after a CNF upgrade operation is performed.

Resolution

This issue is reported in TCA 2.0.1 and is fixed in TCA 2.1 and higher versions .  kindly perform an upgrade to TCA 2.1 and higher to fix the issue.

If you wish to continue with TCA 2.0.1 , kindly perform the steps as mentioned Workaround section to address the issue.

Workaround:

Update the credentials in the DB. 

a. Add the Harbor credentials manually into the TCA Manager database or upgrade TCA to the latest version.

Procedure

1. SSH into the TCA Manager appliance using the admin account.

2. Switch user to root user

su -


NOTE: Backup the DB before update/edit
 

Backup:

mongodump --archive=/tmp/mongoExport/mongo_hybridity.gz --gzip --db hybridity --excludeCollection=audittrail

TO RESTORE (IF APPLICABLE) 

mongorestore --gzip --archive=mongo_hybridity.gz --db hybridity

3. Use the following commands to edit the database:

NOTE: Update harbor username and password for multiple NFs in TCA-M. You will need to replace <<entityId1>> with the actual  entityIds from the DB and run the command for that particular nfInstanceId. Please replace <<harborUsername>> and <<harborPassword>> with the actual values.

mongo hybridity
db.Vnf.updateMany({"id":"<<nfInstanceId>>"},{"$set":{"instantiatedNfInfo.<<entityId1>>.username":"{{harborUsername}}","instantiatedNfInfo.<<entityId1>>.password":"<<harborPassword>>" }})

 

Example : 

db.Vnf.updateMany({"id":"f99dfd9b-a8f3-44f3-a9bf-35e46196ce88"},{"$set":{"instantiatedNfInfo.log-distributor-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.log-distributor-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.reprocessor-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.reprocessor-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.dr-redis-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.dr-redis-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.dr-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.dr-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.output-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.output-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.dr-cmdb-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.dr-cmdb-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.configuration-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.configuration-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.mapper-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.mapper-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.input-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.input-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword","instantiatedNfInfo.kafka-output-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.username":"yourusername","instantiatedNfInfo.kafka-output-stream-f99dfd9b-a8f3-44f3-a9bf-35e46196ce88.password":"samplepassword"}})

Additional Information

Impact/Risks:
Unable to Scale/Reconfigure after an CNF upgrade operation.