High Number of Sessions Generated from Telco Cloud Automation (TCA) to VMware Cloud Director (vCD)

Products

• VMware
• VMware Cloud Director
• VMware Telco Cloud Automation

Issue/Introduction

Symptoms:

Over time, the number of calls being made from the TCA-CP to vCD increases drastically. This is exacerbated in large scale environments where there are many OrgVdcs available.

The vCD database is heavily utilized due to a large number of incoming calls and vCD performance is degraded due to the increased load on the database.

Environment

VMware Telco Cloud Automation 2.0.1
VMware Telco Cloud Automation 2.1
VMware Telco Cloud Automation 2.2
VMware Telco Cloud Automation 2.1.1
VMware Cloud Director 10.x
VMware Telco Cloud Automation 2.0

Cause

Within the TCA-CP appliance, certain inventory pollers are re-queued and cloned after the restart of the app-engine service.

Resolution

VMware is aware of this issue and investigating on this. Kindly proceed with the Workaround to mitigate the issue.

Workaround:

There are two workarounds that will help alleviate this issue, one is done on vCD while the second is done on the TCA-CP.

Reduce Audit Trail retention on vCD

Follow the procedure outlined in Limit audit_trail to improve vCloud Director DB Performance (89933) .

Remove duplicate inventory pollers from TCA-CP

1. SSH into the TCA-CP appliance.

2. Switch user to root.

su -

3. Stop the app-engine service by running the following command:

systemctl stop app-engine

4. Log into MongoDB by running the following command:

mongo hybridity

5. Delete Jobs in Mongo with the following commands:

db.Job.find({"jobType":"VcdInventoryJobType","jobData.entityType":"adminOrgVdcStorageProfile"}).count()
db.Job.deleteMany({"jobType":"VcdInventoryJobType","jobData.entityType":"adminOrgVdcStorageProfile"})

6.Steps for Service Config change:

a. Change directory to /opt/vmware/deploy/zookeeper/

b. Edit file VcdInventoryService.zql to change the frequency value under statsFeeder to 86400. This value is in seconds.

c. Edit file VcStatsService.zql to change both, the frequency value (under statsFeeder) and monitorIntervalInSecondsUnder to 86400. This value is in seconds.

d. Under the same directory edit the file createZkNode.sh and change rmr to deleteall.

e. Save the file.

f. Under the same directory execute the following command:
./createZkNode.sh VcdInventoryService

7. Restart the app-engine service with the following command:

systemctl restart app-engine

Note: Do not restart the app-engine service on the TCA-CP appliance frequently as this will create the cloned jobs again.

Additional Information

Impact/Risks:
Impacts vCD 10.3.x and TCA 2.0.x, 2.1.x, 2.2.x