Provisioning Hosts in a Cell Site Group taking a long time to complete
Article ID: 325400
Updated On:
Products
VMware
VMware Telco Cloud Automation
Issue/Introduction
The purpose of this patch is to resolve the DVS lock handling logic in the TCF manager service in order to allow concurrent host addition to the cell site group.
Symptoms:
When adding hosts to a cell site group they remain stuck in a provisioning state for over 3 hours.
Symptoms:
When adding hosts to a cell site group they remain stuck in a provisioning state for over 3 hours.
Environment
VMware Telco Cloud Automation 2.0
VMware Telco Cloud Automation 2.0.1
VMware Telco Cloud Automation 2.0.1
Cause
The root cause for this issue is that TCF manager service maintains an internal lock while adding hosts to a DVS so that this operation is serialized across different domains. The intent was to not overload the vCenter with bulk requests over time.
However, this inadvertently causes delays when one host gets stuck on a reconfigure dvs task for a long time and subsequent hosts that are added will not be able to acquire the lock as it is not free.
However, this inadvertently causes delays when one host gets stuck on a reconfigure dvs task for a long time and subsequent hosts that are added will not be able to acquire the lock as it is not free.
Resolution
Resolved in Telco Cloud Automation 2.1.
Workaround:
Apply the patch on the TCA manager to prevent this problem from occurring using the following procedure.
1. Create a Snapshot of the TCA Manager.
2. Download the patch file:
vmware-tcf-manager_patch_1.0.tar.gz
3. SSH into the TCA Manager as admin.
4. Back up the folder “/opt/vmware/tcf” in tcf-manager container:
5. Issue the following command:
docker exec -u root -it tcf-manager /bin/bash
6. Change directory to /opt/vmware/
cd /opt/vmware/
7. Backup the tcf folder.
cp -R tcf tcf.bak
8. Copy the backup tcf to /home/admin.
scp -r tcf.bak admin@<TCA-manager>:/home/admin
9. Exit from the container and change directory to /home/admin
10. Change ownership on the file.
chown admin:admin tcf.bak
11. Switch user to root and perform the following steps:
su root
systemctl stop tcf-manager
docker rm tcf-manager
docker image rm vmware-tcf-manager
12. Change directory and issue the following command:
cd /common/tca-repository/tcf-manager
curl -ko tcf-manager.tar.gz <path to the patch file>
13. Restart the following services:
systemctl restart tcf-manager-deploy
systemctl start tcf-manager
Workaround:
Apply the patch on the TCA manager to prevent this problem from occurring using the following procedure.
1. Create a Snapshot of the TCA Manager.
2. Download the patch file:
vmware-tcf-manager_patch_1.0.tar.gz
3. SSH into the TCA Manager as admin.
4. Back up the folder “/opt/vmware/tcf” in tcf-manager container:
5. Issue the following command:
docker exec -u root -it tcf-manager /bin/bash
6. Change directory to /opt/vmware/
cd /opt/vmware/
7. Backup the tcf folder.
cp -R tcf tcf.bak
8. Copy the backup tcf to /home/admin.
scp -r tcf.bak admin@<TCA-manager>:/home/admin
9. Exit from the container and change directory to /home/admin
10. Change ownership on the file.
chown admin:admin tcf.bak
11. Switch user to root and perform the following steps:
su root
systemctl stop tcf-manager
docker rm tcf-manager
docker image rm vmware-tcf-manager
12. Change directory and issue the following command:
cd /common/tca-repository/tcf-manager
curl -ko tcf-manager.tar.gz <path to the patch file>
13. Restart the following services:
systemctl restart tcf-manager-deploy
systemctl start tcf-manager
Additional Information
After applying the patch, the service account information for the pre-deployed CDC/RDC in TCA Automated Infrastructure reverts to administrator. If something other than the administrator SSO user was used for the service account, then the procedure outlined below is required to be followed to restore the configuration.
Both the Service Account username and password will need to be updated in the tcf-manager. Note: If the administrator SSO user was used these steps are not required.
To update the Service Account username, follow this procedure:
1. Run the following command:
docker exec -it tcf-manager /bin/bash
2. Change directory to /opt/vmare/tcf/rest_api/ and backup the tca_web_rest_client.py file
3. Edit tca_web_rest_client.py and update the username in line number 38
vc_sso_username = parser.getVcSsoUsername(central_site_mgmt_domain)
Example:
vc_sso_username = “username”
To update the Service Account Password follow this procedure:
1. Change directory to /opt/vmware/tcf/ and backup the appliance_config.py file
2. Edit appliance_config.py and go to line number 1167 to update the password:
post_body = {"username": self.username, "password": self.password}
Example:
post_body = {"username": self.username, "password": “password” }
3. Issue the following command:
docker restart tcf-manager
Both the Service Account username and password will need to be updated in the tcf-manager. Note: If the administrator SSO user was used these steps are not required.
To update the Service Account username, follow this procedure:
1. Run the following command:
docker exec -it tcf-manager /bin/bash
2. Change directory to /opt/vmare/tcf/rest_api/ and backup the tca_web_rest_client.py file
3. Edit tca_web_rest_client.py and update the username in line number 38
vc_sso_username = parser.getVcSsoUsername(central_site_mgmt_domain)
Example:
vc_sso_username = “username”
To update the Service Account Password follow this procedure:
1. Change directory to /opt/vmware/tcf/ and backup the appliance_config.py file
2. Edit appliance_config.py and go to line number 1167 to update the password:
post_body = {"username": self.username, "password": self.password}
Example:
post_body = {"username": self.username, "password": “password” }
3. Issue the following command:
docker restart tcf-manager
Attachments
Feedback
Yes
No
Powered by
