VCF Automation deployments utilizing the content library fail if the deployments take longer than 15 minutes to provision in vCenter
Article ID: 397004
Updated On:
Products
Issue/Introduction
This KB provides steps to resolve an issue that may be encountered on Content Library based deployments.
Symptoms:
When provisioning VMs using templates from vCenter content library, the requests fail in VCF Automation with the following error:
Provisioning operation failed. Error deploying from library: [java.util.concurrent.CompletionException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: [POST] [https://vcenter.server/rest/vcenter/vm-template/library-items/4ce1b73e-c39e-443b-ab64-b6400d2c958c?action=deploy] [[Lorg.apache.http.Header;@2fcca8f5]: Failed with exception: [Connection is closed]Environment
VCF Automation 9.0
Cause
This is caused by the internal VCF Automation proxy connection timing out the vAPI connection to vCenter. The internal proxy service of VCF Automation has a default active connection timeout of 15 minutes.
If it takes more than 15 minutes for a VM to be deployed in vCenter, the proxy connection is closed resulting in the failure of the deployment request.
The provisioning-service-app.log would show the following error:
####-##-##T##:##:##.382Z WARN provisioning [host='provisioning-service-app-##########-#####' thread='xn-index-updates-17' user='' org='' trace='' parent='' span=''] c.v.a.r.c.ComputeProvisionTaskService.handleFailedStagePatch:1282 - [/provisioning/requests/compute-provision-tasks/########-####-####-####-############] Task failed with: Provisioning operation failed. Error deploying from library: [java.util.concurrent.CompletionException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: [POST] [https://vcenter.server/rest/vcenter/vm-template/library-items/4ce1b73e-c39e-443b-ab64-b6400d2c958c?action=deploy] [[Lorg.apache.http.Header;@2fcca8f5]: Failed with exception: [Connection is closed]Resolution
Increase the connection timeout of the internal VCF Automation proxy service from the default 15 minutes to something appropriate to the longer running VM deployments in the environment.
Prerequisites:
1. Create a backup of the VCFA component
2. Prepare SSH credentials for the VCFA nodes
Remediation steps
1. SSH into one of the nodes of the VCFA appliance
2. Execute the following command replacing the <timeout-in-minutes> placeholder with the appropriate value (must be a positive integer).
base64 -d <<<"/Td6WFoAAATm1rRGBMDtDeUwIQEWAAAAAAAAAB+H4rbgGGQG5V0AEYhCRj30GGqmZ696n3KPnE7ymtBruh8SVHTB/E8V42qo2cf1kRcwYlxKc1vh0hkEkahoP9Y6s4a0UtFM6eaZLvHxsoEvsHvhGlOTLndpzFg368F46H6UgVdPO4Gy4Egkd76aLVnMkH32C0mkxC6xz6IJEkqggqt1yv5lExkPmdeQ0zmP94UjSpKhSFKECc98Q2hZghENHl23eAfwNKyzFfbQIgztdenbzv3HnAshWrNok+B85dc9z7i8xnwI42IggCj0d3/a8Vk5lqZOMPQAsy3eD540L9dJWzXFgmge2VdDQ41zYSSB2Uqfntuv5tWcEAje0wfyf6Mh8R/W1VahCy7TzVjcOKZXoPqsYIfV5A648j8LRFP6/EZGbkSac10RlUq8LWeUKAegAv76GKoeQfFAM0LJEHQQkmnb9LD98GNTCJOTUC1H2NSF45j5Lwlym4g3T+OuYTaN1nR6LiJLCojzz9GTSUCgByv9wOuM4+AHkLjT997XIJH8EPxUBc6sH40T4acvHc2YE6SZUKgDlqyzXqybkzmv/kD7Bs51KHWODlFd33neCfzhKAnMJrCFiOXWuup/eWTAYcc0E5vovhrAy7rnCkh8ahzI8hMsaBJ2hw088T8QTx+1jWP+00WsLlXAlSKPvHWINlXZ9YDO7xIaAoRz6E2/OZrm4faGBPx+IH57wmnzWpj710H0hoNsWPDeyC4UPOCxlNp+sFOW2fAX2aVEXPh7Es8nuwWKWVgpK5327SOJ3x5olxvzaDl+1ZvCf4Tj+JuKhwREROwrxeMag6rCe6yIhUhBaIxjc9ui8bxbZwYm6aQ5F1Dcu1UkMnR2twj8iW0QfMqRXPfCzGLNOd5CYUghlO+AzHN81QKlmWYDwH3eXYN02dCqcuG+Qiwfby2pROGPHAWxLtqxccbbBPhxXIPEFjUtP2EepVAqy060GLXRGIruNm6i1srbSdACVEpFSwyPXiZs6sKZksuuYzjfhg+gOL1atCX4ZLxI6GedDBJfe2MpSEdMIfuA2UVc53Ffy21FVMm4YXNiP1rBSGT3lYBtReS8JBQEsKe2pypaUGNvkmoxQCnQcXJ5n5jj7xK9cMJbr9bd0nStQRO/d8XZNNXfN0XVwrXjVceGxVQmaKyyk7mZ0iauXEYGSptk11gZP1kDUlS7LB7sOmH7iS2hpQgwHLAnLOQ1FbesitLxm02FNBX9VVKbet8METQcJgwIpP4uDo4kNFol3Ktttdh++/Ab3cR1Lkajfq0oSvHwDasrn1GGewKuMDU1OZfeLQBYN2iQlV3j/YZkZ8uizp58PbT7v3tnw/Ly2H60/A3+fZRazhgSMtilF1jVuiO48OV8ynupId8mMAEPvQU60s8amwanL0Mil8kNUEbyKcaXEfj6TDZglwGnvJptzO5flq0iUbL/1j5tJ/qLsi+K0gc+Gh4gIX9GSypG7P68YD8sF7M43w675DEyxVREzK8Ht9Oh/hQsu9o5j/PZbE45fNcGq5oF+OsumjM62bw4xHGNEiU/pHAZ9JR8aTt94psVSPV2NpC1wB07aYLOF6QyJHQl2CStkxqmbJT7h0kRGAqL2WohFtinqbKBlsUhxrhnTP9LIv3lTIIEcVDmOVfydHAbef1ycSLVJnMftULMx12a1aG++FlzhwBKXYTNEi1AZSUDNIOEHNP6kK0bvxm01kNiDm8+9ppo6VoNvSFdDZH5eMf1YsURZ+ahHdAjwEvcUw0YAHXPLb8GS3ALYOx2axDj8l3fvQyY2EkfwibOe6l4u+k0aoaCZu/2pDtFhFp34MuCexhPf2AEc1g2v8bzfyh/DskaqLzQxsQg+D1rZHPtBGe7BspsJ7pqtB1b5MSlf6ioN3VjfiNtxMfBJWc66fAe0Fr5CBca76G8nfSb/isu5/h9ZKNGhuDJa9E/6BcdhDnTbsgemP0SW4Chba74dkbTRArcEP31hLK34DYwOaJpSIyQU0hJJhlnfjCzwCdJpcOvJFs1HiqxQHaGHUmMBSXbaDnNoafEfsZJiJGnp8oGncGjTcpFALTogg655nc4aa1Ec3w3WPyxzsY/bMp/B+qAzmco9UAXgxbIMpIgQuYyBuX5sVZjCja/Dh+f1CpUTEDuyJ9s1YMEadSI2dsaXmvrL0YNIDe0OeEBIKnRJFSdRZ0uOGwPgXdg/e/5vIvDkIYMoKtKccnBQt7kQcQAVacgvzywBxjY6Seo1/SCWc01ZMbYrVZqtvMA5WKSajok8ldZVYrirJpCh9o/YAvP7S64JbHtDuARD6SqV87jmDfd1jjSiEEeWC30OmhjroRMvxpBjdUrIsVI4LFzvDTVFpz/VxuuuPrc/wO3v3DJkHu+MQAAAACGDd+i+Dg4HwABiQ7lMAAANnsQgLHEZ/sCAAAAAARZWg==" | unxz | sudo bash -s <timeout-in-minutes>Feedback
