During the "Deploy and configure VCF Automation" (VCFA) step in a VCF 9.x deployment, the task becomes stuck at 2/8 Completed for an extended period.
The following logs are observed:
kube-vip-vcf-vsp01-#####/kube-vip/console-log-##########.log:<TIMESTAMP> stderr F <TIMESTAMP> ERROR lost leadership, restarting kube-vip<TIMESTAMP> stderr F E0611 <TIMESTAMP> 1 clusterLeaderElection.go:246] "Observed a panic" panic="" stacktrace...
vcf-fleet-lcm-db-0/postgres/console-log-##########.log:<TIMESTAMP> stdout F <TIMESTAMP>ERROR: ObjectCache.run K8sConnectionFailed('No more API server nodes in the cluster')
vcf-fleet-build-service-fleetbuild-##########-#####/fleetbuild/console-log-##########.log<TIMESTAMP> stdout F <TIMESTAMP> WARN Fleet Build 53 [lcm@4413 scPath="ops" thread="pool-26-thread-1" logger="c.zaxxer.hikari.pool.ProxyConnection" method="checkException:L"] [] - HikariPool-1 - Connection org.postgresql.jdbc.PgConnection@79a7009e marked as broken because of SQLSTATE(08006), ErrorCode(0)<TIMESTAMP> stdout F org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
VCF 9.1
The deployment workflow stalls when an unexpected kube-vip restart cuts connectivity to the vcf-fleet-lcm-db. As a result, the fleet build service cannot update the database with the VCFA deployment task status. These issues are often made worse by network instability during deployment.
1. Verify Database Health:
Check the status of the "vcf-fleet-lcm-db" members to ensure a leader is present and there is no significant replication lag. (See details in KB 445455)kubectl -n vcf-fleet-lcm exec vcf-fleet-lcm-db-0 -- patronictl list
2. Restart Services:
If the task remains stuck, refer to KB 435569 to restart the necessary VCF services.
3. Retry Deployment:
Once the services are restarted and the stuck task formally fails in the UI, ensure the physical network is stable and retry the "Deploy and configure VCF Automation" task.