CSE cluster operations do not start and tasks are stuck at 1 %

Products

• VMware Cloud Director

Issue/Introduction

• While trying to create CSE clusters, initiation of VM/vApp do not start and task gets stuck at 1%
• Below error snippets are seen in cse.log:

  goroutine 1 [running]:
  main.main()
          /app/main.go:129 +0x1bf6
  I1016 08:30:37.527750    2617 main.go:55] Connecting to [https://SYSLOG_SERVER_FQDN] as [system:cseadmin]
  I1016 08:30:37.528297    2617 auth.go:52] Using VCD OpenAPI version [36.0]
  I1016 08:30:37.704236    2617 auth.go:76] Running module as sysadmin [true]
  I1016 08:30:37.704375    2617 client.go:204] Client is sysadmin: [true]
  panic: unable to create new connection: [dial tcp xx.xx.xx.xx:514: connect: connection timed out]

• Ping connectivity from CSE to Syslog host fails with packet loss.

Environment

VMware Cloud Director 10.5.1.x

Container Service Extension 4.2

Cause

If the Syslog is configured and if the connectivity between CSE and Syslog fails, CSE cluster operations are blocked.

Resolution

Check if there is any firewall rule on syslog server that blocks traffic (TCP port 514) from CSE server and also from routed org VDC network to Syslog server.

As a workaround, remove the Syslog configuration and continue with the usual CSE operations.

• Log in to VMware Cloud Director provider portal and from the top navigation bar, select More > Kubernetes Container Clusters.
• Click CSE Management > Setting up the Configuration for CSE Server > Start.
• In VMware Cloud Director UI, in the Configure Settings for CSE Server window, in the Set Up Prerequisites section, click Start.
• Click Next. In the Set Configuration Parameters section, remove the entry for the Syslog.

Ideally Syslog server connection issue should not block Cluster CRUD operations. VMware by Broadcom is aware of this issue and working to resolve this in a future release, specifically in CSE version 4.3.0.

Additional Information

Configure the VMware Cloud Director Container Service Extension Server