Cloud Director database size increases unexpectedly after configuring Container Service Extension

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• The VMware Cloud Director(VCD) database is consistently increasing in size.
• The VCD database backups also keeps increasing in size.
• Backing up the VCD database is taking longer than usual, and can fail due to the NFS being full in size as a result of the database having increased in size.
• In the cse.log you see entries similar to:
  
  
  • E0316 15:01:22.461384   12056 auth.go:58] failed to authenticate using refresh token
panic: error logging into VCD: [unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set >authorization header: [Post "https://example/oauth/provider/token": dial tcp: lookup <fully_qualified_domain_name> on xxx.x.x.xx:xx: read >udp xxx.x.x.xx:xx:xxxxx->xxx.x.x.xx:xx: i/o timeout]]]]
    
    
  • error logging into VCD: [unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set >authorization header: [error decoding token text: invalid character '<' looking for beginning of value]]]]
    
    
• The event table in the VCD database is high in size.

Environment

• VMware Cloud Director 10.x
• Container Service Extension(CSE)

Cause

The VCD event table will log all login events.
CSE runs as a service which attempts 2 logins every 5 seconds. Therefore, if there are logins that are continuously failing, it results in the VCD database to grow in size.

Resolution

To resolve the issue you need to reduce the impact that CSE is having on VCD. You will need to make a change in the cse.service file:

1. Edit the cse.service file using the following command: vi /etc/systemd/system/cse.service
2. Change RestartSec from 5s to a larger value of  "5min".
3. Reload the service unit files: systemctl daemon-reload.
4. Restart the CSE service: systemctl restart cse.service.

The VCD database should stop growing in size.

It is also strongly recommended to set up the right credentials for the CSE server to allow the logins succeed. 
If the above workaround  do not help, contact technical support and note this Knowledge Article ID (312278) in the problem description. For more information, see How to Submit a Support Request.