VMware Cloud Director database event table has increased in size after configuring the Container Service Extension plugin.

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• The VMware Cloud Director(VCD) database is consistently increasing in size.
• Backing up the VCD database is taking longer than usual, and can fail due to the NFS being full in size as a result of the database having increased in size.
• In the cse.log you see entries similar to:
  • E0316 15:01:22.461384   12056 auth.go:58] failed to authenticate using refresh token
    panic: error logging into VCD: [unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set >authorization header: [Post "https://example/oauth/provider/token": dial tcp: lookup <fully_qualified_domain_name> on xxx.x.x.xx:xx: read >udp xxx.x.x.xx:xx:xxxxx->xxx.x.x.xx:xx: i/o timeout]]]]
  • error logging into VCD: [unable to get swagger client from secrets: [unable to get bearer token from secrets: [failed to set >authorization header: [error decoding token text: invalid character '<' looking for beginning of value]]]]
• The event table in the VCD database is high in size. 
  • Run the following query to check the size of the event table:  

select count(*), event_type from event where cell_id isnull group by event_type;

Environment

VMware Cloud Director 10.x

Cause

The VCD event table will log all login events
CSE runs as a service which attempts 2 logins every 5 seconds. Therefore, if there are logins that are continuously failing, it results in the VCD database to grow in size.

Resolution

To reduce the impact that CSE is having on VCD you need to make a change in the cse.service file:

1. Edit the cse.service file using the following command: vi /etc/systemd/system/cse.service
2. Change RestartSec from 5s to a larger value of "5min"
3. Reload the service unit files: 'systemctl daemon-reload'
4. Restart the CSE service: 'systemctl restart cse.service'

The VCD database should stop growing in size.

It is also strongly recommended to set up the right credentials for the CSE server to allow the logins succeed.