The vc-ws1a-broker service fails to start after creating a backup of the service's application.properties file and restarting the service

Products

• VMware vCenter Server

Issue/Introduction

• When attempting to perform the Resolution steps described in KB 396939, copying the application.properties file into the same directory and then restarting the vc-ws1a-broker service causes the service to fail to start.
• Even after deleting the backup file of application.properties, the vc-ws1a-broker service still fails to complete the startup process.
• The following message is output to /var/log/vmware/ws1-init-container/install.log:

XXXX-XX-XXTXX:XX:XX.XXXXXXZ Starting installation...
XXXX-XX-XXTXX:XX:XX.XXXXXXZ Preserving encrypted client secret for service: federation
XXXX-XX-XXTXX:XX:XX.XXXXXXZ Preserving encrypted client secret for service: usergroup
XXXX-XX-XXTXX:XX:XX.XXXXXXZ Preserving encrypted client secret for service: token
XXXX-XX-XXTXX:XX:XX.XXXXXXZ Preserving encrypted client secret for service: accesscontrol
XXXX-XX-XXTXX:XX:XX.XXXXXXZ Preserving encrypted client secret for service: crypto
XXXX-XX-XXTXX:XX:XX.XXXXXXZ ERROR Failed to copy the default service configuration. Exit code is 1

Environment

vCenter Server 8.0

Cause

When the vc-ws1a-broker service starts, the process of copying configuration files, including application.properties, is executed by a non-root user.

If the root user places a file (such as a backup) in the same directory as application.properties before the service starts, the copy operation subsequently fails due to a permission error for the non-root user. Consequently, the vc-ws1a-broker service attempts to start with incomplete configuration information, which leads directly to the service startup failure.

Resolution

Perform the following steps to resolve the issue:

1. Delete the Backup File Causing the Issue

Example:
rm /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/13/fs/opt/vmware/idm/initc/services/token/config/application.properties.backup

Note: Ensure the path specified is the location of the file you actually created.

2. Execute the following commands to back up the existing service data:

mkdir /var/core/ws1_backup
cp -r /var/lib/ws1/* /var/core/ws1_backup

3. Delete the following files to allow the service to recreate the correct configuration state upon restart:

rm /var/lib/ws1/accesscontrol/config/application.properties
rm /var/lib/ws1/crypto/config/application.properties
rm /var/lib/ws1/crypto/config/masterkeys.bcfks
rm /var/lib/ws1/crypto/config/masterkeys.pass
rm /var/lib/ws1/token/config/application.properties
rm /var/lib/ws1/federation/config/application.properties
rm /var/lib/ws1/usergroup/config/application.properties
rm /var/lib/ws1/masterapp-credentials.json

Note: Ignore any error messages that indicate a file does not exist.

4. Restart the service using the following command:

service-control --start vc-ws1a-broker

5. Confirm that the vc-ws1a-broker service is running successfully:

service-control --status vc-ws1a-broker

6. Once the vc-ws1a-broker service has successfully started, delete the temporary data backup created in Step 2:

rm -rf /var/core/ws1_backup

Important Note
The filesystem where application.properties is located is an overlay filesystem essential for the vc-ws1a-broker service configuration. Including unnecessary files in this location will corrupt the service's configuration information.

Do not create files (such as backups or temporary files) on the same filesystem where application.properties is located. Please utilize other directories such as  /var/core for temporary operations.

Additional Information

Japanese article: vc-ws1a-broker サービスの application.properties バックアップファイルを作成してサービスの再起動を行うと、vc-ws1a-broker サービスの起動に失敗する

vCenter services experiencing OutOfMemoryError: Java heap space, causing the swap memory to be 100% utilized and due to which vCenter backup fails with " Error: database or disk is full"