Some configuration changes to a deployment or tile were lost or not persisted in Ops Manager

Products

• VMware Tanzu Application Service

Issue/Introduction

Configuration changes to a deployment were not persisted in Ops Manager.  The configuration changes were executed via pipeline using Platform Automation.  The configuration changes were lost even though the configure-task pipeline task completed successfully without any error.  In the Ops Manager production.log file, all the related PUT requests to the resource_config API endpoint also showed a 200 OK response code.

Environment

Tanzu Platform for Cloud Foundry

Operations Manager

Platform Automation Toolkit for Tanzu

Cause

It appears that the data loss was due to overlapping or simultaneous PUT requests to the Ops Manager resource_config API endpoint, which are not supported due to how the configuration state is backed by an on-disk YAML file (installation.yml).  This particular API endpoint is called when the configure-product task is executed.

In the case of overlapping write requests, a request that was first to complete is likely to "lose" the changes because another request could have read the file prior to the first request having made changes.  So, the first request would have succeeded and its changes written but would have been overwritten immediately when the second request completes.

The overlapping write requests can be confirmed to have occurred by reviewing the Ops Manager production.log file, wherein it would show simultaneous multiple PUT requests to the resource_config API endpoint.

Resolution

As a workaround, running the configure-product task on the product again (while there are no overlapping write requests) would address the issue.

For a more permanent solution, the pipelines should be coordinated to prevent simultaneous or overlapping write requests.