RabbitMQ On Demand Smoke Test Fail with error one or more resources within could not be deleted

Products

• VMware Tanzu Application Service

Issue/Introduction

When installing RabbitMQ for Tanzu Application Service (TAS), smoke tests are run during installation to confirm system health. The smoke tests perform the following for each available service plan:

• Targets the org system and creates a space to run the tests.
• Deploys an instance of the CF RabbitMQ Example App to this space
• Creates a RabbitMQ for Tanzu Application Service (TAS) service instance and binds it to the CF RabbitMQ Example App
• Checks that the CF RabbitMQ Example App can write to and read from the Tanzu RabbitMQ for Tanzu Application Service (TAS) service instance
• Cleans up all deployed application and all its service bindings.
• Finally, the cf space is deleted.

Cause

When attempting to perform the final step of the smoke test, the space can't be deleted with error:

32m[2020-07-09 19:02:10.98 (UTC)]> cf delete-space -f -o system rmq-smoke-tests-1-SPACE-abcdefghi [0m
Deleting space rmq-smoke-tests-1-SPACE-abcdefghi in org system as tile_installer...
Job (1234567-8910-1112-1314-1516171819) failed: Deletion of space rmq-smoke-tests-1-SPACE-abcdefghi failed because one or more resources within could not be deleted.
An operation for service instance rmq-smoke-test-instance-xxxxxxxxxx is in progress.
An operation for service instance rmq-smoke-test-instance-xxxxxxxxxx is in progress.
FAILED
[32m[2020-07-09 19:02:14.17 (UTC)]> cf logout [0m
Logging out tile_installer...
OK

[38;5;243m/var/vcap/packages/cf-rabbitmq-smoke-tests/src/rabbitmq-smoke-tests/tests/smoke_tests_suite_test.go:47[0m
[38;5;9m[FAILED] No future change is possible. Bailing out early after 3.173s.
Failed to delete space

Resolution

This error is intermittent and results from a delay in deleting vm's in the backend IaaS due to resource consumption. Once the resource is free, the smoke test errand can be run again and completes successfully.