Error: "Server is busy. Please try again later" when accessing Cost Drivers in VCF Operations 9.x

Products

• VCF Operations

Issue/Introduction

In VCF Operations 9.x, cost calculations fail across data centers. When navigating to Infrastructure Operations ➔ Configurations ➔ Cost Drivers, the Cost Drivers tab displays the following error:

Server is busy. Please try again later. If issue still persists, contact the system administrator.

Following entries are found in /storage/log/vcops/log/bm-controller.log:

java.util.concurrent.CompletionException: java.lang.NullPointerException: Cannot invoke "java.lang.Double.doubleValue()" because the return value of "com.vmware.vrbc.expenses.document.OSLicenseExpense.getRate()" is null
at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.postFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.exec(Unknown Source) ~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source) ~[?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) ~[?:?]
at java.util.concurrent.ForkJoinPool.scan(Unknown Source) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source) ~[?:?]
Caused by: java.lang.NullPointerException: Cannot invoke "java.lang.Double.doubleValue()" because the return value of "com.vmware.vrbc.expenses.document.OSLicenseExpense.getRate()" is null
at com.vmware.vrbc.costengine.costcalculation.costdriver.server.LicensesCostDriver.getTotalCost(LicensesCostDriver.java:713) ~[?:?]
at com.vmware.vrbc.costengine.costcalculation.costdriver.server.LicensesCostDriver.lambda$calculate$5(LicensesCostDriver.java:173) ~[?:?]

Environment

• VCF Operations 9.x

Cause

This issue occurs when cost drivers are imported from a 8.18.x product version environment, causing the VCF / VVF policy types to be written with NULL values in the database. When the application attempts to perform calculations on a policy type that contains a NULL cost value, the calculation fails.

Resolution

To resolve this issue, log into the database and update the NULL cost values to 0.0:

1. Take a snapshot of the VCF Operations cluster nodes before performing database modifications.
2. Log in to the primary node as root via SSH.
3. Switch to the postgres user and access the vcopsdb database by running the following commands: 

   su - postgres
   /opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433

4. Verify the presence of the NULL cost rows inside the refliboscost table: 

   select * from refliboscost where cost is NULL;

5. Update the NULL values to 0.0 by executing the following query: 

   update refliboscost set cost = 0.0 where cost is NULL;

6. Return to the VCF Operations user interface and verify that the Cost Drivers tab loads correctly and calculations complete successfully.

Additional Information

 It is not an HTTP POST payload limit issue. Modifying the payload limit (such as the steps outlined in KB Unable to save cost drivers, "Server is busy. Please try again later. If issue still persists, contact the system administrator." is seen when attempted. (367511)) will not resolve this error.