The VMware Aria Operations product user interface becomes inaccessible, displaying an "HTTP Status 500 - Internal Server Error
" message.
While the Admin UI remains accessible, the cluster status in the Admin UI shows "Online" and the nodes are stuck in a "Waiting for Analytics" state.
The following errors would be observed in the /storage/vcops/log/casa/casa.log file:
2025-04-22T11:20:27,322+0000 ERROR [ajp-nio-127.0.0.1-8011-exec-14] [WT0001bX] casa.status.RebalanceService:60 - Suite API is unavailable for rebalance status request; assume rebalance not in progress
2025-04-22T11:20:27,591+0000 WARN [ajp-nio-127.0.0.1-8011-exec-18] [WT0001c2] casa.exception.CasaControllerExceptionHandler:273 - general runtime exception com.vmware.vcops.casa.exception.CasaException: Can't get clusters info from the suite api.
2025-04-22T11:21:01,305+0000 INFO [ajp-nio-127.0.0.1-8011-exec-10] [WT0001dy] sysadmin.cluster.DocumentCasaClusterManager:516 - IP address for the uuid localhost is not found.
2025-04-22T11:21:01,308+0000 INFO [ajp-nio-127.0.0.1-8011-exec-10] [WT0001dy] casa.status.EnvironmentRequirementService:194 - Disk partition /data/db space free: 9 GB of 205 GB: 4 %
2025-04-22T11:21:01,308+0000 INFO [ajp-nio-127.0.0.1-8011-exec-10] [WT0001dy] casa.status.EnvironmentRequirementService:194 - Disk partition /data/log space free: 5 GB of 19 GB: 26 %
2025-04-22T11:21:01,260+0000 WARN [ajp-nio-127.0.0.1-8011-exec-11] [WT0001dq] casa.suiteapi.SuiteApiTokenService:266 - Failed to get auth token, Request Id null: org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 500: "{"message":"Internal Server error, cause unknown.","httpStatusCode":500,"apiErrorCode":500}"
2025-04-22T11:21:01,267+0000 WARN [ajp-nio-127.0.0.1-8011-exec-11] [WT0001dq] casa.suiteapi.SuiteApiInternalService:442 - Exception calling suite API GET casa/clusters/state; Request Id null: org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 500: "<!DOCTYPE html><EOL><html><EOL> <head><EOL> <title>Internal Server Error</title><EOL> </head><EOL> <body><EOL> We apologize. Internal server error occurred.<EOL> </body><EOL></html>"
2025-04-22T11:21:01,267+0000 WARN [ajp-nio-127.0.0.1-8011-exec-11] [WT0001dq] sysadmin.status.AnalyticsFinder:53 - Could not retrieve analytics state:com.vmware.vcops.casa.exception.CasaSuiteApiInternalServerErrorException: Internal Server Error from Suite API
The primary cause of this issue is insufficient disk space on the /storage/db
partition of the Analytics nodes.
The casa.log files indicate that API calls necessary for retrieving cluster and analytics state are failing due to this lack of storage.
Add additional storage on the Analytics nodes (Primary, Replica, Data) by following the steps outlined in the following KB: Adding additional storage to a node in Aria Operations.
Note: Please refer the following KB for further troubleshooting: Troubleshooting Storage Issues in Aria Operations.
The casa.log
file on the Aria Operations nodes will show errors related to the Suite API being unavailable and failures to retrieve cluster and analytics state information.
Check the disk space utilization of the /storage/db
partition using df -h
command on the Analytics nodes via the command line.