Error "An internal server error was encountered" while retrieving data from the large OSE bucket folder in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

• The system can load data from smaller folders (30-40 GB), but encounters issues with larger folders (300-400 GB).
  
  
  
  
• Performing a sync from VMware Cloud Director (VCD) tenant portal fails with the error below:
  
  com.vmware.vcloud.api.presentation.service.BadRequestException 
  
  
• Running the command 'ose config validate' on the VMware Cloud Director Object Storage Extension (OSE) successfully validates the configuration of OSE.
  
  
• In /opt/vmware/voss/support/ose-default.log on the OSE server 'NullPointerException' entries are present with a stack trace similar to:
  
  [DefaultQuartzScheduler_Worker-1] ERROR org.quartz.core.JobRunShell - Job DEFAULT.UserKeyBroadcastJob threw an unhandled Exception:
java.lang.NullPointerException: Cannot invoke "String.startsWith(String)" because the return value of "com.vmware.voss.data.model.UserKeyInfo.getOwnerId()" is null
        at com.vmware.voss.service.impl.RegionMgmtServiceImpl.lambda$broadcastUserKey$26(RegionMgmtServiceImpl.java:908)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921

Environment

VMware Cloud Director 10.x
Object Storage Extension 3.x

Cause

In previous versions of OSE, Spring Hibernate V5 was used to efficiently handle large datasets in PostgreSQL. This involved generating SQL queries and retrieving estimated object counts. However, when OSE upgraded to Spring Hibernate V6 in version 3.x, this function was broken.

Resolution

This issue has been fixed in Object Storage Extension 3.1.0.1. VMware Cloud Director Object Storage Extension 3.1.0.1 Release Notes

This is a known issue affecting earlier versions of Object Storage Extension.

Workaround

To minimize the impact, it is possible to turn off count estimation following below steps:

1. Disable count estimation on the OSE server with the provided command: 
   
   ose args set -k oss.object.count.estimate -v false
   
   
2. Restart OSE service:
   
   ose service restart

The workaround will dynamically calculate the total object count when you list objects in a folder.