After adding additional storage to Log Management subsequent task starts to fail.

Products

VCF Operations

Issue/Introduction

User successfully adds storage to Log Management however any subsequent task that triggers helm upgrade fails.

Package Deployment fall into a Failed state and the Helm Release into an Unknown state.

Environment

VCF Operations 9.1.0

Cause

Resize storage of Log Management to any value in Gi which is not divisible by 256 causes milli-byte error.

Any additional storage added to Log Management triggers volume-resizer however the volume resizer python script throws an error in subsequent volume resizer check.

For example:
Considering recent storage extension of 3500 GB to Log Management.

Fleet Manager translates 3500Gi into raw bytes: 3758096384000 = 3500*1024*1024*1024 and sends this payload to the VMSP endpoint which formats raw bytes into human-readable strings (like Gi or Ti). It takes 3758096384000, divides it by 1024^4, gets the output "3.41Ti".

The ops-logs config map volume resizer script reads "3.41Ti" from the Package Deployment, passes it to Kubernetes, Kubernetes API server converts 3.41Ti to bytes which results in a fractional/decimal value. As Kubernetes refuses to store fractional bytes, it converts the value to "milli-bytes" to maintain precision as a whole integer, resulting in the value "2682808371773440m".

The ops-logs config map volume resizer script reads "3.41Ti" from the Package Deployment, passes it to Kubernetes, and Kubernetes creates the milli-byte disaster.

Any subsequent task that triggers a Helm upgrade again invokes the volume resizer python script throwing an error.

The volume-resizer Python script crashes while reading "milli-byte" string ("2682808371773440m") caused volume-resizer pods to enter in a crash loop and the Package Deployment fall into a Failed state and the Helm Release into an Unknown state.

Resolution

This issue will be fixed in future release.

To resolve the issue, follow below steps:

Login into VCF Operations and go to Lifecycle > Build > VCF Management > Component > VCF services runtime and Identify a Control Plane node.
Establish an SSH session to one of these nodes, login with the vmware-system-user account, and elevate to root using below command:
```
sudo su-
```
Check whether the volume-resizer pod is in failed state.
```
kubectl get po -n ops-logs
```
Delete any Running or Queued tasks for ops logs,
```
kubectl get tasks -A | grep -E "Running|Queued"
```

Check payload of task to confirm its ops-logs related task,

kubectl get secrets  task-xxxxxxxxx -n vmsp-platform -o json | jq -r '.data.apidata' | base64 -d | jq

Delete these tasks

kubectl delete task <e.g.-configure-component-vcbul23jwrfybiw4w3ed4hje6y> -n vmsp-platform

Verify no tasks running or in pending state,

kubectl get tasks -A | grep -E "Running|Queued"

Disbale drift detection temporarily

kubectl annotate cm ops-logs-config -n ops-logs helm.toolkit.fluxcd.io/driftDetection=disabled

Keep a backup of volume-resizer-config-map

kubectl get cm volume-resizer-config-map -n ops-logs -o yaml --show-managed-fields=false > volume-resizer-config-map.yaml

Edit volume-resizer-config-map. Put a try/catch around, old_size_num = self.from_iec_to_number(old_size)

kubectl edit cm volume-resizer-config-map -n ops-logs

       try:
        old_size_num = self.from_iec_to_number(old_size)
      except:
        old_size_num = 500
        pass

Edit package deployment

kubectl edit pd operations-logs -n ops-logs

Update logStoreLogStore.capacity to as per customers input value e.g. 3500Gi. Now the volume resizer pod should successfully completed.

Validation

kubectl get pd operations-logs -n ops-logs -o jsonpath='{.spec.values.storage.volumeGroups.logStoreLogStore.capacity}'
kubectl get sts log-store -n ops-logs -o jsonpath='{.spec.volumeClaimTemplates[0].spec.resources.requests.storage}'

Re-Enable back the drift detection

kubectl annotate cm ops-logs-config -n ops-logs helm.toolkit.fluxcd.io/driftDetection=enabled --overwrite

Additional Information

Recommend to configure a storage size that resolves to a whole integer in bytes, avoiding fractional byte conversions in Kubernetes. Ensure your desired volume size is perfectly divisible by 256.
Use the following formula:

Volume size = N * 256