NULL Bytes observed in files written two SMB Volume Services on Jammy Stemcell

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

When writing files to an SMB volume shared via diego volume services we will observe random null bytes in large batches. There is a known bug in kernel version 6.5 which results in padded zeros being appended to misaligned write operations.  

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2049634

 

vcap@7d771110-7c7e-43c3-4d8b-e4ff:/var/smb/public$ grep -Pa '\x00' test.txt-0 | xxd
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000100: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000110: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000120: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000130: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000160: 2d61 3b23 305e 2d70 7c72 567d 4d66 5134  -a;#0^-p|rV}MfQ4

Resolution

This problem is observed on jammy Stemcell using ubuntu HWE v6.5 kernel. Use the following "uname" command on BOSH deployed Diego cell to know what kernel version your Stemcell is using.  BOSH Jammy Stemcell version 1.404+ resolves this issue given we have reverted the kernel to v5.15 LTS release line. The v5.15 LTS release line is more stable and does not observe this behavior.  Upgrading the latest jammy patch release should resolve this problem. 

version 6.5 indicates the stemcell is vulnerable to smb corruption.  If your stemcell shows version 5.5 then you are not exposed to this bug

~$ uname -r
6.5.0-18-generic