HCX - Root Partition behavior during system Upgrade Cycle

Products

• VMware HCX
• VMware Cloud on AWS

Issue/Introduction

Post an upgrade, user may notice multiple partitions for a given HCX Connector OR Cloud Manager as shown below:

admin@hcx-enterprise [ ~ ]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        5.9G     0  5.9G   0% /dev
tmpfs           5.9G   64K  5.9G   1% /dev/shm
tmpfs           5.9G  796K  5.9G   1% /run
tmpfs           5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/sda3       5.7G  4.4G  983M  82% / >>>>>>>>>>>>>>>
/dev/sda1       237M  228K  224M   1% /recovery
/dev/sda6        44G   26G   17G  61% /common
/dev/sda2       5.7G  4.9G  466M  92% /slot1 >>>>>>>>>>
tmpfs           1.2G     0  1.2G   0% /run/user/1000

Or,

admin@hcx [ ~ ]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        5.9G     0  5.9G   0% /dev
tmpfs           5.9G   64K  5.9G   1% /dev/shm
tmpfs           5.9G  796K  5.9G   1% /run
tmpfs           5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/sda2       5.7G  4.4G  983M  82% / >>>>>>>>>>>>>>>
/dev/sda1       237M  228K  224M   1% /recovery
/dev/sda6        44G   26G   17G  61% /common
/dev/sda3       5.7G  4.4G  983M  82% /slot2 >>>>>>>>>>
tmpfs           1.2G     0  1.2G   0% /run/user/1000

Resolution

HCX Manager uses /sda2 & /sda3 filesystems for root.
During HCX Manager version upgrade, we perform round robin between /sda2 and /sda3 for root partitions.
Note: On a brand new install, /sda2 filesystem always represented as "/" and /sda3 remains completely empty.

When user upgrade an existing deployment very first time, the /sda2 becomes "/slot1 or  2" and /sda3 gets the new OS and becomes "/",
Similarly, when the user upgrade the system second time, the slot gets swapped back as /sda2 becomes "/", whereas /sda3 is now your "/slot1 or 2" and so on.

IMPORTANT: "/slot1 or 2" is a backup of the root filesystem state prior to upgrade and not used in any workflow. It is a placeholder for the next time when user performs an upgrade.

Note: If "/slot1 or 2" is observed to be full after an upgrade, then it implies some log file would've filled "/" prior to the last upgrade cycle.
Refer to VMware Knowledge Base 89739 for one of possible condition when "/slot" can reach 100%.

It is fully safe to unmount the partition "/slot1 or 2" from a running HCX system.
Below steps can be performed for unmount:

• Login to HCX Manager admin and then switch to root shell.
• Use "umount /slot1" or "umount /slot2" to remove the partition.

admin@hcx-enterprise [ ~ ]$ su -
Password: 
root@hcx-enterprise [ ~ ]# umount /slot1

root@hcx-enterprise [ ~ ]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        5.9G     0  5.9G   0% /dev
tmpfs           5.9G   64K  5.9G   1% /dev/shm
tmpfs           5.9G  796K  5.9G   1% /run
tmpfs           5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/sda3       5.7G  4.4G  983M  82% /
/dev/sda1       237M  228K  224M   1% /recovery
/dev/sda6        44G   26G   17G  61% /common
tmpfs           1.2G     0  1.2G   0% /run/user/1000

Note: The above step will help to remove "/slot1 or 2" from "df -h" output, but the same slot can be recovered during every reboot cycle. To prevent this, user is suggested to follow below additional steps:

• Go to root shell again.
• Go to "/etc/fstab" directory.
• Edit the file and comment out the line which contains "/slot1 or 2" entry.

root@hcx-enterprise [ ~ ]# vi /etc/fstab
#system mnt-pt type options,nosuid,nodev dump fsck
/dev/sda1 /recovery ext3 rw,nosuid,nodev,exec,auto,nouser,async 0 0
/dev/sda5       swap    swap    defaults        0       0
/dev/sda3 / ext3 defaults 1 1
/dev/sda2 /slot1 ext3 rw,nosuid,nodev,exec,auto,nouser,async 1 2 >>>>>>>>>>
/dev/sda6 /common ext3 rw,nosuid,nodev,exec,auto,nouser,async 1 2
/common/logs/ossec /var/ossec/logs none defaults,bind 0 0

Use VI Linux Editor to edit and add "#" at the beginning to comment out that line.

#/dev/sda2 /slot1 ext3 rw,nosuid,nodev,exec,auto,nouser,async 1 2

IMPORTANT: When user performs upgrade to HCX system in the given state, the same process will occur in the backend and system will re-add "/slot1 or 2" for the backup purposes again, which is completely normal and expected.