Vmware Cloud Foundation SDDC Manager upgrade to 4.5 fails and upgrade pre-checks fail with "sudo" permissions issue

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

VCF SDDC Manager upgrade to 4.5 fails and upgrade pre-checks fail with "sudo" permissions issue.

/opt/vmware/vcf/lcm/lcm-debug.log has following error:
"sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?"

Environment

VMware Cloud Foundation 4.5.2
Vmware Cloud Foundation 4.5.1
Vmware Cloud Foundation 4.5

Cause

The issue happens because Common Appliance Platform (CAP) and Non-CAP based SDDC manager upgrades are interleaved in following manner:

1. Upgrade attempt to 4.5 fails (CAP based upgrade). SDDC Manager is not restored to previously taken backup. CAP based upgrade workflow has added 2 new disks (alt-root and lvm-snapshot) to the SDDC Manager VM.
2. Instead of upgrading to VCF 4.5 customer decides to upgrade SDDC Manager to VCF 4.4.1.1 (Non CAP based upgrade). The upgrade is successful. This upgrade runs STIG Hardening scripts that add "nosuid,nodev" mount options including to the newly added disks alt-root and lvm-snapshot.
3. Upgrade to VCF 4.5 is attempted again. Since the alt-root and lvm-snapshot disks are already present, CAP workflow tries to re-use them. As part of the CAP based upgrade to 4.5, the newly added disk becomes the / file system after reboot. But since the disk had "nosuid,nodev" options added by STIG Hardening, no command with "sudo" permissions can be run on it any more, resulting in the above mentioned error and upgrade failure.

Resolution

Currently there is no resolution to the issue.

Workaround:

If upgrade to 4.5 fails and then a successful upgrade to 4.4.1.1 has been done, there is a chance that "alt-root" and "lvm-snapshot" disks exist on the system and have "nosuid,nodev" mount options set. In this case open /etc/fstab file as "root" and update the mount options. Please follow the below mentioned steps to update the mount options:

1. Open /etc/fstab as "root".

2. The options will be as mentioned below:

UUID=051c8c3a-f2e3-4523-98b1-e8ece2f621ee /  ext4 ,nosuid,nodev
UUID=74c9842b-9abe-4d09-8023-d310cbd54275 /storage/lvm_snapshot ext4 ,nosuid,nodev

Change it to:
UUID=051c8c3a-f2e3-4523-98b1-e8ece2f621ee /  ext4 defaults 1 1
UUID=74c9842b-9abe-4d09-8023-d310cbd54275 /storage/lvm_snapshot ext4 defaults 1 1

Save and close the file.

3. Run "findmnt --verify" to check for correctness of the changes done.

4. Reboot SDDC manager.