SDDC Manager operations greyed/failed and report error - “Cannot complete the operation in the current state. Try again later.” & "Unable to determine if functionality is allowed/blocked."

Products

• VMware Cloud Foundation

Issue/Introduction

The same error can be reported on multiple appliances during the pre-check, including but not limited to NSX-T, ESXi, etc.
Password remediation, password rotation, APIs, changing NTP settings, host commissioning, etc., options are greyed out and report “Cannot complete the operation in the current state. Try again later.” & "Unable to determine if functionality is allowed/blocked."
The subscription licensing feature was disabled by following the KB SDDC Manager incorrectly reports subscription read-only mode after upgrade to VCF 4.5

Pre-check Error - 



NTP Settings - 


Password Operations Failure - 


In the LCM logs, it reports the error regarding blocked functionality - /var/log/vmware/vcf/lcm/lcm-debug.log

YYYY-MM-DDT10:24:04.991+0000 ERROR [vcf_lcm,258d448af834479a,c5dd,precheckId=########-####-####-####-##########f5,resourceType=PSC,resourceId=########-####-####-####-##########67] [c.v.e.s.l.p.i.vcenter.VCenterCommon,Precheck-6] Exception occurred while performing password validation task for VCenter vcenter_fqdn
com.vmware.evo.sddc.lcm.model.error.LcmException: Password validation execution failed with exception: 403 : "{"message":"Unable to determine if functionality is allowed/blocked.","status":403}".

/var/log/vmware/vcf/operationsmanager/operationsmanager.log

YYYY-MM-DDT10:24:04.987+0000 ERROR [vcf_om,33873c73e247986e,75dd] [c.v.v.f.t.i.FunctionalityToggleApiFilter,http-nio-127.0.0.1-7300-exec-7] Functionality blocked:
com.vmware.vcf.functionality.toggle.error.FunctionalityToggleException: Unable to determine if functionality is allowed/blocked.
Caused by: com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException

/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log

YYYY-MM-DDT10:24:04.982+0000 ERROR [common,33873c73e247986e,75dd] [c.v.v.r.s.util.LicensingStateUtil,http-nio-127.0.0.1-7100-exec-5] Failed to get licensing state. Response code 404
YYYY-MM-DDT10:24:04.983+0000 ERROR [common,33873c73e247986e,75dd] [c.v.v.r.r.a.c.v.ResourceFunctionalitiesController,http-nio-127.0.0.1-7100-exec-5] Exception occurred when retrieving the resource functionalities.
com.vmware.evo.sddc.common.core.error.InvalidStateException: Error encountered while fetching system licensing state.

Environment

VMware Cloud Foundation 4.5.x

Cause

The commonsvcs service of the SDDC manager determines whether the functionality is allowed or not based on the licensing state. The licensing info is fetched through the feature flag VCF_SUBSCRIPTION, i.e, feature.vcf.plus.subscription. Now, if the flag "feature.vcf.plus.subscription" is set to false, the commonsvcs service fails to determine if the functionality is allowed because it fails to retrieve the licensing state.

Resolution

NOTE - Take a snapshot of the SDDC Manager 

1. SSH into the SDDC Manager and elevate to root with su.
2. Add the following to the /home/vcf/feature.properties file or modify the below property to true in case it is set to false.  (Create the file if it does not already exist)
   • feature.vcf.plus.subscription=true
3. Update the permissions on the feature.properties file.
   • chown vcf:vcf /home/vcf/feature.properties
4. Restart the operationsmanager service
   • systemctl restart operationsmanager
5. Retry the operation from the SDDC Manager UI.