get-tagassignments and get-spbmstoragepolicy powerCLI commands fail with error com.vmware.vapi.std.errors.unauthenticated

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Unable to run tag related powerCLI commands.
• Other powerCLI commands work fine against the same vCenter.
• Error message on powerCLI prompt:

Get-TagAssignment com.vmware.vapi.std.errors.unauthenticated {'messages': [com.vmware.vapi.std.localizable_message {'id':com.vmware.vapi.endpoint.method.authentication.required, 'default_message': Authentication required., 'args': [], 'params': , 'localized':}], 'data': , 'error_type':UNAUTHENTICATED, 'challenge': Basic realm="VAPI endpoint",SIGN realm=93ba3b703f05e09c9f973b9f9d74be4e51a74aac,service="VAPI endpoint",sts=https://vcfqdn/sts/STSService/vsphere.local}.

At line:1 char:15
+ Get-vm $GVM | Get-TagAssignment
+ ~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-TagAssignment], CisException
+ FullyQualifiedErrorId : VMware.VimAutomation.ViCore.Impl.V1.Service.Tagging.Cis.TaggingServiceCisImpl.GetTagAssignment.Error,VMware.VimAutomation.ViCore.Cmdlets.Commands.Tagging.GetTagAssignment

 

Environment

7.0.x

Cause

• When a client (powercli) makes a call to the vapi-endpoint's introspection service (the com.vmware.vapi.std.introspection.provider vapi service), the vapi-endpoint on its turn makes calls to the introspection service of all the vapi providers behind, including the "infraprofile" vapi provider. We make that call in this release through the sidecar proxy at localhost:1080/infraprofile.
• The infraprofile service though is not in a healthy state. In fact it is a zombie because it has failed with an Out Of Memory error earlier and does not respond to the requests.

Resolution

Increase the memory allocated to the vmware-infraprofile service.

• Get the current memory allocation value for the vmware-infraprofile service:

  cloudvm-ram-size -l | grep vmware-infraprofile

• Increase the memory allocated to the infraprofile service: 

  cloudvm-ram-size -C <newMemValue> vmware-infraprofile 

  Note: Increase <newMemValue> as needed if OOM errors are still encountered.
• Confirm the new memory allocation value:

  cloudvm-ram-size -l | grep vmware-infraprofile

• Restart the infraprofile service:

  service-control --stop vmware-infraprofile && service-control --start vmware-infraprofile

   

 

Workaround

• Restart the vmware-infraprofile service

  service-control --stop vmware-infraprofile && service-control --start vmware-infraprofile

Note: This will only address issue until the service hits another OOM condition.

Additional Information

Error/exception in infraprofile logs:

Apr 27, 2023 11:03:41 PM org.abc.jsse.provider.ProvTrustManagerFactorySpi getDefaultTrustStore
INFO: Initializing with trust store at path: /usr/java/jre-vmware/lib/security/cacerts
Exception in thread "http-nio-127.0.0.1-11098-Poller" java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap$KeySet.iterator(HashMap.java:918)
        at java.util.HashSet.iterator(HashSet.java:173)
        at java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1041)
        at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1040)
        at org.apache.tomcat.util.net.NioEndpoint$Poller.timeout(NioEndpoint.java:954)
        at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:756)
        at java.lang.Thread.run(Thread.java:748)
Apr 27, 2023 11:05:48 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor run
SEVERE: Unexpected death of background thread [ContainerBackgroundProcessor[StandardEngine[Tomcat]]]
java.lang.OutOfMemoryError: Java heap space

Exception in thread "ContainerBackgroundProcessor[StandardEngine[Tomcat]]" java.lang.OutOfMemoryError: Java heap space