Protocol Endpoints are not visible in the host - TransportError [Cannot connect to VP, status code=400, status message=Bad Request]

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

1. Hosts do not see Protocol Endpoints

2. Some clusters in the datacenter can access the same virtual volume whereas the other clusters cannot

3. Storage providers in the vCenter keeps circling and does not load

4. Unregistering and re-registering the VASA provider may fix the issue

[root@ESXi:] esxcli storage vvol vasaprovider list
StorageArray-ct0
VP Name: StorageArray-ct0
URL: https://10.#.#.#:8084/version.xml
Status: Offline: TransportError [Cannot connect to VP, status code=400, status message=Bad Request]
VASA Version: 7
Max Concurrent Operations: 4
Arrays:
Array Id: com.purestorage:8539####-####-4038-####-ffcd1cd####
Is Active: true
Priority: 210
VirtualHost:
IP: 10.#.#.#
Port: 0
SNI: vasa-vhost-2e37####-####-####-####-afb41662####.purestorage.com

Environment

VMware ESXi
VMware vCenter Server

Cause

Below log excerpts indicate that the ESXi host is unable to create a VASA session with the Storage array/VASA provider. This is causing the vVol datastores to be marked as inaccessible.

2025-04-26T14:46:01.576Z In(166) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] GetEndPoint: with url https://10.#.#.#:8084/version.xml
2025-04-26T14:46:01.576Z In(166) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] GetEndPoint Settting  Virtual host vasa-vhost-2e37d8c0-46c5-4674-86c4-afb4166264b5.purestorage.com:8084 url https://10.#.#.#:8084/version.xml
2025-04-26T14:46:01.587Z Wa(164) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] status code=400,status message=Bad Request
2025-04-26T14:46:01.588Z Wa(164) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] VasaSession::GetEndPoint: failed to get endpoint, err=Cannot connect to VP, status code=400, status message=Bad Request, using default.Changing session state:TransportError
2025-04-26T14:46:01.588Z In(166) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] VasaSession::Initialize url is empty 
2025-04-26T14:46:01.588Z Wa(164) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] VasaSession::DoSetContext: Empty VP URL for VP (StorageArray-ct0)!
2025-04-26T14:46:01.588Z Er(163) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] VP node is not set, ignoring stats 
2025-04-26T14:46:01.588Z In(166) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] Initialize: Failed to establish connection https://10.#.#.#:8084/version.xml
2025-04-26T14:46:01.588Z Er(163) VVold[2099157]: [Originator@6876 sub=Default OpId=Session] Initialize: Unable to init session to VP StorageArray-ct0 state: 3

NOTE: The first step to create a vasa session is to fetch and parse the "version.xml". This is the file hosted at URL: "https://10.#.#.#:8084/version.xml."  This is failing with status code=400. The VASA Provider is failing this request.

Resolution

Please submit a case to Broadcom to enable our Virtual Volume specialists to investigate the issue at hand. They will assist you in gathering logs by modifying certain log properties.

NOTE: The below changes must be made under the supervision of a Virtual Volumes specialist from Broadcom before generating logs.

1. Increase the sms.threadpool.queueSize in the /usr/lib/vmware-vpx/sps/conf/sms.properties file.

The current value is 2. You can set this to anything you would like, but we recommend setting it to 500. This won't affect anything as it's merely a queue size that permits more thread pools to be queued.

2. Backup the log4j2.properties file - /usr/lib/vmware-vpx/sps/conf/log4j2.properties to /usr/lib/vmware-vpx/sps/conf/log4j2.properties.bak

Edit the log4j2.properties file using vi editor and modify the below properties -

A. logger.sms.level to DEBUG

B. logger.wire.level, logger.wireheader.level, and logger.wirecontect.level to be TRACE

C. appender.rolling.policies.size.size to 100MB (the default is 10MB).

3. Restart SPS service - service-control --stop vmware-sps && service-control --start vmware-sps

4. Wait until the storage providers load in vCenter and collect the vCenter and ESXi log bundles.

After collecting the logs, revert the settings changed in log4j2.properties file

In the /usr/lib/vmware-vpx/sps/conf/log4j2.properties file, change the logger.wire.level, logger.wireheader.level and logger.wirecontect.level back to ERROR and set the appender.rolling.policies.size.size back to 10 MB.

TIP: Command to check the storage provider status from ESXi CLI - wget localhost:22000/sms/HealthStatus ; cat HealthStatus ; rm HealthStatus