HTTPS Health Monitor failure "No Resources" and no health checks are set to backend pool servers on monitored port

Products

VMware Avi Load Balancer

Issue/Introduction

HTTPS heatlh monitors can be configured with or without SSL attributes (SSL Profile). This gives the user the flexibility to use the SSL profile defined on the pool for the TLS/SSL health monitor's health check.

If the SSL profile is not set/defined on the health monitor and no SSL profile is selected on the pool there is a must-check that does not allow you to save this configuration.

However, in one specific sequence of configuration updates this must-check is bypassed which causes the health monitor to fail with 'No Resources' and the health checks are not sent to the monitored port.

Configure Pool with a ssl_profile.
Apply a HTTPs HM with no ssl_attributes under this Pool.
Delete ssl_profile from the pool.

Health Monitor Failure from UI:

Health Monitor failure from CLI:

> show pool <POOL_NAME> server hmonstat filter disable_aggregate se

| server_hm_stat[1]               |                                                                                  |
|   server_name                   | x.x.x.x:443                                                                   |
|   oper_status                   |                                                                                  |
|     state                       | OPER_DOWN                                                                        |
|         reason[1]               | Marked down by HTTPS-no-ssl-attrb [Out of resources]                             |
|   last_transition_timestamp_3   | Thu Aug 21 18:26:53 2025 ms(860147) UTC                                          |
|   last_transition_timestamp_2   | Thu Aug 21 18:20:18 2025 ms(890696) UTC                                          |
|   last_transition_timestamp_1   | Thu Aug 21 18:18:42 2025 ms(440669) UTC                                          |
|   shm_runtime[1]                |                                                                                  |
|     health_monitor_name         | HTTPS-no-ssl-attrb                                                               |
|     health_monitor_type         | HEALTH_MONITOR_HTTPS                                                             |
|     last_transition_timestamp_3 | Thu Aug 21 18:26:53 2025 ms(860056) UTC                                          |

In the all service Engines hosting the virtual service you will find the update where the SSL profile was removed from the pool and the SSL error "Upstream SSL memory ref object not found"

/var/lib/avi/log/glog/se_trace.INFO

C255 18:26:34.000000 I0821 18:26:34.871214  2373 se_agent_rpc_service.cc:260] Update POOL[EXAMPLE-POOL]
C255 18:26:34.000000 I0821 18:26:34.871301  2373 se_agent_rpc_service.cc:260] Delete SSLPROFILE[System-Standard-PFS]
C255 18:26:34.000000 I0821 18:26:34.871522  2373 se_agent_dataplane_intf.cc:312] ## RPC Request[-13252] Delete SSLPROFILE[System-Standard-PFS]
C255 18:26:34.886863 [se_ag_pool_update_object:4114] Pool update completed with status 0 [0 of 1]
C00 18:26:43.860003 [ngx_http_hm_ssl_init:1897] error [0] - Upstream SSL memory ref  object not found
C00 18:26:53.860056 [ngx_http_hm_ssl_init:1897] error [0] - Upstream SSL memory ref  object not found
C00 18:27:03.860063 [ngx_http_hm_ssl_init:1897] error [0] - Upstream SSL memory ref  object not found
C00 18:27:13.868011 [ngx_http_hm_ssl_init:1897] error [0] - Upstream SSL memory ref  object not found
C00 18:27:23.868056 [ngx_http_hm_ssl_init:1897] error [0] - Upstream SSL memory ref  object not found

If you take a virtual service packet capture (only health monitor traffic) you will not find any health checks for the pool's SSL port.

Environment

Affects Version(s):

22.1.1 - 22.1.7-2p10
30.1.1
30.1.2 - 30.1.2-2p3
30.2.1 - 30.2.1-2p6
30.2.2 - 30.2.2-2p6
30.2.3 - 30.2.3-2p4
30.2.4
31.1.1 = 31.1.1-2p3

Cause

This issue has been identified as product issue where the must-check does not cover the deletion of the SSL profile on the pool if the HTTPS health monitor has no SSL attributes.

Resolution

This product issue will be fixed in the next GA releases of the VMware Avi Load Balancer. Please look for the ID below in the product release notes. VMware Avi Load Balancer

ID: AV-246360

Workaround(s):

Add/Configure the SSL profile to the affected pool or the HTTPS health monitor.

You can use the bash script 'get_httpshm_pool_noSSLprofile.sh' attached to find all the HTTPS health monitors with SSL attributes and pools with no SSL attributes using these health monitors.

Steps:

Create a backup of your configuration and save it in the /tmp directory. Please name the file 'avi_config'

ssh to the controller leader node with the admin user and launch the CLI "shell" with admin
execute the following CLI command to create a configuration backup:
```
> export configuration full_system file /tmp/avi_config
```
Exit the CLI "shell" by typing 'exit' on the prompt

NOTE: Ensure there are no other 'avi_config' named files in the /tmp/ directory of the controller.
Create or scp the script on the /tmp directory on the leader controller node, this is where the 'avi_config' file is located.
Modify the scritps permissions with the following command:
```
sudo chmod 777 get_httpshm_pool_noSSLprofile.sh
```

Run the script:

admin@controller:/tmp$ ./get_httpshm_pool_noSSLprofile.sh

Here is an example of the output:

Health Monitors with no SSL attributes:
=======================================
HTTPS-no-ssl-attrb
https-testing
System-GSLB-HTTPS
System-HTTPS


List of Pools that have HMs with no SSL attributes and no SSL Profiles configured:
==================================================================================
HM: HTTPS-no-ssl-attrb
"EXAMPLE-POOL"
=============
HM: https-testing
"pool-A"
"pool-B"
=============
HM: System-GSLB-HTTPS
=============
HM: System-HTTPS
=============


DONE!

Attachments

get_httpshm_pool_noSSLprofile.sh get_app