vCenter HTTP Sessions expiring sooner than configured
search cancel

vCenter HTTP Sessions expiring sooner than configured

book

Article ID: 336100

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

vCenter sessions are sometimes being timed out sooner than the configured session timeout under vmacore/soap/sessionTimeout in /etc/vmware-vpx/vpxd.cfg and is breaking some customer workflow.
These suspicions can be confirmed by observing the vpxd logs under /var/log/vmware/vpxd/. If these logs correlate with observed issues then they are likely caused by a new feature called "session idle timeout penalties". Here are some examples:

Note: The exact numbers will differ between configurations, as they are based on the configured session limit.

  • Logs on current session saturation state in service
    These can be fetched by searching for "Session saturation state" (e.g. grep -A 2 "Session saturation state" /var/log/vmware/vpxd/*)

  • If this is the latest state log, or no state logs can be found, then the idle timeouts are set to the configured value under vmacore/soap/sessionTimeout and premature session closure is impossible. During periods where the session saturation state is "normal", any observed problems cannot have been caused by session idle timeout penalties – the issue lies elsewhere

    [Originator@XXXX sub=HTTP session map opID=XXXXXXX] Session saturation state is now normal.
    --> Session count is 1299 (64.95% of the limit).
    --> Session timeout durations are back to their normal values.

  • If this is the latest state log, then the idle timeouts are shortened and premature session closure is possible.

    [Originator@XXXX sub=HTTP session map opID=XXXXXXX] Session saturation state is now saturated.
    --> Session count is 1401 (70.05% of the limit).
    --> Session timeout durations are decreased.

  • If this is the latest state log, then the idle timeouts of all sessions are very short and premature session closure is very likely.

    Originator@XXXX sub=HTTP session map opID=XXXXXXX] Session saturation state is now critical.
    --> Session count is 1801 (90.05% of the limit).
    --> Session timeout durations are at their minimum.

  • Logs on the per-user session usage

  • This log will be emitted whenever a specific user is using a disproportionately high number of sessions at the current time. It is noticed that session exhaustion problems usually arise from a single user, so these logs should help identifying the culprit.

    [Originator@XXXX sub=SessionCounter opID=XXXXXXX] Unusually high amount of sessions (1700) opened by user VSPHERE.LOCAL\XXXXXXXXX

  • This log will be emitted when the session limit is reached. It's a dump of the session usage for all users.

    [Originator@XXXX sub=HTTP session map] Out of HTTP sessions: Limited to 100
    [Originator@XXXX sub=SessionCounter] Count of sessions per user:
    --> VSPHERE.LOCAL\Administrator: 93
    --> com.vmware.vim.eam: 2
    --> com.vmware.vcIntegrity: 1
    --> VSPHERE.LOCAL\vpxd-extension-ac2725ab-####-####-####-########6db: 3
    --> com.vmware.vim.vsm: 1
    ....



Environment

VMware vCenter Server 8.0.0

Cause

A feature has been added to vCenter 8.0 that improves service resilience by dynamically reducing the timeout of sessions when their count approaches the configured limit.
This limit can be found under the configuration vmacore/soap/maxSessionCount in /etc/vmware-vpx/vpxd.cfg. If it's not found there, it takes on the default value of 500.

This means that, as of 8.0, session timeout is no longer guaranteed to be as long as configured under vmacore/soap/sessionTimeout, especially under heavy workloads.

Resolution

This is an issue users can potentially face, it's their responsibility to fix their workflows to accomodate this resilience-oriented feature. Thus, it's up to them when/whether it will be resolved.


Workaround:

To work around the issue, please follow the below mentioned steps:

A new configuration option is added under the name vmacore/soap/enableSessionTimeoutPenalties in /etc/vmware-vpx/vpxd.cfg. Setting this boolean option to "false" will disable the new functionality and this should resolve the issue. This can be done:

  • Through advanced options in the UI
    Note: In the UI the option is called "config.vmacore.soap.enableSessionTimeoutPenalties".

  • Through command-line
    Note: This option requires restarting vpxd to apply the changes.
    python /usr/lib/vmware-vpx/py/xmlcfg.py -f /etc/vmware-vpx/vpxd.cfg set vmacore/soap/enableSessionTimeoutPenalties false service-control --restart vpxd

However, this will also disable the added resilience for vCenter services. Therefore, the feature should only be disabled temporarily, until the user has made sure to amend their scripts/workflow to not rely on a static timeout for the sessions.

This might involve always checking if their critical sessions are still open, never assuming they are, and reopening sessions if they have been closed prematurely.
If disabling the feature does not fix the observed issues, please go ahead and re-enable it. To do so, you may use the same methods as described above for disabling the feature, but set the value to "true".

 

Powered by Wolken Software