Portal session logout and VSEs not loading in Portal
search cancel

Portal session logout and VSEs not loading in Portal

book

Article ID: 449042

calendar_today

Updated On:

Products

Service Virtualization

Issue/Introduction

Users of DevTest Solutions (Service Virtualization) may experience the following issues:

  • Portal Session Logouts: Users are intermittently kicked out of their Portal sessions.
  • VSEs Not Loading: Virtual Service Environments (VSEs) fail to appear in the Portal or show up as 'Unresponsive' despite the underlying pods/services being in a 'Running' state.

Environment

  • Product: DevTest Solutions / Service Virtualization
  • Versions: 10.8.4, 10.8.4.1
  • Components: Portal, VSE, Registry, ActiveMQ

Cause

The issue is caused by a combination of two primary factors:

  1. ActiveMQ Message Pile-up: A lack of Time-To-Live (TTL) handling for RMI request messages causes them to accumulate indefinitely if a consumer becomes unresponsive. In ActiveMQ 5.16.8 (used in DevTest 10.8.4), enabling message expiration (lisa.net.expire.messages) triggered a Null Pointer Exception (NPE) in the broker.
  2. VSE Registration Race Condition: Ungraceful shutdowns of compute nodes lead to stale VSE registrations. Upon restart, VSEs encounter a duplicate registration error and fail to enter a healthy state.

Resolution

These issues are resolved in DevTest version 10.8.4.2 and newer.

Steps to Apply

  1. Upgrade to 10.8.4.2: Pull and deploy the 10.8.4.2 service pack images for the Portal, LISA (Registry/VSE), Config Server, and Enterprise Dashboard from sv-docker.packages.broadcom.com.
  2. Configure Message Expiration: Update your values.yaml (for Helm) or local.properties to enable TTL. This ensures stale messages are discarded rather than piling up.
    • Set lisa.net.expire.messages (or isMessageProducerTimeToLiveEnabled) to true.
    • Set lisa.net.messages.expiry.ms (or messageProducerTimeToLiveInMilliseconds) to 60000 (60 seconds).
  3. Verify Stale Registration Fix: The 10.8.4.2 version includes a fix for the VSE registration retry loop, allowing VSEs to recover automatically after ungraceful restarts.
  4. Monitor Messaging Health: Use the messaging.log to monitor AMQ stats. Ensure maxEnqueueTime remains low (under 1000ms) during normal operation.