VMware VeloCloud SD-WAN specific user persistently unable to access remote diagnotics

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

Customers may encounter an issue where the remote diagnostic page fails to load, displaying the error message: "Connection to the edge for running diagnostic tests is closed. Try reconnecting..." immediately upon clicking the button. This problem is isolated to a specific account authenticated by RADIUS. The remote diagnostic page fails to load for all edges associated with this account, while all other accounts function correctly.

Environment

All supported VMware VeloCloud SD-WAN orchestrator versions

Cause

1. As issue happenes for all edges, it is not a DP issue. Check F12 in Chrome and found websocket opened successfully.

2. Check portal log in VCO and found below log is immediately generated once clicking "remote diag" button:

2024-11-26T02:10:23.825Z - error: [realtime-req.173255223.276942] [55353] Enterprise WebSocket connection limit exceeded
2024-11-26T02:10:23.825Z - error: [realtime-req.173255223.276942] [55353] Closing websocket connection, reason: Enterprise WebSocket connection limit exceeded
2024-11-26T02:10:23.828Z - error: [realtime-req.173255223.276942] [55353] Enterprise WebSocket connection limit exceeded
2024-11-26T02:10:23.828Z - error: [realtime-req.173255223.276942] [55353] Closing websocket connection, reason: Enterprise WebSocket connection limit exceeded
2024-11-26T02:10:23.831Z - error: [realtime-req.173255223.276942] [55353] Enterprise WebSocket connection limit exceeded
2024-11-26T02:10:23.831Z - error: [realtime-req.173255223.276942] [55353] Closing websocket connection, reason: Enterprise WebSocket connection limit exceeded
2024-11-26T02:10:23.837Z - error: [realtime-req.173255223.276942] [55353] Websocket connection closed.

3. redis-cli monitor | grep -E 'RemDiagWsUserLimitKey|RemDiagWsEdgeLimitKey' output:

vcadmin@vco01:~$ redis-cli monitor | grep -E 'RemDiagWsUserLimitKey|RemDiagWsEdgeLimitKey'
1735527541.588468 [0 127.0.0.1:27466] "set" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true" "EX" "5" "NX"
1735527541.588766 [0 127.0.0.1:27466] "set" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true" "EX" "5" "NX"
1735527541.589038 [0 127.0.0.1:27466] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>"
1735527541.589145 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true"
1735527541.589161 [0 lua] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.589165 [0 lua] "del" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.589250 [0 127.0.0.1:27466] "set" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true" "EX" "5" "NX"
1735527541.589332 [0 127.0.0.1:27466] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b"
1735527541.589393 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true"
1735527541.589406 [0 lua] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.589411 [0 lua] "del" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.590611 [0 127.0.0.1:27466] "set" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true" "EX" "5" "NX"
1735527541.590853 [0 127.0.0.1:27466] "set" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true" "EX" "5" "NX"
1735527541.590947 [0 127.0.0.1:27466] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>"
1735527541.591012 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true"
1735527541.591046 [0 lua] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.591049 [0 lua] "del" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.591120 [0 127.0.0.1:27466] "set" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true" "EX" "5" "NX"
1735527541.591176 [0 127.0.0.1:27466] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b"
1735527541.591232 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true"
1735527541.591244 [0 lua] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.591249 [0 lua] "del" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.592441 [0 127.0.0.1:27466] "set" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true" "EX" "5" "NX"
1735527541.592553 [0 127.0.0.1:27466] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>"
1735527541.592662 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true"
1735527541.592676 [0 lua] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.592680 [0 lua] "del" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.592811 [0 127.0.0.1:27466] "set" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true" "EX" "5" "NX"
1735527541.593018 [0 127.0.0.1:27466] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b"
1735527541.593093 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true"
1735527541.593107 [0 lua] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.593113 [0 lua] "del" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.598541 [0 127.0.0.1:27466] "set" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true" "EX" "5" "NX"
1735527541.598619 [0 127.0.0.1:27466] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>"
1735527541.598683 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true"
1735527541.598694 [0 lua] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.598698 [0 lua] "del" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.598779 [0 127.0.0.1:27466] "set" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true" "EX" "5" "NX"
1735527541.598832 [0 127.0.0.1:27466] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b"
1735527541.598904 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true"
1735527541.598916 [0 lua] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.598921 [0 lua] "del" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.608700 [0 127.0.0.1:27466] "set" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true" "EX" "5" "NX"
1735527541.608812 [0 127.0.0.1:27466] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>"
1735527541.608892 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock" "true"
1735527541.608911 [0 lua] "get" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.608916 [0 lua] "del" "RemDiagWsUserLimitKey:1:<Problematic Account Name>:lock"
1735527541.609014 [0 127.0.0.1:27466] "set" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true" "EX" "5" "NX"
1735527541.609076 [0 127.0.0.1:27466] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b"
1735527541.609135 [0 127.0.0.1:27466] "eval" "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end" "1" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock" "true"
1735527541.609148 [0 lua] "get" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"
1735527541.609152 [0 lua] "del" "RemDiagWsEdgeLimitKey:5eefce8a-bd99-4bb0-8433-7687ac34a02b:lock"

 

Resolution

1. Scan the account name, check if there is any existing key value for RemDiagWsUserLimitKey. Below is an real live example account named op1949

xxx@xx-fra1:~$ redis-cli --scan --pattern '*' | grep '<account name>'

2. If there is any key in wrong format, just delete it. 

3. Check the TTL of "RemDiagWsUserLimitKey:1:op1949", TTL=-1 means that the key has no expiration time set. This is an unexpected key value. A TTL value of -1 signifies that the key is permanent and will not expire automatically. In this way, the key value would accumulate and reach the limit eventually. This is the reason why remote diagnostic not working.

4. Delete the key RemDiagWsUserLimitKey:1:op1949:

5. Check if the value is reset and TTL is corrected. TTL

6. Retry remote diagnostic in SD-WAN orchestrator web portal and it should work now: