Controller UI Dashboard shows Service Timeout
Article ID: 426146
Updated On:
Products
Issue/Introduction
Avi UI Applications->Dashboard page can fail with "Error: Service Timeout". Need to click OK multiple times to complete listing all VSs. Depending on the total number of VS, Dashboard can take 1 minute or more to display all VSs.
HAR logs show "GET /api/virtualservice-inventory" getting error code "504 Gateway Timeout".
HAR (HTTP Archive) logs are JSON-formatted files that record a web browser's interaction with a website, including all network requests, responses, headers, cookies, and timing data.
For example, to view HAR logs in chrome browser, before opening the page, right click on the page, choose "Inspect". In the HAR window, click "Network" tab
When Avi UI is loading the Dashboard, internally, it executes API call "GET /api/virtualservice-inventory"
Login to Avi controller. From linux bash, open file - "vi /var/lib/avi/log/portal_performance.log" to view time taken to execute API call "GET /api/virtualservice-inventory" and which step consumed the most time
In the example below, API call "GET /api/virtualservice-inventory" took 58375msec and "get_shared_vip_faults" consumed the most 49213 msec out of the total 58375msec.
[2025-11-07 22:05:55,534] INFO [api_perf._done:88] API GET /api/virtualservice-inventory/?step=300&limit=6&sort=name&page_size=144&page=1&include=config%2Cruntime%2Chealth_score%2Calert%2Cfaults&include_name=true took 58375.849009 msec @ Fri Nov 7 22:04:57 2025:
Initial: 43.905497 Final : 13.613701
Step get : 58318.329811 (1) 1762553097.201829
Step check_tenant : 17.621279 (2) 1762553097.222058
Step check_user : 10.670424 (2) 1762553097.230458
Step check_user_tenant_resource : 0.384331 (2) 1762553097.231514
Step get_view_data : 317.045212 (1) 1762553097.220515
Step migrate_version : 0.005960 (1) 1762553097.237672
Step list : 294.478178 (1) 1762553097.238308
Step custom_get : 0.008821 (1) 1762553097.238326
Step get_queryset : 0.357389 (1) 1762553097.238344
Step get_queryset_from_role : 0.244856 (1) 1762553097.238451
Step update_queryset_with_role_filters : 0.072718 (1) 1762553097.238461
Step clean_cached_data : 252.790689 (1) 1762553097.279930
Step join_list : 0.003576 (1) 1762553097.532798
Step join_subresources_list : 0.002384 (1) 1762553097.532806
Step reverse_migrate_version : 0.002861 (1) 1762553097.532815
Step get_collection_healthscore : 75.496674 (1) 1762553099.235813
Step get_health_score_data : 71.198463 (1) 1762553099.235978
Step parse_health_scores : 2.879143 (144) 1762553099.311078
Step get_alert_data : 349.175215 (144) 1762553155.497004
Step get_faults : 51653.048038 (144) 1762553155.502503
Step get_vs_sdb_obj : 44270.920515 (6371) 1762553155.504466
Step _vs_state_db_cache : 19.676447 (6371) 1762553155.504469
Step get_server_faults : 131.264925 (137) 1762553155.470955
Step _server_state_db_cache : 0.773191 (137) 1762553155.470960
Step get_scaleoutin_faults : 1.998425 (137) 1762553155.471530
Step get_shared_vip_faults : 49213.411331 (137) 1762553155.471546 <========= look for the time consumed
Step get_ssl_cert_expiry_faults : 243.816614 (144) 1762553155.509151
Step get_ssl_cert_ocsp_status_faults : 239.181280 (144) 1762553155.509805
Step get_debug_trace_status : 171.315432 (144) 1762553155.510989
Step check_debug_for_children : 193.942070 (144) 1762553155.511307
Environment
All AVI Versions
Cause
For every VS displayed in the Dashboard, Avi checks health of the VS before displaying them
This issue is observed if the VS is a shared VIP - where multiple VS share same VIP IP but each VS has a different service port number.
In shared VIP environments, loading Dashboard can consume additional time as Avi checks if there is an asymmetric placement in shared VIP VS.
Resolution
Please contact Avi support for the workaround solution.
Feedback
