Unable to login to VCF Operations for Networks with FlinkContainer and other services not running

Products

• VCF Operations for Networks

Issue/Introduction

• VCF Operations for Networks (formerly Aria Operations for Networks) UI is inaccessible with "Unable to login" error in GUI
  
  
  
  
  
• Executing the show-service-status command across platform nodes reveals that critical services, are not running including but not limited to: FlinkContainer as seen in the example below for platform node 1:
  

  ubuntu@platform1:~$ ./check-service-health.sh -d -p

  ElasticSearch is running and healthy.
ElasticSearch statistics:
Uptime:26:21
HRegionServer is running and healthy.
Uptime:27:41
Kafka is running and healthy.
Kafka statistics:
Uptime:26:03
NodeManager is running and healthy.
Uptime:26:51
Problem: SaasListener javaservice is not running.
Restapilayer is running but not healthy.
Restapilayer statistics:
TSDB is running
TSDB statistics:
Uptime:00:01
DataNode is running and healthy.
Uptime:28:50
Problem: Launcher javaservice is not running.
VIPService is running and healthy.
VIPService statistics:
Uptime:23:48
Problem: DatabusGateway javaservice is not running.
Problem: FlinkContainer javaservice is not running.
HMaster is running and healthy.
Is Master:True
Uptime:28:17
QuorumPeerMain is running and healthy.
QuorumPeerMain statistics:
Uptime:30:14
JournalNode is running and healthy.
Uptime:29:34
NameNode is running and healthy.
HDFS is healthy.
"State" : "active",
IsMaster:True
Under replicated blocks: 0
Uptime:29:24
DFSZKFailoverController is running
Uptime:30:00
ResourceManager is running and healthy.
Uptime:27:35
Problem: Federation javaservice is not running.
Nginx is running and healthy.
Nginx statistics:
Uptime:30:40
ExpressJSApp is running
Uptime:01:04:12
NTPSEC is running and healthy.
Uptime:31:18
FoundationDB is running but not healthy.
FoundationDB statistics:
Uptime:30:12

• In Platform1, running the following query fdbcli --exec="status json" | grep Recovery returns the following error "RecoveryStoppedTooManyOldGenerations" as seen in the example output below:
  
  ubuntu@platform1:~$  fdbcli --exec "status json" | grep Recovery
                        "description" : "RecoveryStoppedTooManyOldGenerations at Tue Mar  3 11:02:50 2026",
                        "raw_log_message" : "\"Severity\"=\"40\", \"Time\"=\"1772535770.133028\", \"DateTime\"=\"2026-03-03T11:02:50Z\", \"Type\"=\"RecoveryStoppedTooManyOldGenerations\", \"ID\"=\"0000000000000000\", \"OldGenerations\"=\"100\", \"Reason\"=\"Recovery stopped because too many recoveries have happened since the last time the cluster was fully_recovered. Set --knob_max_generations_override on your server processes to a value larger than OldGenerations to resume recovery once the underlying problem has been fixed.\", \"Backtrace\"=\"addr2line -e fdbserver.debug -p -C -f -i 0x19ed48c 0x19ecc40 0x19ecd21 0xf471de 0xf4990f 0x6c2728 0xf699e3 0xf69b53 0xa07840 0xa07fa3 0xa01da8 0xa01f6a 0xa01258 0xa0170a 0xa01896 0x6c2728 0x737fde 0x6c2728 0xa01258 0x9f7703 0xa01258 0x9f7b55 0xa01258 0x9f8b5d 0xa01258 0x9f6c01 0xa01258 0xa0a33b 0x1938053 0x1938155 0x8018f0 0x1a2b270 0x6784f9 0x7a4d0d429d90\", \"XXXXXXXXXXXXXXXXX\", \"LogGroup\"=\"default\", \"xxxxxxxxxx"",
                        "type" : "RecoveryStoppedTooManyOldGenerations"
  

Environment

VCF Operations for Networks 6.13.0
VCF Operations for Networks 6.14.x

Cause

FoundationDB (FDB) instability prevents the FlinkContainer and dependent Java services from initializing and starting successfully. Such instability can occur when the platform node is shut down abruptly or ungracefully.

Resolution

This is a known issue that requires intervention under the guidance of Broadcom Support.

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article. For more information, see Creating and managing Broadcom support cases.