Portal swarm not able to start completely following the upgrade: failed_to_initialize_feature_flags_registry

Products

• CA API Developer Portal

Issue/Introduction

Performed a portal upgrade from version 5.2.1 to version 5.2.3 and after launching the ./portal.sh script and waiting more than 20 minutes, the swarm is still not fully started. It seems there's a problem with the rabbitmq instance that crashes with the following error showed in docker journalctl logs:

Mar 21 12:36:18 apimportalqa01.corpo.stm.info cf7617c27977[10895]: 2024-03-21 16:36:18.710589+00:00 [notice] <0.44.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}}

Portal Rabbitmq log contains error message: 

[email protected]    | [38;5;6mrabbitmq 19:07:10.63  [0m [38;5;2mINFO  ==> ** Starting RabbitMQ **
[email protected]    | 2024-03-21 19:07:13.554741+00:00 [error] <0.230.0> Feature flags: `user_limits`: required feature flag not enabled! It must be enabled before upgrading RabbitMQ.
[email protected]    | 2024-03-21 19:07:13.576458+00:00 [error] <0.230.0> Failed to initialize feature flags registry: {disabled_required_feature_flag,
[email protected]    | 2024-03-21 19:07:13.576458+00:00 [error] <0.230.0>                                               user_limits}
[email protected]    | 2024-03-21 19:07:13.594283+00:00 [error] <0.230.0> 
[email protected]    | 
[email protected]    | BOOT FAILED
[email protected]    | ===========
[email protected]    | Error during startup: {error,failed_to_initialize_feature_flags_registry}

 

Environment

Docker/kunernete container portal 

Cause

Upgrade skipped version (version 5.2.1 to version 5.2.3 ), though the document stated that upgrade from 5.2.1. to 5.2.3 is supported. 

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-developer-portal/5-2/install-configure-and-upgrade/install-portal-on-docker-swarm/upgrade-api-developer-portal/upgrade-to-api-portal-version-5-0.html

Resolution

Stop portal

• Docker swarm

          docker volume ls

            to find out the volum name for <rabbitmq-volume>  (normally it should be portal_rabbitmq-volume)

             docker volume rm <rabbitmq-volume>

           restart the portal

• Kubernetes

  Delete the PersistentVolumeClaim (PVC) associated with RabbitMQ using Kubernetes commands:

  $ kubectl delete pvc <rabbitmq_pvc_name> -n <namespace>

   restart the portal

Additional Information

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-developer-portal/5-3/release-notes-api-developer-portal/known-issues.html