SAM Domain Managers show Roles as UNKNOWN on the failover

Products

• VMware Smart Assurance
• Network Observability

Issue/Introduction

Symptoms:

• The SAM domain managers appear missing on active broker & standby broker (brcontrol output) on initialization of the failover setup. 
• The Failover Actions log show the following sample errors on resync_actions:

resync_actions:  <Date & Time>; Standby XML Import 'failover-standby-<Hostname>-<SAM Domain>.xml' (remote):  
resync_actions:  <Date & Time>; XML import of standby configuration into '<Hostname>:<Port>/<SAM Domain>' failed (remote), error: rc={65280}; ''

Environment

Smarts 10.1.X, 24.3.X

Cause

As per the review of the errors and functionality of Failover Manager, the error occurred during XML configuration import occurs when there is configuration mismatch between standby and active server, since during roles assignment configuration will be applied, if there is a mismatch in configuration, error will be thrown which is the case here. However in logs it is unsure why the mismatch would have happened, in ideal scenario if configurations are correct then the roles would have assigned properly.
Once the roles goes to UNKNOWN state, there is a need to restart SAM domains and failover domain both in order to fix the roles even if the configuration files have been copied.

Resolution

If the configurations are setup correctly, and Active SAM domain managers are registered on the broker correctly, yet the roles show UNKNOWN, please report to Broadcom Support for further assistance.