Smarts Failover: OI version mismatch in Smarts Failover; see the following errors in failover manager log: failover-parse-config.asl: WARN: Reason: determine_version: ... Mismatch in version of servers: activeversion 9.4 standbyversion 9.2.2

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

I'm getting the following error in the Smarts Failover manager logs

 

failover-parse-config.asl: Determining version of service INCHARGE-OI

failover-parse-config.asl: WARN:version lookup failed for INCHARGE-OI

failover-parse-config.asl: WARN: Return Code 1

failover-parse-config.asl: WARN: Reason: determine_version: ... Mismatch in version of servers: activeversion 9.4 standbyversion 9.2.2

When running the following dmctl command to check the Feature Version we found the following:  

# dmctl -s active-host:1234/INCHARGE-OI getInstances ICS_FeatureVersion_OI
ICS-FeatureVersion-OI-9.2.2
ICS-FeatureVersion-OI-9.4

The standby side report only 9.2.2 

# dmctl -s standby-host:5678/INCHARGE-OI getInstances ICS_FeatureVersion_OI
ICS-FeatureVersion-OI-9.2.2

Environment

VMware Smart Assurance - SMARTS

Cause

As per Smarts Failover documentation- Both, the Active and Standby components must be running the same version of EMC Smarts software.

After checking that the versions were the same, we found that the old version of the bootend.conf file had been copied to the local folder.  The domain was using the 9.2.2 version of the bootend.conf file and not the 9.4.1 version that it should have been.

This caused the mismatch and the errors listed above.  The bootend.conf file sets the version:  

In this scenario the OI server present in smarts/conf/icoi folder should have entry that is similar to this below:  
 
# create the versioning object and migrate as needed
ICS_FeatureVersion_OI::ICS-FeatureVersion-OI-9.4 
{
        version = "9.4"
}
 
call ICS-FeatureVersion-OI-9.4 performMigration
 
which is responsible for creating the version object

Resolution

Smarts is working as designed.  As we can see based on the above error. activeversion 9.4 standbyversion 9.2.2, both the active and standby are not the same.

To resolve this issue, please make sure both the Active and Standby servers are running the same version of Smarts.

Use the command:  On the impacted installations.

sm_server --version

After confirming both servers are running the same version of Smarts,

1. Check the failover.conf file, where the primary and standby domain locations are specified, as they could be different and or not set correctly.   For failover, it is a requirement that the Active and Standby directories match (see the note below for the document link). If you need to make changes, open and edit the /local/conf/failover/failover.conf file.
2. If the failover.conf file is correct, then the next steps are to review the bootend.conf file.  Especially if this is an upgraded installation from 9.2.2. to 9.4.1.Review the SITEMOD, and compare the BASEDIR bootend.conf file to the local or SITEMOD used and the bootend.conf in that SITEMOD folder.  
3. Look for the versioning info in the bootend.conf.   See the BaseDIR/smarts/conf/icoi  folder.  This snippet of the bootend.conf is from OI 9.4.1 file.

# create the versioning object and migrate as needed
ICS_FeatureVersion_OI::ICS-FeatureVersion-OI-9.4 
{
        version = "9.4"
}
 
call ICS-FeatureVersion-OI-9.4 performMigration
 
which is responsible for creating the version object

Additional Information

For further details regarding Smarts Failover please refer to the Failover Documentation - here
Review the Section on Deployment Considerations.