VMDIR in Read-Only state due to conflict between VCSA FBBR and VM Snapshot level backups
search cancel

VMDIR in Read-Only state due to conflict between VCSA FBBR and VM Snapshot level backups

book

Article ID: 388465

calendar_today

Updated On:

Products

VMware vCenter Server 8.0

Issue/Introduction

  • License related operations fails on vCenter Server with error message "License not available to perform the operation".
  • License logs (/var/log/vmware/cis-license/license.log) shows below errors and following the KB Configuring vSphere HA fails with error "License not available to perform the operation" helps to resolve the issue temporarily but the same re-occurs in few days :

    Caused by: com.vmware.cis.license.embedded.server.core.management.dao.licstorage.CisLicenseStorageDaoException: com.vmware.cis.license.embedded.server.common.repository.RepositoryException: com.vmware.identity.interop.ldap.UnwillingToPerformLdapException: Server is unwilling to perform
    LDAP error [code: 53]

  • There is scheduled FBBR backup and snapshot level backup configured for the VCSA VM.
  • FBBR backup log (/var/log/vmware/applmgmt/backup.log) shows below log entries while taking VMDIR DB backup, it shows the state change to 2 (read-only) instead of setting to state 3 (Normal):

    YYYY-MM-DDThh:mm:ss [] [LotusBackup:PID-<pid>] [Lotus::BackupLotus:Lotus.py:74] INFO: stdout: VdcBackupDB: Setting vmdir state to VMDIRD_READ_ONLY
    VdcBackupDB: Backing up: /storage/db/vmware-vmdir//data.mdb
    VdcBackupDB: Backing up: /storage/db/vmware-vmdir//lock.mdb
    VdcBackupDB: Setting vmdir state to (2)

  • VMDIR logs (/var/log/vmware/vmdird/vmdird.log) shows below pattern while executing scheduled FBBR backup

    YYYY-MM-DDThh:mm:ss info vmdird  t@139708586596096: VmDir State (2)
    YYYY-MM-DDThh:mm:ss info vmdird  t@139708586596096: Srv_RpcVmDirSetState: VmDir State (2)
    YYYY-MM-DDThh:mm:ss info vmdird  t@139708586596096: VmDir State (2)
    YYYY-MM-DDThh:mm:ss info vmdird  t@139708586596096: Srv_RpcVmDirSetState: VmDir State (2)
    YYYY-MM-DDThh:mm:ss info vmdird  t@139708586596096: VmDir State (3)
    YYYY-MM-DDThh:mm:ss info vmdird  t@139708586596096: Srv_RpcVmDirSetState: VmDir State (3)
    YYYY-MM-DDThh:mm:ss info vmdird  t@139708586596096: VmDir State (2)
    YYYY-MM-DDThh:mm:ss info vmdird  t@139708586596096: Srv_RpcVmDirSetState: VmDir State (2)
    YYYY-MM-DDThh:mm:ss err vmdird  t@139708298389248: _VmDirConsumePartner: vmdir is in READ_ONLY state
    YYYY-MM-DDThh:mm:ss err vmdird  t@139708298389248: _VmDirConsumePartner: vmdir is in READ_ONLY state
    YYYY-MM-DDThh:mm:ss err vmdird  t@139708298389248: _VmDirConsumePartner: vmdir is in READ_ONLY state

  • Log file (/var/log/vmware/vmdir/vmdirBackupRestore.log) shows below duplicate entries for VMDIR backup :

    YYYY-MM-DDThh:mm:ss:INFO:Running command: ['/opt/likewise/bin/lwsm', 'status', 'lwreg']
    YYYY-MM-DDThh:mm:ss:INFO:Service lwsmd already running
    YYYY-MM-DDThh:mm:ss:INFO:Running command: ['/opt/likewise/bin/lwsm', 'status', 'vmdir']
    YYYY-MM-DDThh:mm:ss:INFO:vmdir is running
    YYYY-MM-DDThh:mm:ss:INFO:Running command: ['/usr/lib/vmware-vmdir/bin/vdcbackup', '/storage/db/vmware-vmdir', '/storage/db/vmware-vmdir/tmp/']

  • VM Tools pre-freeze script logs on VCSA (/var/log/prefreeze/prefreeze.log) shows entries for script execution during the FBBR backup window:

    YYYY-MM-DDThh:mm:ss: starting prefreeze scripts
    YYYY-MM-DDThh:mm:ss: Starting script: '/usr/lib/prefreeze/001-create-partner-list.sh'
    YYYY-MM-DDThh:mm:ss: Finished running script: '/usr/lib/prefreeze/001-create-partner-list.sh' with return code: 0
    YYYY-MM-DDThh:mm:ss: Starting script: '/usr/lib/prefreeze/002-vpxd-generate-backup-timestamp.sh'
    YYYY-MM-DDThh:mm:ss: Finished running script: '/usr/lib/prefreeze/002-vpxd-generate-backup-timestamp.sh' with return code: 0
    YYYY-MM-DDThh:mm:ss: Starting script: '/usr/lib/prefreeze/099-pg-prefreeze.sh'
    YYYY-MM-DDThh:mm:ss: Finished running script: '/usr/lib/prefreeze/099-pg-prefreeze.sh' with return code: 0
    YYYY-MM-DDThh:mm:ss: done with prefreeze scripts: success

Cause

This issue is caused when the scheduled VCSA FBBR backup and the snapshot level backup of vCenter Server VM (by third party backup software) triggers at same time.

FBBR backup takes backup of VMDIR DB by following below steps :

  1. Read the current state of VMDIR
  2. Set the DB to Read-Only
  3. Takes the DB backup
  4. Change the VMDIR state backup to the same state from Step 1

When quiesced snapshot level backup is configured for the vCenter Server VM, the pre-freeze scripts on the VCSA does the exact same steps mentioned above to take the VMDIR DB backup. If both FBBR and snapshot level backup happens around same time, the state can continue to be in Read-Only due to a race condition between both the backups.

Resolution

Change any of the backup schedules to defer the trigger by at least 5 minutes so that the both the backups will not happen at same time.

Powered by Wolken Software