This is due to the stale entries in the vdsIdPortGroupStateMap of the logical switch config.
If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.
For more information, see Creating and managing Broadcom support cases.