VM backups fail due to missing custom attribute in vCenter Server.

Products

• VMware vCenter Server

Issue/Introduction

• Virtual Machine backups are failing due to missing "custom attribute" in vCenter Server.
• From the /var/log/vmware/vpxd/vpxd.log, following events can be observed for manual deletion of "custom attribute" :

YYYY-MM-DDThh:mm:ss.130Z verbose vpxd[07124] [Originator@6876 sub=Default opID=mhzutss1-18753982-auto-b5yo4-h5:71407364-37] [VpxVmomi] Invoking; <<52239409-cd49-19bf-6491-8b1b18a7d69d, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 40518'>>, CustomFieldsManager, vim.CustomFieldsManager.removeFieldDefinition, <vim.version.v8_0_3_0, internal, 8.0.3.0>, {stm: {<io_obj p:0x00007ff1b066d620, h:99, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 43366'>>, id: 1782725, state(in/out): 3/1}, session: <52239409-cd49-19bf-6491-8b1b18a7d69d, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 40518'>>, req: {POST, /sdk}}>
YYYY-MM-DDThh:mm:ss.147Z trivia vpxd[07124] [Originator@6876 sub=AuthorizeManager opID=mhzutss1-18753982-auto-b5yo4-h5:71407364-37] [CheckEntityPrivByUD] entity=group-d1, priv = Global.ManageCustomFields, res = true, user = <SSO-domain\User-name>

YYYY-MM-DDThh:mm:ss.177Z info vpxd[08922] [Originator@6876 sub=vpxLro opID=mhzutss1-18753982-auto-b5yo4-h5:71407364-37] [VpxLRO] -- BEGIN lro-213949125 -- CustomFieldsManager -- vim.CustomFieldsManager.removeFieldDefinition -- 52017a5c-1d2c-5922-9d77-17be270e453b(52212c15-e581-b5c0-94f6-6263ae6deed0)
YYYY-MM-DDThh:mm:ss.199Z info vpxd[08922] [Originator@6876 sub=vpxLro opID=mhzutss1-18753982-auto-b5yo4-h5:71407364-37] [VpxLRO] -- FINISH lro-213949125

Note: The above events validates, when the "custom attribute" was deleted and by which user.

Cause

• Custom Attributes are stored in the vCenter Database (VCDB) as key-value pairs. So when custom attribute is deleted an API call is made using the method vim.CustomFieldsManager.removeFieldDefinition. This method programmatically removes the field definition from the vCenter database.

Resolution

To restore backup functionality, the attribute must be manually re-added to the vCenter inventory:

1. Log in to the vSphere Client.
2. Navigate to Menu > Tags & Custom Attributes.
3. Click the Custom Attributes tab.
4. Click Add and create the attribute name.
5. Set the Type to Virtual Machine.
6. Ensure the attribute is assigned the correct value for the affected VMs.

Note : Identifying the specific user or service account responsible for deleting custom attributes requires Trivia-level logging to be active prior to the event. Because vCenter defaults to "Info" logging, this granular data will not be captured. VMware does not recommend trivia/verbose logging to be enabled in a production environment due to potential performance impacts.

Caution: Enabling trivia or verbose logging for a longer duration cause performance degradation on vCenter Server. Only enable trivia or verbose logging for troubleshooting purposes. VMware recommends to revert to default logging (info level) immediately after the troubleshooting is complete.