Unable to assign tags to Virtual machines.
search cancel

Unable to assign tags to Virtual machines.

book

Article ID: 316452

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • Unable to attach tags after vCenter upgrade to version 7.x.
  • The VCSA may have been restored from backup.
  • /var/log/vmware/vpxd-svcs/vpxd-svcs.log shows similar entries
YYYY-MM-DDTHH:MM:SSZ [dataservice-0 WARN com.vmware.cis.authorization.impl.AclPrivilegeValidator opId=e9ce3c2e-6891-4b32-bdd8-af0bd1960b2b IS] User AC.TI.CH\SIGIVAD does not have privileges [InventoryService.Tagging.AttachTag] on object urn%3Avmomi%3AInventoryServiceTag%3A3566d0ae-74d1-4e6c-bd6f-4bd687b648b3%3AGLOBAL

YYYY-MM-DDTHH:MM:SSZ [dataservice-0 ERROR com.vmware.vim.vmomi.server.impl.InvocationTask opId=e9ce3c2e-6891-4b32-bdd8-af0bd1960b2b IS] Method invocation threw unexpected exception!java.lang.NullPointerException at
com.vmware.cis.core.tagging.internal.impl.TagAssociationManagerImpl.isPassCardinality(TagAssociationManagerImpl.java:549)
 
Note: The preceding log excerpts are only examples. Date,time and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 7.0.x

Cause

In specific environments, tags associated with active objects in vCenter may be removed during the vCenter upgrade process, but the details of these deleted tags may not be reflected back in vCenter. As a result, the vCenter inventory may display valid objects linked to invalid tags.

When attempting to attach tags back to an object, it retrieves the IDs of all tags associated with the virtual machine from the VCDB, including those that are invalid. Hence when the function “isPassCardinality” attempts to access information related to an invalid tag ID, it results in a null pointer exception.

Resolution

NOTE: 


Steps to replicate as an example:
 
  • Created two VMs.
  • Entering dcli interactive terminal: dcli +i
  • It requires AD username and password.
  • Note the VM IDs from dcli.
dcli> vcenter vm list Username: [email protected]
Password: ********
Do you want to save credentials in the credstore? (y or n) [y]:y
|---------------|-----|---------|-----------|---------|
|memory_size_MiB|vm |name |power_state|cpu_count|
|---------------|-----|---------|-----------|---------|
|128 |vm-1|Test_1 |POWERED_ON |1 |
|128 |vm-2|Test_2  |POWERED_ON |1 |
|1024 |vm-3|Test_3  |POWERED_OFF|1 |
|1024 |vm-4|Test_4 |POWERED_OFF|1 |
|---------------|-----|---------|-----------|---------|
  • Create tags and attach it to available VMs:
dcli> tagging category create --name cat-1 --description "cat info" --cardinality SINGLE urn:vmomi:InventoryServiceCategory: ########-48ea-####-####-############:GLOBAL
 
dcli> tagging tag create --name tag-1 --description "tag info" --category-id urn:vmomi:InventoryServiceCategory:########-48ea-####-####-############:GLOBAL
urn:vmomi:InventoryServiceTag:########-1234-####-####-############:GLOBAL

dcli> tagging category create --name cat-2 --description "cat info 2" --cardinality MULTIPLE
urn:vmomi:InventoryServiceCategory:########-424c-####-########-####:GLOBAL

dcli> tagging tag create --name tag-2 --description "tag info 2" --category-id urn:vmomi:InventoryServiceCategory:########-424c-####-########-####:GLOBAL
urn:vmomi:InventoryServiceTag:########-5678-####-####-############:GLOBAL
 
dcli> tagassociation attach --id vm-3 --tag-id urn:vmomi:InventoryServiceTag:########-1234-####-####-############:GLOBAL --type VirtualMachine

dcli> tagassociation attach --id vm-4 --tag-id urn:vmomi:InventoryServiceTag:########-5678-####-####-############:GLOBAL --type VirtualMachine 

 

  • Create one more tag and attach it later. Create it for a category with SINGLE cardinality else the creation task would fail.
dcli> tagging tag create --name tag-cat-single --description "tag info - cat single" --category-id urn:vmomi:InventoryServiceCategory:########-48ea-####-####-############:GLOBAL
urn:vmomi:InventoryServiceTag:########-6f35-####-########-####:GLOBAL
  • Check VCDB values:
    • /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
    • select * from VPX_ENTITY_TAG_XREF;
entity_id | tag_id
-----------+--------
5 | 1
6 | 2
(2 rows)
    • select * from VPX_TAG_DEF;

      id | tag_uuid | type_id
      ----+--------------------------------------+---------
      1 | ########-1234-####-####-############ | 0
      2 | ########-5678-####-####-############ | 0
      (2 rows)
       
  • Add invalid tags to VPX_TAG_DEF and attach it to Entity 3, which is the "Test_3" in VPX_ENTITY_TAG_XREF
    • insert into VPX_TAG_DEF (id, tag_uuid, type_id) values (5, ‘########-####-4186-####-############', 0);
    • insert into VPX_TAG_DEF (id, tag_uuid, type_id) values (6, '########-####-4817-####-############', 0);
    • insert into VPX_ENTITY_TAG_XREF (entity_id, tag_id) values (3, 5);
  • To quit Postgres: \q
  • Reload VPXD because tag definition and associations are maintained in memory: service-control --restart vpxd
  • Attempt to attach a tag ( urn:vmomi:InventoryServiceTag:########-6f35-####-########-####:GLOBAL ) to vm-3 and vm-4, which should fail for vm-3
dcli> tagassociation attach --id vm-3 --tag-id urn:vmomi:InventoryServiceTag:########-6f35-####-########-####:GLOBAL --type VirtualMachine

dcli> tagassociation attach --id vm-4 --tag-id urn:vmomi:InventoryServiceTag:########-6f35-####-########-####:GLOBAL --type VirtualMachine
  • We can see failure in var/log/vmware/vpxd-svcs/vpxd-svcs.log:
grep -ri -A 10 B 10 NullPointerException

./vpxd-svcs.log-YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-112 INFO com.vmware.cis.server.util.PerfLog opId=] Requesting LDAP connection
./vpxd-svcs.log-YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-112 DEBUG com.vmware.cis.lotus.LdapConnectionFactory opId=] Got connection; com.vmware.identity.interop.ldap.LdapConnection@3281dfc2
./vpxd-svcs.log-YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-112 INFO com.vmware.cis.server.util.PerfLog opId=] Connection type: LDAP, Max allowed connections: 24, Number of active connections: 1, Number of idle connections: 1
./vpxd-svcs.log-YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-112 DEBUG com.vmware.cis.lotus.LdapConnectionFactory opId=] Releasing connection; com.vmware.identity.interop.ldap.LdapConnection@3281dfc2
./vpxd-svcs.log-YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-112 WARN com.vmware.vapi.internal.bindings.ApiMethodSkeleton opId=] Implementation method reported unexpected exception: java.lang.NullPointerException
./vpxd-svcs.log-java.lang.NullPointerException
./vpxd-svcs.log- at com.vmware.cis.core.tagging.internal.impl.TagAssociationManagerImpl.isPassCardinality(TagAssociationManagerImpl.java:549)
./vpxd-svcs.log- at com.vmware.cis.core.tagging.internal.impl.TagAssociationManagerImpl.attach(TagAssociationManagerImpl.java:229)
./vpxd-svcs.log- at com.vmware.cis.core.tagging.vapi.TagAssociationProviderImpl.attach(TagAssociationProviderImpl.java:59)
 
 
Steps to resolve the issue by deleting stale tags:
  • Access the VCDB
    • SSH to the vCSA
    • /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
  • View and export the current tagging associations stored in the vCenter Database:
    • \copy (select * from VPX_ENTITY_TAG_XREF) to ./vcdb-vpx-entity-tag-xref.csv CSV HEADER;
    • \copy (select * from VPX_TAG_DEF) to ./vcdb-vpx-tag-def.csv CSV HEADER;
    • \q
  • Collect visible tag associations with respective to LDAP. This is to compare data between tag associations in the VCDB and tag associations valid from the LDAP perspective. Two ways to execute the Python script:
    • Using session: Use this API to get session: http://vmware.github.io/vsphere-automation-sdk-rest/6.5/operations/com/vmware/cis/session.create-operation.html
      • python3 collect-tag-associations.py --vcip '<vc ip>' --session '<session>'
    • Using AD username and password:
      • python3 collect-tag-associations.py --vcip '<vc ip>' --username '<AD username>' --password '<AD password>'


This will generate a json file named 'tag-associations.json'

  • Use all three files are generated, to generate Postgres delete commands run: python3 find-invalid-tags-migrated.py --json tag-associations.json --txt_tag vcdb-vpx-tag-def.csv --txt_entity_ref vcdb-vpx-entity-tag-xref.csv

Output:
valid vms : 2
valid tags from ldap : 4
From vcdb valid tags : 3
From vcdb invalid tags : 2
vpx_tag_entities_dict_valid_entity : 4
vpx_tag_entities_dict_invalid_entity : {}
valid_vms_found : 2
2
  • Following DCLI commands should fail for validation:
    • tag get --tag-id urn:vmomi:InventoryServiceTag:########-####-4186-####-############:GLOBAL
    • tag get --tag-id urn:vmomi:InventoryServiceTag:########-####-4817-####-############:GLOBAL
  • Postgres command to delete invalid tags from vpx-entity-tag-xref table: delete from VPX_ENTITY_TAG_XREF where tag_id in ( 3, 4 );
  • Postgres command to delete invalid tags from vpx-tag-def table: delete from VPX_TAG_DEF where id in ( 3, 4);
  • Restart the VPXD service to allow for the deletion to take effect: service-control --restart vpxd

find-invalid-tags-migrated.py python command will also generate a JSON file named “valid-vms-with-invalid-tags.json,” which will contain individual delete Postgres command for each tag associations with respect to an entity.
  • For example, Consider a vm named “Test” with the VM ID vm-1234
  • To confirm VM ID, run the following dcli commands: vcenter vm list --name Test
  • Test vm had only two visible tags in the UI but in vcdb-vpx-entity-tag-xref.csv we can see 6 associations :
>1234,1001
>1234,1002
>1234,1003
>1234,1004
>1234,1005
>1234,1006
  • If we check valid-vms-with-invalid-tags.json file for vm 1234, we can see 4 delete commands generated for vm-1234 :
     
"1234": [
{
"id": "1006",
"postgres cmd for vpx-entity-tag-xref": "delete from VPX_ENTITY_TAG_XREF where tag_id 1006 and entity_id 1234",
"postgres cmd for vpx-tag-def": "delete from VPX_TAG_DEF where id 1006",> "tag-uuid": "b189b263-2a93-40d1-947b-142bf0534acf"
},
{
"id": "1003",
"postgres cmd for vpx-entity-tag-xref": "delete from VPX_ENTITY_TAG_XREF where tag_id 1003 and entity_id 1234",
"postgres cmd for vpx-tag-def": "delete from VPX_TAG_DEF where id 1003",
"tag-uuid": "ebbd994f-38e9-4831-8dad-6b98ae201a60"
},
{
"id": "1004",
"postgres cmd for vpx-entity-tag-xref": "delete from VPX_ENTITY_TAG_XREF where tag_id 1004 and entity_id 1234",
"postgres cmd for vpx-tag-def": "delete from VPX_TAG_DEF where id 1004",
"tag-uuid": "b8b71e43-1d88-48ed-bd74-fe5ceb5cf6fa"
},
{
"id": "1005",
"postgres cmd for vpx-entity-tag-xref": "delete from VPX_ENTITY_TAG_XREF where tag_id 1005 and entity_id 1234",
"postgres cmd for vpx-tag-def": "delete from VPX_TAG_DEF where id 1005",
"tag-uuid": "45562f3e-f040-4bc9-8505-f270dac8c86c"
}
]
  • Run the following Postgres cmds,
delete from VPX_ENTITY_TAG_XREF where tag_id 2201 and entity_id 1234;
delete from VPX_ENTITY_TAG_XREF where tag_id 10501 and entity_id 1234;
delete from VPX_ENTITY_TAG_XREF where tag_id 10801 and entity_id 1234;
delete from VPX_ENTITY_TAG_XREF where tag_id 10901 and entity_id 1234;



Workaround:
Unregister and re-register the VM in VC inventory



Additional Information

Impact/Risks: Requires restart of vCenter service.

Attachments

tagging-scripts get_app
Powered by Wolken Software