ERROR: "CPI 'set_vm_metadata' method responded with error: CmdError{"type":"Unknown","message":"Segment port with attachment id: XXXX-XXXX-XXXX-XXXX not found" encountered on TKGI on NSX-T

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

SCENARIO:

You are deploying the following VMware technologies via a Concourse pipeline:

- OpsManager v2.10.46 or greater

- TKGI 1.11.4 or greater

- And you are utilizing NSX-T for your cluster networking 
 

ISSUE:

Your pipeline fails while attempting to deploy and build the Bosh Director VM:

You see an error similar to below:

Setting VM metadata to map[created_at:2022-11-03T20:50:16Z deployment:p-bosh director:bosh-init iaas:<IAAS_NAME_HERE> index:0 instance_group:bosh job:bosh name:bosh/0]:
CPI 'set_vm_metadata' method responded with error: CmdError{"type":"Unknown","message":"Segment port with attachment id: XXXXXX-XXXX-XXXX-XXXX-XXXXXXXX not found","ok_to_retry":false}
Exit code 1
===== 2022-11-03 20:55:36 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty create-env /var/tempest/workspaces/default/deployments/bosh.yml"; Duration: 414s; Exit Status: 1
{"type":"step_finished","id":"bosh_product.deploying","description":"Installing BOSH"}
Exited with 1.2022/11/03 20:55:37 installation was unsuccessful

 

CAUSE:

A breaking change was introduced in Opsmanager, starting with version v2.10.46

This is mentioned in the 2.10.46 release notes 

This is a breaking change starting with Opsmanager 2.10.46. Where the Director setting to use the NSX Policy API is enabled by default. This can break Concourse pipelines of earlier versions, because it was not a necessary parameter. Unless you are using the NSX-T Policy API, you will need to update your pipeline to set: nsx_t_use_policy_api: false

EXPLANATION:

If you were not previously using the NSX-T Policy API (or do not plan to), but instead you are using the NSX-T Manager API, then having the default value of “true” will result in the error described.  As this was a newly introduced field, it is also possible your deployment pipeline was also not setting the nsx_t_use_policy_api parameter in your directory config yaml.  So it will default to “true”.

 

Environment

Product Version: 1.14

Resolution

SOLUTION:

As mentioned, the “Use NSX-T Policy API” checkbox  must be unchecked.  Therefore, your pipeline will need to set your bosh director job will need to include the following:

nsx_t_use_policy_api: false

Here is a sample section from a Concourse pipeline bosh director.yml config which addressed this.  Yours may look different.
 

iaas_configuration:

    additional_cloud_properties: {}

    bosh_disk_path: ((properties-configuration_iaas_configuration_bosh_disk_path))

    bosh_template_folder: ((properties-configuration_iaas_configuration_bosh_template_folder))

    bosh_vm_folder: ((properties-configuration_iaas_configuration_bosh_vm_folder))

    datacenter: ((example-vsphere-datacenter))

    disk_type: thin

    ephemeral_datastores_string: ((properties-configuration_iaas_configuration_ephemeral_datastores_string))

    name: your_vcenter_name

    nsx_networking_enabled: true

    nsx_address: ((nsxmgr-address))

    nsx_ca_certificate: ((nsx-ca-cert.certificate))

    nsx_mode: "nsx-t"

    nsx_t_use_policy_api: false

    nsx_username: ((nsx-creds.username))

    nsx_password: ((nsx-creds.password))

    persistent_datastores_string: ((properties-configuration_iaas_configuration_persistent_datastores_string))

    ssl_verification_enabled: false

    vcenter_host: ((your-vsphere-hostname))

    vcenter_username: ((vsphere-creds.username))

    vcenter_password: ((vsphere-creds.password))