Rotating WSA/vIDM password from the SDDC manager fails with error "Failed to get Environment ID by given Host Name"
search cancel

Rotating WSA/vIDM password from the SDDC manager fails with error "Failed to get Environment ID by given Host Name"

book

Article ID: 382816

calendar_today

Updated On:

Products

VMware SDDC Manager VCF - MCOE - SDDC

Issue/Introduction

  • When performing WSA or vIDM rotation on the SDDC manager or when auto password rotation is scheduled, the password activity fails with the message 

Failed to get Environment ID by given Host Name: example.com. 

  • The sddc operationsmanager.log would show the details as below: 

YYYY-MM-DD HH:MM:SS+0000 ERROR [vcf_om,ab98b8027689d008,b51b] [c.v.v.p.helper.Vrslcm8Util,om-exec-30] Error occured in executing the request on VRSLCM APIs
com.vmware.evo.sddc.common.vrealize.vrlcm.VrlcmException: Failed to get Environment ID by given Host Name: example.com
        at com.vmware.evo.sddc.common.vrealize.vrslcm.VrslcmService.getEnvironmentByHostname(VrslcmService.java:1054)
        at com.vmware.evo.sddc.common.vrealize.vrslcm.VrslcmService.getEnvironmentIdByHostname(VrslcmService.java:1124)
        at com.vmware.vcf.passwordmanager.helper.Vrslcm8Util.updatePassword(Vrslcm8Util.java:131)
        at com.vmware.vcf.passwordmanager.update.changers.WsaAPIPasswordChanger.changePassword(WsaAPIPasswordChanger.java:112)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.retryAwareChangePassword(AbstractPasswordChanger.java:1138)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateAsync(AbstractPasswordChanger.java:531)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdate(AbstractPasswordChanger.java:199)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:100)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:88)
        at org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:67)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
YYYY-MM-DD HH:MM:SS+0000 ERROR [vcf_om,ab98b8027689d008,b51b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-30] Failed to get Environment ID by given Host Name: example.com
YYYY-MM-DD HH:MM:SS+0000 ERROR [vcf_om,ab98b8027689d008,b51b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-30] Failed to get Environment ID by given Host Name: example.com
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Failed to get Environment ID by given Host Name: example.com
        at com.vmware.vcf.passwordmanager.helper.Vrslcm8Util.updatePassword(Vrslcm8Util.java:151)
        at com.vmware.vcf.passwordmanager.update.changers.WsaAPIPasswordChanger.changePassword(WsaAPIPasswordChanger.java:112)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.retryAwareChangePassword(AbstractPasswordChanger.java:1138)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateAsync(AbstractPasswordChanger.java:531)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdate(AbstractPasswordChanger.java:199)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:100)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:88)
        at org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:67)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
YYYY-MM-DD HH:MM:SS+0000 DEBUG [vcf_om,ab98b8027689d008,b51b] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-30] Error Message : Failed to get Environment ID by given Host Name: example.com, Error Token : 7QP674, Error Cause : {}

  • We can also observe that the hostname on the vRSLCM/ARIA lifecycle environments shows the hostname as something else than what is reflecting on the sddc manager. 

 

Environment

VCF 4.x, VCF 5.x 

Cause

This occurs because, the original vIDM configuration that was deployed was a clustered set up and the SDDC manager takes either load balancer FQDN or the tenant FQDN for the vidm.

Ther administrator/User decided to change the configuration from clustered to a single node set up. This triggers a sync on the vRSLCM/Aria lifecycle manger, but the details are not synced on the SDDC manager. 

This can be validated in the following manner: 

  • The SDDC manager sends an API to the vRLSCM to get the environments tails the API is: Request URI: https://vrslcm.lab.local/lcm/lcops/api/environments/requests

  • The output of the API is seen in the operations manager with the "Response headers" and it is read by the SDDC manager to looks for the vidm information. The response will have the follwing details for vidm: 

products": [
            {
                "id": "vidm",
                "version": "3.3.7",
                "patchHistory": null,
                "snapshotHistory": null,
                "logHistory": null,
                "clusterVIP": null,
                "nodes": [
                    {
                        "type": "vidm-primary",
                        "properties": {
                            "hostName": "primary_node.example.com",
                            "cluster": "cluster_fqdn.example.com",
                            "esxHost": "esxi1.example.com",
                            "memory": "16",
                            "diskMode": "thin",
                            "vCenterHost": "mgmt.example.com",
                            "storage": "vSanDatasotre",
                            "network": "seg01",
                            "capacity": "110.0",
                            "vidmRootPassword": "*****",
                            "vidmSystemAdminPassword": "*****",
                            "enableTelemetry": "false",
                            "affinityRules": null,
                            "__vMoid": "vm-1010",
                            "vcPassword": "*****",
                            "cpuCount": "8",
                            "resourcePool": "resgroup-34 (sddc-mgmt.example.com)",
                            "vmName": "primary_node.example",
                            "esxHostBuild": "23794027",
                            "vidmSshPassword": "*****",
                            "ip": "10.X.X.X1",
                            "dns": "127.0.0.53",
                            "ESXI Version": "7.0.3",
                            "domain": "lab.local",
                            "vcUsername": "[email protected]",
                            "folderName": null,
                            "gateway": "10.X.X.X",
                            "searchpath": "example.com",
                            "__sshVerified": "true",
                            "__verified": "true",
                            "__sysAdmPwdVerified": "*****",
                            "netmask": "255.255.255.0"
                        }
                    },
                    {
                        "type": "vidm-connector",
                        "properties": {
                            "hostName": "primary_node.example.com",
                            "cluster": "cluster_fqdn.example.com",
                            "esxHost": "esxi101.example.com",
                            "memory": "16",
                            "diskMode": "thin",
                            "vCenterHost": "mgmt.example.com",
                            "masterTenantId": "MASTER_M001",
                            "storage": "vSanDatasotre",
                            "windowsDeployement": "false",
                            "network": "seg01",
                            "capacity": "110.0",
                            "instanceId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
                            "affinityRules": null,
                            "__vMoid": "vm-1010",
                            "vcPassword": "*****",
                            "cpuCount": "8",
                            "resourcePool": "resgroup-34 (sddc-mgmt.lab.local)",
                            "canBeDeleted": "true",
                            "vmName": "primary_node.lab.local",
                            "esxHostBuild": "23794027",
                            "ip": "10.X.X.X1",
                            "dns": "127.0.0.53",
                            "ESXI Version": "7.0.3",
                            "version": "3.3.7",
                            "domainJoined": "true",
                            "port": "8443",
                            "domain": "lab.local",
                            "vcUsername": "[email protected]",
                            "folderName": null,
                            "gateway": "10.X.X.X",
                            "searchpath": "lab.local",
                            "__iwaLinuxConnectorMigrationRequired": "true",
                            "netmask": "255.255.255.0"
                        }
                    }
                ],
                "collectorGroups": null,
                "properties": {
                    "clusterFqdn": "primary_node.example.com",
                    "__enableTenancy": "false",
                    "defaultConfigurationUsername": "admin@local",
                    "certificateChain": "*****",
                    "vidmDomainName": "lab.local",
                    "__isMigrationRequest": "false",
                    "syncGroupMembers": "true",
                    "__isInstallerRequest": "false",
                    "nodeSize": "medium",
                    "isClustered": "false",
                    "__isTenantByPath": "false",
                    "vidmAdminPassword": "*****",
                    "enableTelemetry": "false",
                    "defaultConfigurationPassword": "*****",
                    "adminEmail": "[email protected]",
                    "vidmOAuthServiceClientId": "Service__OAuth2Client",
                    "vidmOAuthServiceClientSecret": "*****",
                    "__isTenancyEnabled": "true",
                    "defaultTenantAlias": "vidm_lb.example.com",
                    "vidmDBType": "postgresql",
                    "__vidmIsExternalDB": "false",
                    "__vidmHasExternalConnectors": "false",
                    "__vidmFormFactor": "true",
                    "fipsMode": "false",
                    "certificate": "*****"

  • From the output above we can see that the hostname is reported as: primary_node.example.com while, the SDDC manager platform DB in the WSA table reports the following: 

(1,4) -, lb_hostname        vidm_lb.example.com
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
id                     creation_time         modification_time                           lb_hostname           primary_node          status                version               secondary_nodes       type
globalenvironment:w…  1731668758242         1731671998332         vidm_lb.example.com    {"id":"a462bef4-122…  ACTIVE                3.3.7-21173100        \N                    VREALIZE

 

 

Resolution

This can be resolved by modifying the database on the SDDC manager to replace the lb_hostname to reflect the primary hostname of the vidm and the password_expiry table from operations manger. 

  • Take snapshot of the SDDC manager. 
  • Login to the platform database: 
    • psql -h localhost -U postgres -d platform
  • Run the following command to fetch the details for the lb_hostname. 
    • select * from wsa; or
    • select lb_hostname from wsa;
  • To change the lb_hostame and reflect the primary node FQDN: 
    • update wsa set lb_hostname='primary_node.example.com' where lb_hostname='vidm_lb.example.com';
  • Switch to the operationnsmanager database: 
    • \c operationsmanager
  • Update the passwordmanager.credential_expiry table by running the following command: 
    • update passwordmanager.credential_expiry set resource_fqdn='primary_node.example.com' where resource_fqdn='vidm_lb.example.com';
  • Exit database: 
    • \q
  • Refresh the browser to check if the password manager is now reflecting the primary node name on the SDDC manager, 
  • Perform a password rotation and check for the completion. 

 

Additional Information

The same error stack could happen for other Aria components as well. 

Another way to remediate this error would be is to remove the components on the Aria lifecycle and reimport it and the sync should happen with the SDDC manager. 

Powered by Wolken Software