vSphere endpoint for Telco cloud automation virtual infrastructure report connection status as degraded

Products

• VMware Telco Cloud Automation

Issue/Introduction

Error: E_OPERATION_FAILED Request to remote cloud failed with server error Forbidden and validation failed when attempting to reconfigure vSphere endpoint 

Unable to instantiate the network function as the vSphere endpoint (vCenter server) reports 'degraded' status

Environment

3.2

Cause

Telco control plane tca-api log (/cnVmLogs/logs/pods/tca-cp-cn_tca-api) report error related to 

2025-07-01T21:49:37.871171645Z stdout F     "result": "<?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S=\"http:\/\/schemas.xmlsoap.org\/soap\/envelope\/\"><S:Body><S:Fault xmlns:ns4=\"http:\/\/www.w3.org\/2003\/05\/soap-envelope\"><faultcode xmlns:ns0=\"http:\/\/docs.oasis-open.org\/wss\/2004\/01\/oasis-200401-wss-wssecurity-secext-1.0.xsd\">ns0:MessageExpired<\/faultcode><faultstring>The time now Tue Jul 01 21:32:23 GMT 2025 does not fall in the request lifetime interval extended with clock tolerance of 600000 ms: [ Tue Jul 01 21:39:37 GMT 2025; Tue Jul 01 22:09:37 GMT 2025). This might be due to a clock skew problem.<\/faultstring><\/S:Fault><\/S:Body><\/S:Envelope>"
2025-07-01T21:49:37.871173993Z stdout F }
2025-07-01T21:49:37.871176958Z stdout F 2025-07-01 21:49:37.871 UTC [http-nio-8443-exec-6, , , TxId: ] WARN  c.v.t.d.AuthenticationServiceByVc- Unable to login with [email protected] user. Status code: 500, Reason: Internal Server Error
2025-07-01T21:49:37.871391941Z stdout F 2025-07-01 21:49:37.871 UTC [http-nio-8443-exec-6, , , TxId: ] ERROR c.v.v.h.a.AccessTokenRestController- Unable to login with [email protected] user. Status code: 500, Reason: Internal Server Error
2025-07-01T21:49:37.87139897Z stdout F org.springframework.security.access.AccessDeniedException: Unable to login with [email protected] user. Status code: 500, Reason: Internal Server Error
2025-07-01T21:49:37.871402454Z stdout F         at com.vmware.tca.directory.AuthenticationServiceByVc.authenticate(AuthenticationServiceByVc.java:240)
.
2025-07-01T21:49:37.871697351Z stdout F 2025-07-01 21:49:37.871 UTC [http-nio-8443-exec-6, , , TxId: ] INFO  c.v.vchs.hybridity.audit.AuditTrail- {"internal":{"threadEnterprise":"No Thread Context","threadUser":"No Thread Context","thread":"http-nio-8443-exec-6","lineNumber":165,"classname":"com.vmware.vchs.hybridity.api.LoginUtil","method":"logAuth"},"userIdentities":[{"username":"[email protected]","tenantId":"default","enterprise":"DEFAULT","organization":"DEFAULT","userRoles":[],"endpointId":"20211213133321794-6649ea74-ce9c-4ea0-bb46-f1e0870c3ca0"}],"tenantIds":["default"],"severity":"CRITICAL","userIdentity":{"username":"[email protected] "},"eventId":"7ad5d52c-8301-49af-83b6-3c0705e109c1","eventTime":1751406577871,"message":"Access Denied","eventName":"Login Failed","service":{"name":"Login Failed"},"restEndpoint":{"uri":"\/hybridity\/api\/sessions","method":"POST","sourceIPAddress":"1.1.1.1"},"requestParameters":{"query":[]},"responseElements":{"isAuthenticated":"false"}}
2025-07-01T21:49:37.873228668Z stdout F 2025-07-01 21:49:37.873 UTC [http-nio-8443-exec-6, , , TxId: ] ERROR c.v.v.h.a.HybridityAccessDeniedHandlerImpl- Sending Response Error 403 for /hybridity/api/sessions
2025-07-01T21:49:40.442442758Z stdout F 2025-07-01 21:49:40.442 UTC [http-nio-8443-exec-4, , , TxId: ] ERROR c.v.v.h.a.JwtAuthenticationProvider- JWT token rejected:
2025-07-01T21:49:40.442465175Z stdout F com.nimbusds.jwt.proc.BadJWTException: Token rejected because of expired/before use time. Server time is 2025-07-01T21:49:40Z
2025-07-01T21:49:40.442471644Z stdout F         at com.vmware.vchs.hybridity.authentication.JwtClaimsVerifier.verifyTokenTime(JwtClaimsVerifier.java:85)

- This issue occurs due to a clock skew problem. Clock skew is the range of time allowed for a server to accept the authentication.

A SAML token contains information about the lifetime of a token. A SAML token uses the NotBefore and NotOnOrAfter attributes of the SAML Conditions element to define the token lifetime. <saml2:Conditions NotBefore="XXXX" NotOnOrAfter="XXX">

During a token’s lifetime, the vCenter Single Sign-On server considers any request containing that token to be valid and the server will perform renewal and validation operations on the token. The lifetime of a token is affected by a clock tolerance value that the vCenter Single Sign-On server applies to token requests. The clock tolerance value accounts for differences between time values generated by different systems in the vSphere environment. The clock tolerance is 10 minutes.

 

Resolution

Client (Telco control plane) and Server (vCenter server) date and time should be in sync 

• Validate if both machines are configured for the same NTP server
• If the NTP server is not in use, sync time with ESXi host

Enable the "Synchronize time periodically" option under VMware Tools settings. When enabled, it allows the virtual machine's (VM) clock to be periodically synchronized with the host machine's clock. This helps maintain accurate timekeeping on the guest OS by compensating for any drift or discrepancies that may occur.

How to enable "Synchronize time periodically":

1. Access VM settings: Open the settings for the specific virtual machine in vSphere Web Client. 
2. Navigate to VMware Tools: Locate the VMware Tools settings within the VM options. 
3. Locate Synchronization options: Find the "Synchronize time with host" or similar option, which might be under an "Advanced" section. 
4. Enable periodic synchronization: Check the box next to "Synchronize time periodically" that enables periodic polling and resyncing with the host's clock.  
5. Apply changes: Save the settings to activate the periodic time synchronization.

Once time is synced, perform the validation of vSphere endpoint, degraded status changes to connected status