Error: "HTTP status 500: An unexpected condition prevented the server from fulfilling the request (0x8FA101F4)" when unable to establish CEM connection.

Products

• IT Management Suite

Issue/Introduction

The client machines in CEM mode are displaying the following error messages when trying to connect to the ITMS environment:

The logs show the following:

Failed to establish main persistent server connection, error: HTTP status 500: An unexpected condition prevented the server from fulfilling the request (0x8FA101F4)
-----------------------------------------------------------------------------------------------------
Date: 10/19/2021 8:28:42 PM, Tick Count: 2553203 (00:42:33.2030000), Size: 386 B
Process: AeXNSAgent.exe (4508), Thread ID: 7376, Module: AeXNSAgent.exe
Priority: 2, Source: Agent
>>> 

[18FC5FD0010, WS: 690, RECV: 3F17F003] UPGRADE request failed, the server might not support WebSocket protocol, error: HTTP status 500: An unexpected condition prevented the server from fulfilling the request (0x8FA101F4)
-----------------------------------------------------------------------------------------------------
Date: 10/19/2021 8:28:42 PM, Tick Count: 2553203 (00:42:33.2030000), Size: 464 B
Process: AeXNSAgent.exe (4508), Thread ID: 19788, Module: AeXNetComms.dll
Priority: 1, Source: SMAIO.WSTransport.Socket
>>>

Policy request failed, COM error: HTTP status 500: An unexpected condition prevented the server from fulfilling the request (0x8FA101F4)
-----------------------------------------------------------------------------------------------------
Date: 10/19/2021 8:28:38 PM, Tick Count: 2549296 (00:42:29.2960000), Size: 366 B
Process: AeXNSAgent.exe (4508), Thread ID: 14312, Module: AeXNSAgent.exe
Priority: 1, Source: ConfigServer
>>>

Operation 'Direct: Head' failed. 
Protocol: HTTPS 
Host: itmgmt01.example.com:443 
Path: /altiris/NS/Agent/GetClientPolicies.aspx 
Connection Id: 15.4508 
Communication profile Id: {C9673F7F-41C8-4C11-8681-5F76D1B569C6} 
Throttling: 0 0 0 
Error type: HTTP error 
Error code: HTTP status 500: An unexpected condition prevented the server from fulfilling the request (0x8FA101F4) 
Error note: 500 
Server HTTPS connection info: 
   Server certificate: 
      Serial number: xxxxxxxxxx e7 fe 8f 80 ab 
      Thumbprint: xxxxxxxxxxx 55 65 c9 ab 99 72 a1 a6 ab 63 ce 
   Cryptographic protocol: TLS 1.2 
   Cipher suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 
   Cipher algorithm: AES 
   Cipher key length: 256 
   Hash algorithm:  
   Hash length: 0 
   Key exchange algorithm: ECDH 
   Key length: 255
-----------------------------------------------------------------------------------------------------
Date: 10/19/2021 8:28:38 PM, Tick Count: 2549296 (00:42:29.2960000), Size: 1.08 KB
Process: AeXNSAgent.exe (4508), Thread ID: 14312, Module: AeXNetComms.dll
Priority: 1, Source: NetworkOperation
>>>

Calling NS server endpoint 'https://itmgmt01.example.com:443/altiris/NS/Agent/GetClientPolicies.aspx', ID: {32FB5E6D-4E8B-4965-9E4F-9642A99A95E6}
-----------------------------------------------------------------------------------------------------
Date: 10/19/2021 8:28:37 PM, Tick Count: 2547984 (00:42:27.9840000), Size: 378 B
Process: AeXNSAgent.exe (4508), Thread ID: 14312, Module: AeXNSAgent.exe
Priority: 4, Source: Agent

The same certificate was being used for everything: SMP, Gateway, and CEM clients. The expected locations are showing the right certificate assignments (Default Website, Symantec Agent Site, Gateway):

• (Note: for this article itmgmt01.example.com and c5 15 da a5 a7 90 7b 6a bd 55 65 c9 ab 99 72 a1 a6 ab 63 ce as example values)
• The certificate is itmgmt01.example.com with a thumbprint of C5 15 DA A5....
• The default website is using port 443 and itmgmt01.example.com with a thumbprint of C5 15 DA A5..
• Symantec Agent site is using port 4726 and itmgmt01.example.com with a thumbprint of C5 15 DA A5....
• Gateway UI is using port 443 and itmgmt01.example.com with a thumbprint of C5 15 DA A5....
• The Gateway policy has the external name itmgmt01.domain.com with a thumbprint of C5 15 DA A5....
• TLS 1.0, 1.1, 1.2 are enabled in the agent communication profile and Site Server communication profiles. 

Confirmed that a static IP address was used for the external IP address for the Internet Gateway.

When the client machine is connected to the internal network it looks fine, both the request configuration and send basic inventory works.

Environment

ITMS 8.x

Cause

There was a misconfiguration and an unusual implementation of how the external name was use. If the standard CEM implementation steps are followed, then you will know that you need to work on getting a unique external name for your Internet Gateway (something other than the actual SMP server name) that is reachable externally by the client machines, and you will need to generate a new gateway certificate that reflects the external name. Then, update the gateway policy with that new name and add the new thumbprint from the gateway.

Resolution

Use the SMP server name as the external name:  itmgmt01.example.com

The gateway policy also had the SMP name because it was the external name to connect to the gateway according to the setup.

The firewall was set up to redirect the external traffic to the gateway but because the client machines were able to resolve the SMP server name directly, they were not trying to connect via the gateway.

Having "itmgmt01.example.com" as the SMP server name, external reachable gateway name, and the name for the primary certificate name, caused some confusion on the client machines in CEM mode. 

Since the gateway is using a static IP address (the actual gateway hostname is not reachable externally) that is reachable externally, it is suggested that you change the gateway policy and add the IP address instead of the name for the gateway. And, since the client machines are using the same certificate for everything, you may not need to change the current certificate and thumbprint on the gateway.