January Cumulative Security updates prevent endpoints from registering to task server
Article ID: 232242
Updated On:
Products
Issue/Introduction
After installing January (and later) OS Updates, endpoints are not able to register with any task server. See the Cause section of this article for a partial list of known MS Updates.
Agent logs may show the following sequence of errors:
HTTPS: Authentication failed, response to server challenge denied, check credentials are correct, error: The logon attempt failed (0x8009030C).
Then goes to:
altiris/TaskManagement/CTAgent/GetClientTaskServers.aspx and gets Error note: Authentication failed, response to server challenge denied, check credentials are correct, error: The logon attempt failed (0x8009030C)
Next is Task Server Connection:
Failed to request 'https://<serverFQDN>.example.com:443/altiris/TaskManagement/CTAgent/GetClientTaskServers.aspx?shares=1&resourceGuid=e6e4801e-98b3-436f-bcce-832e1ba6ccdb&crc=0008000600000896', error: HTTP status 401: The request requires user authentication (0x8FA10191)
Finally ending with Task Server Connection:
Failed to refresh server list or register, error: HTTP status 401: The request requires user authentication (0x8FA10191)
Under Windows Event logs:
LSA (LsaSRV) event 40970 in the System event log
Environment
ITMS 8.x
Cause
Windows Updates released on January 11, 2022, introduced intentional hardening changes that prevent Kerberos to NTLM fallback in specific scenarios.
Related bulletins that cause this issue (This is a sample list):
Server 2022 - KB5009545
Server 2019 - KB5009557, KB5010791
Server 2016 - KB5009546
Server 2012 R2 - KB5009624
Windows 10 - KB5009543, KB5008876, KB5009546, KB5010793
All Out of Band updates released January 17th i.e. KB5010790 - 0794, etc.
All OS updates were released in February.
Resolution
Starting with our ITMS 8.6 RU2 version (released at the end of January), it has a change for automatic fallback to the NTLM if authentication with Kerberos fails because 3-part SPNs are not configured properly. This is a client-side change meaning that upgrading the Symantec Management Agent (SMA) to 8.6 RU2 or later is required.
With our ITMS 8.6 RU3 Release everything should work out of the box. No longer required to make the changes previously suggested to mitigate the changes introduced by Microsoft regarding Kerberos and SPNs.
The Symantec Management Agent 8.6 RU3 version adds SPNs for the servers itself.
Delegation is also set during Site Server configuration on the client side.
And SPN format has been changed in our code to comply with recent Microsoft requirements, so the manual changes recommended originally on this document are not longer needed anymore.
Broadcom developed a set of point fixes for ITMS 8.5 RU4, 8.6 RU1, and 8.6 RU2 where we provide additional changes.
We are also actively evaluating what other authentication protocols we can support in addition to Kerberos and NTLMv2 to cover all product use cases.
Point Fix Availability
Dev created point fixes for 8.5 RU4, 8.6 RU1, and 8.6 RU2. These fixes address the NTLM / Kerberos issues and will require the upgraded Symantec Management Agent to be rolled out to fully resolve this issue.
NOTE: SPNs are still required if Kerberos authentication is to be used. Contact Microsoft or see the information below, and the attached script, for information on creating SPNs to properly set up Kerberos authentication.
8.5 RU4: A point fix for ITMS 8.5 RU4 is currently available. See KB 198337 "CUMULATIVE POST ITMS 8.5 RU4 POINT FIXES"
8.6 RU1: A point fix for ITMS 8.6 RU1 is currently available. See KB 221269 "CUMULATIVE POST ITMS 8.6 RU1 POINT FIXES"
8.6 RU2: A point fix for ITMS 8.6 RU2 is currently available. See KB 235538 "CUMULATIVE POST ITMS 8.6 RU2 POINT FIXES"
Next Best Option (recommended by Microsoft):
Microsoft has released a public KB on the topic. Please refer to:
KB5011233: Protections in CVE-2022-21920 may block NTLM authentication if Kerberos authentication is not successful (microsoft.com)
1. Make sure the January MS patches are installed again if those were removed to avoid this issue on your Task Servers (TS), Symantec Management Platform (SMP), and Domain Controllers (DC).
2. On the Domain Controller, open a command prompt as Administrator and run the following command for each port used (you have to add SPNs for every Site Server you have, even Package Servers (PS)):
NOTE: A PowerShell script has been created to execute the SPN commands below. If you would like to use it simply add the Server Names of your Altiris environment to the $Server variable and execute the script on the Domain Controller. See the Additional Information below for examples. It might be also required to add SMP Alias (if alias name is used) to be added to SPN script OR update the SMP / Task Server Communication Profile to only include the FQDN of the server (remove the entry for the NonFQDN hostnames).
NOTE: A client machine provides SPN in the form HTTP/host:port/SMP or HTTPS/host:port/SMP to the call.
Example format:
<Protocol>/<serverFQDN>:<Port>
--Notice single space between .../SMP and <domain>...
Setspn -S HTTP/<ServerHostname>.<yourDomain.local>:80/SMP <yourDomain>\<ServerHostname>
Setspn -S HTTPS/<ServerHostname>.<yourDomain.local>:443/SMP <yourDomain>\<ServerHostname>
NOTE: You don't need to worry about the correct syntax if you use the attached script.
Microsoft is recommending using the SetSPN command like below to add 3-part SPNs recorded in the LSA 40970 events in the system event logs on the computer account of the Task servers. This suggested change should resolve the authentication issue after this change is replicated across DCs in the domain.
Example:
We will use ITMS-SS-01 and ITMS-SMP-01 as the name of the Task Server and SMP server respectively as examples. ITMS.local is the example domain name. Port 80, 443, and 4726 are the default ones for regular agent and CEM communications.
If your environment uses only short names, or only HTTP or a different port number(s) as a reference of your servers, you will need to include to the SetSPN command.
Setspn -S HTTP/ITMS-SS-01.example.local:80/SMP itms.local\ITMS-SS-01
Setspn -S HTTPS/ITMS-SS-01.example.local:443/SMP itms.local\ITMS-SS-01
Setspn -S HTTPS/ITMS-SS-01.example.local:4726/SMP itms.local\ITMS-SS-01
Setspn -S HTTP/ITMS-SMP-01.example.local:80/SMP itms.local\ITMS-SMP-01
Setspn -S HTTPS/ITMS-SMP-01.example.local:443/SMP itms.local\ITMS-SMP-01
Setspn -S HTTPS/ITMS-SMP-01.example.local:4726/SMP itms.local\ITMS-SMP-01
NOTE: There is a script attached to this KB that can be used to set these SPNs.
For Reference: https://social.technet.microsoft.com/wiki/contents/articles/717.service-principal-names-spn-setspn-syntax.aspx
Now, you should get a registration back to your Task Server:
Please refer to our KB 223575 "How to set up Symantec Management Platform environment to use Kerberos authentication if NTLM is disabled" for further details on what to consider for implementing Kerberos.
NOTE: In some cases when in the Global Agent Settings, the Agent Connectivity Credentials (ACC account) is configured instead of the Application Credentials (App ID), running the SetSPN script is not enough and you should also change the IIS and put the NTLM on top of Negotiate and it is explained below, in workarounds.
Other Possible Workarounds
NOTE: The following are provided as possible workarounds to mitigate the effect of this Microsoft change. These may not work for everyone.
If you can't set the SPNs entries as recommended by Microsoft in the above reference, please try the following steps to set NTLM as the first provider.
Note: In a few implementations, the following steps were also needed even after adding the SPNs references from the SMP server and Task Servers, especially under situations where there are client machines NOT in a Domain, or while in CEM (Cloud-enabled Management) mode.
1. Change the "Providers" order for Windows Authentication in IIS Manager. Using NTLM as the primary authentication provider is not preferred with the recent changes enforced by Microsoft's January patches but it should help until Kerberos is properly configured on environments:
A) On the SMP server and any servers with Task Services installed, place NTLM first and then Negotiate on the following locations in IIS manager:
Default Website/Altiris/TaskManagement (SMP)
Default Website/Altiris/TaskManagement/CTAgent (SMP)
Symantec Agent/Altiris/TaskManagement
Symantec Agent/Altiris/TaskManagement/CTAgent
Moving the "Enabled Providers" order:
- Open IIS Manager, and select the Default Web Site and go to Default Website/Altiris/TaskManagement (do the same with Default Website/Altiris/TaskManagement/CTAgent/ and Default Website/Altiris/ClientTaskServer/ )
- Click on the Authentication module
- Select Windows Authentication and choose Providers from the Actions pane
- Move NTLM to the top of Enabled Providers, click OK
- If Symantec Agent site exists for CEM implementations, do the same for Symantec Agent/Altiris/TaskManagement, Symantec Agent/Altiris/TaskManagement/CTAgent/ and Symantec Agent/Altiris/ClientTaskServer/
- Open IIS Manager, select the Default Web Site and go to Default Website/Altiris/ClientTaskServer
- Click on the Authentication module
- Select Windows Authentication and choose Providers from the Actions pane
- Move NTLM to the top of Enabled Providers, click OK
To test functionality after making the changes above, open up the Symantec Management Agent UI on the Task Server, go to the Task Server Tab, and click the "Reset Agent" button. The agent should register to a Task Server.
NOTE: Even when the main affected area has Task Server connectivity, it may not be enough to change the Task Server-related folders only. You might need to also consider changing the CEM folders, Package Server folders, and possibly some of the Solution folders in IIS as well. This is because it is not just Task Server calls that use authentication but some others may as well. If the errors mentioned in this article are also seen on other pages, i.e. the suggested NTLM order change on those other pages may need to be changed also.
Additional Information
To use the PowerShell Script attached to this KB, this is what needs to be modified with the Hostnames of your SMP, and Task Servers
##Enter HOSTNAMES ONLY BELOW. Do not enter the DOMAIN, that is retrieved from system environment
$Server = @('hostname1','hostname2','etc') ## ONLY EDIT THIS LINE, NOTHING ELSE
Hostname is the NETBIOS / short name i.e. SMP, SiteServer, or MyServer1234.
The script will create Port 80, 443, 4726 entries for each server. If you're not using these today, they will be available in the future should you make any changes.
If you decide later to Remove the SPN's that have been created, change the -S to -D for $arg1-3, then run the script again and the SPN will be deleted.
i.e.: $arg1 = "-S HTTP/" + $SvrFQDN + ":80/SMP $domain\$Svr"
to $arg1 = "-D HTTP/" + $SvrFQDN + ":80/SMP $domain\$Svr"
Attachments
Feedback
