Understanding how Token-based authentication works

Products

• IT Management Suite

Issue/Introduction

You have or are planning on implementing Token-based authentication on your environment based on recommendations from:

Using_ITMS_with_Token_Based_Authentication.pdf

However, you would like to know details about functionality, security considerations, and basic troubleshooting.

Environment

ITMS 8.7.3

Resolution

The following is provided as a overview of things you should consider/know on how it works (that could be helpful for troubleshooting possible issues):

The new token-based authentication method is meant to replace ACC authentication in the long term and free customers from ACC account password change related problems.

Some background process information:

1. SMP Token Authentication protocol is a one or two steps protocol that involves token exchange between a client and a server.
2. If SMP server supports token authentication and the request needs to be authenticated, then SMP server generates JWT token and sends it back to the client.
3. Only that agent can use this token to authenticate on the SMP server. The client identity ensured by using client's SHA-256 RSA2048 key by both sides. The client also saves the token received from the server for further usage while communicating to this SMP server.
4. Client validates three-part JWT token by checking the server signature first. If signature is valid then client extracts nonce value from JWT header and checks that unique identifier from nonce is the same as was sent in the first request. Then client increments the counter in the nonce, joins the new nonce to the three-part JWT token and signs the resulting four-part JWT token using client RSA2048 key. The result JWT token that client sends to the server is five-part JWT token.
5. SMP Server validates five-part JWT token by checking the client signature first. if signature is valid then server checks token expiration time and if token has not expired yet it finally checks if nonce value contains the incremented counter. If all checks succeed then the connection is considered authenticated, all further requests using the same connection will not need to be authenticated again. 
6. The client stores the last token received from the SMP server and tries using it for future communications with the same server. If token is not expired at that time, then the authentication process can become one step authentication, i.e. client send "Authorization" HTTP header right away bearing the stored JWT token and server succeeds validating that token and processes the request immediately without going through 401 response first.
7. JWT tokens are cached in token storage in memory, SMA service restart will cause new tokens to be requested from the SMP server.
8. JWT tokens are associated with the connection path to the particular SMP server, so if path to the SMP server changes, then the new tokens will be requested. 
9. Agent stores tokens, which are checked for expiration and usage periodically. If tokens are not used for 24 hours then they are removed from the storage every 6 hours, which is reflected in the log, see below:
   
10. If 403 error is received and the stored token was used, then that token is removed from the storage. The new token will be stored once authentication using the newly received token is successful.
11. If multiple requests are sent through the same connection, then only the first request gets authenticated, the tokens are not sent on the consecutive requests. 
12. Logs can be used to see what kind of authentication was used for a particular server or proxy authentication., see sample below. NTLM, Negotiate or Token can be specified there.
    
13. IIS module DLL instance is loaded into context of w3wp.exe process. There can be many w3wp.exe instances on the machine depending on the number of IIS application pools, so there can be many IIS module instances on the machine.
14. IIS module interacts with SMA agent via COM interface when it needs to issue the agent token to the client or to validate the agent or ASDK token.
15. SMA agent uses SMP server's Altiris.NS.dll functionality to issue and validate tokens.
16. SMA uses agent token registry to protect against usage of the stolen tokens, each agent token can be used only once.
17. Site Servers doesn't requires anything special that have to meet in order to use Token-based authentication.

Other relevant functionality introduced:

1. The SMP server part of token-based authentication functionality is implemented as part of a new native IIS module. The token-based authentication is required on SMP and Site servers, the same IIS module is used on every server.
2. IIS module is named AeXIISExt.dll and is added to SMA package.  The module does not have any external DLL dependencies that needs to be installed.
3. IIS module is installed, updated and removed during SMA setup, the module is located in SMA directory.
4. SMA setup randomizes the name of DLL by adding a random GUID to the file name, see below:
   
   AeXIISExt.{B4A9A7B7-2F4E-43BE-8591-6FDDDC94C245}.dll
   
   
5. The random name allows SMA setup to upgrade IIS module without the need to restart IIS - the new module is copied with the new name then the old module gets unregistered and removed.
6. When IIS module is registered on the machine then the registry key "HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent\IISModule" gets created or updated.
   
   

7. IIS module is not used by IIS unless token-based authentication or ASDK token attention is enabled. These settings are part of "Site Server" policy.

8. SMA registers IIS module in IIS as soon as one of those settings are on in the policy and unregister them from IIS as soon as every setting is off.

9. IIS module is registered on the default web site (site 1) on the site server or on the main and CEM sites configured in the registry on SMP server.
   
   

10. IIS module registration operates on applicationHost.config file in C:\Windows\System32\inetsrv\config directory.

11. Two new command-line parameters were added to AEXNSAGENT.EXE and AEXAGENTUTIL.EXE, they can be used to manually register IIS module on IIS:

    /registeriismodule - register the current IIS module on IIS. IIS will load IIS module as soon as first HTTP request to '\Altiris web site is received.
    /unregisteriismodule - unregister the current IIS module on IIS. IIS will unload IIS module immediately

12. This sequence below can be used to manually apply module hot fix if we will need to:

    1. Copy new IIS module DLL with a new random name (actually any name will do)
    2. Register DLL by calling REGSVR32 <new IIS module name>
    3. Call AEXNSAGENT /REGISTERIISMODULE
    4. Remove the old IID module DLL

    The IIS will unload the old IIS module and load the new one as a result of the sequence execution.

13. When IIS module is installed on site server then it operates differently as it cannot issue agent tokens itself and must turn to SMP server for agent tokens

14. Overall, the authentication process is the same as for authentication on SMP server. The difference is that instead of using Altiris.NS.dll to generate the tokens in context of the same process, SMA makes a web call to SMP server to generate an agent token.

15. When agent token is received then it is stored in token cache and if the same agent with the same unique identifier in nonce (used on one occasion) requires a new token then the already cached token will be passed to the client.

    Token cache is operated the following way:

    1. Tokens are requested by calling /NS/Agent/GetClientTokens.aspx web page. Each call contains the signature and the hash so SMP server can validate which site server asks for a token and deny request to not legitimate site servers.
    2. If connection to SMP Server uses HTTPS then GetClientTokens.aspx response is not encrypted, if connection uses HTTP then the response is encrypted. The response contains the response hash and it is signed.
    3. The first time a token for a particular agent can be requested from SMP server is when agent is trying to authenticate on site server for the first time.
    4. The received tokens are stored in token cache until they are expired, if token is expired and it has never been requested again by the agent it will be removed.
    5. A token can be updated in the cache if it is to be expired in 15 minutes or less and if it was requested by the client at least once more.
    6. The cache is being periodically validated for expired tokens and tokens that need to be updated. The validation sequence can be observed in the logs:
       
       
       
       
    7. The initial validation period is 5 minutes, then it can be increased to 10 or 30 minutes depending on the current token expiration period that can be modified in ITMS console.
    8. Multiple about to be expired tokens can be requested from SMP server during validation sequence. GetClientTokens.aspx supports batch tokens requests.
    9. Each validation sequence can request up to 4096 tokens from the SMP server.  Each GetClientTokens.aspx request contains requests for up to 128 agents, so multiple requests can be performed to request all 4096 tokens.
    10. If more than 4096 tokens need to be refreshed, then more than one token validation sequence is needed.
    11. Validation sequence uses the single network connection to make all requests.
    12. However, if multiple agents are requesting the tokens for the first time, then SMA can use multiple network connections to perform simultaneous requests to SMP server.
    13. SMA can perform up to 32 simultaneous GetClientTokens.aspx requests to the server. If 33rd request is needed, then SMA will wait until one of the currently running requests is finished.

16. Site server needs the public key of SMP server and all servers from within server's hierarchy in order to be able to validate a token signature. These keys are distributed as part of client registration request or as part of client policy and stored in secure storage and in server key cache in RAM. Look for "Server keys:" record in the log to see which keys were received, the record contains the key id and the key length.

    

    The token validation sequence on site server matches one on SMP server with a minor difference:

    1. Token syntax is checked. Token considered invalid if any of these checks fail.
    2. SMP server or site server key is looked in server keys cache, if key is not found then token is considered invalid.
    3. Server signature of the token is checked. If signature is invalid, then token is considered invalid.
    4. Agent key is looked up in token cache.  If key is not found then token is considered invalid.
    5. If signature is invalid, then agent is considered revoked and will be informed about this via 403 error accompanied by the corresponding error code in X-SMP-NSError header.
    6. Token expiration time is checked. If it is over, then token is considered expired.

    In all cases when token sent by the client is invalid or when token is expired, it will be discarded, and the site server will issue the new token to the client and send to the client via 401 error. The new token will either be taken from token cache or requested from SMP server

17. Token registry content can be seen in the NS/Agent logs, the highlighted records show the counter of the processed token and the final registry content - counter and "window".
    
    

18. The registry gets cleaned up:

    1. When token expires during authentication
    2. Periodically every 3 hours, that cleans up expired tokens that were not used by the agent 

    The cleanup process can be seen in the logs, look for "Token XXXXX is expired and removed" record on the screenshot above.

19. Various places in core SMA are touched during token-based authentication implementation. Basically, every place where ACC authentication could be used before was changed. These places include but no limited to:

    1. P2P functionality is using ACC credentials to perform "basic" authentication for troubleshooting purposes via internet browser, peers do not use "basic" themselves. Peers use custom KMS keys-based challenge - response authorization when performing downloads from one another. 
    2. Push-install service. Although package downloads from SMP server do not require authentication, downloads from Package Servers require one. Both ACC and token authentication can be used. Push install service uses the new permanent ''push install token for token authentication, this is a special type of token that allows access to AEXNSC.exe on servers only, its lifetime is infinite, but it can be revoked if needed.
    3. SMA stores ACC credentials in secure storage (under C:\ProgramData\Symantec\Symantec Agent\Ldb).
       SMATOOL /STORAGE ENUM can be used to see the current credentials under SYSTEM account if troubleshooting password enabled. If ACC authentication is disabled, then these records do not exist. SMATOOL output sample is shown below:
       
       
       
    4. Quick installer should be able to successfully initialize and download packages upon receiving the initial agent policy that either contains or does not contain ACC credentials. Quick Installer should be able to download packages from Package Servers using token authentication
    5. Package download when redirected to another Site servers can require different type of authentication and works transparently when ACC authentication is enabled on one server and token authentication on another.
       
       Note:
       ACC credentials will be the primary authentication method if token-based doesn't work for whatever reason.
       Now, if the package location is a UNC path, token-based will not be used (UNC access it is not supported with that token-based method) and ACC account will be the one in use if "Agent Connectivity Credentials" option is turned on as “fallback mechanism” option. The agent switches to ACC credentials to connect to a UNC location when it detects that it is a UNC codebase.
       We mainly anticipate that the authentication falls back from token-based authentication to ACC credentials if the package destination is a UNC location. Again, Agent tokens do not support UNC code bases.  Use of the ACC account is required for UNC code bases. ACC account must be kept if customer uses UNC locations for packages including DS images. This includes Software Library if configured as UNC for the SMP Server.

New NS and site server user account _SMP_IUSR and group_SMP_USERS

Even when ACC account is not used the Site server or SMP server still needs some real local identity to represent the user logged in as a result of a JWT token authentication.

1. This identity is represented by the new local account named _SMP_IUSR. The new account can be used:
   1. By IIS server to impersonate the logged-on user while executing the request handlers
   2. By Windows security descriptors applied to folders, files or other resources to restrict access to those resources
2. _SMP_IUSR is created and managed by SMA.
3. The account is created as soon as "Site Server" policy with enabled token authentication is received and is removed when token authentication is off.
4. The account's password is randomly generated and not stored anywhere. The account and its password is created or updated when:
   1. SMA starts
   2. "Site Server" policy changes
   3. Periodically every 6 hours.  Controlled by the old registry value "Server Security Environment Refresh Interval (mins)" under HKEY_LOCAL_MACHINE\SOFTWARE\Altiris\Altiris Agent\Servers\<server> key.
5. The password is modified every time the account is updated.
6. The account management actions can be monitored in the log by using Log Viewer filter "_SMP_IUSR"
   
   
   
   
7. The generated password length is chosen depending on Windows password policy settings:
   
   
   • If minimal password length is less than 14,, then 14 characters password is generated
   • If minimal password length is greater than 128,, then 128 characters password is generated. Group Policy Editor limits minimal password length to 128 characters.
   • The password length corresponds to the minimal password length setting in all other cases.
8. The password length is always reported to the event log.
9. If the generated password cannot be used because of Windows restrictions, then SMA will try generating another password up to 32 times and then will give up.
10. A special alert is sent to SMP server in case '_SMP_IUSR' cannot be created or updated.
11. This reference account id can be observed in SMA logs as well.
    
    
    
    
12. Two new commands have been added to SMATOOL.EXE (Symantec Management Agent (Altiris Agent) Troubleshooting)
    
    
    • SMATOOL /SYSTEM DUMP SECURITYSETTINGS
    • SMATOOL /SYSTEM DUMP PASSWORDPOLICY
13. These commands can be used during testing or troubleshooting to validate if SMA is able to detect the password policy settings correctly. The password policy reading method is not 100% documented.
14. '_SMP_USERS' group is also created and managed by SMA on Site servers and SMP server. The group is created to simplify security management when token authentication or ACC authentication are being turned off or on:

• If token authentication is enabled, then '_SMP_IUSR' account is automatically added to '_SMP_USERS' group
• If ACC authentication is enabled, then ACC account is automatically added to '_SMP_USERS' group
• If either authentication type is disabled, then the corresponding account is removed to '_SMP_USERS' group
• SMP uses '_SMP_USERS' group in various security descriptors of file system objects or web sites to allow access to ACC account or clients that use token authentication
• There is no need to rebuild security descriptor in case authentication types get enabled or disables because the new group is always there.
• SMA validates the group periodically and recreates it if needed.

Security considerations

SMP token-based authentication protocol implementation allows new set of remote attacks against the protocol itself, which can include token hijacking, forging and replaying.

While the appropriate measures like usage of asynchronous cryptography to protect against such sort of remote attacks have been implemented, certain local security measures have been slightly loosened to allow token authentication to work.

Such measure include:

1.  We have tightened SMA COM security couple of releases back, so even if malicious code can be injected into our Application Pool, it will not be able to call interfaces SYSTEM and administrators are able to call. Screenshot below shows DCOMCNFG security descriptor for Altiris Agent COM object. These security restrictions are applied only on SMP server and Site server but not on the endpoint clients, so this reduces the danger.

             

2. The new SMA COM interface method that allows caller to request agent token can be called by a regular user on SMP server and Site server now, but even if malicious code called this method and received agent token, it would need the legitimate remote agent's RSA2048 key in order to sign the token and create the valid agent token. Stealing RSA2048 key requires some extra work that is not feasible.

3. The new SMA COM interface method that allows caller to validate token and receive handle to _SMP_IUSR account and gain access to protected resources would require a valid agent or ASDK token. This is hard or impossible to do for agents tokens because of step #2 and replay attack protection, but since ASDK token protection depends solely on customer security measures while distributing ASDK token, validating ASDK token can be achievable with the stolen token, but this should be executed on SMP server or Site server, which are supposed to be protected in the first place.

4. SMA logging. IIS module writes into SMA logs located in %ProgramData%\Symantec\Symantec Agent\Logs folder. The previous security settings for the folder and the content did not allow file deletion for "Users" group, so the code running in context of any user account cannot rotate the log files when the current file is full.
The new security settings are applied as soon as policy with the enabled token-based authentication arrives to the client and reset to the default when token-based authentication is off. The new security rights look like below:

• • SYSTEM and Administrators group- full access
  • Users group - full access expect for file and folder deletion right and security changing rights
  • DefaultAppPool - full access except for security changing rights
  • Symantec Task Server AppPool - full access except for security changing rights
  • Symantec Agent AppPool - full access except for security changing rights
  • SMP Server AppPool - full access except for security changing rights

Few things to consider:

1. Normally when client signature validation fails on the server side, then HTTP error 403 is returned to the client to deny malicious client or disallow stolen tokens to be used.
2. If you have "Allow anonymous access to package codebases" selected (under SMP Console>Settings>Notification Server>Site Server Settings>Site Management>Site Server Settings>Package Service>Package Service Settings), "Access Type" under the "Package" tab on your Package Server will display as "Anonymous"
   
   
   If it is unchecked and you have Token-based and ACC authentication enebled, then "Access Type" will display as "ACC,Token"
   
   
3. Windows and token-based authentication can be simultaneously enabled on SMP or Site server. Multiple authentication scenarios exists when various token-based authentication types get enabled or disabled. No agent, IIS, SMA or SMP server restarts are needed when authentication types change.
4. Agent HTTP transport is tuned to work in such scenarios, for example, when agent sends a cached token in web request to the server, but SMP server requires Windows authentication now.
5. There is no ability on the agent to disable ACC or token-based authentication, if SMP server requests either of them, then the client will try using it.
6. Token-based authentication had preference over Windows authentication.
7. UNC transfer does not work if ACC credentials are not available on the client and transfers should be authenticated. The attempt will fail with the corresponding error description.
8. Proxy authentication including proxied WebSocket connections does not support token authentication.

IIS module troubleshooting

Logging

1. IIS module does not log anything into SMA logs by default in order to not affect IIS performance.
2. IIS module processing results can be observed in IIS logs and HTTP responses by default only.
3. There is an ability to turn on IIS logging by using "LoggingSeverity" registry entry under IIS module key, see below:
   
   
   
   
4. The value is a bit mask, which bits describe various logging levels and matches SMA mask, i.e.:
   
   
   1. Error - 1
   2. Warning - 2
   3. Info - 4
   4. Trace - 8
   5. Verbose - 0x10
   6. Debug - 0x20
5. If IIS logging is turned on then IIS module will write log records into SMA log files.
   
   Note: Both SMA logging level and IIS logging level must be turned on for IIS logging to work
   
   
6. Most of the records are logged on Debug level, IIS module works by intercepting every web request coming to IIS, so the lowest level includes all the steps of request processing as shown on the sample below:
   
   
   
   
7. Each request processing record is proceeded by the request number, which allows user to group them together.
8. The records dumping request and response headers (shown below) also contain IIS connection and request ids, so requests coming on the same connection can be grouped together.
9. The connection id on the sample below is fa0000013000027e, the request id is fa0000014000027f. The same ids can be found in the client-side logs in case "Debug" logging level is turned on for IIS module.
   
   

Useful Altiris Log Viewer filters

The following strings can be used to extract the needed information from the NS/Agent logs:

Filter	Description
NetworkOperation	Use on the client to see which network request the client is performing
IISModule	Use on SMP or site server to see the whole sequence of http request processing by IIS module. Require IIS module logging to be enabled
Token	Use on SMP or site server to see token related operations, like generation, expiration, usage and cleanup.
Request:	Use on SMP or site server to see the request coming to IIS. Require IIS module logging to be enabled
Response:	Use on SMP or site server to see the responses coming from IIS. Require IIS module logging to be enabled
SiteServer	Use on SMP or site server to see token processing by SMA agent.
Token cache	Use on site server to see token requests from SMP server and their caching
Token registry	Use on SMP or site server to see token registration and usage
Token storage	Use on client side to see token cache operations in HTTP transport.  This is a client-side cache as opposed to "Token cache" records that reflect site server's token cache operations.
Server keys:	Use on client to see which SMP and hierarchy server keys are available for usage