How to setup Kerberos Authentication on Windows using AES encryption type.
search cancel

How to setup Kerberos Authentication on Windows using AES encryption type.

book

Article ID: 411265

calendar_today

Updated On:

Products

SITEMINDER CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On Agents (SiteMinder)

Issue/Introduction

This is to demonstrate setting up SiteMinder Kerberos Authentication on Windows environment based on the Official Documentation but using AES encryption.

Cause

Documentation is based on using RC4-HMAC encryption type.

There are more configuration steps required when using AES encryption types.

 

The batch script in the Resolution provides the configuration steps to that match the documentation above but is using AES encryption for demonstration purpose.

 

Resolution

 

@echo off
cls
echo.
echo %~nx0 initial beta release.
echo Please review the configuration steps carefully!


echo [PREREQUISITES]
echo All hostnames/FQDN of servers, Service Accounts are created, Kerberos REALMS and SPN values are pre-determined.
echo.
echo Following values are used in this sample:
echo.
echo KDC hostname                                   : kdc
echo KDC DNS record                                 : kdc.example.com
echo Policy Server hostname                         : pserver
echo Policy Server DNS record                       : pserver.example.com
echo Policy Server Reverse DNS PTR                  : 192.168.0.2
echo Policy Server Service Account Name             : EXAMPLE\krbsvc-smps
echo Access Gateway/Web Server hostname             : webagentname
echo Access Gateway/Web Server DNS record           : www.example.com
echo Access Gateway/Web Server Reverse DNS PTR      : 192.168.0.3
echo Access Gateway/Web Server Service Account Name : EXAMPLE\krbsvc-smwa
echo Kerberos Realm                                 : EXAMPLE.COM
echo Test User Account Name                         : EXAMPLE\testkrb
echo Kerberos TKT Default Encryption Type           : AES256-CTS-HMAC-SHA1-96
echo Kerberos TGS Default Encryption Type           : AES256-CTS-HMAC-SHA1-96
echo.
echo Official Document:
echo https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-9/configuring/policy-server-configuration/authentication-schemes/configure-kerberos-authentication.html
echo.
echo [BEFORE RUNNING THIS SCRIPT]
echo 1. All machines including the PS and AG machines are registered to the AD Domain
echo 2. All machines including the PS and AG have forward and reverse DNS records.
echo.
echo    Both PS and WA servers DNS records (pserver.example.com and www.example.com) must have reverse PTR.
echo    If Reverse DNS is not configured, you can get "Server not found in Kerberos database" error.
echo    "Failed to create delegated GSSAPI token on behalf of HTTP/[email protected] for [email protected]: Minor Status=-1765328377, Major Status=851968, Message=Server not found in Kerberos database"
echo.
echo 3. PS and AG service accounts are created(ex: krbsvc-smps, krbsvc-smwa)
echo 4. Configure KccExt in ACO (Optionally add to IgnoreExt)
echo    KccExt=.kcc
echo 5. Both PS and AG Service Accounts (in the Account tab) need to enable both AES options if you choose to use AES encryption.
echo    This account supports Kerberos AES 128 bit encryption.
echo    This account supports Kerberos AES 256 bit encryption.
echo.
echo    If this is not done you will get following error.
echo    "Failed to create delegated GSSAPI token on behalf of HTTP/[email protected] for [email protected]: Minor Status=-1765328370, Major Status=851968, Message=KDC has no support for encryption type"
echo.
echo 6. (OPTIONAL) TURN OFF FIREWALL on the PS and AG machines if you want this script to copy the keytab and krb5.ini files automatically.
echo    Or please manually copy the krb5.ini file and keytab files to the respective machines.
echo    krb5.ini in "C:\Windows" folder and the keytab file in C:\ (ex: C:\krbsvc-smps.keytab)
echo    If you want to place the keytab file in C:\Windows, please update the krb5.ini file to point the default_keytab_name correctly.
echo 7. UPDATE THIS Batch file variables to match your configuration.
echo.
echo [AFTER RUNNING THIS SCRIPT]
echo 1. Run "setspn -Q HTTP/www.example.com" and see if it correctly returns the user account.
echo 2. Run "setspn -Q smsp/pserver.example.com" and see if it correctly returns the user account.
echo 3. Configure AG service account(krbsvc-smwa) "DELEGATION" tab to select "Trust this user for delegation to specified services only."
echo    It is no longer an option to select "Trust this user for delegation to any service (Kerberos only)" as it will NOT work!
echo    Select "Use Kerberos only" option and "Add" the PS service account(krbsvc-smps) and then you will see the Service Type and Computer".
echo    Select the "smps/pserver.example.com" entry and click "OK" and again "OK" in the parent window.
echo.
echo 4. Get actual SPN added to both keytab files.
echo    Must use klist from the jdk, not the one coming with Windows.
echo    (C:\jdk\bin\)klist -e -k -t C:\krbsvc-smps.keytab
echo    This will return the kvno, SPN(smps/[email protected]), keytype, etc.
echo    (C:\jdk\bin\)klist -e -k -t C:\krbsvc-smwa.keytab
echo    This will return the kvno, SPN(HTTP/[email protected]), keytype, etc.
echo    
echo 5. Add SPN to ACO
echo.
echo    httpserviceprincipal=(This must be the exact SPN value from krbsvc-smwa.keytab from step #4)
echo    ex) httpserviceprincipal=HTTP/[email protected]
echo      If this SPN value does not exactly match, you will see error in the watrace.log
echo      "Kerberos Credential Cache login failed with service principal HTTP/[email protected]: Keytab contains no suitable keys for HTTP/[email protected]"
echo.
echo    smpsserviceprincipal=smps@{PS DNS name}
echo    ex) [email protected]
echo.
echo 6. Create Kerberos Authentication Scheme
echo    Target: /siteminderagent/Kerberos/creds.kcc
echo    Principal Name: (This must be the exact SPN value from krbsvc-smps.keytab from step #4)
echo      If this SPN value does not exactly match, you will see error in the smtracedefault.log
echo      "Kerberos Credential Cache login failed with service principal smps/[email protected]: Keytab contains no suitable keys for smps/[email protected]"
echo    User DN Lookup: %{UID}
echo    Realm Name - Domain Name:   example.com     EXAMPLE.COM
echo    Realm Name - Domain Name:  .example.com     EXAMPLE.COM
echo.
echo.
echo [NOTICE]
echo This Batch file must be run from the Domain Controller.
echo Only after you have correctly updated this script, run the script with "go" switch.
echo ex) %~nx0 go
echo.
echo [REFERENCES]
echo Kerberos Key Types:
echo    https://www.iana.org/assignments/kerberos-parameters/kerberos-parameters.xhtml
echo    You can choose from the following list for default_tkt_enctypes and default_tgs_enctypes
echo    but would recommend using AES128 or AES256.
echo    - aes256-cts-hmac-sha1-96 
echo    - aes128-cts-hmac-sha1-96 
echo    - des3-cbc-sha1 
echo    - arcfour-hmac-md5 
echo    - camellia256-cts-cmac 
echo    - camellia128-cts-cmac 
echo    - des-cbc-crc 
echo    - des-cbc-md5 
echo    - des-cbc-md4
echo.
echo Check if the Service Account has the matching kvno.
echo    (Run from DomainController) dsquery * -filter sAMAccountName=krbsvc-smwa -attr msDS-KeyVersionNumber
echo    This should return the kvno value and must match the kvno from klist output.
echo.
echo Use Fiddler to capture the kerberos authentication: (SAML Tracer or Developer mode does NOT capture the actual HTTP 401 and the 
echo    client submitting Authorization Negotiate header correctly.
echo.
echo Submit the captured "YIIxxxxxxxxx" Authorization Negotiate header value to https://asn1js.eu/ and click "decode"
echo    This will give you the decoded content. Note the following.
echo        1.3.6.1.5.5.2.0.2.0.14.3.1 REALM Name
echo        1.3.6.1.5.5.2.0.2.0.14.3.2.1 Service Name/SPN(HTTP/www.example.com)
echo        1.3.6.1.5.5.2.0.2.0.14.3.3.0 Encryption Type (of SPN above)
echo        1.3.6.1.5.5.2.0.2.0.14.3.3.1 KVNO (of SPN above)
echo        1.3.6.1.5.5.2.0.2.0.14.4.0 Authenticator Encryption Type (of SPN above)
echo.
echo.


IF /I "%1"=="go" (
 GOTO COMMON
) ELSE (
 GOTO END
)


:COMMON
REM AD DOMAIN NAME in Upper Case, Kerberos Realm
SET KERB-REALM=EXAMPLE.COM
REM NETBIOS Name
SET NETBIOS=EXAMPLE
SET ENC-TYPE=AES256-CTS-HMAC-SHA1-96
REM COOKIEDOMAIN1 is HTTP cookie domain/AD Domain in lower case starting with a leading dot
SET COOKIEDOMAIN1=.example.com
REM COOKIEDOMAIN2 is HTTP cookie domain/AD Domain in lower case starting without a leading dot
SET COOKIEDOMAIN2=example.com
REM KDC-HOST is the short hostname of AD Domain Controller
SET KDC-HOST=kdc
REM User Password of service accounts
SET PWD={{password}}

 


:PS
REM PS-HOST is the short hostname of Policy Server in the DNS record.
SET PS-HOST=pserver
REM samaccountname for Policy Server Service
SET PS-USER=krbsvc-smps
REM krb5-ps full filename
SET KRB5INI-PS=C:\krb5-ps.ini
REM Keytab full filename for PS
SET KEYTAB-PS=C:\krbsvc-smps.keytab

 


:AG
REM AGENT-HOST is the short hostname of the Access Gateway or Web Server in the DNS record.
SET AGENT-HOST=www
REM samaccountname for Access Gateway Server Service
SET AG-USER=krbsvc-smwa
REM krb5-ag full filename
SET KRB5INI-WA=C:\krb5-smwa.ini
REM Keytab full filename for AG
SET KEYTAB-WA=C:\krbsvc-smwa.keytab

 


:KEYTAB
del /F C:\*.keytab
ktpass -out %KEYTAB-PS% -princ smps/%PS-HOST%%COOKIEDOMAIN1%@%KERB-REALM% -ptype KRB5_NT_PRINCIPAL -mapuser %NETBIOS%\%PS-USER% -mapOp set -pass %PWD% -crypto All
ktpass -out %KEYTAB-WA% -princ HTTP/%AGENT-HOST%%COOKIEDOMAIN1%@%KERB-REALM% -ptype KRB5_NT_PRINCIPAL -mapuser %NETBIOS%\%AG-USER% -mapOp set -pass %PWD% -crypto All


:INI
echo [libdefaults]> %KRB5INI-PS%
echo default_realm = %KERB-REALM%>> %KRB5INI-PS%
echo default_keytab_name = %KEYTAB-PS%>> %KRB5INI-PS%
echo dns_lookup_realm = false>> %KRB5INI-PS%
echo dns_lookup_kdc = false>> %KRB5INI-PS%
echo default_tkt_enctypes = %ENC-TYPE%>> %KRB5INI-PS%
echo default_tgs_enctypes = %ENC-TYPE%>> %KRB5INI-PS%
echo default_ccache_name = FILE:%%{TEMP}\krbcc_%%{uid}>> %KRB5INI-PS%
echo forwardable=true>> %KRB5INI-PS%
echo.>> %KRB5INI-PS%
echo [realms]>> %KRB5INI-PS%
echo %KERB-REALM% = {>> %KRB5INI-PS%
echo kdc = %KDC-HOST%%COOKIEDOMAIN1%:88>> %KRB5INI-PS%
echo default_domain = %KERB-REALM%>> %KRB5INI-PS%
echo }>> %KRB5INI-PS%
echo.>> %KRB5INI-PS%
echo [domain_realm]>> %KRB5INI-PS%
echo %COOKIEDOMAIN1% = %KERB-REALM%>> %KRB5INI-PS%
echo %COOKIEDOMAIN2% = %KERB-REALM%>> %KRB5INI-PS%
echo.
echo.
echo.
echo [libdefaults]> %KRB5INI-WA%
echo default_realm = %KERB-REALM%>> %KRB5INI-WA%
echo default_keytab_name = %KEYTAB-WA%>> %KRB5INI-WA%
echo dns_lookup_realm = false>> %KRB5INI-WA%
echo dns_lookup_kdc = false>> %KRB5INI-WA%
echo default_tkt_enctypes = %ENC-TYPE%>> %KRB5INI-WA%
echo default_tgs_enctypes = %ENC-TYPE%>> %KRB5INI-WA%
echo default_ccache_name = FILE:%%{TEMP}\krbcc_%%{uid}>> %KRB5INI-WA%
echo forwardable=true>> %KRB5INI-WA%
echo.>> %KRB5INI-WA%
echo [realms]>> %KRB5INI-WA%
echo %KERB-REALM% = {>> %KRB5INI-WA%
echo kdc = %KDC-HOST%%COOKIEDOMAIN1%:88>> %KRB5INI-WA%
echo default_domain = %KERB-REALM%>> %KRB5INI-WA%
echo }>> %KRB5INI-WA%
echo.>> %KRB5INI-WA%
echo [domain_realm]>> %KRB5INI-WA%
echo %COOKIEDOMAIN1% = %KERB-REALM%>> %KRB5INI-WA%
echo %COOKIEDOMAIN2% = %KERB-REALM%>> %KRB5INI-WA%
echo.


:COPY
echo copy /Y %KEYTAB-PS% \\%PS-HOST%\c$\
copy /Y %KEYTAB-PS% \\%PS-HOST%\c$\
echo copy /Y %KEYTAB-WA% \\%AGENT-HOST%\c$\
copy /Y %KEYTAB-WA% \\%AGENT-HOST%\c$\
echo copy /Y %KRB5INI-PS% \\%PS-HOST%\c$\Windows\krb5.ini
copy /Y %KRB5INI-PS% \\%PS-HOST%\c$\Windows\krb5.ini
echo copy /Y %KRB5INI-WA% \\%AGENT-HOST%\c$\Windows\krb5.ini
copy /Y %KRB5INI-WA% \\%AGENT-HOST%\c$\Windows\krb5.ini


:END
echo The End.
pause

 

 

When accessing Kerberos Authentication Scheme protected resource without a valid session then browser will redirect to Kerberos Authentication Scheme URL and HTTP 401 is expected requesting kerberos token.

 

HTTP/1.1 401 401
Date: Thu, 09 Oct 2025 04:06:43 GMT
Server: Apache
WWW-Authenticate: Negotiate
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: text/html; charset=ISO-8859-1
Proxy-Support: Session-Based-Authentication
Content-Length: 32

Negotiated Credentials Required 

 

 

Then the client must submit the Authorization header as shown below.

GET https://www.example.com/siteminderagent/Kerberos/creds.kcc?&SMAGENTNAME=-SM-exPODbbR1fMlHWgUeic34vq9l9%2bUym%2fobRkvUhhxZaQuWNjCxygOelrae%2fjU2Wpl&TARGET=-SM-https%3a%2f%2fwww%2eexample%2ecom%2fprotected%2f HTTP/1.1
Host: www.example.com
Connection: keep-alive
Authorization: Negotiate YIIG5AYGKwYBBQUCoIIG2DCCBtSgMDAuBgkqhkiC9xIBAgIGCSqGSIb3EgECAgYKKwYBBAGCNwICHgYKKwYBBAGCNwICCqKCBp4EggaaYIIGlgYJKoZIhvcSAQICAQBuggaFMIIGgaADAgEFoQMCAQ6iBwMFACAAAACjggTDYYIEvzCCBLugAwIBBaENGwtFWEFNUExFLkNPTaIiMCCgAwIBAqEZMBcbBEhUVFAbD3d3dy5leGFtcGxlLmNvbaOCBH8wggR7oAMCARKhAwIBA6KCBG0EggRpxTZSlc/JssXsRUs4NcZTZ8V+tdKUIA/vf5xLhVOng6d6AzVUEmZL4l+LpUCqOALCX+vklV82VafmAoP9R5NcD09nylcz+cA2567z3bvlF1uO/yxvAR84fLSN0fOQy0bjDRZ4cP/uYgn7GGPlF3E/EqGuk4LA5RJ55lc9Kt1akiF4mJWTU997PMuS9EFQ6foDqhCBsBEF3zdnMA2zKOwCpZhuYNZbyaFDamcyCQso3jwqwcQRhqDusgzerrYqfZdcAh5CoGuZP42n2IP2hU3TRQSoOhBx0R/E0xWbadrmeumB9euKmbwKp74PGpueGNZu4eoGipZRDkA7th+4NNNGRjXTGhWJzNwGX/ytw4aq6yrKL+0HNUgl85o8bLigQJFW4CP9rb0XazdXzf/TBmHTb2JrM3LR2M/wZct4GinLB8V98FnD5ZuxkO0M5QU1mgzy4voXLwIENNGxyp5EGIOO/BuAkduPu6taEtiu/Y39mzn+dVO5RCFsV9sZB9x4/mSnIsKXlO08hWxMOSEt+sddYuoFITRw0/yn9xcj6/mA/Bs3z9ZLeNyzqpBBgckBwGNWUPRqpLPnN9onaV9IOnikVLnSpj2g2CfKb48qfrn+4rfmK5MyrBF1/eli3iAPMHMWFEUp/sxdXtSQME1MtaB4b1l8Q+sbPk3F5faFDMkw3DtKFyQfa9MQzAop7zqH9Pa5SBhNwW9sW1Itlg628AbahySru4fo7i5pyFKLW0XsPc3OOvZAPlE7oqCTavw+XVmdFWPED143ZvywKN6/PooA5tzy/ARSgi0WyN30y5ECuabVwYJC84Kk9ObhHYV0YpOmvs5V9bUAGHRoY+stj4otQ264xj8mb4Dm1R6As5Ko1TSSqowVJRyNQqrjxXn4grH4xGAQeqZQ4HauzmDc45ADgbq9n74QC8P738OdQ7T3sCybghvZl1o7Z0bzRrY/7qufOVp59WQp+iTyYwzwPKAPLUv0tPwZGhqXNR8xLV6/bKyh71wnN20fQSJGzUoLs52vcy+7gJSSx9GO4pxwjS5sXCCmvd+ylPrH04BGd3Vll5QK9dL+ccxWDQnCOXqQPSpV8CYPSvm/q5BHShZWx6w9/FpGOObvxcienedR5tpIallK0SxsdW/EMmDFLVE6o3dxW3BBugROtoHMYomkFmVO89Yh5sLgwhzJkotWL73rH65BTn4HbVdxHerkBGfEfpIU5Ag6xFx9iv+oTPDzCwipgeJXGNIFrMkWCHXRd+0DxHjQyxO7XrzPblEa6pr3D+ifVkby3tKAj7lfKiClzd+d4mpHOW53pAp/+t0F/avb/lCWo8a+nUi/CGuLmjOfylpZXR1kMqcbUafTcjrJHRmAOIlTsdje/IeVV6LI+Bhqks/VA0e/0C5t7tAzRwrFuJWqMzzXMcmbkG8f+2XnOyTIXcNo6GbTzlH8uUc5nMItjgh7pqWYBiJJ+lpCYA1enV5QlTyIfVXx5CLRCgxcaoGb6McEUH/ytpM3aqSCAaMwggGfoAMCARKiggGWBIIBkpqrBIoqNt2Vqno1glgB07KaiPegfGSJbNo9ca+/f1Y7+vo4IpkGEHbc6TL0/p/ISCo2WusbW/5pVT5XADUImCASagfKHNCdRpwCpFvZnF2CU7HH4QVxlbG/7GS664ySDv3IIaMxVZzKveRClOaLQu/jAlnucM+FtiMSQqOQKi3l3W9cHKvOb8oGT6NUavs6By+FLLnzIdfobqnLh8CkVN6CFo6fjYXMgzClih6bJR3a6XRUrVTTxV7PkAvEBrSY2Dyj6ePq8yf/mVXoCiTB9UAQG0J6oOKQHHhsR5MjwGKeiExEm8oBnO48hjRTgWxUz76hosMv1IvZ8BW5Jmpfy9VDGklps+NzFF5hyffS2PkBo3lZEaZA4rsdfk0EAPdMWkepRljTWCdZvkuwnltr+x20OpespJxhzpWN2U7Yne2aLb6IbKblt0w7bD+79FUfrfEvM3UEzwcZYPKkQ3TD2787PQPclCx4kZAOXpv22f32SAQOAAK3wFaam6L8P6+lGMzmvVNaa5g4qrsKKz/UXghJ5A==
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
sec-ch-ua: "Chromium";v="134", "Not:A-Brand";v="24", "Microsoft Edge";v="134"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Referer: https://www.example.com/
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Cookie: ASPSESSIONIDSABASSTD=ANDFGKJDJKABEBFNADNDKMHH

 

Copy the Authorization header value starting with YII until end of the header and paste it to https://asn1js.eu and click "decode"

This is equivalent to:

You can find the following
- Realm
- ServiceName
- EncryptionType
- KeyVersionNumber

SampleLogs.zip file contains the fiddler capture and logs sample.

Attachments

SampleLogs.zip get_app