kerberos in GPCC - troubleshooting guide
Article ID: 296599
Updated On:
Products
Issue/Introduction
When setting up kerberos authentication for GPCC check the standard guide: https://docs.vmware.com/en/VMware-Greenplum/5/greenplum-database/admin_guide-kerberos.html
Note: Please be aware that setting up the kerberos authentication on Windows and Linux (or Mac OSX) is slightly different. The Linux/MAC environment use kinit to set up an authentication token. This token is used to access GPCC and greenplum (if greenplum is also kerberized and usually is)
On Windows, you can use kinit to get a token and this will allow you to use windows client to connect to greenplum BUT if you try to connect to GPCC, the browser will use the current windows log in user to authenticate to kerberos, not your token you generated with kinit. If your windows user is not in kerberos database it will fail.
Troubleshooting guide:
1) How to add a debugging to kinit
Kinit by default doesn't give you much information:
$ kinit -k -t /Users/#######/kerberos/gpdb-kerberos.keytab [email protected]
kinit: krb5_init_creds_set_keytab: Failed to find [email protected] in keytab FILE:/Users/######/kerberos/gpdb-kerberos.keytab (unknown enctype)
If you want to see a trace log add KRB5_TRACE=/dev/stdout to the command in linux/OSX.
Same command with trace:
$ KRB5_TRACE=/dev/stdout kinit -k -t /Users/######/kerberos/gpdb-kerberos.keytab [email protected]
2022-11-21T16:29:53 set-error: -1765328243: no cache for 8B05159C-4441-4189-94AD-841761BC88E1
2022-11-21T16:29:53 set-error: -1765328243: no cache for A6418DE6-C712-4D2D-A27A-A02CEFE77981
2022-11-21T16:29:53 set-error: -1765328243: no cache for 31050DDC-3D42-4A32-B4EB-7CB294067731
2022-11-21T16:29:53 set-error: -1765328242: Reached end of credential caches
2022-11-21T16:29:53 set-error: -1765328243: Principal [email protected] not found in any credential cache
2022-11-21T16:29:53 set-error: -1765328243: no credential for 18221C7B-BDA5-49AA-B1F9-5A0AF8520B44
2022-11-21T16:29:53 set-error: 2: keytab /Users/#####/kerberos/gpdb-kerberos.keytab access failed: No such file or directory
2022-11-21T16:29:53 set-error: 2: Failed to find [email protected] in keytab FILE:/Users/#####/kerberos/gpdb-kerberos.keytab (unknown enctype)
kinit: krb5_init_creds_set_keytab: Failed to find [email protected] in keytab FILE:/Users/######/kerberos/gpdb-kerberos.keytab (unknown enctype)
If you want same trace in Widows add KRB5_TRACE = "CON" in System variables:
How it looks like when it is working:
Example of windows commands:
cd "C:\Program Files\Greenplum\greenplum-clients\bin"
set KRB5CCNAME=%USERPROFILE%\krb5cache
kinit -k -t C:\Users\IEUser\Desktop\gpdb-kerberos.keytab [email protected]
set PGGSSLIB=gssapi
klist.exe
"c:\Program Files\Greenplum\greenplum-clients\greenplum_clients_path.bat"
psql -h mdw-lab2 -p 3070 -U "gpmon" template1
Example of OSX commands
$ KRB5_TRACE=/dev/stdout kinit -k -t /Users/##########/kerberos/gpdb-kerberos.keytab [email protected]
2022-11-21T16:54:36 set-error: -1765328243: no cache for 8B05159C-4441-4189-94AD-841761BC88E1
2022-11-21T16:54:36 set-error: -1765328243: no cache for A6418DE6-C712-4D2D-A27A-A02CEFE77981
2022-11-21T16:54:36 set-error: -1765328243: no cache for 31050DDC-3D42-4A32-B4EB-7CB294067731
2022-11-21T16:54:36 set-error: -1765328243: no cache for 74BD85AC-60BE-490C-AECA-D6AE1F5B02D8
2022-11-21T16:54:36 set-error: -1765328242: Reached end of credential caches
2022-11-21T16:54:36 set-error: -1765328243: Principal [email protected] not found in any credential cache
2022-11-21T16:54:36 set-error: -1765328243: no credential for 626709D5-B6D6-4042-A2FF-AB6BEA7EA13E
2022-11-21T16:54:36 Adding PA mech: ENCRYPTED_CHALLENGE
2022-11-21T16:54:36 Adding PA mech: ENCRYPTED_TIMESTAMP
2022-11-21T16:54:36 krb5_get_init_creds: loop 1
2022-11-21T16:54:36 KDC sent 0 patypes
2022-11-21T16:54:36 fast disabled, not doing any fast wrapping
2022-11-21T16:54:36 Trying to find service kdc for realm GPDB.KRB flags 0
2022-11-21T16:54:36 configuration file for realm GPDB.KRB found
2022-11-21T16:54:36 submissing new requests to new host
2022-11-21T16:54:36 host_create: setting hostname gpdb-kdc
2022-11-21T16:54:36 connecting to host: udp 10.225.28.79:88 (gpdb-kdc) tid: 00000001
2022-11-21T16:54:36 writing packet: udp 10.225.28.79:88 (gpdb-kdc) tid: 00000001
2022-11-21T16:54:36 reading packet: udp 10.225.28.79:88 (gpdb-kdc) tid: 00000001
2022-11-21T16:54:36 host completed: udp 10.225.28.79:88 (gpdb-kdc) tid: 00000001
2022-11-21T16:54:36 krb5_sendto_context GPDB.KRB done: 0 hosts 1 packets 1 wc: 0.192283 nr: 0.004262 kh: 0.002108 tid: 00000001
2022-11-21T16:54:36 krb5_get_init_creds: loop 2
2022-11-21T16:54:36 krb5_get_init_creds: processing input
2022-11-21T16:54:36 KDC sent 1 patypes
2022-11-21T16:54:36 KDC sent PA-DATA type: 19 (ETYPE_INFO2)
2022-11-21T16:54:36 dropped pa-mech ENCRYPTED_CHALLENGE since not running under FAST
2022-11-21T16:54:36 pa-mech trying: ENCRYPTED_TIMESTAMP, searching for 2
2022-11-21T16:54:36 Stepping pa-mech: ENCRYPTED_TIMESTAMP
2022-11-21T16:54:36 PA type ENCRYPTED_TIMESTAMP returned 0: Success
2022-11-21T16:54:36 final pamech done step
2022-11-21T16:54:36 krb5_get_init_creds: extracting ticket
2022-11-21T16:54:36 krb5_get_init_creds: wc: 0.199225
2022-11-21T16:54:36 set-error: -1765328243: Deleted credential not found
######-a01:~ ########$ klist
Credentials cache: API:F97B4C19-0ED4-4250-9B9C-EA04024548F8
Principal: [email protected]
Issued Expires Principal
Nov 21 16:54:36 2022 Nov 22 16:54:36 2022 krbtgt/[email protected]
and once you access gpcc using browser:
klist
Credentials cache: API:C3981F23-757D-4038-88EC-9354D989F2D8
Principal: [email protected]
Issued Expires Principal
Nov 23 11:36:20 2022 Nov 24 11:36:19 2022 krbtgt/[email protected]
Nov 23 14:12:06 2022 Nov 24 11:36:19 2022 HTTP/[email protected]
2) How to check if GPCC is using kerberos:
Check the current gpcc configuration in app.conf
[gpadmin@mdw-lab2 conf]$ cat /usr/local/greenplum-cc-4.13.0/conf/app.conf
(...)
enable_kerberos = true
(...)
[Kerberos]
webserver_url = mdw-lab2
krb_service_name = postgres
krb_mode = gpmon_only
keytab = /home/gpadmin/kerberos/gpdb-kerberos.keytab
Form app.conf we can see the enable_kerberos = true which indicates the kerberos is used.
Please note: The gpcc has a keytab specified in app.conf (in this example /home/gpadmin/kerberos/gpdb-kerberos.keytab) this doesn't have to be the same keytab file as used by greenplum.
For greenplum you can find the used keytab with:
$ grep krb_server_keyfile $MASTER_DATA_DIRECTORY/postgresql.conf
krb_server_keyfile='/home/gpadmin/kerberos-gpdb/gpdb-kerberos.keytab'
3) How to setup browser to use kerberos
In the documentation : "The user's web browser must be configured to use the SPNEGO protocol so that it offers the user's Kerberos principal to the web browser. The method for configuring web browsers varies with different browsers and operating systems. Search online to find instructions to enable SPNEGO with your browser and OS"
Each browser has this implemented in a different way and also there might be a difference between same browser and different operating systems. At the moment of writing this article the google chrome requires windows registry entry in windows environment. You can also try to run chrome in windows with whithlist command center server name
example:
"C:\Program Files\Google\Chrome\Application\chrome.exe" -auth-server-whitelist="mdw-lab2"
For Firefox it is easier, you have to set up network.negotiate-auth.trusted-urisPlease note: The way you configure a browse can change so please google it how you should set it up at the time you configure.
4) How to troubleshoot browser kerberos connection:
We will use chrome browser for this example but most modern browser would have a network tab to trace the requests.
a) Open a browser, navigate to gpcc hostname:port
b) right click anywhere in the page and chose inspect
c) navigate to a Network tab
d) chose "access" entry
e) The "Status Code" should give you first hint what is the problem
You can also check the gpcc log file:
tail /usr/local/greenplum-cc-4.13.0/logs/gpccws.log
2022/11/21 12:39:24.600 [I] [value.go:476] Kerberos service name is HTTP/mdw-lab2
2022/11/21 12:39:24.600 [I] [value.go:476] Kerberos Info: no input token provided
2022/11/21 12:39:24.765 [I] [router.go:833] WebsocketController register user , role map[]
In this situation there is no authentication token. You can clarify this by using klist on a computer that is trying to use browser to access gpcc
$ klist
klist: Cache not found: API:18221C7B-BDA5-49AA-B1F9-5A0AF8520B44
Potential scenarios:
1) incorrect hostname for keytab entry
In the gpcc log file we could see:
2022/11/17 08:39:44.892 [I] [value.go:476] Kerberos service name is HTTP/bdtbasr5n1
2022/11/17 08:39:44.900 [I] [value.go:476] Kerberos Info: Request ticket server HTTP/mdw[email protected] not found in keytab (ticket kvno 22)
The keytab file contained the HTTP request for correct host but it was incorrect realm. Once we used the one reported by the gpcc (HTTP/mdw[email protected]) and generated a new keytab with this entry and gpmon it worked
2) out of date tokens
In the gpcc log file we could see:
2022/11/17 10:00:53.626 [I] [value.go:476] Kerberos service name is HITTP/mdw
2022/11/17 10:00:53.636 [I] [value.go:476] Kerberos Info: Request ticket server HTTP/mdw[email protected] kvno 22 not found in keytab; keytab is likely out of date
The new keytab was generated for gpcc containing gpmon user and HTTP request for the correct host.
The error was a result of generated keytab not being synchronised. The solution was to wait 30min and try again
3) User name capital letter
- The windows user is having a capital letter in the username (for example K123456)
- in order to use command center the same role has to exist in greenplum
- the corresponding role exist in greenplum but with small letter (for example k123456) as a result you will see in a log file:
2022/11/17 10:57:43.252 [E] [logon.go:103] Get Role got error pq: role "K123456" does not exist
- the issue was a result of creating a role without double quotes
example:
gpadmin=# create role K123456
will create role k123456
gpadmin=# create role "K123456"
will create role K123456
The solution was to capitalise the existing roles with
ALTER ROLE k123456 RENAME TO "K123456";
4) missing /etc/krb5.conf
$ KRB5_TRACE=/dev/stdout kinit -k -t /Users/########/kerberos/gpdb-kerberos.keytab [email protected]
2022-11-21T16:45:06 set-error: -1765328243: no cache for 8B05159C-4441-4189-94AD-841761BC88E1
2022-11-21T16:45:06 set-error: -1765328243: no cache for A6418DE6-C712-4D2D-A27A-A02CEFE77981
2022-11-21T16:45:06 set-error: -1765328243: no cache for 31050DDC-3D42-4A32-B4EB-7CB294067731
2022-11-21T16:45:06 set-error: -1765328243: no cache for 74BD85AC-60BE-490C-AECA-D6AE1F5B02D8
2022-11-21T16:45:06 set-error: -1765328242: Reached end of credential caches
2022-11-21T16:45:06 set-error: -1765328243: Principal [email protected] not found in any credential cache
2022-11-21T16:45:06 set-error: -1765328243: no credential for 18221C7B-BDA5-49AA-B1F9-5A0AF8520B44
2022-11-21T16:45:06 Adding PA mech: ENCRYPTED_CHALLENGE
2022-11-21T16:45:06 Adding PA mech: ENCRYPTED_TIMESTAMP
2022-11-21T16:45:06 krb5_get_init_creds: loop 1
2022-11-21T16:45:06 KDC sent 0 patypes
2022-11-21T16:45:06 fast disabled, not doing any fast wrapping
2022-11-21T16:45:06 Trying to find service kdc for realm GPDB.KRB flags 0
2022-11-21T16:45:06 configuration file for realm GPDB.KRB not found
(...)
5) Ticket expire:
Command Center log file gpccws.log :
2022/11/21 16:59:12.861 [W] [router.go:833] WebsocketController cannot read message: websocket: close 1001 (going away)
2022/11/21 16:59:13.414 [I] [value.go:476] Kerberos service name is HTTP/mdw-lab2
2022/11/21 16:59:13.415 [I] [value.go:476] Kerberos Info: no input token provided
2022/11/21 16:59:13.671 [I] [value.go:476] Kerberos service name is HTTP/mdw-lab2
2022/11/21 16:59:13.746 [E] [logon.go:103] Connect db error: pq: GSSAPI continuation error: Ticket expired
2022/11/21 16:59:13.746 [E] [logon.go:139] GSSAPI continuation error: Ticket expired
2022/11/21 16:59:13.807 [I] [router.go:833] WebsocketController register user , role map[]
Environment
Resolution
solution: Check with klist your tickets and worth to try if you can psql to greenplum as gpmon user from this host using kerberos.
Feedback
