Zowe Certificate Troubleshooting using ACF2

Products

• ACF2 - z/OS
• ACF2
• ACF2 - MISC

Issue/Introduction

ACF2 KEYRING and certificates have been set up using ZWEKRING from ZWESAMP JCL library. However, when trying to log in from Zowe desktop various error messages are seen:

(prepare-instance.sh,global_validate:34) Error 0: Could not contact z/OS MF on 'https://example.com:30443/zosmf/info' - 403

ZWED0179W - Unable to retrieve the list of certificate authorities from the keyring=ZWESTC_RING owner=ZWESTC Error: TypeError: keyring_js.listKeyring is not a function

org.zowe.zlux.auth.safsso: ZSS 502

{"messages":[{"messageType":"ERROR","messageNumber":"ZWEAM511E","messageContent":"The certificate of the service accessed using URL '/api/v1/zss/login' is not trusted by the API Gateway: Received fatal alert: bad_record_mac","messageKey":"org.zowe.apiml.common.tlsError"}]}

This document outlines how to collect documentation needed to diagnose Zowe certificate issues on a system secured by ACF2.

Resolution

Follow the normal ACF2 certificate troubleshooting steps first to rule out typical certificate issues:

1. The Server/Client log showing the error messages related to the KEYRING/certificates.
   
   
2. To verify KEYRING setup get a LIST of the KEYRING that is being used. From TSO: 
   
   ACF
   SET PROFILE(USER) DIV(KEYRING) 
   LIST ringowner.suffix 
   
   
3. To verify the certificate status, keys and signing chain for the PERSONAL Certificate issue a CHKCERT command. From TSO:   
   
   ACF     
   CHKCERT certdata.suffix CHAIN
   
   
4. To verify the server/client task has the authority to access the Keyring and the private key of the PERSONAL certificate, run the ACFRPTRV report against the SMF active at the time of the error (sample JCL below).
   
   
5. To verify that the Keyring, certificates and PERSONAL certificate’s Private Key are being returned, perform a SECTRACE for R_datalib calls. Enable the SECTRACE, recreate the error, and then turn off the trace. The output will be in the started task joblog and the syslog. Commands for this are listed below.

Sample RV Report JCL:
//REPORT  EXEC PGM=ACFRPTRV                        
//SYSPRINT DD SYSOUT=*                             
//HEXDUMP  DD SYSOUT=*                             
//* RECMAN1  DD DSN=IFASMF.STREAM,DISP=SHR,   
//*             SUBSYS=(LOGR,IFASEXIT)             
//RECMAN1  DD DISP=SHR,DSN=SYS1.MAN1               
//RECMAN2  DD DISP=SHR,DSN=SYS1.MAN2               
//RECMAN3  DD DISP=SHR,DSN=SYS1.MAN3               
//SYSIN    DD *                                    
TITLE(ACFRPTRV)

To turn on SECTRACE:
SECTRACE SET,ID=OMVSTRC,TYPE=OMVS,SFUNC=RDATALIB,END

To turn off SECTRACE:
SECTRACE DELETE,ID=OMVSTRC

Zowe specific certificate troubleshooting:

1. Review the entirety of ZWEKRING JCL and zowe-setup-keyring-certificates.env. These will need to be compared to the previously provided certificate documentation.
   
   
2. If it is still not clear why the certificate issue is occurring, here is how to get a full TLS debug log for Zowe:

• Locate the start.sh script for the gateway in the <zowe-runtime-dir>/components/gateway/bin/start.sh directory.
  • Edit the script to
    • Djavax.net.debug=all
• To enable traces of TLS in ZSS, add the following lines to the instance.env file
  
  • export GSK_TRACE=0xFF
  • export GSK_TRACE_FILE=/path/to/file.trc
• Recycle the task, recreate the error.
• Review the joblog and the /path/to/file.trc file. This is a binary file.
  
  Note: There will be a lot of output. If ICSF is a suspected culprit, you can search on ICSF. For example, a resource rule deny would look like this:
  
  12/10/2021-11:09:32 Thd-0 ERROR crypto_aes_gcm_decrypt_ctx(): ICSF service failure: CSFPSKD retCode = 0x8, rsnCode = 0x3e80
  12/10/2021-11:09:32 Thd-0 ERROR gsk_decrypt_v3_record(): AES GCM Decryption failed: Error 0x03353084
  
  In this example, rules needed to be written for ICSF CSFSERV resource profiles CSF1SKD and CSF1SKE.

Notes regarding z/OSMF and Zowe:

The Z/OSMF public key must be in the Zowe keyring in order for Zowe to trust z/OSMF. 

The TLS Web Client Authentication (1.3.6.1.5.5.7.3.2) value needs to be in the Extended Key Usage section in order for the Zowe certificate to also act as the client to the z/OSMF server. ACF2 cannot set this value.  To verify this value was properly set outside of ACF2 the SAFCRRPT report can be ran with the EXT parameter:

//SAFRPTCR EXEC PGM=SAFCRRPT,REGION=0M,                                
//       PARM='TITLE(CERTIFICATE UTILITY REPORT)'                      
//SYSUDUMP DD SYSOUT=*                                                 
//SYSPRINT DD SYSOUT=*                                                                                                        
//SYSIN DD *
USER(owner)
DETAIL EXT